System and method for on-road detection of a vehicle using knowledge fusion

Information

  • Patent Grant
  • 7639841
  • Patent Number
    7,639,841
  • Date Filed
    Monday, December 19, 2005
    18 years ago
  • Date Issued
    Tuesday, December 29, 2009
    14 years ago
Abstract
The present invention is directed to a system and method for on-road vehicle detection. A video sequence is received that is comprised of a plurality of image frames. A potential vehicle appearance is identified in an image frame. Known vehicle appearance information and scene geometry information are used to formulate initial hypotheses about vehicle appearance. The potential vehicle appearance is tracked over multiple successive image frames. Potential motion trajectories for the potential vehicle appearance are identified over the multiple image frames. Knowledge fusion of appearance, scene geometry and motion information models are applied to each image frame containing the trajectories. A confidence score is calculated for each trajectory. A trajectory with a high confidence score is determined to represent a vehicle appearance.
Description
FIELD OF THE INVENTION

The present invention is directed to a system and method for vehicle detection, and more particularly, to a system and method for on-road vehicle detection using knowledge fusion.


BACKGROUND OF THE INVENTION

With the decreasing cost of optical sensors and increasing computing power of microprocessors, vision-based systems have been widely accepted as an integral part of the feasible solutions to driver assistance. The ability of detecting other vehicles on the road is essential to sensing and interpreting driving environments, which enables important functions like adaptive cruise control and pre-crash sensing. Vehicle detection requires effective vision algorithms that can distinguish vehicles from complex road scenes accurately. A great challenge comes from the large variety of vehicle appearance as well as different scenarios of driving environments. Vehicles vary in size, shape and appearance, which lead to considerable amount of variance in the class of vehicle images. Illumination changes in outdoor environments introduce additional variation in vehicle appearance. Meanwhile, unpredictable traffic situations create a wide range of non-stationary backgrounds with complex clutters. Moreover, high degrees of reliability and fast processing are required for driver assistance tasks, which also increase the difficulty of the task.


Known vision techniques have been used in vehicle detection. A number of approaches use empirical knowledge about vehicle appearance, such as symmetry, horizontal and vertical occluding edges around vehicle boundaries to detect the rear-view appearance of vehicles. These methods are computationally efficient but lack robustness because the parameters (e.g., thresholds) involved in edge detection and hypothesis generation are sensitive to lighting conditions and the dynamic range in image acquisition. To achieve reliable vehicle detection, several appearance-based methods exploit machine learning and pattern classification techniques to obtain elaborated classifiers that separate the vehicle class from other image patterns. Bayesian classifiers have also been used for classification in which a mixture of Gaussian filters and histograms were used to model the class distribution of vehicles and non-vehicles. Another method uses neural network classifiers that are trained on image features obtained from local orientation coding. Still other methods use Support Vector Machines (SVMs) that are trained on wavelet features.


Many of the methods mentioned above use partial knowledge for vehicle detection. For example, appearance-based methods mainly utilize the knowledge about vehicle and non-vehicle appearance, while motion-based detectors focus on the knowledge about relative vehicle motion. To make a detection system reliable, all the available knowledge should be utilized in a principled manner. There is a need for a vehicle detection system which is capable of fusing multiple sources of data over multiple image frames in order to more consistently and more accurately detect a vehicle.


SUMMARY OF THE INVENTION

The present invention is directed to a system and method for on-road vehicle detection. A video sequence is received that is comprised of a plurality of image frames. A potential vehicle appearance is identified in an image frame. Known vehicle appearance information and scene geometry information are used to formulate initial hypotheses about vehicle appearance. The potential vehicle appearance is tracked over multiple successive image frames. Potential motion trajectories for the potential vehicle appearance are identified over the multiple image frames. Knowledge fusion of appearance, scene geometry and motion information models are applied to each image frame containing the trajectories. A confidence score is calculated for each trajectory. A trajectory with a high confidence score is determined to represent a vehicle appearance.





BRIEF DESCRIPTION OF THE DRAWING

Preferred embodiments of the present invention will be described below in more detail, wherein like reference numerals indicate like elements, with reference to the accompanying drawings:



FIG. 1 is a system block diagram of a system for detecting preceding vehicles in accordance with the present invention;



FIG. 2 illustrates an example of appearance trajectories in accordance with the present invention;



FIG. 3 is a block diagram of an integrated framework for knowledge fusion in accordance with the present invention;



FIG. 4 shows examples of vehicle and non-vehicle training samples in accordance with the present invention;



FIG. 5 illustrates an example of a geometry constraint on appearance size and location in accordance with the present invention;



FIGS. 6
a and 6b illustrate empirical error rates of car classifier and truck classifier in accordance with the present invention; and



FIG. 7 illustrates examples of vehicle detection results in accordance with the present invention.





DETAILED DESCRIPTION

The present invention is directed to an integrated framework for on-road vehicle detection that uses knowledge fusion of appearance, scene geometry and vehicle motion. FIG. 1 illustrates a block diagram of a system for implementing the present invention. A camera 102 is used to capture images of a road and its surroundings. As would be expected with a typical road image, the image includes background images, such as buildings, trees, and houses, and vehicles driving on the road. The images are communicated to a processor 104 which analyzes the image intensity and image motion to detect vehicles in front of the ego-vehicle.


Appearance, geometry and motion information are fused over multiple image frames. The knowledge of vehicle/non-vehicle appearance, scene geometry and vehicle motion is utilized through prior models obtained by learning, probabilistic modeling and estimation algorithms. The prior models are stored in database 108. Once a vehicle is identified at a sufficient confidence level, the vehicle is identified via an output device 106. The output device 106 provides an output signal which communicates to the user or following modules the presence of the vehicle as well as its location and size within an image frame. The output signal may be an audible signal or other type of warning signal. The output device 106 may also include a display for viewing the detected vehicles. The display provides a view of the images taken by the camera 102 which are then enhanced to indicate vehicles that have been detected and which are being tracked. The detection of a vehicle can also be incorporated with other vehicle features such as automatic cruise control and collision avoidance systems.


On-road vehicle detection is different than detecting vehicles in still images. In an on-board vision system, preceding vehicles appear in multiple image frames consistently. The information of vehicle appearance, vehicle motion as well as scene geometry can be exploited jointly to ensure robust and reliable detection. Appearance information provides strong discrimination for distinguishing vehicles from non-vehicles. Motion information has the ability of associating vehicle appearance over time. With temporal data association, detection becomes more robust against isolated errors made by appearance detectors. The knowledge about scene geometry induces strong constraints on where a vehicle on the road would appear on the image plane. Incorporating geometry information into detection can reduce certain errors such as detecting vehicles in the sky or on a tree.


In accordance with the present invention, it is important to detect consistent vehicle appearance over multiple image frames. If {I1,I2, . . . , Im} denotes m consecutive image frames, and (xk,sk) is the vehicle location (xk=[x,y]k′) and size (sk) in the k-th frame, and Ik (xk,sk) as the image patch of size sk at location xk of the k-th frame (k=1, . . . , m). Essentially, {(x1,s1), . . . , (xm,sm)} defines a trajectory of vehicle appearance on the image plane. Given the observation of m consecutive image frames {Ik}k=1m and the knowledge of scene geometry, the likelihood of consistent appearance of an on-road vehicle on the image plane is expressed as











p
m



(


(


x
1

,

s
1


)

,









(


x
m

,

s
m


)


|

I
1


,





,

I
m


)


·




k
=
1

m





p
g



(


(


x
k

,

s
k


)

|

scene





geometry


)


·




k
=
1

m




P
a



(



I
k



(


x
k

,

s
k


)



vehicle

)









(
1
)







The first term pm((x1,s1), . . . (xm,sm)|I1, . . . , Im) defines the likelihood of the appearance of the trajectory {(xl,s1), . . . , (xm,sm)} being consistent. The subscript m is used in the notation because this term incorporates motion information to determine temporal association of object appearance. The second term,









k
=
1

m




p
g

(


(


x
k

,





s
k


)

|







scene geometry) defines the likelihood of an on-road vehicle appearing on an admissible trajectory {(x1,s1, ), . . . , (xm,sm))} given the knowledge of scene geometry. The subscript g is used in the notation to indicate geometry information being exploited. The third term









k
=
1

m




P
a



(



I
k



(


x
k

,

s
k


)



vehicle

)







vehicle) defines the probability that the image patches Ik(xk,sk) (k=1, . . . , m) belong to the vehicle class, where the subscript a in the notation indicates the use of appearance information.


An example of appearance trajectories is illustrated in FIG. 2. A vehicle 202 is shown over time in a number of subsequent image frames 208, 210, 212 (i.e., time t, t+1, t+2). In each subsequent image frame, the possible appearance trajectories 204, 206 are shown. As more information about the vehicle is obtained, the correct trajectory can be identified. Over time the possible appearance trajectories are maintained and the probability of their likelihood is calculated. If the probability falls below a predetermined threshold, the trajectory is dropped from consideration.


Using the above probabilistic formulation, an integrated framework of knowledge fusion in accordance with the present invention is shown in FIG. 3. The prior models of appearance Pa 308, geometry pg 310 and motion pm 312 are used to fuse and propagate information over time. To detect on-road vehicles in an image sequence, the appearance and geometry models Pa, pg are used to generate initial hypotheses of vehicle appearance. Using the motion model pm, the initial hypotheses are tracked over successive image frames 302-306. Consequently, the initial hypotheses evolve into hypotheses of vehicle appearance trajectories.


After a number of image frames, the likelihood of consistent appearance being a vehicle is compared with a threshold to decide whether the appearance trajectory represents a vehicle or non-vehicle. In accordance with the present invention, strong geometry and motion constraints are exploited to improve the reliability of the over-all detection system. Note that the use of motion information from multiple frames causes delayed decisions. However, in practice, a small number of frames can be used (e.g., <10 frames) to avoid significant delay.


In accordance with the present invention, prior knowledge of vehicle and non-vehicle appearance provides discriminant information for separating the vehicle class from the non-vehicle class. A machine learning algorithm, AdaBoost, is adopted to learn appearance priors from vehicle and non-vehicle examples. The boosting technique has been shown to be very effective in learning binary classifiers for object detection. Examples of vehicle and non-vehicle training samples are shown in FIG. 4. Images 402-410 show training samples for vehicles and images 412-420 show training samples for non-vehicles.


The appearance model is obtained from image examples through learning. In general, any learning algorithm can be used as well to construct an appearance model from image examples. Here, the Adaboost algorithm is used as an example of learning an appearance model from image examples. An image sample, denoted by I and its class label by l(lε{+1.−1)}. The method finds a highly accurate classifier H(I) by combining many classifiers {hj,(I)} with weak performance.











H


(
I
)


=

sign


(

f


(
I
)


)



;


f


(
I
)


=



j




α
j




h
j



(
I
)









(
2
)







where hj(I)ε(+1,−1)


Given a set of labeled training samples {(Ii,li)}, the Adaboost algorithm chooses {αj} by minimizing an exponential loss function Σi exp(−liΣjhj(Ii)) which is determined by the classification error on the training set. Simple image features are used to define weak classifiers. Feature values are thresholded to produce weak hypotheses. The optimal thresholds are automatically determined by the boosting algorithm. An additional procedure of joint optimization on {αj} is performed to further reduce the error rate of the final classifier. In accordance with the present invention, separate classifiers are used to classify cars and trucks from the non-vehicle class. Vehicle samples collected from traffic videos captured in various driving scenarios are used. Non-vehicle samples collected from image regions containing background clutters and extended through the bootstrap procedure are also used.


A posteriori probability can be derived from the classifier response f(I).











P
a



(

l
=

1
|
I


)


=




f


(
I
)








-

f


(
I
)




+



f


(
I
)














(
3
)








Class labels for vehicles and non-vehicles are +1 and −1 respectively. The probability term Pa in (I) can be evaluated as













k
=
1

m




P
a

(



I
k



(


x
k

,

s
k


)



vehicle

)


=




k
=
1

m






f


(


I
k



(


x
k

,

s
k


)


)







-

f


(


I
k



(


x
k

,

s
k


)


)




+



f


(


I
k



(


x
k

,

s
k


)


)










(
4
)








In general, other learning methods such as Support Vector Machines, Neural Networks can be adopted to obtain appearance models, as long as a proper probabilistic model Pa(l=1|I) is derived by these methods.


Scene context plays an important role in improving the reliability of a vehicle detection system. Strong constraints on where vehicles are likely to appear can be inferred from the knowledge of scene geometry. Through perspective projection, points in the 3D world pw are mapped to points on the 2D image plane pim.

pim=Tpw
T=TinternalTperspectiveTexternal  (5)


The entire image formation process comprises perspective projection and transformation induced by internal and external camera calibration. Assuming that a vehicle is on a flat road plane, vehicle size in the world sw is known and the internal and external camera parameters θinternalθexternal are available. Given the location of vehicle appearance on the image plane xim, the size sim of the vehicle appearance on the image plane can be easily determined as a function of xim and θ={sw, θinternal, θexternal}.

sim=g(xim,sw, θinternal, θexternal)  (6)


In practice, the flat road assumption may be violated, vehicles vary in size and the camera calibration may not be very accurate. To address such variance of the parameters, a probability model is used to characterize the geometry constraint. The conditional distribution of vehicle size sim on the image plane given its location can be modeled by a normal distribution N(•; μ,σ2) with mean μ=g(xim,sw, θinternal, θexternal) and variance σ2 determined by the variance of the parameter set θ={sw, θinternal, θexternal} and the deviation of the road surface from the planar assumption.

p(sim|xim)=N(sim; μ,σ2) μ=g(xim, θ)  (7)
σ22(xim, σθ2)

Given the geometry constraint, the likelihood of a vehicle being present at location xim with size sim on the image plane is given by

p(xim,sim)=p(xim)p(sim|xim)=c·N(sim; g(xim, θ), σ2(xim, σθ2))  (8)

where c is a constant.


A uniform distribution is assumed for the prior probability of the vehicle location xim. Consequently, the geometry model pg in (1) is formulated as















k
=
1

m




p
g



(


x
k

,

s
k


)



|

scene





geometry


)

=

κ





k
=
1

m



N


(






s
k

;

g


(


x
k

,
θ

)



,







σ
2



(


x
k

,
θ

)





)








(
9
)








Information about the road geometry can be used to refine the distribution model of xim. An example of a geometry constraint on size and location is illustrated in FIG. 5. The location and the size of vehicle appearance in the image plane is dependent. The size of a vehicle appearance in the image plane is constrained given the image location of the vehicle, and vice versa For instance, if a vehicle is observed in the lower portion of the image frame, its appearance is larger than a vehicle that appears in the upper portion of the image.


To derive the motion model in (1), the Markov property of vehicles in the image plane is assumed, i.e., given the vehicle location and size (xt,st) at time t, future location and size (xt+k,st+k) (k≧1) are independent of past observations {I1, . . . , It−1}. In accordance with this assumption, the motion model pm used in the fusion framework (1) can be written as
















p
m



(





(


x
1

,

s
1


)

,





,









(


x
m

,

s
m


)







|

I
1


,





,






I
m







)


=



p
m



(




(


x
1

,

s
1


)






|

I
1





)







k
=
1


m
-
1





p
m



(




(


x

k
+
1


,

s

k
+
1



)







|

(


x
k

,

s
k


)


,







I

k
+
1


,

I
k





)











=


c


·




k
=
1


m
-
1





p
m



(



(


x

k
+
1


,

s

k
+
1



)

|

(


x
k

,

s
k


)


,


I

k
+
1




I
k



)




















(
10
)








where c′ is a constant.


The product term pm ((xk+1,sk+1)|(xk,sk),Ik+1,Ik) represents the likelihood of a vehicle moving from location xk, size sk in frame Ik to location Xk+1, size sk+1 in frame Ik+1 given that {Ik, Ik+1} are observed.


To solve the likelihood term pm((xk+1,sk+1)|(xk,sk), Ik+1, ik) the motion estimation algorithm is extended to estimate a special form of affine motion with translation u=[ux, uy] and scaling parameter a. Under the brightness consistency assumption,

Ik(x)=Ik+1(ax+u) x=[x,y]′; u=[ux,uy]′  (11)

the optical flow equation is generalized to

[∇xTIk(xx]α+∂xIk(x)ux+∂yIk(x)uy=[∇xTIk(xx]−∂tIk(x)  (12)

where ∇xIk(x)=[∂xI(x), ∂yI(x)]′ and ∂tI(x) are spatial and temporal derivatives at image location x. An unbiased estimation of scaling and translation vector can be obtained by solving a least square problem.
















[



a
,

u
x

,

u
y




]

k


=

E


{




[

a
,

u
x

,

u
y


]

k


|

x
k


,

I

k
+
1


,

I
k


]



}






=



(


A
k




A
k


)


-
1




A
k




B
k













A
k

=

[






x
T





I
k



(

x
1

)


·

x
1









x




I
k



(

x
1

)













y




I
k



(

x
1

)





















x
T





I
k



(

x
N

)


·

x
N









x




I
k



(

x
N

)














y




I
k



(

x
N

)











]









B
k

=


[




k




I
k



(

x
1

)



,





,



x




I
k



(

x
N

)




]








(
13
)








where N is the number of pixels in the local image region used for estimation. The covariance of the unbiased estimate [a,ux,uy]′k can be derived as follows.











Cov


{


[



a
,

u
x

,

u
y




]

k


}


=




σ
^

2



(


A
k




A
k


)



-
1










σ
^

2

=


1

N
-
3









A
k

·


[

a
,



u
x

,

u
y




]

k



-

B
k




2






(
14
)








Given the vehicle location xk and size sk in the k-th frame as well as the observed image frames {Ik,Ik+1}, the vehicle location and size in the k-th frame can be estimated through the affine transform












[


x

k
+
1


,

y

k
+
1


,

s

k
+
1



]



=


C
k

·


[

a
,

u
x

,

u
y


]

k











C
k

=

[




x
k



1


0





y
k



0


1




1


0


0



]






(
15
)







E


{


[


x

k
+
1


,

y

k
+
1


,

s

k
+
1



]



}


=




C
k

·
E





{


[

a
,

u
x

,

u
y


]

k


}











Cov


{


[


x

k
+
1


,

y

k
+
1


,

s

k
+
1



]



}


=



C
k

·
Cov




{


[

a
,

u
x

,

u
y


]

k


}

·


C
k













(
16
)








Given the unbiased estimate [a,ux,uy]′k and its covariance Cov{[a,ux,uy]′k} obtained by the motion estimation algorithm, the likelihood term pm(xk+1|xk,Ik,Ik−1) can be modeled as a multivariate normal distribution.














p
m



(



(


x

k
+
1


,

s

k
+
1



)






(


x
k

,

s
k


)

,

I

k
+
1


,

I
k


)


=

N
(



(


x

k
+
1


,

s

k
+
1



)

;

μ

k
+
1



,



k
+
1



)








μ

k
+
1



=


C
k

·


[


a
,

u
x

,

u
y



]

k













k
+
1




=



C
k

·
Cov



{



[


a
,

u
x

,

u
y



]

k


·

C
k











(
17
)








Consequently, the motion model (10) is expressed as












p
m



(



(


x
1

,

s
1


)


,





,




(


x
m

,

s
m


)






I
1

,





,

I
m


)


=

κ





k
=
1


m
-
1








N


(



(


x

k
+
1


,

s

k
+
1



)





;









μ

k
+
1


,




k
+
1




)







(
18
)







In accordance with the present invention, the prior models of appearance, geometry and motion have been described as well as method for obtaining these prior models. Using these prior models, knowledge fusion is performed on the image frame level. Initially, appearance and geometry models are used to generate hypotheses of vehicle appearance. From equations (4) and (9), the likelihood of a vehicle appearance, i.e., length-1 trajectory, is given by

l1 α pg((x1,s1)|scene geometry)·Pa(I1(x1,s1)εvehicle  (19)

The initial hypotheses are pruned and trajectories of high likelihood are kept. Hypotheses are updated sequentially over time using appearance, geometry and motion information.

lk+1α lk·pm((xk+1,sk+1)|(xk,sk),Ik+1,Ik)·pg ((xk+1,sk+1)|scene geometry)·Pa(Ik+1(x+1,s+1)εvehicle)  (20)

where the trajectories are extended into a new image frame Ik+1.

(xk+1,sk+1)=argmax(x,s) pm((x,s)|(xk,sk), I+1,Ikpg((x,s)|scene geometry)·Pa(Ik+1(xk+1,sk+1)εvehicle)  (21)


For computational efficiency, trajectories with low likelihood values are terminated during the fusion process. After the information is accumulated over a number of frames, decisions are made by thresholding the likelihood values.










{


(


x
1

,

s
1


)

,





,

(


x
m

,

s
m


)


}



{







ε





vehicle









l
m

>
τ










ε





non


-


vehicle







otherwise









(
22
)







In accordance with the present invention, an example of how the method may be used will now be described. In the current example, the camera is calibrated. Examples of rear view images of cars and trucks are collected. Separate classifiers are trained to detect cars and trucks. Classifier performance is shown in FIG. 6. If 150 simple features are used, the composite error rate (i.e., miss detection rate plus false alarm rate) of the car classifier is approximately 10−4 on the training data set, and the composite error of the truck classifier is approximately 10−3 on the training data set. The number shows that truck appearance is more difficult to classify compared to cars due to different degrees of within-class variance. The number of frames used in fusion can be adjusted according to requirements on response time. During testing, a large degree of performance improvement was observed by fusing appearance, geometry and motion information. FIG. 7 shows examples of false detection eliminated by the fusion approach.


Having described embodiments for a system and method for detecting vehicles using a knowledge fusion framework, it is noted that modifications and variations can be made by persons skilled in the art in light of the above teachings. It is therefore to be understood that changes may be made in the particular embodiments of the invention disclosed which are within the scope and spirit of the invention as defined by the appended claims. Having thus described the invention with the details and particularity required by the patent laws, what is claimed and desired protected by Letters Patent is set forth in the appended claims.

Claims
  • 1. A method for on-road vehicle detection comprising the steps of: receiving a video sequence comprised of a plurality of image frames;identifying a potential vehicle appearance in an image frame;using known vehicle appearance information and scene geometry information to formulate initial hypotheses about vehicle appearance;tracking the potential vehicle appearance over multiple successive image frames;identifying potential motion trajectories for the potential vehicle appearance over the multiple image frames;applying knowledge fusion by assigning a confidence score to each trajectory of vehicle appearance, where the confidence score is defined as the product of probabilities obtained from appearance model, scene geometry model and motion model determined for each image frame containing the trajectories; anddetermining that a trajectory with a high confidence score represents a vehicle appearance.
  • 2. The method of claim 1 wherein a potential vehicle appearance is determined by hypothesis testing using a probability model.
  • 3. The method of claim 2 wherein the probability model is defined as the probability that a vehicle appears in an observed image patch.
  • 4. The method of claim 2 wherein the probability model is obtained from known vehicle and non-vehicle training samples.
  • 5. The method of claim 2 wherein the probability model is formed by using a set of image features as weak classifiers that characterize various vehicle aspects.
  • 6. The method of claim 1 wherein the scene geometry models impose strong constraints on a location of a vehicle in a given image frame.
  • 7. The method of claim 6 wherein the scene geometry model is a probability model defined as the joint probability distribution function of vehicle location and vehicle size inside an image frame.
  • 8. The method of claim 7 wherein the joint probability distribution function of vehicle location and vehicle size in a given image frame is obtained as the product of probability distribution function of vehicle location in a give image frame and conditional probability distribution function of vehicle size given its location in a given image frame.
  • 9. The method of claim 8 wherein the probability distribution function of vehicle location in a give image frame is a uniform distribution.
  • 10. The method of claim 7 wherein the vehicle size in a given image frame is a function of vehicle location in a given image frame, vehicle size in the world coordinate system, and elevation of the road surface where the vehicle touches the road.
  • 11. The method of claim 10 wherein given a vehicle location in a given image frame, the vehicle size in the world coordinate system is described as a random variable with a normal distribution, and elevation of the road surface is described as a random variable with a normal distribution.
  • 12. The method of claim 8 wherein the conditional probability distribution function of vehicle size given its location in a given image frame is defined as a normal distribution.
  • 13. The method of claim 12 wherein mean and covariance for the normal distribution are derived from mean and covariance of vehicle size in the world coordinate system and mean and covariance of elevation of the road surface where the vehicle touches the road.
  • 14. The method of claim 1 wherein the motion models impose constraints on the movement of the potential vehicle appearance in a subsequent frame.
  • 15. The method of claim 14 wherein the motion model is defined as conditional probability distribution function of vehicle location and vehicle size in an image frame given its location and size in the previous image frame and the two consecutive image frames.
  • 16. The method of claim 15 wherein the conditional probability of the conditional probability distribution function of vehicle location and vehicle size is a normal distribution.
  • 17. The method of claim 16 wherein mean of the normal distribution is an unbiased estimate of vehicle location and size given its location and size in a previous image frame and two consecutive image frames.
  • 18. The method of claim 16 wherein covariance of the normal distribution is the covariance of an unbiased estimate of the vehicle location and size given its location and size in a previous image frame and two consecutive image frames.
  • 19. The method of claim 1 wherein the video sequence is received from a camera mounted to a vehicle.
  • 20. The method of claim 1 wherein trajectories of vehicle appearance are extended into each subsequent image frames using estimated image motion of the vehicle appearance.
  • 21. The method of claim 1 wherein the step of applying knowledge fusion is applied to trajectories of vehicle appearance in each image frame of the video sequence.
  • 22. The method of claim 1 wherein the appearance, scene geometry and motion information models are used to update the confidence score for each trajectory in each subsequent image frame in the video sequence.
  • 23. The method of claim 1 wherein if the confidence score for a particular trajectory falls below a predetermined value, the particular trajectory is not tracked in subsequent image frames.
  • 24. A system for on-road vehicle detection comprises: at least one camera for capturing a video sequence of image frames of background dynamics;a processor associated with the at least one camera, the processor performing the following steps: i). identifying a potential vehicle appearance in an image frame;ii). using known vehicle appearance information and scene geometry information to formulate initial hypotheses about vehicle appearance;iii). tracking the potential vehicle appearance over multiple successive image frames;iv). identifying potential motion trajectories for the potential vehicle appearance over the multiple image frames;v). applying knowledge fusion by assigning a confidence score to each trajectory of vehicle appearance, where the confidence score is defined as the product of probabilities obtained from appearance model, scene geometry model and motion model determined for each image frame containing the trajectories;andvi). determining that a trajectory with a high confidence score represents a vehicle appearance.
  • 25. The system of claim 24 wherein a potential vehicle appearance is determined by hypothesis testing using a probability model.
  • 26. The system of claim 25 wherein the probability model is defined as the probability that a vehicle appears in an observed image patch.
  • 27. The system of claim 25 wherein the probability model is obtained from known vehicle and non-vehicle training samples.
  • 28. The system of claim 25 wherein the probability model is formed by using a set of image features as weak classifiers that characterize various vehicle aspects.
  • 29. The system of claim 24 wherein the scene geometry models impose strong constraints on a location of a vehicle in a given image frame.
  • 30. The system of claim 29 wherein the scene geometry model is a probability model defined as the joint probability distribution function of vehicle location and vehicle size inside an image frame.
  • 31. The system of claim 30 wherein the joint probability distribution function of vehicle location and vehicle size in a given image frame is obtained as the product of probability distribution function of vehicle location in a give image frame and conditional probability distribution function of vehicle size given its location in a given image frame.
  • 32. The system of claim 31 wherein the probability distribution function of vehicle location in a give image frame is a uniform distribution.
  • 33. The system of claim 31 wherein the vehicle size in a given image frame is a function of vehicle location in a given image frame, vehicle size in the world coordinate system, and elevation of the road surface where the vehicle touches the road.
  • 34. The system of claim 33 wherein given vehicle location in a given image frame, the vehicle size in the world coordinate system is described as a random variable with a normal distribution, and elevation of the road surface is described as a random variable with a normal distribution.
  • 35. The system of claim 31 wherein the conditional probability distribution function of vehicle size given its location in a given image frame is defined as a normal distribution.
  • 36. The system of claim 35 wherein mean and covariance for the normal distribution are derived from mean and covariance of vehicle size in the world coordinate system and mean and covariance of elevation of the road surface where the vehicle touches the road.
  • 37. The system of claim 24 wherein the motion models impose constraints on the movement of the potential vehicle appearance in a subsequent frame.
  • 38. The system of claim 37 wherein the motion model is defined as conditional probability distribution function of vehicle location and vehicle size in an image frame given its location and size in the previous image frame and the two consecutive image frames.
  • 39. The system of claim 38 wherein the conditional probability distribution function of vehicle location and vehicle size is a normal distribution.
  • 40. The system of claim 39 wherein mean of the normal distribution is an unbiased estimate of vehicle location and size given its location and size in a previous image frame and two consecutive image frames.
  • 41. The system of claim 39 wherein covariance of the normal distribution is the covariance of an unbiased estimate of the vehicle location and size given its location and size in a previous image frame and two consecutive image frames.
  • 42. The system of claim 24 the at least one camera is mounted to a vehicle.
  • 43. The system of claim 24 wherein trajectories of vehicle appearance are extended into each subsequent image frames using estimated image motion of the vehicle appearance.
  • 44. The system of claim 24 wherein the step of applying knowledge fusion is applied to trajectories of vehicle appearance in each image frame of the video sequence.
  • 45. The system of claim 24, wherein the appearance, scene geometry and motion information models are used to update the confidence score for each trajectory in each subsequent image frame in the video sequence.
  • 46. The system of claim 24 wherein if the confidence score for a particular trajectory falls below a predetermined value, the particular trajectory is not tracked in subsequent image frames.
CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application Ser. No. 60/637,804 filed on Dec. 21, 2004, which is incorporated by reference in its entirety.

US Referenced Citations (211)
Number Name Date Kind
4739401 Sacks et al. Apr 1988 A
4868871 Watson, III Sep 1989 A
4926346 Yokoyama May 1990 A
4931937 Kakinami et al. Jun 1990 A
4969036 Bhanu et al. Nov 1990 A
4970653 Kenue Nov 1990 A
5036474 Bhanu et al. Jul 1991 A
5159557 Ogawa Oct 1992 A
5161632 Asayama Nov 1992 A
5233541 Corwin et al. Aug 1993 A
5253050 Karasudani Oct 1993 A
5369590 Karasudani Nov 1994 A
5373456 Ferkinhoff et al. Dec 1994 A
5390133 Sohie Feb 1995 A
5410346 Saneyoshi et al. Apr 1995 A
5434927 Brady et al. Jul 1995 A
5487116 Nakano et al. Jan 1996 A
5500904 Markandey et al. Mar 1996 A
5515448 Nishitani May 1996 A
5521633 Nakajima et al. May 1996 A
5530420 Tsuchiya et al. Jun 1996 A
5530771 Maekawa Jun 1996 A
5535144 Kise Jul 1996 A
5537511 DeAngelis et al. Jul 1996 A
5555312 Shima et al. Sep 1996 A
5555555 Sato et al. Sep 1996 A
5590217 Toyama Dec 1996 A
5592567 Kilger Jan 1997 A
5600731 Sezan et al. Feb 1997 A
5612686 Takano et al. Mar 1997 A
5617085 Tsutsumi et al. Apr 1997 A
5621645 Brady Apr 1997 A
5638116 Shimoura et al. Jun 1997 A
5742699 Adkins et al. Apr 1998 A
5761326 Brady et al. Jun 1998 A
5765116 Wilson-Jones et al. Jun 1998 A
5777690 Takeda et al. Jul 1998 A
5892855 Kakinami et al. Apr 1999 A
5910817 Ohashi et al. Jun 1999 A
5929785 Satonaka Jul 1999 A
5937078 Hyland et al. Aug 1999 A
5937079 Franke Aug 1999 A
5969755 Courtney Oct 1999 A
5991428 Taniguchi Nov 1999 A
6035053 Yoshioka et al. Mar 2000 A
6044166 Bassman et al. Mar 2000 A
6072889 Deaett et al. Jun 2000 A
6122597 Saneyoshi et al. Sep 2000 A
6141435 Naoi et al. Oct 2000 A
6185314 Crabtree et al. Feb 2001 B1
6263088 Crabtree et al. Jul 2001 B1
6285393 Shimoura et al. Sep 2001 B1
6298144 Pucker et al. Oct 2001 B1
6301542 Kirchberger et al. Oct 2001 B1
6327522 Kojima et al. Dec 2001 B1
6327536 Tsuji et al. Dec 2001 B1
6370261 Hanawa Apr 2002 B1
6380934 Freeman et al. Apr 2002 B1
6430303 Naoi et al. Aug 2002 B1
6445809 Sasaki et al. Sep 2002 B1
6466684 Sasaki et al. Oct 2002 B1
6477260 Shimomura Nov 2002 B1
6531959 Nagaoka et al. Mar 2003 B1
6542621 Brill et al. Apr 2003 B1
6549642 Sakurai Apr 2003 B1
6553130 Lemelson et al. Apr 2003 B1
6556692 Gavrila Apr 2003 B1
6570998 Ohtsuka et al. May 2003 B1
6590521 Saka et al. Jul 2003 B1
6590999 Comaniciu et al. Jul 2003 B1
6594583 Ogura et al. Jul 2003 B2
6597801 Cham et al. Jul 2003 B1
6597816 Altunbasak et al. Jul 2003 B1
6628835 Brill et al. Sep 2003 B1
6636257 Harada et al. Oct 2003 B1
6643387 Sethuraman et al. Nov 2003 B1
6683969 Nishigaki et al. Jan 2004 B1
6687577 Strumolo Feb 2004 B2
6694044 Pavlovic et al. Feb 2004 B1
6704621 Stein et al. Mar 2004 B1
6718259 Khosla Apr 2004 B1
6731204 Lehmann May 2004 B2
6731777 Nishigaki et al. May 2004 B1
6734787 Ikeda May 2004 B2
6737963 Gutta et al. May 2004 B2
6744380 Imanishi et al. Jun 2004 B2
6757571 Toyama Jun 2004 B1
6765480 Tseng Jul 2004 B2
6795014 Cheong Sep 2004 B2
6813370 Arai Nov 2004 B1
6826292 Tao et al. Nov 2004 B1
6842531 Ohtsuka et al. Jan 2005 B2
6845172 Furusho Jan 2005 B2
6847894 Hasegawa Jan 2005 B1
6853738 Nishigaki et al. Feb 2005 B1
6873912 Shimomura Mar 2005 B2
6879705 Tao et al. Apr 2005 B1
6879706 Satoh et al. Apr 2005 B2
6901152 Lee et al. May 2005 B2
6906620 Nakai et al. Jun 2005 B2
6937746 Schwartz Aug 2005 B2
6944317 Pavlovic et al. Sep 2005 B2
6954544 Jepson et al. Oct 2005 B2
6963657 Nishigaki et al. Nov 2005 B1
6963661 Hattori et al. Nov 2005 B1
6973201 Colmenarez et al. Dec 2005 B1
6990216 Yamamura Jan 2006 B2
6993159 Ishii et al. Jan 2006 B1
6999600 Venetianer et al. Feb 2006 B2
7034742 Cong et al. Apr 2006 B2
7035431 Blake et al. Apr 2006 B2
7038577 Pawlicki et al. May 2006 B2
7042389 Shirai May 2006 B2
7046822 Knoeppel et al. May 2006 B1
7068815 Chang et al. Jun 2006 B2
7069130 Yopp Jun 2006 B2
7072494 Georgescu et al. Jul 2006 B2
7088846 Han et al. Aug 2006 B2
7124027 Ernst et al. Oct 2006 B1
7127083 Han et al. Oct 2006 B2
7132933 Nakai et al. Nov 2006 B2
7132959 Seabury et al. Nov 2006 B2
7149327 Okamoto et al. Dec 2006 B2
7167578 Blake et al. Jan 2007 B2
7171023 Kim et al. Jan 2007 B2
7190809 Gutta et al. Mar 2007 B2
7221777 Nagaoka et al. May 2007 B2
7224735 Porikli et al. May 2007 B2
7248718 Comaniciu et al. Jul 2007 B2
7263209 Camus et al. Aug 2007 B2
7263472 Porikli Aug 2007 B2
7266454 Takahama et al. Sep 2007 B2
7274801 Lee Sep 2007 B2
7286707 Liu et al. Oct 2007 B2
7336803 Mittal et al. Feb 2008 B2
7352880 Kim et al. Apr 2008 B2
7356408 Tsuchiya et al. Apr 2008 B2
7386163 Sabe et al. Jun 2008 B2
7397929 Nichani et al. Jul 2008 B2
7409092 Srinivasa Aug 2008 B2
7433496 Ishii et al. Oct 2008 B2
7436980 Sigal et al. Oct 2008 B2
7450736 Yang et al. Nov 2008 B2
7463754 Yang et al. Dec 2008 B2
7466841 Bahlmann et al. Dec 2008 B2
7466842 Tuzel et al. Dec 2008 B2
7486802 Hougen Feb 2009 B2
7499571 Han et al. Mar 2009 B1
7519471 Shibata et al. Apr 2009 B2
7526101 Avidan Apr 2009 B2
7542835 Takahama et al. Jun 2009 B2
7561720 Miyahara Jul 2009 B2
7561721 Miyahara Jul 2009 B2
7596243 Paniconi et al. Sep 2009 B2
20010028729 Nishigaki et al. Oct 2001 A1
20020087269 Sasaki et al. Jul 2002 A1
20020134151 Naruoka et al. Sep 2002 A1
20020159616 Ohta Oct 2002 A1
20020159627 Schneiderman et al. Oct 2002 A1
20030053658 Pavlidis Mar 2003 A1
20030053659 Pavlidis et al. Mar 2003 A1
20030055563 Lars et al. Mar 2003 A1
20030091228 Nagaoka et al. May 2003 A1
20030108220 Jepson et al. Jun 2003 A1
20030123703 Pavlidis et al. Jul 2003 A1
20030151664 Wakimoto et al. Aug 2003 A1
20030156737 Ohtsuka et al. Aug 2003 A1
20030160866 Hori et al. Aug 2003 A1
20030169340 Kamijo et al. Sep 2003 A1
20030185421 Okamoto et al. Oct 2003 A1
20030210807 Sato et al. Nov 2003 A1
20030219146 Jepson et al. Nov 2003 A1
20030235327 Srinivasa Dec 2003 A1
20040057599 Okada et al. Mar 2004 A1
20040096082 Nakai et al. May 2004 A1
20040107033 Rao et al. Jun 2004 A1
20040131233 Comaniciu et al. Jul 2004 A1
20040151342 Venetianer et al. Aug 2004 A1
20040151343 Lee Aug 2004 A1
20040183905 Comaniciu et al. Sep 2004 A1
20040197010 Lee et al. Oct 2004 A1
20040234136 Zhu et al. Nov 2004 A1
20040252863 Chang et al. Dec 2004 A1
20050002572 Saptharishi et al. Jan 2005 A1
20050004762 Takahama et al. Jan 2005 A1
20050093697 Nichani et al. May 2005 A1
20050102070 Takahama et al. May 2005 A1
20050104959 Han et al. May 2005 A1
20050104960 Han et al. May 2005 A1
20050104962 Han et al. May 2005 A1
20050125121 Isaji et al. Jun 2005 A1
20050143887 Kinoshita Jun 2005 A1
20050169501 Fujii et al. Aug 2005 A1
20050175219 Yang et al. Aug 2005 A1
20050196020 Comaniciu et al. Sep 2005 A1
20050228587 Kobayashi et al. Oct 2005 A1
20050237385 Kosaka et al. Oct 2005 A1
20050248654 Tsujino et al. Nov 2005 A1
20050285937 Porikli Dec 2005 A1
20050286767 Hager et al. Dec 2005 A1
20060002587 Takahama et al. Jan 2006 A1
20060111819 Serapio et al. May 2006 A1
20060140449 Otsuka et al. Jun 2006 A1
20060153459 Zhang et al. Jul 2006 A1
20060165277 Shan et al. Jul 2006 A1
20060171563 Takashima et al. Aug 2006 A1
20060182312 Nakai et al. Aug 2006 A1
20060262959 Tuzel et al. Nov 2006 A1
20070086621 Aggarwal et al. Apr 2007 A1
20070211917 Nakano et al. Sep 2007 A1
20080025568 Han et al. Jan 2008 A1
Related Publications (1)
Number Date Country
20060177099 A1 Aug 2006 US
Provisional Applications (1)
Number Date Country
60637804 Dec 2004 US