The present invention is directed to a system and method for vehicle detection, and more particularly, to a system and method for on-road vehicle detection using knowledge fusion.
With the decreasing cost of optical sensors and increasing computing power of microprocessors, vision-based systems have been widely accepted as an integral part of the feasible solutions to driver assistance. The ability of detecting other vehicles on the road is essential to sensing and interpreting driving environments, which enables important functions like adaptive cruise control and pre-crash sensing. Vehicle detection requires effective vision algorithms that can distinguish vehicles from complex road scenes accurately. A great challenge comes from the large variety of vehicle appearance as well as different scenarios of driving environments. Vehicles vary in size, shape and appearance, which lead to considerable amount of variance in the class of vehicle images. Illumination changes in outdoor environments introduce additional variation in vehicle appearance. Meanwhile, unpredictable traffic situations create a wide range of non-stationary backgrounds with complex clutters. Moreover, high degrees of reliability and fast processing are required for driver assistance tasks, which also increase the difficulty of the task.
Known vision techniques have been used in vehicle detection. A number of approaches use empirical knowledge about vehicle appearance, such as symmetry, horizontal and vertical occluding edges around vehicle boundaries to detect the rear-view appearance of vehicles. These methods are computationally efficient but lack robustness because the parameters (e.g., thresholds) involved in edge detection and hypothesis generation are sensitive to lighting conditions and the dynamic range in image acquisition. To achieve reliable vehicle detection, several appearance-based methods exploit machine learning and pattern classification techniques to obtain elaborated classifiers that separate the vehicle class from other image patterns. Bayesian classifiers have also been used for classification in which a mixture of Gaussian filters and histograms were used to model the class distribution of vehicles and non-vehicles. Another method uses neural network classifiers that are trained on image features obtained from local orientation coding. Still other methods use Support Vector Machines (SVMs) that are trained on wavelet features.
Many of the methods mentioned above use partial knowledge for vehicle detection. For example, appearance-based methods mainly utilize the knowledge about vehicle and non-vehicle appearance, while motion-based detectors focus on the knowledge about relative vehicle motion. To make a detection system reliable, all the available knowledge should be utilized in a principled manner. There is a need for a vehicle detection system which is capable of fusing multiple sources of data over multiple image frames in order to more consistently and more accurately detect a vehicle.
The present invention is directed to a system and method for on-road vehicle detection. A video sequence is received that is comprised of a plurality of image frames. A potential vehicle appearance is identified in an image frame. Known vehicle appearance information and scene geometry information are used to formulate initial hypotheses about vehicle appearance. The potential vehicle appearance is tracked over multiple successive image frames. Potential motion trajectories for the potential vehicle appearance are identified over the multiple image frames. Knowledge fusion of appearance, scene geometry and motion information models are applied to each image frame containing the trajectories. A confidence score is calculated for each trajectory. A trajectory with a high confidence score is determined to represent a vehicle appearance.
Preferred embodiments of the present invention will be described below in more detail, wherein like reference numerals indicate like elements, with reference to the accompanying drawings:
a and 6b illustrate empirical error rates of car classifier and truck classifier in accordance with the present invention; and
The present invention is directed to an integrated framework for on-road vehicle detection that uses knowledge fusion of appearance, scene geometry and vehicle motion.
Appearance, geometry and motion information are fused over multiple image frames. The knowledge of vehicle/non-vehicle appearance, scene geometry and vehicle motion is utilized through prior models obtained by learning, probabilistic modeling and estimation algorithms. The prior models are stored in database 108. Once a vehicle is identified at a sufficient confidence level, the vehicle is identified via an output device 106. The output device 106 provides an output signal which communicates to the user or following modules the presence of the vehicle as well as its location and size within an image frame. The output signal may be an audible signal or other type of warning signal. The output device 106 may also include a display for viewing the detected vehicles. The display provides a view of the images taken by the camera 102 which are then enhanced to indicate vehicles that have been detected and which are being tracked. The detection of a vehicle can also be incorporated with other vehicle features such as automatic cruise control and collision avoidance systems.
On-road vehicle detection is different than detecting vehicles in still images. In an on-board vision system, preceding vehicles appear in multiple image frames consistently. The information of vehicle appearance, vehicle motion as well as scene geometry can be exploited jointly to ensure robust and reliable detection. Appearance information provides strong discrimination for distinguishing vehicles from non-vehicles. Motion information has the ability of associating vehicle appearance over time. With temporal data association, detection becomes more robust against isolated errors made by appearance detectors. The knowledge about scene geometry induces strong constraints on where a vehicle on the road would appear on the image plane. Incorporating geometry information into detection can reduce certain errors such as detecting vehicles in the sky or on a tree.
In accordance with the present invention, it is important to detect consistent vehicle appearance over multiple image frames. If {I1, I2, . . . Im} denotes m consecutive image frames, and (xk, sk) is the vehicle location (xk=[x, y]k′) and size (sk) in the k-th frame, and Ik (xk, sk) as the image patch of size sk at location xk of the k-th frame (k=1, . . . , m). Essentially, {(x1, s1), . . . (xm, sm) defines a trajectory of vehicle appearance on the image plane. Given the observation of m consecutive image frames {Ik}k=1m and the knowledge of scene geometry, the likelihood of consistent appearance of an on-road vehicle on the image plane is expressed as
The first term pm((x1, s1), . . . (xm, sm)|I1, Im) defines the likelihood of the appearance of the trajectory {(x1, s1), . . . (xm, sm)} being consistent. The subscript m is used in the notation because this term incorporates motion information to determine temporal association of object appearance. The second term,
scene geometry) defines the likelihood of an on-road vehicle appearing on an admissible trajectory {(x1, s1, ), . . . (xm, sm) )} given the knowledge of scene geometry. The subscript g is used in the notation to indicate geometry information being exploited. The third term
vehicle) defines the probability that the image patches Ik(xk, sk) (k=1, . . . , m) belong to the vehicle class, where the subscript a in the notation indicates the use of appearance information.
An example of appearance trajectories is illustrated in
Using the above probabilistic formulation, an integrated framework of knowledge fusion in accordance with the present invention is shown in
After a number of image frames, the likelihood of consistent appearance being a vehicle is compared with a threshold to decide whether the appearance trajectory represents a vehicle or non-vehicle. In accordance with the present invention, strong geometry and motion constraints are exploited to improve the reliability of the over-all detection system. Note that the use of motion information from multiple frames causes delayed decisions. However, in practice, a small number of frames can be used (e.g., <10 frames) to avoid significant delay.
In accordance with the present invention, prior knowledge of vehicle and non-vehicle appearance provides discriminant information for separating the vehicle class from the non-vehicle class. A machine learning algorithm, AdaBoost, is adopted to learn appearance priors from vehicle and non-vehicle examples. The boosting technique has been shown to be very effective in learning binary classifiers for object detection. Examples of vehicle and non-vehicle training samples are shown in
The appearance model is obtained from image examples through learning. In general, any learning algorithm can be used as well to construct an appearance model from image examples. Here, the Adaboost algorithm is used as an example of learning an appearance model from image examples. An image sample, denoted by I and its class label by l(lε{+1.−1)}. The method finds a highly accurate classifier H(I) by combining many classifiers {hj,(I)} with weak performance.
where hj(I)ε(+1,−1)
Given a set of labeled training samples {(Ii, Ii)}, the Adaboost algorithm chooses {αj} by minimizing an exponential loss function Σi exp(−liΣjhj(Ii)) which is determined by the classification error on the training set. Simple image features are used to define weak classifiers. Feature values are thresholded to produce weak hypotheses. The optimal thresholds are automatically determined by the boosting algorithm. An additional procedure of joint optimization on {αj} is performed to further reduce the error rate of the final classifier. In accordance with the present invention, separate classifiers are used to classify cars and trucks from the non-vehicle class. Vehicle samples collected from traffic videos captured in various driving scenarios are used. Non-vehicle samples collected from image regions containing background clutters and extended through the bootstrap procedure are also used.
A posteriori probability can be derived from the classifier response f(I).
Class labels for vehicles and non-vehicles are +1 and −1 respectively. The probability term Pa in (I) can be evaluated as
In general, other learning methods such as Support Vector Machines, Neural Networks can be adopted to obtain appearance models, as long as a proper probabilistic model Pa(l=1|I) is derived by these methods.
Scene context plays an important role in improving the reliability of a vehicle detection system. Strong constraints on where vehicles are likely to appear can be inferred from the knowledge of scene geometry. Through perspective projection, points in the 3D world pw are mapped to points on the 2D image plane pim.
pim=TpwT=TinternalTperspectiveTexternal (5)
The entire image formation process comprises perspective projection and transformation induced by internal and external camera calibration. Assuming that a vehicle is on a flat road plane, vehicle size in the world sw is known and the internal and external camera parameters θinternalθexternal are available. Given the location of vehicle appearance on the image plane xim, the size sim of the vehicle appearance on the image plane can be easily determined as a function of xim and θ={sw, θinternal, θexternal)
sim=g(xim, sw, θinternal, θexternal) (6)
In practice, the flat road assumption may be violated, vehicles vary in size and the camera calibration may not be very accurate. To address such variance of the parameters, a probability model is used to characterize the geometry constraint. The conditional distribution of vehicle size sim on the image plane given its location can be modeled by a normal distribution N(·;μ,σ2) with mean μ=g(xim, sw, θinternal, θexternal) and variance σ2 determined by the variance of the parameter set θ={sw, θinternal, θexternal} and the deviation of the road surface from the planar assumption.
p(sim|xim)=N(sim; μ,σ2) μ=g(xim, θ) (7)
σ2=σ2(xim, σθ2)
Given the geometry constraint, the likelihood of a vehicle being present at location xim with size sim on the image plane is given by
p(xim, sim)=p(xim)p(sim|xim)=c·N(sim;g(xim, θ), σ2(xim, σθ2)) (8)
where c is a constant.
A uniform distribution is assumed for the prior probability of the vehicle location xim. Consequently, the geometry model pg in (1) is formulated as
Information about the road geometry can be used to refine the distribution model of xim. An example of a geometry constraint on size and location is illustrated in
To derive the motion model in (1), the Markov property of vehicles in the image plane is assumed, i.e., given the vehicle location and size (xt,st) at time t, future location and size (x1+k, st+k) (k≧1) are independent of past observations {I1, . . . , It−1}. In accordance with this assumption, the motion model pm used in the fusion framework (1) can be written as
where c′ is a constant.
The product term pm ((xk+1, sk+1)|(xk, sk), Ik+1, Ik) represents the likelihood of a vehicle moving from location xk, size sk in frame Ik to location Xk+1, size sk+1 in frame Ik+1 given that {Ik, Ik+1} are observed.
To solve the likelihood term pm((xk+1, sk+1,)|(xk, sk), Ik+1, ik) the motion estimation algorithm is extended to estimate a special form of affine motion with translation u=[ux, uy] and scaling parameter a. Under the brightness consistency assumption,
Ik(x)=Ik+1(ax+u) x=[x, y]′; u=[ux, uy]′ (11)
the optical flow equation is generalized to
[∇xTIk(x)·x]α+∂xIk(x)ux+∂yIk(x)uy=[∇xTIk(x)·x]−∂, Ik(x) (12)
where ∇xIk(x)=[∂yI(x), ∂yI(x)]′ and ∂,I(x) are spatial and temporal derivatives at image location x. An unbiased estimation of scaling and translation vector can be obtained by solving a least square problem.
where N is the number of pixels in the local image region used for estimation. The covariance of the unbiased estimate [a, ux, uy]′k can be derived as follows.
Given the vehicle location xk and size sk in the k-th frame as well as the observed image frames {Ik, Ik+1}, the vehicle location and size in the k-th frame can be estimated through the affine transform
Given the unbiased estimate [a,ux, uy]′k and its covariance Cov{[a,ux, uy]′k} obtained by the motion estimation algorithm, the likelihood term pm(xk+1|xk, Ik, Ik−1) can be modeled as a multivariate normal distribution.
Consequently, the motion model (10) is expressed as
In accordance with the present invention, the prior models of appearance, geometry and motion have been described as well as method for obtaining these prior models. Using these prior models, knowledge fusion is performed on the image frame level. Initially, appearance and geometry models are used to generate hypotheses of vehicle appearance. From equations (4) and (9), the likelihood of a vehicle appearance, i.e., length-1 trajectory, is given by
l1 a pg((x1, s1)|scene geometry)·Pa(I1(x1, s1)εvehicle (19)
The initial hypotheses are pruned and trajectories of high likelihood are kept. Hypotheses are updated sequentially over time using appearance, geometry and motion information.
Ik+1 a lk·pm((xk+1, sk+1)|(xk, sk),Ik+1, Ik)·p8 ((xk+1, sk+1)|scene geometry)·Pa(Ik+1(x+1, s+1)εvehicle) (20)
where the trajectories are extended into a new image frame Ik+1.
(xk+1, sk+1)=argmax(x,s) pm((x,s)|(xk, sk), I+1, Ik)·pg((x, s)|scene geometry)·Pa(Ik+1(xk+1, sk+1)εvehicle) (21)
For computational efficiency, trajectories with low likelihood values are terminated during the fusion process. After the information is accumulated over a number of frames, decisions are made by thresholding the likelihood values.
In accordance with the present invention, an example of how the method may be used will now be described. In the current example, the camera is calibrated. Examples of rear view images of cars and trucks are collected. Separate classifiers are trained to detect cars and trucks. Classifier performance is shown in FIG.6. If 150 simple features are used, the composite error rate (i.e., miss detection rate plus false alarm rate) of the car classifier is approximately 10−4 on the training data set, and the composite error of the truck classifier is approximately 10−3 on the training data set. The number shows that truck appearance is more difficult to classify compared to cars due to different degrees of within-class variance. The number of frames used in fusion can be adjusted according to requirements on response time. During testing, a large degree of performance improvement was observed by fusing appearance, geometry and motion information.
Having described embodiments for a system and method for detecting vehicles using a knowledge fusion framework, it is noted that modifications and variations can be made by persons skilled in the art in light of the above teachings. It is therefore to be understood that changes may be made in the particular embodiments of the invention disclosed which are within the scope and spirit of the invention as defined by the appended claims. Having thus described the invention with the details and particularity required by the patent laws, what is claimed and desired protected by Letters Patent is set forth in the appended claims.
This application claims the benefit of U.S. Provisional Application Ser. No. 60/637,604 filed on December 21, 2004, which is incorporated by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
60637604 | Dec 2004 | US |