1. Technical Field
The present invention relates to driver assistance systems and in particular to collision warning systems
2. Description of Related Art
During the last few years camera based driver assistance systems (DAS) have been entering the market; including lane departure warning (LDW), Automatic High-beam Control (AHC), pedestrian recognition, and forward collision warning (FCW).
A core technology behind conventional forward collision warning (FCW) camera based driver assistance systems and headway distance monitoring is vehicle detection in the image frames. Assume that reliable detection of vehicles in a single image a typical forward collision warning (FCW) system requires that a vehicle image be 13 pixels wide, then for a car of width 1.6 m, a typical camera gives initial detection at 115 m and multi-frame approval at 100 m. A narrower horizontal field of view (FOV) for the camera gives a greater detection range however; the narrower horizontal field of view (FOV) will reduce the ability to detect passing and cutting-in vehicles. A horizontal field of view (FOV) of around 40 degrees was found by Mobileye (to be almost optimal (in road tests conducted with a camera) given the image sensor resolution and dimensions. A key component of a conventional forward collision warning (FCW) algorithm is the estimation of distance from a camera and the estimation of time-to-contact/collision (TTC) from the scale change. as disclosed for example in U.S. Pat. No. 7,113,867.
Various methods are provided herein for computing a time-to-contact (TTC) of a vehicle with an object. The object includes a light source. The method is performable by a camera connectible to a processor. Multiple images of the object are captured. A spot, including an image of the light source is tracked between the images and a tracked spot is produced. Time-to-contact is computed responsive to change in brightness of the tracked spot. The time-to-contact may be computed from energy of the spot in the image. The energy may be computed by summing pixel values from a multiple pixels of the spot. The time-to-contact may be computed from change in the energy over time. A function of the energy may be fit to a function of the time-to-contact. The reciprocal of the square root of the energy may be fit to a linear function of the time-to-contact. It may be determined whether the vehicle and the object are on a collision course responsive to image motion of the tracked spot.
Various systems are provided herein for computing a time-to-contact (TTC) of a vehicle with an object. The object includes a light source. The system mountable in the vehicle includes a camera and a processor configured to capture the camera a plurality of images of the object. The processor is operable to track a spot between the images to produce a tracked spot. The spot includes an image of the light source. The processor is operable to compute time-to-contact (TTC) responsive to changes in brightness of the tracked spot. The time-to-contact may be computed from energy of the spot in the image. The energy may be computed by summing pixel values from multiple pixels of the spot. The time-to-contact may be computed from change in the energy over time. A function of the energy may be fit to a function of the time-to-contact. The camera may be a camera from a stereo pair of cameras. The time-to-contact may computed from a sensor input to the processor from a combination of the camera with a radar system. The time-to-contact may be computed from a sensor input provided to the processor from a combination of the camera with a lidar system.
The foregoing and/or other aspects will become apparent from the following detailed description when considered in conjunction with the accompanying drawing figures.
The invention is herein described, by way of example only, with reference to the accompanying drawings, wherein:
Reference will now be made in detail to features of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The features are described below to explain the present invention by referring to the figures.
Before explaining features of the invention in detail, it is to be understood that the invention is not limited in its application to the details of design and the arrangement of the components set forth in the following description or illustrated in the drawings. The invention is capable of other features or of being practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.
By way of introduction, features of the present invention are directed to an accurate computation of time-to-contact (TTC) with an object. The accurate computation of time-to-contact (TTC) may be used to provide a warning to a driver of a vehicle to prevent a collision with the object. The object may be a light source such as a taillight of a lead vehicle or a headlight of an oncoming vehicle, for example. The accurate computation may be derived from the changes in brightness of a tracked spot of the light source in an image or multiple images acquired in a forward view of the vehicle. The accurate computation may be used among other applications, for forward collision warning (FCW) and/or collision mitigation by braking (CMbB).
Reference is now made to
Although embodiments of the present invention are presented in the context of driver assistance applications, embodiments of the present invention may be equally applicable in other real time vision processing applications such as machine vision.
Reference is now made to
If t′ is the expected contact time with the object, e.g. lead vehicle 40, and time t is current time, then time-to-contact (TTC) between vehicle 18 and the object is the time interval between current time t and the expected contact time t′ and may be given by equation 1.
TTC=t′−t (1)
Distance Z to the object is given by equation 2
Z=Vrel*TTC (2)
where Vrel is the relative speed between vehicle 18 and lead vehicle 40, assuming constant relative speed.
The measured energy (e) of a taillight of in image sensor 12 may be given in equation 3, where K is some constant.
Combining equations 1-3 above gives:
for some constant A. Time-to-contact TTC may be estimated according to the square root of spot energy e (or brightness).
Referring now to
With reference to the equations above and
In practice, a dynamic threshold for computing the energy of spot 42 performs better than a fixed threshold such as the fixed threshold used to first detect spot 42.
If pixels in spot 42 are saturated, the saturated pixels do not contribute correctly to an energy estimation of spot 42. Saturated pixels may be avoided in a number of ways such as by use of:
As an example for calculating the constant A and time-to-contact TTC, the most recent 16 frames 15 may be used. For image frames 15, there are couples of energy ei, ti where the index i varies between −15 and 0. All possible pairs of frames 15 may be chosen for the 16 frames. However, it was found that pairs with consecutive image frames 15 may be discarded since it was found that consecutive image frames 15 in practice may not give accurate results. For each i, j pair of selected image frames 15, the following two equations may be solved, where Ti, Tj are time measurements and ei and ej are energy measurements for frames i and j respectively. Constant A and time offset t are estimated:
The above two equations (5) (6) are linear equations with an unknown slope A and offset t.
Ti and Tj are the corresponding times of the measurements where the time of image frame 15 (i=−15) is set to be zero.
Constant A may be extracted is by solving equations 5 and 6:
where dt is the difference between the time-to-contact TTCi for frame i and the time-to-contact TTCj for frame j.
When the history for spot 42 is less than sixteen image frames 15, fewer than sixteen frames may be used such as eight frames. However, calculating time-to-contact (TTC) with a history less than eight frames may not achieve sufficiently accurate results.
Multiple measurements may be combined together. Multiple measurements may be combined as weighted least squares where each measurement provides one equation to a large number of linear equations in the same variables slope A and offset time t. The equation for each measurement may be weighted according to when in time the measurements of time T and energy e are taken. It is advantageous to use robust estimation techniques. Iteratively re-weighted least squares may be used. The value of slope A may be computed as the most frequent in a distribution as follows:
Compute Constant A as Most Frequent in a Distribution
A histogram of the solutions for slopes A may be calculated using up to 50 bins from a minimum slope Amin to a maximum slope Amax. The slopes A received from image frames 15 which are further apart in time, may receive a higher weight and contribute more to the histogram, for instance in the following manner:
The bin with the maximal number is identified. For slope A with the maximal number, the mean offset or intercept t is calculated over the 16 data points. In the simplest form, the following equation may be solved for intercept t as a least squares.
The spread of intercept t values may then computed. A tighter spread means intercept t is calculated to a higher confidence.
The calculation of the mean offset t may be made more robust, by rejecting equations from image frames 15 which did not appear in any of the inlier pairs used to estimate A. It would also be possible to use the L1 norm or other known robust methods to improve the calculation of the mean offset t.
Offset t is computed in the time frame of frame i=−15. The subtraction T15−Ti adjusts the offset to the current ith image frame 15.
Multi-frame model
In the single frame (SF) model as described above, each image frame 15 and fifteen prior image frames 15 are used to compute slope A and intercept t as described above. A time-to-contact (TTC) warning could be triggered by a single frame (SF) model showing time-to-contact TTC below a threshold. However, a spot 42 is typically tracked for significantly more than 16 image frames 15 before the time-to-contact (TTC) becomes critical. Each image frame 15 typically gives a new single frame (SF) model. A multi-frame model (MF) accumulates information from the single frame models over time:
An example is now shown for detecting collision course, step 309 of
Otherwise, spot 42 is not on a collision course with lead vehicle 40 and tracking of spot 42 continues with step 503.
Method 501 checks whether spot 42 is predicted to leave in the y direction through the bottom of image rectangle 44, indicating a collision; rather than spot 42 leaving the side of rectangle 44 in the x direction indicating no collision. The choice of the image dimensions for the rectangle 44 is rather arbitrary. The algorithm above can be fine-tuned by adjusting rectangle 44, assuming the target taillight of lead vehicle 40 is about the same height as the host vehicle 18 headlamp. Rectangle 44 may be adjusted as follows:
Steps 1-3 above may be used define the rectangle 44 in image space. Two rectangles may be defined based on the minimum and maximum expected heights of the target taillights.
Results
Collision course algorithm 501 was applied to night time pre-crash test-track scenes using spot 42 based time-to-contact (TTC) method 301, according to an embodiment of the present invention with other methods of vehicle detection and collision warning disabled.
There were 106 relevant test cases.
For the 106 cases, a time-to-contact (TTC) signal was generated on 81 cases (77%). where all 25 misses of threat assessment had only blinking hazard lights on but with taillights of leading vehicle 40 were off.
Real World Examples
There were 35 false time-to-contact (TTC) warnings giving a Mean Time Between Failure (MTBF) of 10 hours. Of these, 13 were stationary light spots on a curved tunnel wall as shown in
As shown in
Combination with Other Driver Assistance Systems
Combination with Stereo
Consider a classic two camera stereo system with a focal length of f=950 pixels and a baseline b=0.2 m. Approaching a single light target with a closing speed of 60 km/h. A time-to-contact (TTC) of 2.5 seconds would be at a distance Z=41 meters. At that distance the stereo disparity d, would be:
In order to compute the time-to-contact (TTC) from a classic stereo system, the disparity may be calculated at multiple time steps. The change in disparity over time gives an estimate of the time-to-contact (TTC). Alternatively, disparity may be converted to distance and reason in metric space. In any case, the change in disparity over 0.25 secs will be about 1/10 of the value d computed above or about 0.46 pixels. Given the slight differences between cameras it may be often difficult to determine stereo disparity on a spot with the required accuracy.
A solution is to apply the time-to-contact (TTC) from method 301 to each spot where the disparity indicates a distance below v×T, where v is the host vehicle 18 velocity and T is some time threshold such as T=2.5 sec.
The time-to-contact (TTC) method 301 can be applied to one camera 12, to both cameras 12 individually or to both cameras 12 together, summing up the energy from the corresponding spot 42 in the left and right images.
Combination with Radar
The radar system gives accurate range and range rate information using a Doppler shift. Computing time-to-contact (TTC) from radar range and range rate of change is straightforward. However, the angular resolution of the radar may be poor and thus the lateral distance accuracy is weak. It may be advantageous to combine radar information with camera 12 information. A key task is matching radar targets and vision targets. Matching may be performed using various known techniques based on angle or based on angle plus range.
Time-to-contact (TTC) from spots 42 information enables matching using angle and time-to-contact (TTC). Both radar and vision system 16 provide a list of targets with angle and time-to-contact (TTC) value.
1. For each radar target find all vision targets that are within a given angular range.
2. Find the vision target that maximizes:
where: σ is the angle of the target found by radar, θv is the angle of the target found by vision, TTCr is the time-to-contact from radar, TTCv is the time-to-contact from spots.
σθ, σTTC are standard deviations of the difference in angle and time-to-contact as calculated respectively using radar and vision. α and β are weight constants.
After matching, the more accurate vision angle can be combined with the accurate radar range and time-to-contact (TTC) to provide improved collision course detection and forward collision warning (FCW) 22 using brightness of spots or to provide collision mitigation by braking (CMbB) 21.
Combination with Lidar
The Lidar system typically provides better angular resolution than radar and provides an accurate range but does not provide range rate directly. Range rate may be provided by differentiating distance to target Z. The same algorithm can be used as for radar for matching using angle but with different weight constants. Following the matching, the time-to-contact (TTC) values from Lidar and vision can be combined.
The term “inlier” as used herein is in the context of the estimation of parameters of a mathematical model from a set of observed data. Data whose distribution can be explained by some set of the mathematical model parameters are known as “inliers” as opposed to “outliers” which are data which does not fit the mathematical model parameters.
The term “energy” as used herein for an imaged spot of a light source in real space is proportional to the sum of the grey scale or color intensity values of the picture elements (pixels) of the imaged spot. The term “pixel value” refers to the grey scale and/or color intensity value of the picture element.
The indefinite articles “a”, “an” is used herein, such as “a light source”, “a spot” have the meaning of “one or more” that is “one or more light sources” or “one or more spots”.
Although selected features of the present invention have been shown and described, it is to be understood the present invention is not limited to the described features. Instead, it is to be appreciated that changes may be made to these features and combinations between the various embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and the equivalents thereof.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/IB2013/051166 | 2/13/2013 | WO | 00 |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO2013/121357 | 8/22/2013 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
20070152803 | Huang | Jul 2007 | A1 |
20080046181 | Koike | Feb 2008 | A1 |
20100172542 | Stein | Jul 2010 | A1 |
20120287276 | Dwivedi | Nov 2012 | A1 |
Number | Date | Country |
---|---|---|
1837803 | Sep 2007 | EP |
Entry |
---|
International Search Report and Written Opinion in corresponding International Application No. PCT/IB2013/051166. |
Number | Date | Country | |
---|---|---|---|
20150085119 A1 | Mar 2015 | US |
Number | Date | Country | |
---|---|---|---|
61598916 | Feb 2012 | US |