Patent Application
20190092340
The present invention relates generally to a vehicle vision system for a vehicle and, more particularly, to a vehicle vision system that utilizes one or more cameras at a vehicle.
Use of imaging sensors in vehicle imaging systems is common and known. Examples of such known systems are described in U.S. Pat. Nos. 5,949,331; 5,670,935 and/or 5,550,677, which are hereby incorporated herein by reference in their entireties.
The present invention provides a driver assistance system or vision system or imaging system for a vehicle that utilizes one or more camera modules or cameras (preferably one or more CMOS cameras) to capture image data representative of images exterior of the vehicle, including a camera module configured for attachment at a windshield of a vehicle equipped with the vision system. The system uses a forward viewing camera and a forward sensing non-imaging sensor and determines if the non-imaging sensor is functioning properly. If the system determines that the non-imaging sensor is not detecting an object associated with a camera-sensed vehicle (ahead of the equipped vehicle), then the system may be disabled following a period of time and if the vehicle is traveling above a threshold speed. If the system determines that the non-imaging sensor is detecting an object associated with a camera-sensed vehicle, but the angular offset between the two sensors is above a threshold level, the system may correct the offset or may disable the driver assistance system function (depending on how great the offset is).
These and other objects, advantages, purposes and features of the present invention will become apparent upon review of the following specification in conjunction with the drawings.
A vehicle vision system and/or driver assist system and/or object detection system and/or alert system operates to capture images exterior of the vehicle and may process the captured image data to display images and to detect objects at or near the vehicle and in the predicted path of the vehicle, such as to assist a driver of the vehicle in maneuvering the vehicle in a rearward direction. The vision system includes an image processor or image processing system that is operable to receive image data from one or more cameras and provide an output to a display device for displaying images representative of the captured image data. Optionally, the vision system may provide a top down or bird's eye or surround view display and may provide a displayed image that is representative of the subject vehicle, and optionally with the displayed image being customized to at least partially correspond to the actual subject vehicle.
Referring now to the drawings and the illustrative embodiments depicted therein, a vehicle 10 includes an imaging system or vision system that includes a forward viewing camera module 12 that is disposed at and views through the windshield 14 of the vehicle and captures image data of the scene exterior and forward of the vehicle (
The camera system or camera module of the present invention may utilize aspects of the systems and/or modules described in International Publication Nos. WO 2013/123161 and/or WO 2013/019795, and/or U.S. Pat. Nos. 8,256,821; 7,480,149; 7,289,037; 7,004,593; 6,824,281; 6,690,268; 6,445,287; 6,428,172; 6,420,975; 6,326,613; 6,278,377; 6,243,003; 6,250,148; 6,172,613 and/or 6,087,953, and/or U.S. Publication Nos. US-2015-0327398; US-2014-0226012 and/or US-2009-0295181, which are all hereby incorporated herein by reference in their entireties. Optionally, the vision system may include a plurality of exterior facing imaging sensors or cameras, such as a rearward facing imaging sensor or camera, a forwardly facing camera at the front of the vehicle, and sidewardly/rearwardly facing cameras at respective sides of the vehicle, which capture image data representative of the scene exterior of the vehicle.
The system also includes at least one other forward sensing sensor 16, such as a radar sensor or laser sensor or lidar sensor (such as a 3D lidar sensor capturing a three dimensional point cloud of sensing data) or ultrasonic sensor or the like. The other forward sensor captures sensing data that is processed by a data processor of the system to detect or determine the presence of objects ahead of the vehicle.
Many advanced driver assistance systems (ADAS) require information regarding the location and motion of subject and preceding vehicles and the environment of the roadway these vehicles are traveling. Determination of object location and motion may utilize various sensing technologies including, for example, imaging, light, electromagnetics, ultrasound, and/or the like. In many applications, two sensors are utilized to reduce the risk of false detections by means of redundancy. These sensors may be mounted inside and/or outside of the vehicle. Typically, a forward viewing camera is utilized in ADAS applications where forward looking sensing is required. Such a camera is typically located behind the windshield and views through the windshield of the vehicle. Radars and/or lasers are typically located in areas forward of the vehicle radiator.
Combining captured data from two sensors that are mounted at significantly different locations can be challenging. The sensor data processing needs to comprehend sensor location differences based on a vehicle reference point and occurrence of sensor misalignment. Sensor misalignment determination and correction has a great potential of occurrence for the sensor mounted in an area that can be subject to impacts, such as, for example, in front of the vehicle radiator. Such misalignment may result in incorrect association of object information derived from sensors mounted in different vehicle locations.
To avoid these situations, features that utilize sensors located in different vehicle locations require sensor offset determination and correction. Also, because these sensors may be very sensitive to angular alignment, this offset can cause errors in the measurement of the lateral position of surrounding objects and thus can lead to unintended behavior of the ADAS (which functions responsive to a determined location and movement of an object detected in a field of view of a camera and field of sensing of a non-imaging sensor), associated with not taking action when it is appropriate or a false action such as active emergency braking when it is not warranted. If the magnitude of the correction exceeds the performance capability of the sensor the associated ADAS application is disabled.
The present invention utilizes an object association algorithm which assigns objects from one sensor (such as the camera 12) to objects from another sensor (such as the non-imaging sensor 16) and integrates sensor offset compensation functionality. The compensation functionality checks if a sensor offset is present and corrects the sensor offset if necessary. Sensor misalignment is calculated and corrected by continuously calculating the mean average m of the angle measurement difference between both sensors (θ1, θ2) for the closest in-path vehicle (CIPV) according to Equation 1 below.
where n represents the number of measurements that is incorporated in the mean value, mk+1 is the mean value of the current time step, and mk represents the mean value of the previous time step.
Assuming that the average difference of angle measurements between both sensors for a sufficiently long time period is zero, in case no misalignment is present, sensor misalignment can be detected by comparing the average error to a threshold constant K1. This information may be used to disable certain functions or to compensate the sensor offset by adding the calculated offset to the sensor measurement.
After a valid sensor offset has been calculated (i.e., n>K_Fusion_Offst_TmStps and m<K_Fusion_Offst_Thr1 or Thr2), the mean angle error m and the number of measurements n are reset to 0 and the calculation is repeated. In case no valid radar object is associated with the camera CIPV, a counter is incremented. If this counter exceeds a threshold level (K_disable_Thr) and the vehicle speed is greater than a threshold speed (K_min_speed), the system is disabled. Otherwise, the algorithm checks again for a valid camera CIPV and a valid association. The Counter is reset when a radar object is associated with camera CIPV. The system is disabled because the radar is not detecting an object associated with a camera CIPV for a prolonged period of time, and thus, either the radar may be misaligned beyond a correctable state or is otherwise not functioning properly. Responsive to such determination of radar malfunctioning, the sensing system may be disabled or the driver assistance function may be disabled or the driver assistance function may only utilize processing of image data captured by the camera and may not rely on processing of sensor data sensed by the radar sensor. Similarly, if the system determines that image processing of image data captured by the camera fails to detect the presence of a vehicle or object detected via processing of sensor data sensed by the radar sensor, the system may disable the camera or image processor or the driver assistance function may only utilize processing of sensor data sensed by the radar sensor and may not rely on processing of image data captured by the camera.
Thus, the present invention is operable to disable the ADAS function or sensing system when a determination is made that there is a sensor offset that may result in erroneous detections or measurements. The system determines if there is a valid radar object detection associated with a camera-detected closest in path vehicle. If the system determines that there is no such valid radar object detection, then the sensing system (at least the radar part of the sensing system) is disabled after a threshold number of such determinations and if the vehicle speed is greater than a threshold value. If the system determines that there is such a valid radar object detection associated with a camera-sensed CIPV, but the angle error is above a threshold level, then the ADAS function may be disabled to avoid erroneous actions by the ADAS function.
The system thus uses a vision-based (camera) system and a non-vision-based (e.g., radar or lidar or the like) system. The image data captured by the camera(s) and the sensor data sensed by the radar sensor(s) are provided to a control, and the image data and sensor data may be fused at the control. The control, responsive to processing of image data captured by the camera, is operable to detect the presence of a vehicle exterior of the equipped vehicle and in the forward field of view of the camera and to determine a vision-based angle of the determined vehicle relative to the equipped vehicle. Also, the control, responsive to processing of sensor data sensed by the non-imaging sensor, is operable to detect the presence of the vehicle (the vehicle determined to be present via image processing of captured image data) exterior of the equipped vehicle and to determine a sensor-based angle of the determined vehicle relative to the equipped vehicle. The angles may represent a determined angle between a current directional heading of the vehicle and a center of the detected vehicle or any other point on the detected vehicle that can be determined by both image processing of captured image data and processing of sensed sensor data.
The control determines an angle difference between the vision-based angle (the angle of the determined vehicle relative to the equipped vehicle as determined via processing of image data captured by the camera) and the sensor-based angle (the angle of the determined vehicle relative to the equipped vehicle as determined via processing of sensor data sensed by the non-imaging sensor). The control disables a driver assistance system function at least in part responsive to the determined angle difference being greater than a disable threshold level (such as greater than, for example, a two degree difference or greater than, for example, a five degree difference or any suitable or selected threshold offset). The control may also disable the non-imaging sensor system at least in part responsive to the control not determining, when processing of sensor data sensed by the non-imaging sensor, the presence of the determined vehicle for a period of time.
In other words, if both sensing systems detect a vehicle, but indicate that the vehicle is at a different location (offset angle) relative to the equipped vehicle (so as to be at a different angle relative to the vehicle), with the different angle being greater than a disable threshold angle, then the control disables the driver assistance system function, since this is indicative of one of the systems being misaligned such that the captured image data and sensed sensor data cannot be trusted. Also, if both sensing systems detect the vehicle, but indicate that the vehicle is at a different angle relative to the vehicle, with the different angle being less than the disable threshold angle, but greater than a correction threshold angle (such as greater than about 1 degree but less than about 5 degrees or any other suitable or selected range of offset), then the system provides an offset correction (which comprise an average of the vision-based angle and the sensor-based angle over a period of time that is used for determining the location of the detected vehicle or object relative to the equipped vehicle, or optionally which may be used to adjust or calibrate the camera and/or the non-imaging sensor to accommodate or correct for the determined offset) and continues to process captured image data and sensed sensor data.
For example, if a driver assistance system function (such as automatic emergency braking or adaptive cruise control or lane departure warning or the like) is responsive to a detected vehicle and location and motion of that vehicle relative to the equipped vehicle, the system of the present invention may disable that function when the angle error or difference (between the vision-based angle and the sensor-based angle) is greater than a threshold amount, but may use an average of the vision-based angle and the sensor-based angle for determining the location of the detected vehicle for that driver assistance system function when the angle error or difference is less than the disable threshold amount. Moreover, if the non-imaging sensor fails to even detect the presence of the vehicle detected by the imaging system, then the system disables the non-imaging sensing system, since this is indicative of the non-imaging sensor not functioning properly, such that the sensed sensor data cannot be trusted.
Thus, in accordance with the present invention, image data captured by the forward viewing camera that views forward through the vehicle windshield is provided to the control, wherein, via image processing of captured image data by an image processor of the control, the control controls a plurality of driver assistance functions, including, for example, at least three selected from (i) automatic headlamp control, (ii) lane keeping assist, (iii) traffic sign recognition, (iv) forward collision warning, (v) automatic emergency braking, and (vi) pedestrian detection. For driver assistance functions such as forward collision warning, automatic emergency braking and pedestrian detection, the control preferably also receives and processes sensor data sensed by a non-imaging sensor (such as a radar sensor or a lidar sensor) of the equipped vehicle. For driver assistance functions such as automatic headlamp control, lane monitoring/lane keeping, and traffic sign recognition, these driver assistance functions are provided by image processing of captured image data without need to process non-image data sensed by any non-imaging sensor of the equipped vehicle.
In accordance with the present invention, when the control determines that the angle difference between the vision-based angle and the sensor-based angle is greater than the disable threshold level, the control disables one or more driver assistance functions (such as forward collision warning, automatic emergency braking and pedestrian detection) and preferably alerts the driver that such driver assistance functions have been disabled, so that the driver relies on his/her own driving abilities for safe maneuvering of the equipped vehicle. When the likes of an automatic emergency braking driver assistance function is disabled because of unreliability in angle measurement/estimation from the equipped vehicle to a detected other vehicle, other driver assistance functions that do not rely on or require such angle determination (such as automatic headlamp control or lane departure warning or traffic sign recognition) need not be disabled. Thus, when the data being processed may not be reliable, the system disables driver assistance functions such as forward collision warning, automatic emergency braking and pedestrian detection that rely on sensing or sensor fusion of image data and sensor data, and may not disable other driver assistance functions that do not rely on image data and sensor data.
The system may utilize sensors, such as radar or lidar sensors or the like. The sensing system may utilize aspects of the systems described in U.S. Pat. Nos. 8,027,029; 8,013,780; 6,825,455; 7,053,357; 7,408,627; 7,405,812; 7,379,163; 7,379,100; 7,375,803; 7,352,454; 7,340,077; 7,321,111; 7,310,431; 7,283,213; 7,212,663; 7,203,356; 7,176,438; 7,157,685; 6,919,549; 6,906,793; 6,876,775; 6,710,770; 6,690,354; 6,678,039; 6,674,895 and/or 6,587,186, and/or International Publication No. WO 2011/090484 and/or U.S. Publication No. US-2010-0245066 and/or U.S. provisional applications, Ser. No. 62/330,557, filed May 2, 2016, Ser. No. 62/313,279, filed Mar. 25, 2016, Ser. No. 62/303,546, filed Mar. 4, 2016, and/or Ser. No. 62/289,441, filed Feb. 1, 2016, which are hereby incorporated herein by reference in their entireties.
The camera or sensor may comprise any suitable camera or sensor. Optionally, the camera may comprise a “smart camera” that includes the imaging sensor array and associated circuitry and image processing circuitry and electrical connectors and the like as part of a camera module, such as by utilizing aspects of the vision systems described in International Publication Nos. WO 2013/081984 and/or WO 2013/081985, which are hereby incorporated herein by reference in their entireties.
The system includes an image processor operable to process image data captured by the camera or cameras, such as for detecting objects or other vehicles or pedestrians or the like in the field of view of one or more of the cameras. For example, the image processor may comprise an image processing chip selected from the EYEQ™ family of image processing chips available from Mobileye Vision Technologies Ltd. of Jerusalem, Israel, and may include object detection software (such as the types described in U.S. Pat. Nos. 7,855,755; 7,720,580 and/or 7,038,577, which are hereby incorporated herein by reference in their entireties), and may analyze image data to detect vehicles and/or other objects. Responsive to such image processing, and when an object or other vehicle is detected, the system may generate an alert to the driver of the vehicle and/or may generate an overlay at the displayed image to highlight or enhance display of the detected object or vehicle, in order to enhance the driver's awareness of the detected object or vehicle or hazardous condition during a driving maneuver of the equipped vehicle.
The vehicle may include any type of sensor or sensors, such as imaging sensors or radar sensors or lidar sensors or ladar sensors or ultrasonic sensors or the like. The imaging sensor or camera may capture image data for image processing and may comprise any suitable camera or sensing device, such as, for example, a two dimensional array of a plurality of photosensor elements arranged in at least 640 columns and 480 rows (at least a 640×480 imaging array, such as a megapixel imaging array or the like), with a respective lens focusing images onto respective portions of the array. The photosensor array may comprise a plurality of photosensor elements arranged in a photosensor array having rows and columns. Preferably, the imaging array has at least 300,000 photosensor elements or pixels, more preferably at least 500,000 photosensor elements or pixels and more preferably at least 1 million photosensor elements or pixels. The imaging array may capture color image data, such as via spectral filtering at the array, such as via an RGB (red, green and blue) filter or via a red/red complement filter or such as via an RCC (red, clear, clear) filter or the like. The logic and control circuit of the imaging sensor may function in any known manner, and the image processing and algorithmic processing may comprise any suitable means for processing the images and/or image data.
For example, the vision system and/or processing and/or camera and/or circuitry may utilize aspects described in U.S. Pat. Nos. 8,694,224; 7,005,974; 5,760,962; 5,877,897; 5,796,094; 5,949,331; 6,222,447; 6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202; 6,201,642; 6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452; 6,822,563; 6,891,563; 6,946,978; 7,859,565; 5,550,677; 5,670,935; 6,636,258; 7,145,519; 7,161,616; 7,230,640; 7,248,283; 7,295,229; 7,301,466; 7,592,928; 7,881,496; 7,720,580; 7,038,577; 6,882,287; 5,929,786 and/or 5,786,772, and/or International Publication Nos. WO 2011/028686; WO 2010/099416; WO 2012/061567; WO 2012/068331; WO 2012/075250; WO 2012/103193; WO 2012/0116043; WO 2012/0145313; WO 2012/0145501; WO 2012/145818; WO 2012/145822; WO 2012/158167; WO 2012/075250; WO 2012/0116043; WO 2012/0145501; WO 2012/154919; WO 2013/019707; WO 2013/016409; WO 2013/019795; WO 2013/067083; WO 2013/070539; WO 2013/043661; WO 2013/048994; WO 2013/063014, WO 2013/081984; WO 2013/081985; WO 2013/074604; WO 2013/086249; WO 2013/103548; WO 2013/109869; WO 2013/123161; WO 2013/126715; WO 2013/043661; WO 2013/158592 and/or WO 2014/204794, which are all hereby incorporated herein by reference in their entireties. The system may communicate with other communication systems via any suitable means, such as by utilizing aspects of the systems described in International Publication Nos. WO/2010/144900; WO 2013/043661 and/or WO 2013/081985, and/or U.S. Publication No. US-2012-0062743, which are hereby incorporated herein by reference in their entireties.
The imaging device and control and image processor and any associated illumination source, if applicable, may comprise any suitable components, and may utilize aspects of the cameras (such as various imaging sensors or imaging array sensors or cameras or the like, such as a CMOS imaging array sensor, a CCD sensor or other sensors or the like) and vision systems described in U.S. Pat. Nos. 5,760,962; 5,715,093; 6,922,292; 6,757,109; 6,717,610; 6,590,719; 6,201,642; 5,796,094; 6,559,435; 6,831,261; 6,822,563; 6,946,978; 7,720,580; 8,542,451; 7,965,336; 7,480,149; 5,550,677; 5,877,897; 6,498,620; 5,670,935; 5,796,094; 6,396,397; 6,806,452; 6,690,268; 7,005,974; 7,937,667; 7,123,168; 7,004,606; 6,946,978; 7,038,577; 6,353,392; 6,320,176; 6,313,454 and/or 6,824,281, and/or International Publication Nos. WO 2009/036176; WO 2009/046268; WO 2010/099416; WO 2011/028686 and/or WO 2013/016409, and/or U.S. Pat. Publication Nos. US 2010-0020170 and/or US-2009-0244361, which are all hereby incorporated herein by reference in their entireties.
The camera module and circuit chip or board and imaging sensor may be implemented and operated in connection with various vehicular vision-based systems, and/or may be operable utilizing the principles of such other vehicular systems, such as a vehicle headlamp control system, such as the type disclosed in U.S. Pat. Nos. 5,796,094; 6,097,023; 6,320,176; 6,559,435; 6,831,261; 7,004,606; 7,339,149 and/or 7,526,103, which are all hereby incorporated herein by reference in their entireties, a rain sensor, such as the types disclosed in commonly assigned U.S. Pat. Nos. 6,353,392; 6,313,454; 6,320,176 and/or 7,480,149, which are hereby incorporated herein by reference in their entireties, a vehicle vision system, such as a forwardly, sidewardly or rearwardly directed vehicle vision system utilizing principles disclosed in U.S. Pat. Nos. 5,550,677; 5,670,935; 5,760,962; 5,877,897; 5,949,331; 6,222,447; 6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202; 6,201,642; 6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452; 6,822,563; 6,891,563; 6,946,978 and/or 7,859,565, which are all hereby incorporated herein by reference in their entireties, a trailer hitching aid or tow check system, such as the type disclosed in U.S. Pat. No. 7,005,974, which is hereby incorporated herein by reference in its entirety, a reverse or sideward imaging system, such as for a lane change assistance system or lane departure warning system or for a blind spot or object detection system, such as imaging or detection systems of the types disclosed in U.S. Pat. Nos. 7,881,496; 7,720,580; 7,038,577; 5,929,786 and/or 5,786,772, which are hereby incorporated herein by reference in their entireties, a video device for internal cabin surveillance and/or video telephone function, such as disclosed in U.S. Pat. Nos. 5,760,962; 5,877,897; 6,690,268 and/or 7,370,983, and/or U.S. Publication No. US-2006-0050018, which are hereby incorporated herein by reference in their entireties, a traffic sign recognition system, a system for determining a distance to a leading or trailing vehicle or object, such as a system utilizing the principles disclosed in U.S. Pat. Nos. 6,396,397 and/or 7,123,168, which are hereby incorporated herein by reference in their entireties, and/or the like.
Optionally, the vision system may include a display for displaying images captured by one or more of the imaging sensors for viewing by the driver of the vehicle while the driver is normally operating the vehicle. Optionally, the vision system (utilizing the forward facing camera and a rearward facing camera and other cameras disposed at the vehicle with exterior fields of view) may be part of or may provide a display of a top-down view or birds-eye view system of the vehicle or a surround view at the vehicle, such as by utilizing aspects of the vision systems described in International Publication Nos. WO 2010/099416; WO 2011/028686; WO 2012/075250; WO 2013/019795; WO 2012/075250; WO 2012/145822; WO 2013/081985; WO 2013/086249 and/or WO 2013/109869, and/or U.S. Publication No. US-2012-0162427, which are hereby incorporated herein by reference in their entireties.
Changes and modifications in the specifically described embodiments can be carried out without departing from the principles of the invention, which is intended to be limited only by the scope of the appended claims, as interpreted according to the principles of patent law including the doctrine of equivalents.
The present application is a continuation of U.S. patent application Ser. No. 15/291,319, filed Oct. 12, 2016, now U.S. Pat. No. 10,137,904, which claims the filing benefits of U.S. provisional application Ser. No. 62/241,299, filed Oct. 14, 2015, which is hereby incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 62241299 | Oct 2015 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15291319 | Oct 2016 | US |
| Child | 16199479 | US |
