Patent Application
20110122249
The present invention relates to imaging systems for vehicles and, more particularly, to reverse aid imaging systems with a rearward facing imaging device or camera and a display. However, aspects of the present invention are equally suitable for other vehicle imaging systems, such as side object detection systems, and forward facing imaging systems and the like.
The advent of low cost, reliable imaging devices, based on a variety of silicon technologies, and in particular CMOS technology, combined with an improved cost/performance ratio for displays capable of meeting automotive specifications, and an increasing application rate of video monitor displays for automotive navigation systems or as part of the driver interface to a wide variety of vehicle systems, has lead to an increasing use of cameras or imaging sensors designed to give the driver a view of those areas around the vehicle which are not in the normal direct field of view of the driver, typically referred to as “blind spats”. These areas include the region close to the front Of the vehicle, typically obscured by the forward structure of the vehicle, the region along the passenger side of the vehicle, the region along the driver side of the vehicle rearward of the driver, and the area immediately rearward of the vehicle which cannot be seen directly or indirectly through the rear view Mirror system. The camera or imaging sensor may capture an image of the rearward (or sideward or other blind spot area) field of view, and the image may be displayed to the driver of the vehicle to assist the driver in backing up or reversing or otherwise driving or maneuvering the vehicle. The use of electronic cameras in these applications significantly increases the driver's knowledge of the space immediately surrounding the vehicle, which may be of importance prior to and during low speed maneuvers, and thus contributes to the safe completion of such maneuvers. However, in order to provide user controls or user inputs to allow the driver or user to control various functions of camera at the rear or side or front of the vehicle, additional wiring or connections or communication links may be needed between a control and user input in the vehicle and the camera or imaging device at the rear or side or front of the vehicle.
It is known to provide a headlamp control system having an imaging sensor positioned on a vehicle and having a forward field of view. The system may detect headlamps and taillights in the images captured by the imaging sensor. It is also known to provide a lane departure warning system that includes an imaging sensor positioned on a vehicle and having a forward field of view. The lane departure warning system detects lane markers and the like along the road surface in front of the vehicle and determines when the vehicle is drifting out of the lane, and may provide an alert to the driver of the vehicle when such drifting is detected. Such systems typically are separate systems with different, independently operable controls and image processors and imaging sensors.
The present invention provides an imaging and display system or vision system for a vehicle that captures images of a scene occurring exteriorly of the vehicle, such as rearward of the vehicle, and displays the captured images at a display device in the vehicle. The imaging and display or vision system includes an imaging device or camera that is positioned at the vehicle with an exterior field of view (such as a rearward field of view) for capturing images of the exterior scene. The display device is positioned within the vehicle and remote from the camera and is operable to display the captured images in the vehicle where they are readily viewable by the driver or occupant of the vehicle. The vision system includes one or more user inputs at the display device and is operable to adjust or control at least one function or mode or feature of the camera in response to actuation of the user input or inputs. The vision system is operable to communicate imaging signals from the imaging device to the display device and to communicate camera control signals from the display device to the imaging device along common connections or wiring or communication links between the imaging device and the display device.
According to an aspect of the present invention, a vision system for a vehicle includes an imaging device having an imaging sensor, a camera microcontroller, a display device having a display element, a display microcontroller, and at least one user input selectively actuatable by a user. The user input is selectively actuatable by a user. The imaging device communicates an image signal to the display device via a communication link. The display microcontroller affects the image signal in response to the at least one user input. The camera microcontroller monitors the image signal on the communication link and adjusts a function of the imaging device in response to a detection of the affected image signal.
The imaging sensor may have a field of view exteriorly of the vehicle, such as rearwardly of the vehicle, for capturing an image of the scene occurring exteriorly of the vehicle. The vision system may include control circuitry at or near or associated with an interior rearview mirror assembly of the vehicle or a windshield electronics module or accessory module of the vehicle,
The display microcontroller may affect the image signal by disabling a video termination at the display device, and the imaging device may stop communicating the image signal when the video termination is disabled. The display microcontroller may selectively apply at least two voltage levels across the communication link in response to at least two user inputs, and the camera microcontroller may adjust a function of the imaging device in response to a respective one Of the voltage levels.
The imaging device may include a video encoder that communicates the image signal to the display device via the communication link. The communication link may comprise a video plus and a video return wire.
According to another aspect of the present invention, a vision system for a vehicle includes an imaging device having an imaging sensor, a camera microcontroller, a display device having a display element, a display microcontroller, and at least one user input selectively actuatable by a user. The imaging device communicates an image signal to the display device via a communication link, and the display microcontroller communicates a control signal to the imaging device via the communication link in response to the user input. The camera microcontroller receives the control signal and adjusts a function of the imaging device in response to the control signal. The image signal and the control signal utilize a common link between the imaging device and the display device.
According to another aspect of the present invention, an imaging and display system or vision system may be operable in combination with or in conjunction with an-auxiliary or triggering or initial sensing device or system or distance sensing/measuring/determining system. The auxiliary sensing device or system may provide additional sensing areas to cover blind spots that may not be encompassed by the imaging sensor or camera. Optionally, the auxiliary sensing system, such as ultrasonic sensors, radar, lidar, and the like, may detect an object exteriorly of the vehicle, and may determine or measure or detect the distance to the object, whereby the imaging and display system may be adjusted or controlled to provide enhanced imaging of the exterior scene or enhanced processing of the captured images or enhanced displaying of the images of the exterior scene and the detected object, in response to such an object detection by the auxiliary sensing system. For example, the imaging and display system may adjust a camera setting to provide enhanced imaging of the detected object, or may adjust the processor or control to provide enhanced processing of a portion of the image data that corresponds to the region of the scene at which the object was detected, or may adjust the display to highlight or enhance the displayed images of the detected object or of the region of the display that corresponds to the region of the exterior scene at which the object was detected.
Therefore, the present invention provides an imaging and display system or vision system that includes an imaging device and a display device connected via video connections or communication links. The imaging device includes user inputs and the imaging and display system is operable to selectively control or adjust the imaging device in response to the user inputs. The imaging and display system or vision system is operable to communicate imaging signals from the imaging device to the display device and to communicate camera control signals from the display device to the imaging device along common connections or wiring or communication links between the imaging device and the display device. The imaging and display system or vision system thus may provide various optional features or functions or modes without having to change or add wiring or connections between the imaging device and the display device.
The present invention also provides an imaging system or vision system for a vehicle that is operable to capture images of a scene occurring exteriorly of the vehicle, such as forward of the vehicle, and that captures and/or processes selective image data differently than other image data to extract and analyze the desired data for different applications or systems or accessories. The imaging system includes an imaging device or camera that is positioned at the vehicle with an exterior field of view (such as a forward field of view) for capturing images of the exterior scene. The imaging system may be operable to selectively or intermittently capture and/or process some image data in a manner suitable for a lane detection and/or to capture and/or process other image data in a Manner suitable for headlamp detection and/or to capture and/or process other image data in a manner suitable for ambient light detection and/or other functions, systems or features or the like.
According to another aspect of the present invention, an imaging system for a vehicle includes an image sensor and a control. The image sensor is positioned at a vehicle and has an exterior field of view. The image sensor is operable to capture images of the exterior field of view and to generate image data. The control is operable to adjust the image sensor to at least two settings, and to process the image data via at least two processing techniques. The control is operable to synchronize the image sensor settings and the processing techniques to extract respective or appropriate information from the captured images for at least two applications of the imaging system.
The image sensor may have a field of view forwardly of the vehicle for capturing an image of the scene occurring forwardly of the vehicle.
The control may adjust the image sensor setting to capture image data suitable for a particular application, and may process those captured images via a processing technique suitable for the particular application. The control may adjust the image sensor to other settings and may correspondingly process those captured images via other processing techniques depending on the desired or appropriate or particular or respective application or function of the imaging system. For example, the control may selectively or intermittently or occasionally adjust the image sensor setting and processing technique to extract information froth the captured image data for a headlamp control and/or a lane departure warning system and/or a rain sensor and/or a navigational system and/or an ambient light sensor and/or a collision avoidance system and/or a driving separation indicator and/or a back up aid and/or an object detection system and/or the like.
Therefore, the present invention also provides an imaging system or vision system that includes an imaging sensor and a control that are operable to provide multiple functions or control of multiple accessories or the like via selective capturing of image data and processing of the captured image data. The control is thus operable to selectively capture images and process image data to provide multiple functions or applications with a common image sensor and image processor. The imaging system may synchronize the settings of the image sewer with the processing techniques applied to the image data by the image processor, in order to perform and/or optimize two or more functions of the imaging system. The present invention thus provides a multitasking capability to an image sensor and image processor while providing enhanced capturing of the images and processing of the image data for the desired functions or applications.
These and other objects, advantages, purposes and features of the present invention will become apparent upon review of the following specification in conjunction with the drawings.
Referring now to the drawings and the illustrative embodiments depicted therein, an image capture system or vision system or imaging and display system 10 is positioned at an exterior portion 12a of a vehicle 12, such as at a rearward portion 12a of the vehicle 12, and is operable to capture an image of a scene occurring exteriorly of the vehicle, such as rearwardly of the vehicle, and to display the image at a display device or display system 14 of the vehicle which is viewable by a driver of the vehicle (
Imaging system or imaging device 16 may be positioned at the exterior portion of the vehicle and directed generally exteriorly of the vehicle for capturing images of the exterior scene to assist the driver in maneuvering or driving the vehicle. Vision system 10 and/or imaging system 16 may utilize principles of other vehicle vision or imaging systems, such as a vision or imaging system of the types disclosed in U.S. Pat. Nos. 6,757,109; 6,717,610; 6,396,397; 6,201,642; 5,550,6′77; 5,670,935; 5,877,897; 5,796,094; 6,097,023; and 6,498,620, and U.S. patent applications, Ser. No. 09/441,341, filed Nov. 16, 1999 by Schofield et al. for VEHICLE HEADLIGHT CONTROL USING IMAGING SENSOR, now U.S. Pat. No. 7,339,149; and/or U.S. provisional applications, Ser. No. 60/614,644, filed Sep. 30, 2004 by Camilleri et al. for CAMERA DISPLAY SYSTEM; and Ser. No. 60/618,686, filed Oct. 14, 2004 by Laubinger for VEHICLE IMAGING SYSTEM, which are hereby incorporated herein by reference. In the illustrated embodiment, the vision system is operable to capture and display images of the rearward area immediately behind the vehicle, so as to assist the driver in backing up or otherwise driving or maneuvering the vehicle rearwardly. However, the vision system may be operable to capture and display images of other areas exteriorly of the vehicle to provide images of other blind, spot areas around or near the vehicle, without affecting the scope of the present invention. The vision system thus may be operable to captures images of the scene immediately rearward of the vehicle to assist the driver of the vehicle in backing up or maneuvering the Vehicle in reverse. The back up assist system may be operable in response to the reverse gear of the vehicle being selected.
Preferably, the display element or display device May be located at an upper windshield area of the vehicle so as to be readily viewable by the driver of the vehicle. The display element may be located at or near the interior rearview mirror assembly of the vehicle or may be associated with the interior rearview mirror assembly of the vehicle. Optionally, the display element may be located at or near an accessory module or pod or windshield electronics module of the vehicle or may be associated with the accessory module or pod or windshield electronics module of the vehicle.
The imaging system may process the captured image data to detect objects or the like in the exterior scene. Such data processing may utilize aspects of the image processing techniques described in U.S. Pat. Nos. 6,353,392 and 6,313,454; and/or U.S. patent application Ser. No. 10/427,051, filed Apr. 30, 2003 by Pawlicki et al. for OBJECT DETECTION SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,038,577; and/or U.S. provisional application, Ser. No. 60/638,687, filed Dec. 23, 2004 by Higgins-Luthman for OBJECT DETECTION SYSTEM FOR VEHICLE, which are hereby incorporated herein by reference. The display device or element may display the image of the scene or may display the detected objects and/or may display graphic overlays or the like (such as distance measurements, icons, text or the like) to enhance the display for viewing by the driver or occupant of the vehicle. The vision system may he selectively operable to provide different functions or modes or features as selected by the user via one or more user inputs, as discussed below.
The imaging system 16 of vision system 10 thus may be operable in two or more modes and/or may provide various functions or features that are selectable by a user of the system (such as the driver of the vehicle) via one or more user inputs at or near or associated with the display system. For example, the imaging system may be selectively operable in a color mode or a black and white mode or a daytime or nighttime mode, or may be selectively operable to zoom in on a detected object or to pan across a scene or to adjust the gain, exposure and/or white balance in the captured images, in response to actuation of a respective user input by a user within the vehicle. Optionally, the imaging system may be selectively operable to process the images and display the images of the exterior scene in an outline form, where only the outlines of detected objects are shown at the display element, such as in a black and white line format. The image thus may be manipulated to provide an outline form and thus may be suitable for display on a low cost display element, such as a dot matrix display or the like, such as the type described in U.S. patent application Ser. No. 10/418,486, filed Apr. 18, 2003 by McMahon et al. for VEHICLE IMAGING SYSTEM, now U.S. Pat. No. 7,005,974, which is hereby incorporated herein by reference. Also, or otherwise, the vision system may provide user inputs that enable a user to selectively actuate an overlay image to overlay images, such as distance measurements, text, icons, an image representation of the back of the vehicle, or other overlay images or icons, onto or over the image of the exterior scene, in order to enhance the display of the exterior scene for the user.
As shown in
As shown in
Display system or display device 14 includes a display element 28 and a display microcontroller 34. The display system 14 receives the encoded video signal 26a from the imaging system 16 via the video+ and video return connections 20e; 20d, and displays the captured image on the display element 28, such as an LCD panel or the like (or other types of displays or display elements as discussed below). As shown in
Display system 14 also includes display microcontroller 34 that is operable to receive a user input signal 18a from one or more user inputs 18 and to adjust or control the display element 28 and/or to enable or disable the video termination 30 in response to the user input signal 18a. When the video termination is disabled, the camera microcontroller 24 detects the change in the signal along the video+ and video return connections 20c, 20d, and may adjust or control the imager 22 and/or video encoder 26 in response to such detection.
During normal operation of the imaging system, the imaging system 16 may output the NTSC video signal (or-any other video signal protocol whether analog or digital) on the video+ and video− or return connections to the display system 14. The camera microcontroller 24 may continuously or substantially continuously monitor the NTSC video output signal 26a for a normal output that is about one volt peak into a 75 ohm load (provided at the video buffer and termination 30). When an appropriate user input [such as a button or switch or the like at or near or associated with the display element or a voice command or a consequent of a user action (such as selection of a reverse gear of the vehicle or the like) or a vehicle action or condition or characteristic or status (such as in response to the vehicle reaching a particular or threshold forward or reverse vehicle speed or the vehicle encountering a braking condition or the like) or other input or condition or characteristic or status or the like] is activated by a user action (such as a user input for selectively adjusting a camera function or mode, such as a digital zoom function or pan function or and adjustment of a gain or exposure or white balance function or a selection of a color mode or a black and white mode or a night vision mode or the like), the display microcontroller 34 disables the video termination 30 at the display system 14 via a signal 34a. The disabling of the video termination 30 at the display system 14 causes the video encoder 26 of the imaging system 16 to stop outputting the NTSC video signal 26a. The camera microcontroller 24 senses the change in the output signal or voltage along the connections 20c, 20d, and may control or adjust the imager to a different or new camera setting in response to such a detected change in signal output/voltage, or may control or adjust the video encoder if appropriate in response to such a detected change in the video signal along the connections 20c, 20d.
Optionally, when multiple functions or modes are desired, the display microcontroller 34 may disable the video termination and apply a particular voltage across the connections 20e, 20d (via signal 34a) that corresponds to the particular function or adjustment selected by the user at the user inputs, and the camera microcontroller 24 may be operable to determine the voltage applied on the connections and to adjust or control the imager and/or encoder accordingly. The camera microcontroller 24 may be preset or programmed to recognize the particular voltage across the connections and to control or adjust or activate the respective function or feature or mode in response to such detection and recognition. The camera adjustments or settings (such as activation of a digital zoom feature, a panning feature, a gain or exposure or white balance function or a black and white mode or night vision mode or the like) thus may be predetermined and based on the DC voltage level applied across the connections 20c, 20d by the display microcontroller 34 and sensed across the connections 20c, 20d by the camera microcontroller 24.
After the camera adjustment is made by the camera microcontroller, the display microcontroller may enable the video termination and remove the applied DC voltage if applicable, so that the encoder 26 will again communicate the video signal 26a to the display system 14. Optionally, the display microcontroller 34 may wait a predetermined amount of time before enabling the video termination again and removing the DC voltage level. This delay allows the camera microcontroller enough time to sense that the termination has been removed, that the new programmed settings have been uploaded to the imager, and that the settings have taken affect: Once the delay is completed, the video termination is enabled at the display, and the camera microcontroller and video encoder resume the NTSC signal to the display system to communicate the capture image data to the display system.
This process may be repeated for invoking and removal of special camera modes or functions. For example, for different user inputs and associated functions, the display microcontroller may disable the video termination and may apply a different voltage to the connections 20c, 20d. The camera microcontroller may then determine the voltage in the connections 20c, 20d and may adjust the imager and/or video encoder appropriately and in response to the detected voltage. The number of special modes of operation or functions that are desired to control or activate/deactivate at the display will dictate the number of voltages that are to be resolved by the camera microcontroller at the camera in order to activate or control or adjust the multiple camera settings from the display inputs. For example, if four different functions or modes and associated user inputs were desired, the display microcontroller may apply different voltages, such as at ¼ volt increments or the like, for the different inputs and functions, and the camera microcontroller may be operable to discern or resolve the different voltages and control or adjust the imager and/or video decoder according to the detected voltage.
The vision system of the present invention thus provides user selectable control or adjustment of an exterior facing camera via user inputs at an interior display of the vehicle, without requiring additional wiring for the connection between the inputs and the camera. The present invention thus is highly suitable for retrofitting imaging and display systems or vision systems and/or selecting an appropriate or desired camera and/or display device or system that provides the desired features without concerns as to the wiring or connections between the camera and imaging system and the display element and display system. A user may select an optional camera that has desired features and may install the camera at the vehicle and connect to the existing wiring, which may be the base wiring for a base camera that may not have the features of the new camera. The present invention thus provides for upgrading of a camera and/or display and/or vision system either as an option installed at the vehicle assembly plant, or as an aftermarket device, without the cost and difficulties associated with rewiring or wiring new connectors or links between a user input or inputs and the camera. Different optional cameras and display systems and imaging systems thus be selected by an owner of a vehicle and may be readily installed in the vehicle without having to re-wire the vehicle or provide complicated communication links between the camera and the display device or system. The present invention thus allows for selection of a camera and display device or system having the desired features or content, without the costs and difficulties of implementing the high content cameras, since any selected camera and display device or system may utilize the same connections or communication links.
Optionally, the vision system may include or may be associated with an ultrasonic or radar device that determines the distance from the rear of the vehicle to a detected object. Optionally, other means for determining the distance to a detected object may be implemented, such as the means described in U.S. patent applications, Ser. No. 10/427,051, filed Apr. 30, 2003 by Pawlicki et al. for OBJECT DETECTION SYSTEM FOR VEHICLE, now U.S. Pat. No 7,038,577; and/or Ser. No. 11/105,757, filed Apr. 14, 2005 by Schofield et al. for IMAGING SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,526,103; and/or. U.S. provisional application, Ser. No. 60/607,963, filed Sep. 8, 2004 by Schofield for IMAGING SYSTEM FOR VEHICLE, which are hereby incorporated herein by reference. The vision system may detect the distance to objects in various zones or regions behind the vehicle and may provide a graphic or video overlay of the displayed image to indicate to the driver the distance to one or more detected objects. The overlay may be enhanced, such as by flashing or the like, to enhance the viewability of the overlay, such as when a hazardous condition is encountered, such as when the vehicle is within a threshold distance to a detected object rearward of the vehicle and in the vehicle's path of travel. The graphic overlay may be initiated or activated in response to the vehicle being shifted into the reverse gear or may be selectively activated via a user input or the like, without affecting the scope of the present invention.
Optionally, the imaging system and/or camera and/or display system may be operable in conjunction with or in combination with a distance sensing/measuring/determining system, such as an ultrasonic sensing system or the like. For example, and with reference to
Optionally, an ultrasonic sensing system may include a plurality of sensors, such as three or more ultrasonic sensors, positioned and spaced across a rear portion of the vehicle, such as along a rear fender of the vehicle, or along a rear lift gate of the vehicle, or along a rear spoiler of the vehicle, or along a light bar associated with the license plate of the vehicle or the like. For example, and as shown in
The ultrasonic sensors 36, 40 may provide adjacent and overlapping sensing regions to provide a sensing range that encompasses substantially the entire region behind the vehicle and about eight feet or thereabouts rearward of the vehicle. The imaging sensor or camera 16′ may capture images of the rearward exterior scene that is encompassed by the field of view of the camera, and the display may display images of the scene for viewing by the driver of the vehicle when the vehicle is backing up. The display may be adjusted or modified or enhanced in one or more of the regions associated with one or more of the ultrasonic sensors in response to a detection of an object by one or more of the ultrasonic sensors.
For example, the display may be divided into four zones, each zone of the display corresponding to a zone or region covered by a respective ultrasonic sensor. If that ultrasonic sensor detects an object, then the corresponding zone of the display may be adjusted or enhanced to indicate to the driver of the vehicle that an object is detected in a particular zone or region rearward of the vehicle. For example, one or more of the zones in the display may flash or modulate or may change intensity to indicate that an object has been detected in the region or zone rearward of the vehicle that corresponds to the enhanced or flashing or modulating display zone. Optionally, the display may provide a graphic overlay, such as a measurement bar or the like, to show a distance to a detected object.
Optionally, the ultrasonic sensor or auxiliary sensing device may provide a triggering or initial sensing function, and the imaging system may adjust a camera setting or characteristic in response to a detection of an object by one or more of the ultrasonic sensors. For example, the camera may be controlled or adjusted to electronically zoom to the appropriate zone or region of the captured image in response to a detection of an object by a respective ultrasonic sensor. Because the captured image may have distortion due to the wide angle lens or optic of the imaging system and camera, it may be desirable to adjust the imaging sensor or camera to provide enhanced imaging of an object detected in the field of view. The camera thus may be zoomed in on the appropriate zone or region to provide enhanced imaging of that zone or region and to provide reduced distortion of that zone or region and, thus, enhanced imaging and reduced distortion of the detected object when it is displayed on the display. The imaging and display system may optionally provide a dynamic overlay at the display to highlight the detected object in the display, and may display the distance (which may be detected or determined in response to the ultrasonic sensing devices) to the object.
Optionally, the imaging system may process particular zones or regions of the image data corresponding to the exterior scene more than other zones or regions, such as by utilizing aspects of the object detection system described in U.S. patent applications, Ser. No. 10/427,051, filed Apr. 30, 2003 by Pawlicki et al. for OBJECT DETECTION SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,038,577; and/or Ser. No. 11/105,757, filed Apr. 14, 2005 by Schofield et al. for IMAGING SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,526,103; and/or U.S. provisional application, Ser. No. 60/607,963, filed Sep. 8, 2004 by Schofield for IMAGING SYSTEM FOR VEHICLE, which are hereby incorporated herein by reference. Optionally, the imaging system may process particular zones or regions in response to a detection of an object by one or more of the ultrasonic sensors. For example, if one of the ultrasonic sensors initially detects an object at the left rear corner of the vehicle, the control may process the image data that corresponds to that area or region, or may enhance the processing of the image data that corresponds to that area or region, in order to identify or classify or detect the detected object in that area or region. The imaging and display system may then highlight or enhance the display at the detected object, such as by a graphic overlay at the display or an adjustment or modulation of the display in the region of the detected object. The ultrasonic sensors thus may function to detect an object and to direct the image processor to process the image data that corresponds to the area in which the object was detected. The image processor thus may process the image data for the regions that may have another therein, and may provide reduced processing of other regions, and thus may provide efficient and enhanced processing of the captured images.
Optionally, the display may be actuated in response to a detection of an object behind the vehicle when the vehicle is backing up or in reverse. For example, when a driver shifts the vehicle into reverse, the ultrasonic sensors at the rear portion of the vehicle may be activated to sense the area rearward of the vehicle. The imaging and display system may monitor the ultrasonic sensors or may receive an input from the ultrasonic sensors or system, and may initially be in a non-active or non-display mode, where the display is deactivated. When an object rearward of the vehicle is detected by the ultrasonic sensors, the detection by the ultrasonic sensor or sensors may trigger the imaging and display system to be activated or set to an active mode, whereby the imaging sensor may begin capturing images of the rearward scene and the display may begin displaying images of the rearward scene (which may include enhanced imaging or processing or display or highlighting of the detected object such as discussed above). The activation of the display may provide an alert function to the driver of the vehicle that an object is detected behind the vehicle and/or in the path of the vehicle, so as to draw the driver's attention to the activated display and to the object being displayed in the display.
Optionally, the imaging and display system and ultrasonic sensing system may be implemented in conjunction with a movable video display, such as a slide out or extendable/retractable display screen that extends and retracts from the interior rearview mirror assembly or from an accessory module or overhead console or the like within the vehicle cabin and in the field of view of the driver of the vehicle. For example, the video display may comprise a slide out video display such as described in PCT Application No. PCT/US03/40611, filed Dec. 19, 2003 by Donnelly Corp. et al. for ACCESSORY SYSTEM FOR VEHICLE, and published on Jul. 15, 2004 as International Publication No. WO 2004/058540 A3; and/or U.S. provisional applications, Ser. No. 60/630,061, filed Nov. 22, 2004 by Lynam et al. for MIRROR ASSEMBLY WITH VIDEO DISPLAY; Ser. No. 60/667,048, filed Mar. 31, 2005 by Lynam et al. for MIRROR ASSEMBLY WITH VIDEO DISPLAY, which are hereby incorporated herein by reference. The display screen thus may be automatically extended and activated in response to a detection of an object rearward of the vehicle by the ultrasonic sensors, whereby the extension of the display screen provides an alert to the driver so that the driver's attention is readily drawn to the now extended and activated display screen. The imaging and display system and ultrasonic sensing system of the present invention thus may provide a display only when there is an object detected and may provide an alert function to the driver to view the display when an object is detected. In such an application, the video display may not be extended and be activated in response to the shifting into reverse, but would extend and activate after detection of an object exteriorly or rearward of the vehicle by the auxiliary sensor or sensors.
Although described above as ultrasonic sensors or sensing devices, it is envisioned that the imaging and display system of the present invention may be combined with other types of sensing systems or auxiliary or secondary or triggering or initial sensing devices or sensing techniques. For example, the imaging and display system may be operable in conjunction with or in combination with radar devices, sonar devices, laser sensing or lidar devices or laser scanning devices or the like, without affecting the scope of the present invention. The initial or triggering or auxiliary sensing device or devices may initiate or trigger enhanced imaging or zooming or other characteristics of the camera, or may initiate or trigger enhanced processing of the captured images, or may initiate or trigger enhanced display features or the like, in response to a detection of an object rearward of the vehicle by one of the initial or triggering or auxiliary sensing devices. Although shown and described as being combined with or operating in conjunction with a rearward facing camera and associated display, it is envisioned that aspects of the auxiliary sensing devices or system of the present invention may be equally suitable for use in conjunction with sideward facing imaging systems, such as side object detection systems or lane change assist systems or the like, or forward facing imaging systems, such as lane departure warning systems, adaptive speed control, headlamp controls, rain sensors or the like.
Optionally, and desirably, the image capture device or imaging device may be at least partially contained within an imaging module or camera Module, which includes imaging sensor or imager and a lens positioned within a housing which defines a transparent window (which may comprise an at least substantially transparent glass or polycarbonate or acrylic (or other suitable material) window or panel) at the end of lens (such as described in PCT Application No. PCT/US2003/036177, filed. Nov. 14, 2003 by Donnelly Corp, for IMAGING SYSTEM FOR VEHICLE, and published Jun. 3, 2004 as International Publication No. WO 2004/047421 A2, which is hereby incorporated herein by reference). The imaging module may include the circuitry and controls or camera microcontroller and video encoder for the imaging sensor, such as on one or more printed circuit boards contained within the housing.
Optionally, the camera module may comprise a fully sealed module, which protects the imaging device and microcontroller and circuitry from exposure to the elements at the exterior of the vehicle, such as dust, dirt, mud, water; ice, salt and the like. The lens may provide part of the sealing of the Module. For example, the camera module may include a lens that may be purchased or obtained as a subassembly that is separate from the imaging sensor and mounted to the module. The lens may loaded into the module, such as via insertion of the lens or threading of the lens into an aperture or opening in the module wall or adjacent to a transparent cap or cover of the module wall. The lens assembly (which is typically a five element lens set, but may include more or less optic elements without affecting the scope of the present invention) may be positioned at the opening or aperture or cover such that the lens is positioned at the appropriate or precise location relative to the imaging sensor or chip to properly focus the image onto the imaging sensor.
Optionally, the lens may be adjustably positioned, such as via threading into a threaded opening, to precisely position the lens to focus the image onto the imaging sensor. Once positioned at the precise or appropriate location, the lens may be retained or secured in the precise position, and the module may then be sealed with a back plate and gasket to seal the lens and imaging sensor and associated microprocessor and circuitry within the module. The module thus provides enhanced positioning of the lens relative to the imaging sensor, since the lens may be readily adjusted to provide proper focusing relative to a fixed imaging sensor.
The module may include a transparent cap or cover through which the field of view of the imaging sensor and lens is directed. The transparent cap may comprise a molded polycarbonate material or the like and may provide a substantially transparent and durable cover at the lens. Optionally, the transparent cap may include an anti-reflective coating or a hydrophobic or hydrophilic coating or the like, such as the coatings described in PCT Application No. PCT/US2003/036177, filed Nov. 14, 2003 by Donnelly Corp. for IMAGING SYSTEM FOR VEHICLE, and published Jun. 3, 2004 as International Publication No. WO 2004/047421 A2, which is hereby incorporated herein by reference. The transparent cap thus may provide functionality and sealing to the camera module.
Because the lens assembly may be selected and positioned at the imaging sensor as a separate sub-assembly or as an aftermarket sub-assembly, the lens assembly may be selected to provide the desired effect depending on the particular application of the camera module. The lens may then be assembled to the module and the appropriate module, with the desired lens and features or content, may be mounted to the vehicle and connected to the existing connector (such as a five wire connector as described above). The display system may further process the image data or encoded signal to digitally correct for distortion in the image due to lens distortion and the like.
The present invention thus provides a customized imaging system that provides various features or functions or modes or content to the imaging device or system without requiring costly communication or data wires connecting between the imaging device and the user inputs within the vehicle. The present invention thus provides an vision system that is suitable for economically configuring the system to the desired content for a particular application of the imaging system. A user thus may select a vision system with a particular camera and/or display, with little or no affect on the wiring or connections between the camera and the display.
Optionally, the imaging system may be operable to function as a lane departure warning (LDW) system utilizing image processing of the images captured by the rearward facing imaging sensor or camera. The imaging system may include controls and/or circuitry for operating as such a lane departure warning system and thus may process the images to detect the lane markers and the like along the road surface, or the imaging system may provide image data to a separate image processor or microcontroller for processing the image data to detect the lane markers and the like along the road surface. The imaging system may utilize image processing techniques such as those described in. U.S. patent applications, Ser. No. 10/427,051, filed Apr. 30, 2003 by Pawlicki et al. for OBJECT DETECTION SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,038,577; and/or Ser. No. 11/105,757, filed Apr. 14, 2005 by Schofield et al. for IMAGING SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,526,103; and/or U.S. provisional applications, Ser. No. 60/607,963, filed Sep. 8, 2004 by Schofield for IMAGING SYSTEM FOR VEHICLE; and/or Ser. No. 60/638,687, filed Dec. 23, 2004 by Higgins-Luthman for OBJECT DETECTION SYSTEM FOR VEHICLE, which are hereby incorporated herein by reference.
The imager or imaging sensor 22 of the imaging system 16 Of the imaging system 10 may comprise an imaging array sensor or a pixelated imaging array, such as a multi-pixel array such as a CMOS sensor or a CCD sensor or the like, such as the types disclosed in commonly assigned U.S. Pat. Nos. 5,550,677; 5,670,935; 5,796,094; 5,8′77,897; 6,097,023; 6,498,620; and 6,690,268, and U.S. patent applications, Ser. No. 09/441,341, filed Nov. 16, 1999 by Schofield et al. for VEHICLE HEADLIGHT CONTROL USING IMAGING SENSOR, now U.S. Pat. No. 7,339,149, which are hereby incorporated herein by reference, or such as an extended dynamic range camera, such as the types disclosed in PCT Application No: PCT/US2003/036177, filed Nov. 14, 2003 by Donnelly Corp. for IMAGING SYSTEM FOR VEHICLE, and published Jun. 3, 2004 as International Publication No. WO 2004/047421 A2, which is hereby incorporated. herein by reference. For example, the imaging sensor may comprise a CMOS camera, such as the OV7930 single chip CMOS color NTSC camera available from OmniVision Teehnologies Inc. of Sunnyvale, Calif. Such color cameras may have the performance characteristics identified above and may additionally provide RUB and/or YCrCb video signals. Preferably, the color video camera operates at a minimum illumination (3000 K) of less than about 5 lux at fl.2, more preferably of less than about 3 lux at fl.2, and most preferably less than about of less than about 2 lux at fl.2. Such CMOS imaging sensors typically may have a peak sensitivity in the near infrared range, such as at approximately 850 nm to 900 nm.
Such pixelated imaging sensors may include a plurality of pixels, with at least some of the pixels masked or covered with a particular color filter, such that the individual pixels function to capture a particular color, such as red, green and blue colors or the like, such as disclosed in U.S. Pat. Nos. 5,550,677; 5,670,935; 5,796,094; 6,097,023; and/or 6,498,620, referenced above. For example, the imaging sensor may comprise an individual blue or a green or a red color filter over each pixel element of the CMOS multi-pixel element array. The imaging sensor is thus operable to provide color images to the display. Such RGB filters enable the capture of a color image by the CMOS detector, but necessarily result in a reduced or decreased low light level sensitivity for a color camera compared to a monochromatic or black and white camera. Optionally, and preferably, the imaging sensor may be capable of selectively operating in either a color mode, in which a color image may be displayed at display element 28 of display system 14, or a monochromatic or black and white mode, in which a monochromatic or black and white image may be displayed at display element 28 of display system 14, such as by utilizing aspects of the imaging sensor disclosed in U.S. Pat. No. 6,498,620; and/or PCT Application No. PCT/US2003/036177, filed Nov. 14, 2003 by Donnelly Corp. for IMAGING
SYSTEM FOR VEHICLE, and published Jun. 3, 2004 as International Publication No. WO 2004/047421 A2, which are hereby incorporated herein by reference.
Although described as a CMOS type camera, clearly other types of imaging arrays or imaging sensors or cameras may be implemented with the imaging system of the present invention. For example, the imaging sensor may comprise a CCD or other type of sensor, without affecting the scope of the present invention. Preferably, the selected imaging sensor has a low dark current and thus provides enhanced ruggedness and enhanced performance at higher temperatures. Optionally, the dark-current (the current through the pixels when they are not sensing light) may be used to detect the temperature at the imaging sensor, such as described in U.S. patent application Ser. No. 11/105,757, filed Apr. 14, 2005 by Schofield et al. for IMAGING SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,526,103; and U.S. provisional application, Ser. No. 60/607,963, filed Sep. 8, 2004 by Schofield for IMAGING SYSTEM FOR VEHICLE, which is hereby incorporated herein by reference,
Optionally, the imaging device or system may communicate the video signals to the display device or system via other types of signal communicating means. For example, the imaging device or system may communicate the video signals to the display device or system via an LVDS output of the imaging device or system. Optionally, the imaging system and the display system may share common components or circuitry or a common microprocessor to reduce components and cost of the vision system.
Referring now to
Imaging sensor 112 may be positioned at the vehicle and directed or oriented with a field of view generally exteriorly of the vehicle for capturing images of the exterior scene for processing by the control, as discussed below. Imaging system 110 may utilize principles of other vehicle vision or imaging systems, such as a vision or imaging system or control of the types disclosed in U.S. Pat. Nos. 6,757,109; 6,717,610; 6,396,397; 6,201,642; 6,353,392; 6,313,454; 6,396,397; 5,550,677; 5,670,935; 5,796,094; 5,877,897; 6,097,023; and 6,498,620, and U.S. patent applications, Ser. No. 09/441,341, filed Nov. 16, 1999 by Schofield et al. for VEHICLE HEADLIGHT CONTROL USING IMAGING SENSOR, now U.S. Pat. No. 7,339,149, and Ser. No. 10/427,051, filed Apr. 30, 2003 by Pawlicki et al. for OBJECT DETECTION SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,038,577, which are all hereby incorporated herein by reference. In a preferred embodiment, the imaging system 112 may include a lens element or optic between the imaging device 114 and the exterior scene. The optic may comprise an asymmetric optic, which focuses a generally central portion of the scene onto the imaging device, while providing classical distortion on the periphery of the scene or field of view.
In the illustrated embodiment, the image sensor 112 is mounted at or in an accessory module or windshield electronics module or mirror attachment or pod 120 and is arranged to have a field of view 116 forward of the vehicle, such as through the windshield of the vehicle and preferably through a portion of the windshield that is wiped by a windshield wiper of the vehicle. The image sensor 112 thus may capture images of a forward scene as the vehicle is traveling forwardly along a road or street or highway or the like. Optionally, the imaging device or sensor may be positioned elsewhere, such as at or in the interior rearview mirror assembly 122, or at or in an accessory module or windshield electronics module or the like (as discussed below), without affecting the scope of the present invention. Optionally, the image sensor and imaging system may be operable to capture images of other areas exteriorly of the vehicle to provide image data of other areas around or near the vehicle, without affecting the scope of the present invention. The imaging system thus may be operable to capture images of the scene sidewardly of the vehicle for a blind spot or side object detection or lane change assist systems or immediately rearward of the vehicle to detect objects rearward of the vehicle, such as for assisting the driver of the vehicle in backing up or maneuvering the vehicle in reverse.
As shown in
The control may be operable to control various accessories (or to generate an output to a control or circuitry of one or more accessories) in response to the image processing. For example, the control may control or adjust a headlamp high-low beam setting in response to a detection of headlamps or taillights in the captured images, and/or may actuate an alert device, such as a visible display or audible alert or the like, in response to a detection of the vehicle drifting out of its lane along the road, and/or may adjust an interior lighting or display intensity of a light or display within the vehicle in response to an ambient light detection, and/or may generate other outputs or actuate/control other accessories in response to detections of other objects or light sources of interest via such image processing.
The imaging system thus may be operable to function as or in conjunction with a lane departure warning system, and may generate an alert to the driver of the vehicle in response to a detection that the vehicle is drifting or moving out of a detected lane along the road. Such an application may utilize principles of systems of the types described in U.S. Pat. Nos. 6,353,392 and 6,313,454; and/or U.S. patent applications, Ser. No. 10/427,051, filed Apr. 30, 2003 by Pawlicki et al. for OBJECT DETECTION SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,038,577; Ser. No. 11/105,757, filed Apr. 14, 2005 by Schofield et al. for IMAGING SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,526,103; and/or Ser. No. 10/209,173, filed Jul. 31, 2002 by Schofield for AUTOMOTIVE LANE CHANGE AID, now U.S. Pat. No. 6,882,287, and/or U.S. provisional applications, Ser. No. 60/607,963, filed Sep. 8, 2004 by Schofield for IMAGING SYSTEM FOR VEHICLE; and/or Ser. No. 60/638,687, filed Dec. 23, 2004 by Higgins-Luthman for OBJECT DETECTION SYSTEM FOR VEHICLE, which are hereby incorporated herein by reference. Optionally, the imaging system may capture and process images for a headlamp control application, and may utilize principles of systems of the types described in U.S. Pat. Nos. 6,824,281; 5,796,094; 6,097,023; 5,320,176; and 6,559,435, which are hereby incorporated herein by reference, and/or may capture and process images for a rain sensing function or rain sensor application, and may utilize principles of the systems of the types described in U.S. Pat. Nos. 6,824,281; 6,320,176; 6,353,392; 6,313,454; 6,516,664; 6,341,523; and 6,250,148; and/or in U.S. patent application Ser. No. 10/348,514, filed Jan. 24, 2003 by Lynam for RAIN SENSOR MOUNTING SYSTEM, now U.S. Pat. No. 6,968,736, which are all hereby incorporated herein by reference, and/or may capture and process images for an ambient light detection or the like, and may utilize principles of systems of the types described in U.S. Pat. Nos. 5,550,677 and/or 5,670,935, which are hereby incorporated herein by reference, and/or may capture and process images for a collision avoidance system or vehicle separation system, and may utilize principles of systems of the types described in U.S. Pat. Nos. 6,411,204; 6,3-96,397; 6,124,647; 6,291,906;and 6,534,884, and/or U.S. patent application Ser. No. 10/422,512, filed Apr. 24, 2003 by Schofield for DRIVING SEPARATION DISTANCE INDICATOR, now U.S. Pat. No. 7,123,168, which are all hereby incorporated herein by reference, and/or may capture and process images for a navigational system, and may utilize principles of systems of the types described in U.S. Pat. Nos. 6,477,464; 5,924,212; 4,862,594; 4,937,945; 5,131,154; 5,255,442; 6,678,614; and/or 5,632,092, and/or U.S. patent applications, Ser. No. 10/456,599, filed Jun. 6, 2003 by Weller et al. for INTERIOR REARVIEW MIRROR SYSTEM WITH COMPASS, now U.S. Pat. No. 7,004,593; Ser. No. 10/645,762, filed Aug. 20, 2003 by Taylor et al. for VEHICLE NAVIGATION SYSTEM FOR USE WITH A TELEMATICS SYSTEM, now U.S. Pat. No. 7,167,796; Ser. No. 11/105,757, filed Apr. 14, 2005 by Schofield et al. for IMAGING SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,526,103; and Ser. No. 10/422,378, filed Apr. 24, 2003, now U.S. Pat. No. 6,946,978, now U.S. Pat. No. 6,946,978; and/or PCT Application No. PCT/US03/40611, filed Dee. 19, 2003 by Donnelly Corporation et, al. for ACCESSORY SYSTEM FOR VEHICLE, and published Jul. 15, 2004 as International Publication No. WO 2004/058540 A3; and/or PCT Application No. PCT/US04/015424, filed May 18, 2004 by Donnelly Corporation et al. for MIRROR ASSEMBLY FOR VEHICLE, and published Dec. 2, 2004 as International Publication No. WO 2004/103772 A3, and/or U.S. provisional application, Ser. No. 60/60′7,963, filed Sep. 8, 2004 by Schofield for IMAGING SYSTEM FOR VEHICLE, which are all hereby incorporated herein by reference. Other accessories or functions or applications may also or otherwise be provided or controlled or adjusted by the imaging system, without affecting the scope of the present invention.
The control may adjust one or more characteristics or settings of the image sensor or camera and/or may operate an iris and/or an optical zoom and/or a digital zoom or the like so that the image sensor is adapted or set to enhance image capturing for a particular function of the imaging system or control. Optionally, the control may adjust a focus of the image sensor (such as via adjustment of the lens or optic) to provide a clear captured image focused on the desired or appropriate objects for the particular application of the imaging system.
The imager or imaging sensor 114 of the imaging system 110 may comprise an imaging array sensor or a pixelated imaging array, such as a multi-pixel array such as a CMOS sensor or a CCD sensor or the like, such as the types disclosed in commonly assigned U.S. Pat. Nos. 5,550,677; 5,670,935; 5,796,094; 5,877,897; 6,097,023; 6,498,620; and 6,690,268, and U.S. patent application Ser. No. 09/441,341, filed Nov. 16, 1999 by Schofield et al. for VEHICLE HEADLIGHT CONTROL USING IMAGING SENSOR, now U.S. Pat. No. 7,339,149, which are hereby incorporated herein by reference. For example, the multi-pixel array may comprise a multi-pixel array having a 640×480 pixel array or the like. In order to reduce the computational load on the processor, only a desired subset or zone of the multi-pixel array frame may be chosen to be processed, such as for a particular application of the imaging system. For different applications, the objects or light sources of interest may be different objects or categories or classifications of objects and/or may be typically located at different regions of the captured image. For example, for a headlamp control application or function, the objects of interest are light sources, particularly headlamps and taillights, that are typically located at a generally central band or region of interest across the captured images, while for a lane departure warning application or function, the objects of interest are lane markers or road edges or the like that are typically located along a lower region of interest of the captured images (such as along the lower about 30 percent of the captured image or thereabouts). For ambient light detection, the light or region of interest would typically be along the upper region of the captured images (such as about the upper 20 percent of the captured image or thereabouts). The desired or appropriate or chosen subset or zone or target area of the multi-pixel array frame may be processed for the particular application to reduce processing or computational load when additional processing of other subsets or zones may not enhance the performance of the imaging system for that particular application.
Also, for different applications, it may be desirable to capture the respective images with the image sensor being at a different sensitivity or setting. For example, a camera setting or characteristic, such as an exposure time, an amplification, a gain, a white balance and/on the like, of the image sensor may be adjusted for capturing images to optimize the captured image and image data for a particular application or function. Amongst other things, the sensitivity of the image sensor can be adjusted by adjusting the integration time of the pixels and the amplification stage in reading out pixels. This allows the image sensor to be adapted to various lighting conditions at the observed exterior scene. In a closed loop camera control scheme, the image sensor sensitivity can be adjusted as determined by an application module so that the image brightness meets the application specific requirements. For example, the control may adjust the image sensor to be at an increased sensitivity for detecting headlamps and taillights during nighttime conditions, yet may decrease the sensitivity for detecting lane markers along the road surface, particularly during nighttime conditions When the headlamps are activated and the road is thus well illuminated. Other adjustments or settings or characteristics, such as a desired or selected zone of interest in a frame or an optical zoom or digital zoom or panning of or through a frame or the like, may be made or set or adjusted to optimize the image sensor for capturing one or more images for a particular function or accessory control.
The control thus may adjust a characteristic or setting of the imaging sensor for particular captured images and may process the image data of those particular captured images in a manner or processing technique suited for a particular application or function of the control, such as for headlamp control, lane departure warning, rain sensing, vehicle detection, vehicle lighting and/or display control, and/or the like. The control may synchronize the camera settings for particular captured images with the processing technique applied to the image data associated with those captured images, and may switch or alternate between different settings and processing techniques to provide the desired processing and analysis and output for the desired applications.
For example, and with reference to
Multiple applications thus can be operated and controlled using the same imager and processing resources of the imaging system. In a case where only two applications share the system resources and where frames of image data are thus processed alternately by the two application modules using the same processor, the control may alternate every other image for the two applications. For example (and as shown in
Optionally, such as for cases where the two applications do not have to process images at the same frame rate, the frame rate ratio can be adjusted accordingly. For example, if the imager is operable to provide up to 15 frames per second (fps), and the first application needs the control to process five frames per second while the second or other application only needs the control to process one frame per second, then the control may process five frames in a row for the first application, and may then process only one frame for the second application. This processing scheme may be adjusted for any desired frames per second and may be extended to allow more than two applications to execute on the system, without affecting the scope of the present invention.
As shown in
The present invention thus provides an imaging system that may selectively capture images via an image sensor and may selectively process the captured images to determine or detect a desired or appropriate characteristic of the captured images. The imaging system may synchronize a setting or sensitivity or focus of the image sensor with an image processing technique so that the captured image data are selectively captured and/or processed in a particular manner fur a particular application of the imaging system. The imaging system thus may provide multiple functions or applications by utilizing a common imaging sensor and common image processor, and thus may provide a multi-tasking imaging system at a reduced cost and with reduced components. The imaging system may capture multiple frames sequentially, and may elect (such as in a predetermined manner) the processing or treatment of the respective frames or sets of frames. The imaging system thus may focus or harness the processing power to a restricted set of criteria for one subset of frames, and to another restricted set of criteria for another subset of frames. The imaging system of the present invention thus may selectively focus or harness the image sensor capturing capabilities and the image processing power to enhance the image capturing and processing of different frames or sets of frames for different applications.
Although shown and described as a forward facing imaging sensor, it is envisioned that aspects of the present invention may be suitable for selectively capturing and/or processing image data with a rearward facing imaging sensor and control or image processor. For example, the rearward facing imaging system may be operable to function as a lane departure warning (LDW) system utilizing image processing of the images captured by a rearward facing imaging sensor or camera. The imaging system may include controls and/or circuitry for operating as such a lane departure warning system and thus may process the images to detect the lane markers and the like along the road surface, or the imaging system may provide image data to a separate image processor or microcontroller for processing the image data to detect the lane markers and the like along the road surface. The imaging system may utilize image processing techniques such as those described in U.S. patent applications, Ser. No. 10/427,051, filed Apr. 30, 2003 by Pawlicki et al, for OBJECT DETECTION SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,038,577; and/or Ser. No. 11/105,757, filed Apr. 14, 2005 by Schofield et al. for IMAGING SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,526,103; and/or U.S. provisional applications, Ser. No. 60/607,963, filed Sep. 8, 2004 by Schofield for IMAGING SYSTEM FOR VEHICLE; and/or Ser. No. 60/638,687, filed Dec. 23, 2004 by Higgins-Luthman for OBJECT DETECTION SYSTEM FOR VEHICLE, which are hereby incorporated herein by reference. Optionally, the imaging system may selectively capture and process image data of the rearward facing image sensor to provide a bark up aid function, such as by utilizing principles described in U.S. Pat. Nos. 5,550,677; 5,760,962; 5,670,935; 6,201,642; 6,717,610 and/or 6,757,109, and/or in U.S. patent application Ser. No. 10/418,486, filed Apr. 18, 2003 by McMahon et al. for VEHICLE IMAGING SYSTEM, now U.S. Pat. No. 7,005,974, and/or in PCT Application No. PCT/US2003/036177, filed Nov. 14, 2003 by Donnelly Corporation et al. for IMAGING SYSTEM FOR VEHICLE, and published Jun. 3, 2004 as International Publication No. WO 2004/047421 A2; which are all hereby incorporated herein by reference.
The imager or imaging sensor 114 of the imaging system 110 may comprise an imaging array sensor or a pixelated imaging array, such as a multi-pixel array such as a CMOS sensor or a CCD sensor or the like, such as the types disclosed in commonly assigned U.S. Pat. Nos. 5;550,677; 5,670,935; 5,796,094; 5,877,897; 6,097,023; 6,498,620; and 6,690,268, and U.S. patent application Ser. No. 09/441,341, filed Nov. 16, 1999 by Schofield et al. for VEHICLE HEADLIGHT CONTROL USING IMAGING SENSOR, now U.S. Pat. No. 7,339,149, which are hereby incorporated herein by reference, or such as an extended dynamic range camera, such as the types disclosed in PCT Application No. PCT/US2003/036177, filed Nov. 14, 2003 by Donnelly Corp. for IMAGING SYSTEM FOR VEHICLE, and published Jun. 3, 2004 as International Publication No. WO 2004/047421 A2, which is hereby incorporated herein by reference. For example, the imaging sensor may comprise a CMOS camera, such as the OV7930single chip CMOS color NTSC camera available from OmniVision Technologies Inc. of Sunnyvale, Calif. Such color cameras may have the performance characteristics identified above and may additionally provide RGB and/or YCrCb video signals. Preferably, the color video camera operates at a minimum illumination (3000 K) of less than about 0.5 lux at fl.2, more preferably of less than about 3 lux at fl.2, and most preferably less than about of less than about 2 lux at fl.2. Such CMOS imaging sensors typically may have a peak sensitivity in the near infrared range, such as at approximately 850 nm to 900 nm or thereabouts.
Such pixelated imaging sensors may include a plurality of pixels, with at least some of the pixels masked or covered with a particular color filter, such that the individual pixels function to capture a particular color, such as red, green and blue colors or the like, such as disclosed in U.S. Pat. Nos. 5,550,677; 5,670,935; 5,796,094; 6,09′7,023; and 6,498,620, referenced above. For example, the imaging sensor may comprise an individual blue or a green or a red color filter over each pixel element of the CMOS multi-pixel element array. The imaging sensor is thus operable to detect colors, which is useful in determining whether a detected object or item is a headlamp or a taillight, such as described in U.S. Pat. No. 5,796,094, referenced above.
Although described as a CMOS type camera, clearly other types of imaging arrays or imaging sensors or cameras may be implemented with the imaging system of the present invention. For example, the imaging sensor may comprise a CCD or other type of sensor, without affecting the scope of the present invention. Preferably, the selected imaging sensor has a low dark current and thus provides enhanced ruggedness and enhanced performance at higher temperatures. Optionally, the dark current (the current through the pixels when they are not sensing light) may be used to detect the temperature at the imaging sensor, such as described in U.S. patent application Ser. No. 11/105,757, filed Apr. 14, 2005 by Schofield et al. for WAGING SYSTEM FOR VEHICLE, now U.S. Pat. No. 7,526,103; and/or U.S. provisional application, Ser. No. 60/607,963, filed Sep. 8, 2004 by Schofield for IMAGING SYSTEM FOR VEHICLE, which is hereby incorporated herein by reference.
Optionally, a method or system may be implemented to transform a CMOS color imager from its individual color space into a standardized color protocol, such as the CIE color space, in order to enhance the identification of a detected object or light Source. The Centre Internationale dEclairage (CIE) is an international organization that establishes methods for measuring color and that recommends standards and procedures for light and lighting, including colorimetry. The color standards for colormetric measurements are internationally accepted specifications that define color values Mathematically. CIE defines color as a combination of three axes: x, y, and z. CIE color models-are considered device independent because the colors should not differ, theoretically, from one output device to another if properly calibrated.
The CM color spaces thus provide the foundation upon which device-independent color and color management are built. Device profiles, device calibration, and gamut mapping may also be needed for color management. A device profile describes a device's color capabilities including color gamut, color production method, and device operation modes. Device profiles are typically created by color imaging scientists using spectrophotometers, which are instruments that measure the relative intensities of light in different parts of the visible spectrum. The measurements are then used to produce the device profile, such as via various algorithms and the like. Device profiles are used by the color management software to translate color data from one device to another based upon an independent color space or system for ordering colors that respects the relationships or similarity among them.
The CMOS color imager of the imaging system of the present invention is used in an automotive environment where all the road scene objects of interest that may be typically encountered may be classified within the CIE Color System. Digital imaging systems represent color in any given number of ways. The transformation provides an easy method for the identification of CIE color. Classification of the various objects within the road scene can be enhanced using CIE color as well as other image processing methods to provide high confidence that any particular detected object or light source is correctly identified.
The transformation requires an initial characterization of the pixel output of the CMOS imager using four primary calibration targets and a spectroradiometer. The CMOS imager is used to measure the “color” of the calibrated targets. The calibrated targets are also measured using the spectroradiometer. The transformation yields a three by three matrix that is applied to the x, y, z values of the camera's output to provide CIE color. The CIE color of the detected object or light source may then be compared to the Classifications or index of the various objects or light sources of interest to identify the detected object or light source.
Therefore, the present invention provides an imaging system that includes an imaging device and a control that is operable to selectively capture images and process image data to provide multiple functions or applications of a common image sensor and image processor. The imaging system may synchronize the settings of the image sensor with the processing techniques applied to the captured image data to optimize two or more functions of the imaging system. The present invention thus provides a multi-tasking capability to a common image sensor and image processor while providing enhanced capturing of the images and processing of the image data for the desired functions or applications.
Optionally, an imaging sensor or system of the present invention may be associated with an imaging and display system for displaying the captured images to the driver of the vehicle. In such display applications, a variety of display means or display devices may be utilized to visually convey information to the driver of the vehicle, without affecting the scope of the present invention. For example, and such as described in U.S. Pat. No. 6,477,464, which is hereby incorporated herein by reference, a text display may be provided and/or an iconistic display may be provided, such as a display readable through the interior rearview mirror reflective element itself. In this regard, use of a transflective or display on demand (DOD) type display (such as disclosed in commonly assigned, U.S. patent application Ser. No. 10/054,633, filed Jan. 22, 2002 by Lynam et al. for VEHICULAR LIGHTING SYSTEM, now U.S. Pat. No. 7,195,381, and in U.S. Pat. Nos. 6,690,268; 5,668,663 and 5,724,187, the entire disclosures of which are hereby incorporated by reference herein), may be preferred. For example, a video display element or a video display screen or an information display element can be used (such as an elongated alphanumeric/multi-pixel/multi-icon display element and/or such as an LCD display or an emitting display element, such as a multi-pixel electroluminescent display or field emission display or light emitting diode display (organic or inorganic) or the like) which is disposed within the mirror housing of the interior mirror assembly of the vehicle, and located behind the mirror reflective element in the mirror housing, and configured so that the information displayed by the display element (that is positioned to the rear of the reflector of the mirror reflective element) is viewable by the driver through the mirror reflective element. Such a display can be accomplished by partially or wholly removing the reflector in the area of the display or, more preferably, by providing a display on demand type display, whereby the reflective element comprises a transflective element, as discussed below.
Note that other display locations are possible for display of the video image or information display, such as a text message or the like, to the driver or occupant of the vehicle. For example, a video image may be displayed on an LCD video screen of flip-down display (such as is disclosed in U.S. Pat. No. 6,690,268, incorporated above), or on a video screen incorporated into the rearview mirror assembly (such as the types described in U.S. patent application Ser. No. 10/964,512, filed Oct. 13, 2004 by Schofield et at for VEHICLE COMMUNICATION SYSTEM, now U.S. Pat. No. 7,308,341; and/or in PCT Application No PCT/US03/40611, filed Dec. 19, 2003 by Donnelly Corp. et al. for ACCESSORY SYSTEM FOR VEHICLE, and published Jul. 15, 2004 as International Publication No. WO 2004/058540A3, which is hereby incorporated herein by reference). Optionally, for example, a video display located in the front instrument panel can be used, or a video display located in an overhead console (such as an overhead accessory module or system as described in PCT Application No. PCT/US03/40611, and published Jul. 15, 2004 as International Publication No. WO 2004/058540 A3, which is hereby incorporated herein by reference) can be used, without affecting the scope of the present invention. Alternately, a low cost, multi-pixel display (such as the type disclosed in U.S. patent application Ser. No. 10/418,486, filed Apr. 18, 2003 by McMahon et al. for VEHICLE IMAGING SYSTEM, now U.S. Pat. No. 7,005,974, which is hereby incorporated herein by reference), such as a low cost multi-pixel vacuum fluorescent display, a low cost multi-pixel organic light emitting diode (OLED), a low cost multi-pixel field emission display, or any other or similar multi-pixel light emitting display or the like may be utilized, without affecting the scope of the present invention.
Optionally, the imaging sensor and control, and optionally a display device and/or an audible alert device or speaker or the like, may include a display element positioned at a rear portion of the vehicle, such as at the rear window of the vehicle or the like, on which the captured image is projected, such as via a projector within the vehicle. The displayed or projected image may be viewed at the rear portion of the vehicle by the driver of the vehicle, such as by viewing the image in the rearview mirror or by turning to view the rearward image. The image thus may be projected as a reverse image or mirror image display depending on the particular application or desired viewing of the displayed/projected image.
Optionally, the display device and/or an audible alert device or speaker may be positioned at or in or near the interior rearview mirror assembly of the vehicle. The mirror assembly may comprise a prismatic minor assembly, such as a prismatic mirror assembly utilizing aspects described in U.S. Pat. Nos. 6,318,870; 5,327,288;4,948,242; 4,826,289; 4,436,371; and 4,435,042; and PCT Application No. PCT/US04/015424, filed May 18, 2004 by Donnelly Corporation et al. for MIRROR ASSEMBLY FOR VEHICLE; and published Dec. 2, 2004 as International Publication No. WO 2004/103772 A3; and U.S. patent application Ser. No. 10/933,842, filed Sep. 3, 2004, now U.S. Pat. No. 7,249,860, which are hereby incorporated herein by reference. Optionally, the prismatic reflective element may comprise a conventional prismatic reflective element or prism or may comprise a prismatic reflective element of the types described in PCT Application No. PCT/US03/29776, filed Sep. 19, 2003 by Donnelly Corp. et al. for MIRROR REFLECTIVE ELEMENT ASSEMBLY, and published Apr. 1, 2004 as International Publication No. WO 2004/026633 A2; U.S. patent applications, Ser. No. 10/709,434, filed May 5, 2004 by Lynam for MIRROR REFLECTIVE ELEMENT, now U.S. Pat. No. 7,420,756; and Ser. No. 10/993,302, filed Nov. 19, 2004, now U.S. Pat. No. 7,338,177; and U.S. provisional application, Ser. No. 60/525,952, filed Nov. 26, 2003 by Lynam for MIRROR REFLECTIVE ELEMENT FOR A VEHICLE, which are all hereby incorporated herein by reference, without affecting the scope of the present invention. A variety of mirror accessories and constructions are known in the art, such as those disclosed in U.S. Pat. Nos. 5,555,136; 5,582,383; 5,680,263; 6,227,6′75; 6,229,319; and 6,315,421 (the entire disclosures of which are hereby incorporated by reference herein), that can benefit from or be implemented with the present invention.
Optionally, the interior rearview mirror assembly may comprise an electro-optic or electrochromic reflective element or cell, such as an electrochromic mirror assembly and electrochromic reflective element utilizing principles disclosed in commonly assigned U.S. Pat. Nos. 6,690,268; 5,140,455; 5,151,816; 6,178,034; 6,154,306; 6,002,544; 5,567,360; 5,525,264; 5,610,756; 5,406,414; 5,253,109; 5,076,673; 5,073,012; 5,117,346; 5,724,187; 5,668,663; 5,910,854; 5,142,407; and/or 4,712,879, which are hereby incorporated herein by reference, and/or as disclosed in the following publications: N. R. Lynam, “Electrochromic Automotive Day/Night Mirrors”, SAE Technical Paper Series 870636 (198′7); N. R. Lynam, “Smart Windows for Automobiles”, SAE Technical Paper Series 900419 (1990); N. R. Lynam and A. Agrawal, “Automotive Applications of Chromogenic Materials”, Large Area Chromogenics: Materials and Devices for Transmittance Control, C. M. Lampert and C. O. Granquist, EDS., Optical. Engineering Press, Wash. (1990), which are hereby incorporated by reference herein.
The mirror assembly may include one or more displays, such as the types disclosed in U.S. Pat. Nos. 5,530,240 and/or 6,329,925, which are hereby incorporated herein by reference, and/or display-on-demand or transflective type displays, such as the types disclosed in U.S. Pat. Nos. 6,690,268; 5,668,663 and/or 5;724,187, and/or in U.S. patent applications, Ser. No. 11/021,065, filed Dec. 23, 2004 by McCabe et al. for ELECTRO-OPTIC MIRROR CELL, now U.S. Pat. No. 7,255,451; Ser. No. 10/528,269, filed. Mar. 17, 2005, now U.S. Pat. No. 7,274,501; and/or Ser. No. 10/993,302, filed Nov. 19, 2004 by Lynam for MIRROR REFLECTIVE ELEMENT FOR A VEHICLE, now U.S. Pat. No. 7,338,177; Ser. No. 10/533,762, filed May 4, 2005, now U.S. Pat. No. 7,184,190; Ser. No. 11/226,628, filed Sep. 14, 2005 by Karner et al. for MOUNTING ASSEMBLY FOR INTERIOR MIRROR; and/or Ser. No. 10/054,633, filed Jan. 22, 2002 by Lynam et al. for VEHICULAR LIGHTING SYSTEM, now U.S. Pat. No. 7,19.5,381; PCT Application No. PCT/US03/29776, filed Sep. 9, 2003 by Donnelly Corp. et al. for MIRROR REFLECTIVE ELEMENT ASSEMBLY, and published Apr. 1, 2004 as International Publication No. WO 2004/026633 A2; and/or PCT Application No. PCT/US03/40611, filed Dec. 19, 2003 by Donnelly Corp. et al. for ACCESSORY SYSTEM FOR VEHICLE, and published Jul. 15, 2004 as International Publication No. WO 2004/058540 A3, and/or U.S. provisional applications, Ser. No. 60/681,250, filed May 16, 2005; Ser. No. 60/690,400, filed Jun. 14, 2005; Ser. No. 60/695,149, filed Jun. 29, 2005, which are all hereby incorporated herein by reference.
Optionally, the image sensor and control, and optionally a display device and/or alert device, may be positioned at or in an accessory module or windshield electronic module of the vehicle, without affecting the scope of the present invention. The accessory module may comprise any type of accessory module or windshield electronics module or console, such as the types described in U.S. patent applications, Ser. No. 10/355,454, filed Jan. 31, 2003 by Schofield et al. for VEHICLE ACCESSORY MODULE, now U.S. Pat. No. 6,824,281; and Ser. No. 10/456,599, tiled Jun. 6, 2003 by Weller et al. for INTERIOR REARVIEW MIRROR SYSTEM WITH COMPASS, now U.S. Pat. No. 7,004,593, and/or U.S. Pat. Nos. 6,690,268; 6,250,148; 6,341,523; 6,593,565; and 6,326,61′3, and/or PCT Application No. PCT/US03/40611, filed Dec. 19, 2003 by Donnelly Corp. et al, for ACCESSORY SYSTEM FOR VEHICLE, and published Jul. 15, 2004 as International Publication. No. WO 2004/058540 A3, which are all hereby incorporated herein by reference.
Optionally, the imaging system may be selectively operable and switchable to provide the desired function and/or display to the driver of the vehicle. Such switching may occur in response to one or more user inputs in the vehicle, such as at the interior rearview mirror system of the vehicle or the like. The user inputs may be positioned at or near a display, such as along a perimeter region of a flip out or slide out video display of a mirror assembly or along a chin or bezel region of the mirror assembly, and may comprise buttons, switches, proximity sensors, touch sensors, or the like, without affecting the scope of present invention. For example, the user inputs may comprise buttons or switches, such as user inputs of the types described in PCT Application No. PCT/US2003/036177, filed Nov. 14, 2003 by Donnelly Corp. for IMAGING SYSTEM FOR VEHICLE, and published Jun. 3, 2004 as International Publication No. WO 2004/047421 A2, and/or PCT Application No. PCT/US04/015424, filed May 18, 2004 by Donnelly Corporation et al. for MIRROR ASSEMBLY FOR VEHICLE, and published Dec. 2, 2004 as international Publication No. WO 2004/103772 A3, which are hereby incorporated herein by reference, touch sensors or proximity sensing inputs or the like, such as sensors of the types described in U.S. Pat. Nos. 6,001,486; 6,310,611; 6,320,282; 6,627,918; and 5,594,222; and/or U.S. Pat. Publication No. 2002/0044065, published Apr. 18, 2002 by Quist et al, for INTERACTIVE AUTOMOTIVE REAR VISION SYSTEM, now U.S. Pat. No. 7,224,324; and/or U.S. patent application Ser. No. 10/933,842, filed Sep. 3, 2004, now U.S. Pat. No. 7,249,860; and/or PCT Application No. PCT/US03/40611, filed Dec. 19, 2003 by Donnelly Corp. et al. for ACCESSORY SYSTEM FOR VEHICLE, and published Jul. 15, 2004 as International Publication No. WO 2004/058540 A3, which are hereby incorporated herein by reference, or may comprise inputs molded within the bezel of the mirror assembly, such as described in U.S. patent application Ser. No. 11/029,695, filed Jan. 5, 2005, now U.S. Pat. No. 7,253,723, and/or U.S. provisional applications, Ser. No. 60/535,559, filed Jan. 9, 2004 by Lindahl for MIRROR ASSEMBLY; Ser. No. 60/553,517, filed Mar. 16, 2004 by Lindahl et al. for MIRROR ASSEMBLY; Ser. No. 60/556,259, filed Mar. 25, 2004, which are hereby incorporated herein by reference, and/or other types of buttons or inputs at a bezel region, such as the types described in U.S. provisional applications, Ser. No. 60/690,401, filed Jun. 14, 2005 by Uken for MIRROR ASSEMBLY FOR VEHICLE, and/or Ser. No. 60/719,482, filed Sep. 22, 2005 by Uken for MIRROR ASSEMBLY WITH USER INPUTS, which are hereby incorporated herein by reference, or may comprise membrane type switches, such as described in U.S. patent application Ser. No. 11/140,396, filed May 27, 2005, now U.S. Pat. No. 7,360,932, and/or U.S. provisional applications, Ser. No. 60/575,904, filed Jun. 1, 2004 by Uken for MIRROR ASSEMBLY FOR VEHICLE; and Ser. No. 60/624,320, filed Nov. 2, 2004, which are hereby incorporated herein by reference; and/or the like, without affecting the scope of the present invention. The switching or inputs may control or adjust the control and/or imaging sensor, such as by utilizing the principles described above and in U.S. provisional application, Ser. No. 60/614,644, filed Sep. 30, 2004 by Camilleri et al, for CAMERA DISPLAY SYSTEM, which is hereby incorporated herein by reference.
Therefore, the present invention provides an imaging and display system that includes an imaging device and a display device connected via video connections or communication links. The imaging device includes user inputs and the imaging and display system is operable to selectively control or adjust the imaging device in response to the user inputs. The imaging and display system is operable to communicate imaging signals from the imaging device to the display device and to communicate camera control signals from the display device to the imaging device along common connections or wiring or communication links between the imaging device and the display device. The imaging and display system thus may provide various optional features or functions or modes without having to change or add wiring or connections between the imaging device and the display device.
Changes and modifications to the specifically described embodiments may be carried out without departing from the principles of the present invention, which is intended to be limited only by the scope of the appended claims as interpreted according to the principles of patent law.
The present application is a continuation of U.S. patent application Ser. No. 11/239,980, filed Sep. 30, 2005, now U.S. Pat. No. 7,881,496, which claims the benefit of U.S. provisional applications, Ser. No. 60/628,709, filed Nov. 17, 2004; and Ser. No. 60/614,644, filed Sep. 30, 2004, which are all hereby incorporated herein by reference in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| 60628709 | Nov 2004 | US | |
| 60614644 | Sep 2004 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11239980 | Sep 2005 | US |
| Child | 13017353 | US |
