The present invention relates to a system for monitoring surroundings of a vehicle, and more specifically to a system for controlling modes of display according to monitoring of the surroundings of a vehicle.
It is conventional to mount an imager (imaging device) on a vehicle and to display images of the surroundings of the vehicle captured by the imager on a monitor (display device). When one or more specific objects are in the captured image, a warning is provided to draw a driver's attention.
When such warning is provided quite often, the driver would feel bothered. To avoid it, the patent literature 1 identified below describe a scheme of determining a direction of gaze by the driver and a direction needing to be gazed to collect information that is required for driving wherein the frequency of matching of the two directions is determined, and based on the frequency, a level of notification of the information to be provided to the driver is determined. With this scheme, the driver is less bothered and yet is given appropriate information about the situations that the driver is not knowledgeable.
Japanese Patent No. 2929927
When a monitor is provided remote from the direction of the driver's line of sight, large movement of the line of sight is required for the driver to recognize the monitor screen. There is a need for a technology that enables the driver to quickly recognize the information displayed on the monitor screen (enhancement of instantaneous recognition). With enhancement of instantaneous recognition, the drive may be prompted to gaze forward.
Accordingly, it is an object of the present invention to control the volume of information in a monitor screen according to the position of the monitor and to provide display modes that optimize driver's gazing of the monitor screen and of forward direction.
According to one aspect of the present invention, the system for monitoring surroundings of a vehicle comprises a detector that detects objects in the surroundings of the vehicle based on images captured by an imager (imaging device), and a plurality of display device for displaying images produced from the images captured by the imager. The system is configured to alert the driver of the objects via one or more of the display devices. The display devices are placed at a plurality of positions that the driver may gaze. The system is configured to provide less information to a display device that requires a large movement of the driver's line of sight to gaze the display device relative to the driver's line of sight gazing straight forward.
Generally, instantaneous recognition lowers as the information displayed in the monitor screen increases. According to the present invention, the volume of information decreases for the display unit that requires large movement of the driver's line of sight so that instantaneous recognition is enhanced for monitor screen requiring a large movement of the driver's line of sight, whereby the driver's gaze at the monitor screen and gaze at straight forward are optimized.
According to one embodiment of the present invention, the system calculates a time for the vehicle to reach an object. The difference of the volume of information to be displayed on respective monitor screens placed at a plurality of places is set small as the time for the vehicle to reach an object is large.
The larger the time for the vehicle to reach an object, the larger the lead time for recognizing the object is. Therefore, under such conditions, a same amount of information may be displayed on each display device for the convenience of the vehicle's passengers.
According to one embodiment of the invention, the system includes a detector that detects passengers other than the driver in the vehicle. When one or more passengers are detected, the system suppresses reduction of the volume of information to be displayed on a display device placed near the one or more passengers.
According to the present invention, when a passenger is on a front passenger's seat, reduction of information to be displayed on a display device near the passenger is suppressed to enhance usability by the passenger.
The other features and advantages of the present invention will be appreciated from the description below.
Referring to the attached drawings, embodiments of the present invention will be described.
The system for monitoring surroundings of a vehicle is mounted to a vehicle and comprises far infrared cameras 1R, 1L and an image processing unit 2 that detects one or more objects in the surroundings of a vehicle based on image data captured by cameras 1R and 1L. The system further comprises a speaker 3 that produces warning by voice based on the detection results from the image processing unit 2, and a first display device 31 that displays images based on the image captured by camera 1R or 1L. The system also comprises a yaw rate sensor 6 that detects yaw rate of a vehicle, and speed sensor 7 that detects travel speed of the vehicle. The outputs from these sensors are sent to the image processing unit 2.
In the embodiment, as illustrated in
As illustrated in
In this embodiment, a navigation system is installed in the vehicle. The navigation system comprises a navigation unit 5 and a third display device 33. The third display device 33 is placed on a dashboard of the vehicle as shown in
Navigation unit 5 comprises a computer having a central processing unit (CPU) and a memory, and also comprised a communications unit (not shown) that receives GPS signal from artificial satellites, the GPS signal being used to measure the location of the vehicle 10. The navigation unit 5 detects the location of the vehicle based on the GPS signal. The navigation unit 5 overlaps an image showing the current position of the vehicle onto map of the surroundings of the vehicle to display on the third display device 33. The map of the surroundings of the vehicle may be stored in the memory of the navigation system or may be received from a server via the communications unit.
The monitor screen of the third display device 33 may be made of a touch panel. The driver of the passenger of the vehicle may enter a destination into the navigation unit with the use of the touch panel or other input devices such as keys and buttons. The navigation unit 5 determines a best route to the destination and overlaps an image of the best route onto the map to display on the third display device 33.
The navigation unit 5 is connected to a speaker 3, and provides guidance to the driver and passenger about stop signs and crossings by sound or voice via the speaker 3, in addition to providing display on the third display device 33. The navigations devices in the market today include various functions such as supply of traffic information and guidance on nearby facilities, any one of such navigation devices may be used for embodiments of the present invention.
Further, according to one embodiment, a second display device 32 is provided on an instrument panel of the vehicle in the width direction of the vehicle and between the first and the third display devices, as shown in
Image processing unit 2 comprises an A/D convertor circuit that converts input analog signal into digital signal, an image memory that stores digitized image signal, a central processing unit (CPU) for performing various computations, a RAM (random access memory) for temporarily storing data computed by the CPU, a ROM (read only memory) that stores computer programs to be executed by the CPU and data (tables, maps), and output circuit for supplying driving signal to speaker 3 and display signal to the first to third display devices 31, 32 and 33. Output signals from cameras 1R and 1L are converted to digital signals and are sent to CPU.
The first, second and third display devices 31, 32 and 33 are respectively connected to image processing unit 2 of the embodiment, and present images processed by the image processing unit 2. A switching mechanism may be provided to the second and the third display devices for switching the contents to be displayed for example. For the second display device 32, switching may be made between presentation of images from the image processing unit 2 and presentation of regular information, and for the third display device, switching may be made between presentation of images from the image processing unit and presentation of information supplied by the navigation unit 5.
As described above, vehicles today may be provided with a plurality of display devices. For the display device placed remote from the line L1 requires a large movement of the driver's line of sight to gaze the display screen. Thus, enhancement of instantaneous recognition is desired for such a display device.
Accordingly, in one embodiment of the present invention, the image processing unit 2 controls the volume of information in the images to be presented on the display screen based on the results of processing by the image processing unit 2. Specifically, for the display device placed remote from the driver's line of sight gazing straight forward (the above mentioned line L1), the volume of display image is reduced. Thus, as the display device is placed remote from the driver, display content is made feasible for instantaneous recognition. This way, gazing by the driver at the display screen and gazing ahead are optimized (balanced). A specific manner will be described below.
In steps S11 to S13, image processing unit 2 receives output signals from cameras 1R and 1L, performs A/D conversion on the output signals and stores the converted data in an image memory. The stored image data is grayscale image including intensity information.
In step S14, the image data is binary coded with the right image captured by camera 1R functioning as a reference image (alternatively, the left image may be used as a reference image). Specifically, regions having intensity larger than an intensity threshold ITH are coded to “1” (white) and regions having intensity lower than the threshold are coded to “0” (black). The intensity threshold may be determined by an appropriate scheme. With this binary coding, objects such as living bodies having higher temperature than a predetermined or a specified temperature are extracted as white regions.
In step S15, binary coded image data is converted into run length data. Specifically, for the region converted to white with the binary coding, run length data is formed as the length (expressed in terms of the number of pixels) from a starting point of the white region in a line of pixels and to the ending point of the white region. Here, the vertical direction in the image is given y axis and the horizontal direction is given x axis. For example, if the white region is from (x1, y1) to (x3, y1), that is, a line of three pixels, run length data may be expressed as (x1, y1, 3).
In steps S16 and S17, labeling of one or more objects is performed and the objects are extracted(detected). That is, out of the run length coded lines, lines sharing y axis are assumed to belong to an object, to which a label is attached. This way, one or more objects are extracted (detected).
In the following description, the detected objects are pedestrians. After step S17, a determination process may be added to determine if the detected objects are pedestrians. This determination process may be performed with any one of appropriate schemes. For example, a well known pattern matching scheme may be used to calculate similarity between the detected objects and predetermined or specified patterns of pedestrians. High similarity results in determination of pedestrians. Examples of such schemes may be seen in Japanese patent application publication Nos. 2007-241740 and 2007-334751.
In step S18, warning output is made relative to the detected objects by presenting display of the detected objects. Specifically, based on the grayscale image of the objects, a first, second and third images to be displayed on the first, second and third display devices 31, 32 and 33 respectively are produced and presented to the display devices 31, 32 and 33 respectively. The images are produced such that the information volume decreases from the first image to the third image. The information volume corresponds to the image content that a person may recognize from the image. As the features (not only living bodies such as pedestrians, but also architectures, other vehicles and other artificial features) imaged in the image increases, the information volume increases, making it difficult to instantaneously recognize the contents of the image (lowering of instantaneous recognition). The mode where information volume decreases from the first image to the third image is called a first display mode.
There are a number of methods for producing display images according to the first display mode. A first and a second methods are described below.
According to the first method, the first image includes other features than the detected objects as recognizable features, the second image includes only the detected objects as substantially recognizable features, and the third image does not include the detected objects and other features as recognizable features.
Specifically, in the first method, the first image is the above mentioned grayscale image. In the second image, the image region other than the object, that is, the image region other than the image region corresponding to the objects detected in step S17, is made substantially non-recognizable. For example, the difference between the intensity of pixels in the regions other than the objects and the background intensity is decreased to lower the contrast of the regions other than the objects making such regions substantially non-recognizable. Alternatively, the intensity of pixels in the regions other than the object regions may be decreased by a predetermined or a specified value or may be replaced with a predetermined or a specified low intensity. This way, the second image is produced such that substantially the object regions are recognizable (legible).
In the third image, object regions are made non-recognizable by decreasing the intensity of all pixels in the grayscale image by a predetermined or a specified amount or replacing the intensity of all pixels in the grayscale image by predetermined or specified intensity. This way, the third image looks as if no images are captured, or the captured image is not displayed. Alternatively, without converting the intensity of pixels, presentation of the third image may be suppressed.
One example of thus produced images are shown in
From
As an alternative to the first method, the second image may be produced by decreasing the contrast of the entire first image that is a grayscale image. For example, intensity of pixels may be reduced to decrease the difference between the largest intensity and the smallest intensity to produce the second image with decreased contrast. As the contrast is lowered, intensity of all the objects that are imaged approaches the intensity of the background to produce a blurred image as a whole. This means that the volume of recognizable information decreases. However, preferably, the decrease of contrast in the second image should be set to the extent that would make the detected objects recognizable. Thus, the second image may be produced to enable substantial recognition of the object regions only. The third image may be produced by further decreasing the contrast of the second image to present no substantial screen images.
In lieu of decreasing the contrast, intensity of all pixels in the gray scale image may be reduced uniformly by a predetermined or specified value to produce a dark image. For the second image, intensity may be reduced such that only the object region is recognizable.
As a further alternative embodiment, a first image may be produced by increasing the contrast of the gray scale image, which is named a second image. A third image may be produced by decreasing the contrast of the gray scale image. However, high contrast may spoil intermediate colors so that information volume may reduce. Thus, increase of the contrast should be controlled such that information recognizable from the first image is larger than that of the second image.
In a second scheme, the objects detected in step S17 are emphasized in the first image, the second image being without such emphasis. The third image is made such that no objects are recognizable in the screen image.
An example of such a scheme is shown in
Steps S11-S17 are the same as those in
In step S29, whether or not the distance (or the time) thus calculated is larger than a predetermined value is determined. If the decision is negative, the process proceeds to step S30 where the first, second and third images are produced according to the first display mode and are presented on the first, second and third display device respectively. Thus, an alerting output is made for the object. If the determination is positive, the process proceeds to step S31 where the first, second and third images are produced according to a second display mode and are presented on the first, second and third display device respectively. Thus, an alerting output is made for the object.
The first display mode was described with reference to
In
In
When the distance to the pedestrian 101 or the time to reach the pedestrian 101 is larger than a predetermined value, there is a lead time to reach the object. In such a case, difference of information volume may be made smaller to permit certain gazing time for the second and third display devices 32 and 33.
When a passenger is detected by the detecting device 9, a third display mode may be used to produce a first through a third images to present on the display devices 31-33. This process will be described referring to
In step S38, a result of passenger detection by the detecting device 9 is acquired. In step S39, determination is made whether or not a passenger is in the passenger's seat next to the driver's seat. If negative, the process proceeds to step S40 to produce an alerting output in the first display mode. This step is the same as step S18 in
If the determination is positive, the process proceeds to step S41 to produce an alerting output in the third display mode.
The third display mode is a mode for suppressing, as compared to the first display mode, reduction of information volume of the image to be displayed on a display device near the detected passenger. Preferably, the display device that is nearest to the detected passenger is identified, and the information volume of the image to be presented on the nearest display device is not reduced as in the first display mode.
In this embodiment, the nearest display device is the third display device 33. Information volume of the third image is modified. As a specific example, the third image is produced to have the same information volume as the second or the first image, and is presented to the third display device. 33.
As an another example, images of (a7), (b7) and (c7) represent the first, second and third images respectively with the first display mode being
In these examples, the third image in the third display mode is produced the same as the first or the second image. Alternatively, the third image in the third display mode may be produced to have an information volume less than the first or the second image but larger than the third image in the first display mode. For example, in lieu of the third image of
Thus, when a passenger is in the passenger's seat next to the driver's seat, reduction of information volume for the display device that is located nearest to the passenger is suppressed providing a display screen that is easy to recognize. The passenger may view the display screen and advise the driver of the contents.
The embodiment (
In the embodiment, three display devices are used. The present invention may be implemented using two or more display devices. When three or more display devices are provided, at least two display devices should be controlled for reduction of information volume in the first display mode, reduction of difference of information volume in the second display mode, and suppression of reduction of information volume in the third display mode. Thus, not all the display devices need be controlled.
In the above embodiment, a far infrared camera is used. The present invention may be practiced using other cameras (such as visible light cameras). In the above embodiment, a pedestrian is detected. In addition to a pedestrian, an animal may be detected solely or along with the pedestrian.
In the above embodiment, warning for a detected object is presented via one or more display devices. In addition, speaker 3 may be used to inform the driver of the existence of the object.
Specific embodiments of the present invention are described above. The present invention is not limited to such embodiments.
Number | Date | Country | Kind |
---|---|---|---|
2010-096054 | Apr 2010 | JP | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/JP2011/002206 | 4/14/2011 | WO | 00 | 10/8/2012 |