Patent Grant
10453214
The present invention relates to an image capturing device and method. The present invention relates also to a program for causing a computer to execute a process in the image capturing device or method, and a record medium storing the program.
There has been known a conventional image capturing device that performs arithmetic processing on the level of an imaging signal in a central part of an imaging screen and the level of the imaging signal in a peripheral part of the imaging screen, judges whether or not an image is in a state of including a light source or the like in its peripheral part, and makes a correction so that the central part of the image has an appropriate signal level by continuously controlling the gain of a gain control unit depending on the state (see Patent Reference 1, for example).
Patent Reference 1: Japanese Patent Application Publication No. 62-110369 (Page 2, Paragraph 8-Paragraph 9)
Since the aforementioned conventional image capturing device performs backlight compensation by judging whether or not the image is in the state of including a light source or the like in the peripheral part, when a subject to be paid attention to is not at the center of the screen, the device performs exposure control while being incapable of detecting the backlight state of the attention subject. Accordingly, there is a problem in that the attention subject can become invisible due to blocked up shadows or blown out highlights. Further, there is also a problem in that the recognition rate of the attention subject is low when the recognition is carried out by using the captured image.
An object of the present invention, which has been made to resolve the above-described problems, is to restrain the occurrence of the blocked up shadows or blown out highlights to the attention subject and thereby make the attention subject clearly visible.
An image capturing device according to the present invention includes:
an image capturing means that captures an image of a subject situated in an imaging field angle range and thereby generates a captured image;
a detection means that detects an object existing in a detection range at least partially overlapping the imaging field angle range and outputs information representing a direction of the object and information representing distance to the object;
an attention object selection means that selects one or more objects based on the information representing the direction of the object and the information representing the distance outputted from the detection means and outputs information representing position of the selected object in the captured image and information representing the distance to the selected object;
an attention region specification means that specifies a region in the captured image including the selected object as an attention region based on the information representing the position of the selected object and the information representing the distance to the selected object outputted from the attention object selection means;
a brightness calculation means that calculates brightness of the captured image in the attention region specified by the attention region specification means; and
an exposure control means that performs exposure control on the image capturing means based on the brightness calculated by the brightness calculation means.
According to the present invention, there is an effect that the occurrence of the blocked up shadows or blown out highlights to image parts corresponding to the selected object is restrained, and accordingly, the selected object is made clearly visible.
The image capturing device shown in the figure is an image capturing device to be mounted on a vehicle, including a lens 1, an image pickup device 2, a camera signal processing circuit 3, a radar 4, an attention object selection circuit 5, an attention region specification circuit 6, a brightness calculation circuit 7 and an exposure control circuit 8.
The lens 1 guides light incident from a subject situated in an imaging field angle range onto an imaging surface of the image pickup device 2 and thereby forms a subject image on the imaging surface.
The image pickup device 2 photoelectrically converts the subject image formed on the imaging surface and thereby generates an imaging signal representing the captured image. The imaging signal D2 generated by the image pickup device 2 is supplied to the camera signal processing circuit 3.
The following explanation will be given assuming a case where the image pickup device 2 captures video and outputs the imaging signal every frame period.
The image pickup device 2 is provided so as to capture images of the scene in front of the vehicle body of the vehicle on which the image capturing device is mounted (host vehicle), and the direction of an optical axis of the lens 1 is assumed to coincide with the forward direction of the vehicle body of the host vehicle.
The camera signal processing circuit 3 performs a color synchronization process, a signal amplification process, a gradation correction process, a noise reduction process, a contour correction process, a white balance adjustment process and a color correction process on the captured image of each frame outputted from the image pickup device 2, and outputs a time series of images obtained as the result of these processes (captured images after undergoing the signal processing) as an image signal D3. The image signal D3 is outputted from an output terminal 9.
The image signal D3 outputted is used for object recognition in a driving support process, for example.
The camera signal processing circuit 3 also outputs a luminance signal Y3 representing the luminance of each pixel of the captured image of each frame represented by the image signal D3. The luminance signal Y3 is supplied to the brightness calculation circuit 7.
In the aforementioned color synchronization process, color components missing at each pixel position are interpolated. In cases of pixel arrangement of the RGB Bayer type, pixel data of only one of the R pixel, the G pixel and the B pixel exists at each pixel position, and thus pixel data of the colors missing at each pixel position are interpolated by using pixel data of the same colors of vicinal pixels. This makes it possible to generate a complete image signal including pixel data of all the colors R, G and B for each pixel position.
In the aforementioned signal amplification process, the signal is amplified based on the result of detection of image brightness by the exposure control circuit 8 which will be explained later.
In the aforementioned gradation correction process, gradation correction is made by referring to a table to achieve a gradation characteristic according to a gamma characteristic, for example.
In the aforementioned noise reduction process, noise is reduced by performing at least one of a spatial smoothing process and a temporal smoothing process, for example.
In the aforementioned contour correction process, correction is made by enhancing contours diminished by the smoothing process and the like by applying a high-pass filter, for example.
In the aforementioned white balance adjustment process, the detection is performed on an R signal and a B signal (or an R-Y signal and a B-Y signal), for example, their average values are obtained, magnitudes of the average values are compared with each other, and gains are adjusted so that the R signal and the B signal (or the R-Y signal and the B-Y signal) are white balanced, for example.
In the aforementioned color correction process, the hue and the saturation are corrected by performing a matrix operation on the R signal, the G signal and the B signal.
The radar 4 as a detection means detects the positions of one or more objects existing in a detection range. The following explanation will be given assuming that the detection range of the radar 4 is identical with the imaging field angle range of the image pickup device 2.
As shown in
The radar 4 outputs information D4a representing the measured direction (direction information D4a) and information D4b representing the measured distance (distance information D4b) as position information D4 on each object.
The direction represented by the direction information D4a is associated with a position in the captured image in the image pickup device 2. The position in the captured image is represented by a horizontal direction position and a vertical direction position in the captured image, for example.
It is assumed in the following explanation that the object detection by the radar 4 is performed in sync with (i.e., at the same cycles as) and with the same timing as the image capturing by the image pickup device 2.
The position information D4 is supplied to the attention object selection circuit 5.
The attention object selection circuit 5 selects an attention object based on the position information D4 on each object supplied from the radar 4. In the selection of the attention object, an attention degree of each object is evaluated and an object is selected based on the result of the evaluation. For example, an object having the highest attention degree is determined and the determined object is selected.
The attention object selection circuit 5 supplies information D5 representing the position of the selected object to the attention region specification circuit 6.
The information D5 representing the position of the selected object includes information D5a representing the position of the selected object in the captured image and information D5b representing the distance from the host vehicle to the object.
As mentioned earlier, the direction represented by the direction information D4a included in the position information D4 outputted from the radar 4 is associated with a position in the captured image. Therefore, a corresponding position in the captured image can be determined from the direction information D4a on each object outputted from the radar 4.
The attention object selection circuit 5 also extracts the information D5b representing the distance to the selected object from the distance information D4b on each object outputted from the radar 4.
The attention region specification circuit 6 specifies an attention region in the captured image based on the position of the object represented by the information D5 outputted from the attention object selection circuit 5, that is, the position of the object selected by the attention object selection circuit 5, and outputs information D6 representing the specified region (attention region information D6). The attention region information D6 is supplied to the brightness calculation circuit 7.
The brightness calculation circuit 7 performs the detection on the luminance signal Y3 outputted from the camera signal processing circuit 3 based on the information D6 representing the attention region outputted from the attention region specification circuit 6 and by using the attention region as a detection frame. For example, the brightness calculation circuit 7 calculates an average value Yav of luminance values of pixels included in the detection frame (luminance average value) Yav based on the luminance signal Y3 regarding the pixels included in the detection frame, and supplies the result of the calculation to the exposure control circuit 8 as information representing brightness. The luminance average value obtained as above is used as a brightness index value.
The exposure control circuit 8 compares the luminance average value Yav calculated by the brightness calculation circuit 7 with a luminance target value Yrf and performs exposure control based on the result of the comparison.
The exposure control is performed by controlling parameters of exposure conditions, that is, by controlling an exposure time Te for the image pickup device 2 and controlling a signal amplification gain Gs for the camera signal processing circuit 3. The exposure control circuit 8 supplies a control signal Ct for controlling the exposure time Te to the image pickup device 2 while supplying a control signal Cg for controlling the signal amplification gain Gs to the camera signal processing circuit 3.
The exposure control circuit 8 includes a memory 81. The memory 81 stores the value (parameter) of the exposure time Te to be used for the image capturing by the image pickup device 2 and the value (parameter) of the signal amplification gain Gs to be used for the signal amplification by the camera signal processing circuit 3.
When the luminance average value Yav is inputted from the brightness calculation circuit 7, the exposure control circuit 8 makes the adjustment of the exposure time Te and the signal amplification gain Gs.
For example, when the luminance average value Yav is greater than the luminance target value Yrf, adjustment for decreasing the exposure time or adjustment for decreasing the signal amplification gain is made.
Conversely, when the luminance average value Yav is less than the luminance target value Yrf, adjustment for increasing the exposure time or adjustment for increasing the signal amplification gain is made.
Which of the exposure time or the signal amplification gain should be changed is determined in such a manner that the noise becomes lower. For example, when the subject gradually becomes darker, the exposure time is first set long, and after the exposure time reaches the maximum, the signal amplification gain is increased. The maximum value of the exposure time is one frame cycle or a time as a tolerance limit of the motion blur. The length of time as the tolerance limit of the motion blur becomes shorter as the motion of the subject becomes quicker.
Incidentally, it is also possible to carry out the exposure control by controlling the aperture of the lens 1.
When the exposure time Te or the signal amplification gain Gs has been adjusted, the value of the exposure time or the signal amplification gain stored in the memory 81 is overwritten (updated) with the adjusted value.
The adjusted value of the exposure time Te or the signal amplification gain Gs is supplied to the image pickup device 2 and the camera signal processing circuit 3 and used for the exposure control in the image capturing from the next frame period.
By the control of the exposure time of the image pickup device 2 and the control of the signal amplification gain of the camera signal processing circuit 3 performed by the exposure control circuit 8, the images represented by the image signal outputted from the camera signal processing circuit 3 are controlled so that the object in the attention region is visible with appropriate brightness.
A case where the image capturing device of this embodiment is employed for capturing images of an object in front of a vehicle and thereby recognizing the object will be explained below. In this case, the image capturing device is mounted on the vehicle and captures images of the scene in front of the vehicle.
The illustrated example assumes a case where the host vehicle (not shown) is traveling in a passing lane TFL of a road with two lanes on each side and is approaching a tunnel TN.
A small-sized vehicle A is traveling in the passing lane TFL in the tunnel TN, a large-sized vehicle B is traveling in a slow lane TSL before the tunnel TN, and a motorcycle D is traveling after the large-sized vehicle B. A small-sized vehicle C coming out from the tunnel TN is traveling in a slow lane RSL in the opposite direction. No vehicle is traveling in a passing lane RFL in the opposite direction.
Examples of the large-sized vehicle include large trucks and buses. Examples of the small-sized vehicle include passenger cars. Besides, there can also be a pedestrian on the road. In the following description, the aforementioned vehicles A-D can be referred to as objects. Each pedestrian is also handled as a type of object.
The lanes in the traveling direction and the lanes in the opposite direction are separated from each other by a median MD. A lane marking TLM is drawn between the slow lane TSL and the passing lane TFL in the traveling direction. A lane marking RLM is drawn between the slow lane RSL and the passing lane RFL in the opposite direction.
It is assumed in the following explanation that the time period is in the daytime, the weather is sunny, the outside of the tunnel TN is bright, and the inside of the tunnel TN is dark.
In the front scene shown in
In regard to the detection range of
In the detection by the radar 4, when an object is farther than a certain distance and the time it takes for the reflected wave to return is longer then a preset time or the intensity of the reflected wave is lower than a preset value, such an object is handled as not existing or not detected.
In the example shown in
The radar 4 outputs the information (direction information) D4a representing the direction of each of the objects viewed from the host vehicle and the information (distance information) D4b representing the distance from the host vehicle to each of the objects, for example, as the information (position information) D4 representing the position of each of the detected objects.
As the position information D4, information representing the position of the center of each object is outputted.
In this case, the direction of the center of a direction range in which the reflected wave from the same object is received or the direction in which the intensity of the reflected wave from the same object is the strongest is regarded as the direction of the center of the object.
Further, a distance calculated based on the reflected wave from the direction of the center of the object is handled as the distance to the center of the object.
It is assumed that the radar 4 is a radar having relatively low performance and the output of the radar 4 is not sufficient to precisely learn the size and the shape of each object, and thus not sufficient to determine the type of each object.
As mentioned earlier, the attention object selection circuit 5 evaluates the attention degree of each object based on the position information D4 on each object supplied from the radar 4, selects an object having the highest attention degree, and outputs the information D5 representing the position of the selected object. The information D5 representing the position of the object includes the information D5a representing the position of the object in the captured image and the information D5b representing the distance to the object.
The information D5a representing the position of the object in the captured image is information representing the position of the center of the object in the captured image, for example. The information D5b representing the distance to the object is information representing the distance to the center of the object, for example. The outputted information D5 is supplied to the attention region specification circuit 6.
As mentioned earlier, the direction represented by the position information D4 outputted from the radar 4 is associated with a position in the captured image, and it is possible to determine a position in the captured image corresponding to the position of the center of the object detected by the radar 4. Thus, from the information D4a representing the direction of each object outputted from the radar 4, the information D5a representing the position of the same object in the captured image can be generated.
As the information D5b representing the distance to each object, the information D4b representing the distance to the same object outputted from the radar 4 can be used without change.
In the evaluation of the attention degree of each object by the attention object selection circuit 5, the attention degree is increased with the decrease in the distance from the host vehicle, for example.
In the example of
The attention region specification circuit 6 specifies the attention region in the captured image in regard to the object selected by the attention object selection circuit 5. For example, a region centering at the center of the object selected by the attention object selection circuit 5 and having a size corresponding to the distance to the object is specified as the attention region. The attention region is a rectangular region having a pair of sides extending in the horizontal direction and a pair of sides extending in the vertical direction.
The size of the attention region is a size containing the whole of a part of the captured image that is estimated to be occupied by the object, i.e., an image part corresponding to the object, in a case where the selected object is assumed to be of the largest type among presumed plurality of types of objects, namely, a large-sized vehicle. For the estimation of the size, the distance to the selected object is taken into consideration. This is because even the same large-sized vehicle appears in different sizes in the captured image depending on the distance.
The reason why the selected object is assumed to be of the largest type among the presumed plurality of types of objects, namely, a large-sized vehicle, in the estimation of the size of the object appearing in the captured image is to make the attention region contain the part of the captured image corresponding to the object (the object appearing in the captured image) irrespective of the type of the selected object.
The attention region specification circuit 6 generates information specifying the attention region Rd shown in
The brightness calculation circuit 7 calculates the luminance average value Yav by performing the detection on the luminance signal Y3 outputted from the camera signal processing circuit 3 by using the attention region Rd as the detection frame, and supplies the result of the calculation to the exposure control circuit 8.
The exposure control circuit 8 compares the luminance average value Yav calculated by the brightness calculation circuit 7 with the luminance target value Yrf and performs the exposure control based on the result of the comparison.
In the example shown in
As a result, the brightness of the image represented by the image signal D3 outputted from the camera signal processing circuit 3 changes and an image in which the brightness has been optimally controlled in the attention region Rd is obtained. Specifically, an image with high visibility of the object D in the attention region Rd, with no blown out highlights in the image part in the attention region Rd, is obtained.
For example, as shown in
The above explanation has been given of a case where the scene as the image capturing target of the image pickup device 2 and the detection target of the radar 4 is a scene in the vicinity of the entrance of a tunnel. A case where the scene as the image capturing target of the image pickup device 2 and the detection target of the radar 4 is a scene in the vicinity of the exit of a tunnel will be explained below.
In the front scene shown in
The result of the detection by the radar 4 is identical with that shown in
In regard to the objects in
For example, it is assumed that the object D closest to the host vehicle has been selected as the attention object and the attention region Rd corresponding to the object D has been specified as shown in
As shown in
Consequently, as shown in
According to the embodiment described above, an image in which the brightness of the object to be paid attention to has been appropriately controlled can be obtained by the combination of the radar and the image pickup device.
In systems detecting an obstacle by using a radar alone without using an image pickup device, there are cases where it is impossible to determine the type of the obstacle, such as cases where it is even impossible to judge whether the obstacle is a vehicle or not.
In contrast, in the above-described embodiment, the exposure control is performed by the combination of the radar and the image pickup device so that the object of the highest attention degree among the objects detected by the radar is optimally exposed, and thus an effect is achieved in that the object of high attention degree can be viewed without being blocked up to black or being blown out to white. This is advantageous for driving support or accident prevention.
Further, since the radar is only required to obtain information representing the direction and the distance regarding each object, a radar of relatively low performance and low price can be used and the image capturing device can be implemented at a low cost.
Furthermore, since an object closest to the host vehicle among the objects detected by the radar is selected and a region containing the object is specified as the attention region in the above embodiment, the visibility of the closest object increases and that is advantageous for avoiding an impending risk of collision.
The image capturing device of
The operation of the other components is equivalent to that in the explanation of
The traveling direction detection circuit 10 detects the traveling direction of the host vehicle based on the steering direction of a steering wheel of the host vehicle and supplies information D10 representing the traveling direction to the attention object selection circuit 5b.
The attention object selection circuit 5b in
The attention object selection circuit 5b evaluates the attention degree of each object based on the information D10 representing the traveling direction of the host vehicle supplied from the traveling direction detection circuit 10 and the position information D4 (the direction information D4a and the distance information D4b) on each object supplied from the radar 4, selects an object having a relatively high attention degree, such as an object having the highest attention degree, based on the result of the evaluation, and supplies the information D5 representing the position of the selected object to the attention region specification circuit 6.
For example, the attention degree is determined by increasing a direction evaluation value as the direction of each object becomes closer to the traveling direction of the host vehicle and increasing a distance evaluation value with the decrease in the distance to each object and totalizing the direction evaluation value and the distance evaluation value. In this case, the attention degree is increased with the increase in the direction evaluation value and increased with the increase in the distance evaluation value. Then, an object having the highest attention degree is selected.
For example, it is assumed that the radar 4 has detected the objects A-D shown in
The attention object selection circuit 5b notifies the attention region specification circuit 6 of the position of the object A judged to have the highest attention degree.
The attention region specification circuit 6 specifies the attention region in regard to the position of the object A having the high attention degree.
Also in this case, as explained in the first embodiment, a region Ra centering at the center of the object A selected by the attention object selection circuit 5b and having a size determined according to the distance to the object A and on the assumption that the object A is a large-sized vehicle is specified. For example, the region Ra shown in
The brightness calculation circuit 7 calculates the luminance average value Yav by performing the detection on the luminance signal Y3 outputted from the camera signal processing circuit 3 by using the attention region Ra as the detection frame, and supplies the result of the calculation to the exposure control circuit 8.
The exposure control circuit 8 performs the exposure control based on the luminance average value Yav.
The object A is situated in the tunnel TN and the region Ra is situated in a part of the captured image in the tunnel TN.
Since the luminance average value of the region Ra in the tunnel TN is relatively low, the adjustment for increasing the exposure time or the adjustment for increasing the signal amplification gain is made, for example. Consequently, a captured image in which the brightness has been optimally controlled in the attention region Ra is obtained. Specifically, an image with high visibility of the object A in the attention region Ra, with no blocked up shadows in the image part in the attention region Ra, is obtained.
For example, as shown in
The above explanation has been given of a case where the scene as the image capturing target of the image pickup device 2 and the detection target of the radar 4 is a scene in the vicinity of the entrance of the tunnel as shown in
In this case, the positions of the objects detected by the radar 4 are as shown in
As shown in
While the above example assumes a case where the imaging direction of the image pickup device 2 coincides with the forward direction of the vehicle body of the host vehicle, the traveling direction of the host vehicle may be calculated based on the angle made by the imaging direction of the image pickup device 2 with respect to the forward direction of the vehicle body of the host vehicle and the steering direction of the steering wheel in cases where the imaging direction of the image pickup device 2 does not coincide with the forward direction of the vehicle body of the host vehicle.
As above, the exposure control is performed so that an object in a direction closer to the traveling direction of the host vehicle and at a shorter distance from the host vehicle among the objects detected by the radar is optimally exposed, which is advantageous for the prevention of rear-end collisions.
The image capturing device of
The attention object judgment circuit 11 receives the image signal D3 from the camera signal processing circuit 3 while receiving the information D5a representing the position of the attention object in the captured image from the attention object selection circuit 5.
The attention object judgment circuit 11 judges the type of the object by analyzing an image of a region in the image signal D3 at the position represented by the information D5a and having a preset size (analysis region).
The judgment of the type of the object is made to determine which of a large-sized vehicle such as a truck or a bus, a small-sized vehicle such as a passenger car, a two-wheel vehicle, a pedestrian, etc. the object corresponding to an image part included in the analysis region is.
Information D11 representing the result of the judgment is supplied to the attention region specification circuit 6c.
The size of the analysis region is determined to coincide with the size of the image part corresponding to the selected object in the image in the case where the object selected by the attention object selection circuit 5 is assumed to be a large-sized vehicle. Also in this case, the distance to the object is taken into consideration.
The attention region specification circuit 6c receives the information D5 representing the position of the selected object from the attention object selection circuit 5, receives the information D11 representing the type of the selected object from the attention object judgment circuit 11, and specifies the attention region corresponding to the selected object based on these pieces of information.
When the result of the judgment by the attention object judgment circuit 11 indicates that the attention object is a large-sized vehicle, the attention region specified by the attention region specification circuit 6c is the same as the attention region in the first embodiment.
When the result of the judgment indicates that the attention object is an object other than a large-sized vehicle, the attention region specified by the attention region specification circuit 6c is smaller than the attention region in the first embodiment.
The attention region specification circuit 6c supplies the information D6 representing the attention region to the brightness calculation circuit 7.
The attention object selection circuit 5 selects an object having the highest attention degree based on the distance among the objects detected by the radar 4 similarly to the case of
In this case, the attention object judgment circuit 11 receives the information D5a representing the position of the object D in the captured image from the attention object selection circuit 5, analyzes the image part of the image signal D3 corresponding to the position represented by the information D5a, thereby judges that the object D is a two-wheel vehicle, and supplies the information D11 representing the result of the judgment to the attention region specification circuit 6c.
The attention region specification circuit 6c recognizes that the object D is a two-wheel vehicle based on the information D11 and outputs a part of the captured image that is estimated to be occupied by the object D in the case where the object D is a two-wheel vehicle, that is, a region Rmd (
The attention region specification circuit 6c supplies information representing the attention region Rmd to the brightness calculation circuit 7.
The brightness calculation circuit 7 calculates the luminance average value Yav by performing the detection on the luminance signal Y3 outputted from the camera signal processing circuit 3 based on the information supplied from the attention region specification circuit 6c and by using the attention region Rmd as the detection frame, and supplies the result of the calculation to the exposure control circuit 8.
The exposure control circuit 8 performs the exposure control based on the luminance average value Yav of the attention region Rmd calculated by the brightness calculation circuit 7.
In the example shown in
As a result, the brightness of the image represented by the image signal D3 outputted from the camera signal processing circuit 3 changes and a captured image in which the brightness has been optimally controlled in the attention region Rmd is obtained. Specifically, an image with high visibility of the object D in the attention region Rmd, with no blown out highlights in the image part in the attention region Rmd, is obtained.
For example, as shown in
Further, since the region Rmd in the captured image is smaller than the region Rd in the first embodiment, a part other than the object D (a part other than the image part corresponding to the object D) becomes smaller and more appropriate exposure control is carried out for the brightness of the object D.
Consequently, the visibility of the object D increases further.
While the image capturing device of
The image capturing device of
The attention object judgment circuit 11 and the attention region specification circuit 6c are identical with those described in the description of the third embodiment.
The attention object selection circuit 5b determines an object having the highest attention degree among the objects detected by the radar 4 based on the distance and the direction similarly to the case of
In this case, the attention object judgment circuit 11 receives the information D5a representing the position of the object A in the captured image from the attention object selection circuit 5b, analyzes the image part of the image signal D3 corresponding to the position represented by the information D5a, thereby judges that the object A is a small-sized vehicle, and supplies the information D11 representing the result of the judgment to the attention region specification circuit 6c.
The attention region specification circuit 6c recognizes that the object A is a small-sized vehicle based on the information D11 and outputs a part of the captured image that includes the object A and is estimated to be occupied by the object A in the case where the object A is a small-sized vehicle, that is, a region Rma (
The attention region specification circuit 6c supplies information representing the attention region Rma to the brightness calculation circuit 7.
The brightness calculation circuit 7 calculates the luminance average value Yav by performing the detection on the luminance signal Y3 outputted from the camera signal processing circuit 3 based on the information supplied from the attention region specification circuit 6c and by using the attention region Rma as the detection frame, and supplies the result of the calculation to the exposure control circuit 8.
The exposure control circuit 8 performs the exposure control based on the luminance average value Yav of the attention region Rma calculated by the brightness calculation circuit 7.
In the example shown in
As a result, the brightness of the image represented by the image signal D3 outputted from the camera signal processing circuit 3 changes and a captured image in which the brightness has been optimally controlled in the attention region Rma is obtained. Specifically, an image with high visibility of the object A in the attention region Rma, with no blocked up shadows in the image part in the attention region Rma, is obtained.
For example, as shown in
Further, since the region Rma in the captured image is smaller than the region Ra in the second embodiment, a part other than the object A (a part other than the image part corresponding to the object A) becomes smaller and more appropriate exposure control is carried out for the brightness of the object A.
Consequently, the visibility of the object A increases further.
As above, in the third and fourth embodiments, the detection frame is determined according to the size of the object, and thus the size of the detection frame can be made closer to the size of the object, an image with more appropriate brightness control of the object can be obtained, and accordingly, the visibility of the object in the image increases.
The lane detection circuit 12 receives the captured image or an image signal obtained by processing the captured image, such as the image signal D3 outputted from the camera signal processing circuit 3, detects the positions of the lane markings and the median in the captured image represented by the image signal D3, and outputs information representing the detected positions of the lane markings and the median.
The attention object selection circuit 5c is roughly equivalent to the attention object selection circuit 5 in
The attention object selection circuit 5c evaluates the attention degree of each object based on information D12 representing the positions of the lane markings and the median supplied from the lane detection circuit 12 and the position information D4 (the direction information D4a and the distance information D4b) on each object supplied from the radar 4, selects an object having a relatively high attention degree, such as an object having the highest attention degree, based on the result of the evaluation, and supplies the information D5 representing the position of the selected object to the attention region specification circuit 6.
For example, objects situated on the other side of the median are excluded from the targets of evaluation. Among objects situated on the same side of the median, objects in the same lane as the host vehicle are assigned a high lane evaluation value. Further, the distance evaluation value is increased with the decrease in the distance to each object. The attention degree is determined by totalizing the lane evaluation value and the distance evaluation value. In this case, the attention degree is increased with the increase in the lane evaluation value and increased with the increase in the distance evaluation value. Then, an object having the highest attention degree is selected.
The attention region specification circuit 6, the brightness calculation circuit 7 and the exposure control circuit 8 operate similarly to those in the first embodiment.
As above, the object selection is made exclusively from objects situated in lanes in the same traveling direction as the host vehicle by using the information representing lanes and the exposure control is performed so that an object in the same lane and having high possibility of rear-end collision is optimally exposed, and thus the images can be prevented from darkening under the influence of headlights of vehicles traveling in opposite lanes, which is advantageous for the prevention of rear-end collisions.
In the example regarding the first embodiment explained referring to
The image capturing device of
In such a configuration, the attention object selection circuit 5d selects a plurality of objects and outputs information representing the positions of the selected objects. For example, an object having the highest attention degree and an object having the second highest attention degree are selected. In the case where the attention degree is increased with the decrease in the distance, an object at the shortest distance and an object at the second shortest distance are selected. In the example of
The attention region specification circuit 6d specifies regions, each centering at the center of one of the plurality of objects selected by the attention object selection circuit 5d and having a size determined according to the distance to the object, as the attention regions. For example, a region including the object D and a region including the object B are specified as shown in
The brightness calculation circuit 7b calculates the luminance average value Yav by performing the detection on the luminance signal Y3 outputted from the camera signal processing circuit 3 by using a region obtained by combining the attention regions Rb and Rd together as a single detection frame, and supplies the result of the calculation to the exposure control circuit 8. In other words, one luminance average value Yav is determined for the attention regions Rb and Rd as one brightness index value.
The exposure control circuit 8 performs the exposure control based on the luminance average value Yav.
As a result, an image in which the brightness has been optimally controlled in the attention regions Rb and Rd is obtained. Specifically, an image with high visibility of the objects B and D in the attention regions Rb and Rd, with no blown out highlights in the image parts in the attention regions Rb and Rd, is obtained.
For example, as shown in
The sixth embodiment has been described above as a modification to the first embodiment.
A similar modification can be made also to the second to fifth embodiments.
When a plurality of objects are selected and a plurality of regions corresponding to the selected objects are specified as the attention regions in the sixth embodiment, the plurality of attention regions are integrated into a single detection frame, the luminance average value regarding the detection frame is determined, and the exposure control is performed based on the determined luminance average value. Instead, it is also possible to specify attention regions different from each other in different frame periods of the image capturing by the image pickup device 2, determine the luminance average value by using a specified attention region as the detection frame, and employ an exposure condition, specified based on the determined luminance average value, for later image capturing in a frame period in which the same attention region is specified. The following explanation will be given assuming that the number of the selected objects is M (M is an integer larger than or equal to 2) and thus the number of the attention regions is M. In this case, the image capturing device operates by using M frame periods as one operation period or operation cycle.
The counter 14, having a maximum value M equal to the number of frame periods constituting one operation period, has its count value m incremented by 1 every frame period and returned to an initial value 1 when the count value m reaches the aforementioned maximum value M, and repeats the count.
The count value m of the counter 14 is supplied to the brightness calculation circuit 7d and the exposure control circuit 8d.
The attention object selection circuit 5d selects M objects and outputs information representing the positions of the selected objects. For example, objects of the highest attention degree to the M-th highest attention degree are selected. In cases where M equals 2 and the attention degree is increased with the decrease in the distance, an object at the shortest distance and an object at the second shortest distance are selected. In the example of
The attention object selection circuit 5d supplies information D5 representing the position of each of the selected M objects to the attention region specification circuit 6d.
The attention region specification circuit 6d specifies the attention region for each of the M objects selected by the attention object selection circuit 5d. For example, in cases where M equals 2 and the object D and the object B shown in
The attention region specification circuit 6d supplies the attention region information D6 on each of the specified M attention regions to the brightness calculation circuit 7d.
The brightness calculation circuit 7d specifies one of the M attention regions in each frame period and calculates the luminance average value Yav by performing the detection on the luminance signal Y3 outputted from the camera signal processing circuit 3 by using the specified attention region as the detection frame.
Specifically, for a captured image obtained in the m-th frame period in one operation period, the luminance average value Yavm is calculated by using the m-th attention region as the detection frame.
The fact that the captured image as the target of processing is the captured image obtained in the m-th frame period in one operation period can be confirmed from the counter 14's count value being m.
In cases where M equals 2 and the attention regions Rb and Rd have been specified, for example, the luminance average value Yav1 or Yav2 of the attention region Rb or Rd is calculated by alternately using the attention regions Rb and Rd, that is, by using each of the attention regions Rb and Rd every other frame period.
The brightness calculation circuit 7d supplies the result Yavm of the above calculation to the exposure control circuit 8d.
The memory 81 of the exposure control circuit 8d stores exposure condition parameters corresponding to the M attention regions R1-RM, namely, values (parameters) Te1-TeM of the exposure time Te and values (parameters) Gs1-GsM of the signal amplification gain Gs.
In each frame period, the exposure control circuit 8d performs the exposure control based on the luminance average value calculated by using one of the M attention regions as the detection frame.
Specifically, in the image capturing in the m-th frame period in one operation period, the exposure control on the image pickup device 2 and the camera signal processing circuit 3 is performed by using the exposure condition parameters Tem and Gsm stored in the memory 81.
For the captured image obtained in the m-th frame period in one operation period, the brightness calculation circuit 7d calculates the luminance average value Yavm by using the m-th attention region of the captured image as the detection frame, and thus the exposure control circuit 8d calculates new parameters Tem and Gsm (updates the parameters) based on the calculated luminance average value Yavm and the exposure condition parameters Tem and Gsm stored in the memory 81 and stores the calculated new parameters (updated parameters) Tem and Gsm in the memory 81.
The updated parameters Tem and Gsm stored in the memory 81 will be used for the exposure control in the image capturing in the m-th frame period in the next operation period.
The fact that the captured image as the target of processing is a captured image obtained in the m-th frame period in one operation period can be confirmed from the counter 14's count value being m.
As above, a plurality of attention regions are successively selected respectively for frame periods, the luminance average value is determined by using the selected attention region as the detection frame, the new exposure condition parameters corresponding to the selected attention region are calculated based on the determined luminance average value, and the exposure control is performed by using the calculated parameters in a frame period in which the same attention region is selected in the next operation period. Therefore, for the captured image of each frame, an image with high visibility of the selected attention region can be obtained.
For example, in cases where M equals 2 and the attention regions are Rb and Rd, an image with high visibility of the attention region Rb is obtained in a frame period and an image with high visibility of the attention region Rd is obtained in the next frame period.
In the above-described example, the number of selected objects and the number of frame periods constituting one operation period are both M, the same as each other, and the exposure control based on the luminance average value of the attention region corresponding to each object is performed at the same frequency. However, the present invention is not limited to this example. For example, the exposure control based on the luminance average value of the attention region corresponding to each object may be performed at different frequencies.
For example, it is possible to set the number (Mf) of frame periods constituting one operation period larger than the number (Mo) of selected objects and perform the exposure control based on the luminance average value of the attention region corresponding to an object in two or more frame periods in one operation period in regard to some of the selected objects. In short, it is desirable if the exposure condition parameters are updated based on the luminance average value of a certain attention region in the captured image of a certain frame period and the updated parameters are used in the image capturing in a frame period in which the luminance average value of the same attention region is determined next.
The seventh embodiment has been described above as a modification to the first embodiment. A similar modification can be made also to the second to fifth embodiments.
While the third to seventh embodiments have been described by using the operation in the vicinity of the entrance of a tunnel as an example, similar effects can be obtained also in the vicinity of the exit of a tunnel similarly to the explanation in the first and second embodiments.
While the first to seventh embodiments have been described on the assumption that the detection range of the radar 4 is identical with the imaging field angle range of the image pickup device 2, this condition is not essential; it is permissible if these ranges at least partially overlap each other.
While the detection by the radar 4 is carried out in sync with the image capturing by the image pickup device 2 in the first to seventh embodiments, this condition is not essential. In cases where the detection by the radar 4 and the image capturing by the image pickup device 2 are not in sync with each other, it is possible to interpolate either of the position information obtained as the result of the detection by the radar 4 and the captured image obtained by the image pickup device 2 in the temporal direction and thereby generate information and an image at the same timing. For example, it is possible to interpolate the position information obtained by the radar 4 and thereby generate position information at the same timing as the image capturing by the image pickup device 2. Conversely, it is also possible to interpolate the captured image obtained by the image pickup device 2 and thereby generate an image at the same timing as the detection by the radar 4. The interpolation mentioned here includes interpolation of the type directly using the position information or captured image obtained at the nearest time as the interpolated position information or image.
While the present invention has been described above as image capturing devices, image capturing methods executed by the above-described image capturing devices also constitute a part of the present invention.
In the first to seventh embodiments described above, at least part of the component parts of the image capturing device is implemented by a processing circuit. The processing circuit may either be dedicated hardware or a CPU executing a program stored in a memory.
For example, the functions of the parts shown in
In cases where the processing circuit is a CPU, the function of each part of the image capturing device is implemented by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in a memory. The processing circuit implements the function of each part by reading out a program stored in the memory and executing the program. Thus, the image capturing device includes a memory for storing programs to consequently cause the execution of the functions of the parts shown in
It is also possible to implement the functions of the parts of the image capturing device partially by dedicated hardware and partially by software or firmware.
As above, the processing circuit(s) can implement the aforementioned functions by hardware, software, firmware, or a combination of some of these.
The computer shown in
The CPU 51 operates according to a program stored in the memory 52. Specifically, the CPU 51 performs the same processing as the camera signal processing circuit 3 on the imaging signal D2 inputted via the first input interface 53A and outputs the image signal D3 obtained as the result of the processing from the second output interface 54B. Further, the CPU 51 performs the same processing as the attention object selection circuit 5, the attention region specification circuit 6, the brightness calculation circuit 7 or 7b and the exposure control circuit 8 or 8b on the imaging signal D2 inputted via the first input interface 53A and the position information D4 from the radar 4 inputted via the second input interface 53B and supplies a control signal Ct for controlling the exposure time obtained as the result of the processing to the image pickup device 2 via the first output interface 54A.
In the case of the embodiment shown in
The CPU 51 in
Incidentally, the traveling direction detection circuit 10 is a circuit for detecting the traveling direction of the host vehicle based on the steering direction of the steering wheel of the host vehicle and outputting the information D10 representing the traveling direction, and it is also possible to form part of the traveling direction detection circuit 10 by a processing circuit. In cases where the traveling direction detection circuit 10 is a circuit for detecting the traveling direction based on information generated inside the CPU 51, it is also possible to form the whole of the traveling direction detection circuit 10 by a processing circuit.
Effects similar to those described in regard to the image capturing device are achieved also by a program for causing a computer to execute the image capturing method executed by the image capturing device, the processes of the parts of the image capturing device, or the processes in the image capturing method.
1: lens, 2: image pickup device, 3: camera signal processing circuit 3, 4: radar, 5, 5b, 5c, 5d: attention object selection circuit, 6, 6c, 6d: attention region specification circuit, 7, 7b, 7d: brightness calculation circuit, 8, 8d: exposure control circuit, 9: output terminal, 10: traveling direction detection circuit, 11: attention object judgment circuit, 12: lane detection circuit, 41: transmission unit, 42: antenna, 43: reception unit, 50: computer, 51: CPU, 52: memory.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2015-010132 | Jan 2015 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2015/081838 | 11/12/2015 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2016/117213 | 7/28/2016 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 6590521 | Saka et al. | Jul 2003 | B1 |
| 20070073484 | Horibe | Mar 2007 | A1 |
| 20080055412 | Tanaka | Mar 2008 | A1 |
| 20080309517 | Saito | Dec 2008 | A1 |
| 20090051794 | Ando | Feb 2009 | A1 |
| 20150243017 | Fujimoto | Aug 2015 | A1 |
| 20170085771 | Schwager | Mar 2017 | A1 |
| 20170098132 | Yokota | Apr 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| 62-110369 | May 1987 | JP |
| 7-125567 | May 1995 | JP |
| 2001-134769 | May 2001 | JP |
| 2001-183462 | Jul 2001 | JP |
| 2005-339176 | Dec 2005 | JP |
| 2006-175936 | Jul 2006 | JP |
| 2007-96510 | Apr 2007 | JP |
| 2008-70999 | Mar 2008 | JP |
| 2009-134455 | Jun 2009 | JP |
| 2010-127717 | Jun 2010 | JP |
| 2014-135611 | Jul 2014 | JP |
| 2014-135612 | Jul 2014 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20180012374 A1 | Jan 2018 | US |
