Patent Application
20210321062
One disclosed aspect of the embodiments relates to an image processing device, an imaging device, an image processing method, and a recording medium.
While driving a vehicle that follows a large-size vehicle, a driver may not be able to ascertain the situation of a field of view blocked by the large-size vehicle. For example, even if a driver desires to pass a preceding large-size vehicle because its traveling speed is slow, the driver's field of view is blocked by the large-size vehicle and the driver cannot confirm whether there is a preceding vehicle in front of the large-size vehicle, and therefore the passing is not possible. In addition, the blockage of the field of view delays the detection of pedestrians appearing in front of the large-size vehicle, which may lead to an accident. Japanese Patent Laid-Open No. 2002-166750 discloses a vehicle that displays a video captured by a camera installed in front of the vehicle in its traveling direction on a display installed behind the vehicle.
According to the vehicle disclosed in Japanese Patent Laid-Open No. 2002-166750, a following vehicle only recognizes a video displayed on the display of a preceding vehicle, and therefore it is not possible to ascertain a forward situation more accurately than in the case of visual observation. For example, from the video reflected in the display, it is not possible to accurately ascertain a distance between the large-size vehicle and a vehicle preceding it, or to know whether there is sufficient space for a host vehicle to enter in front of the large-size vehicle. In addition, if a pedestrian appears in front of the large-size vehicle and the pedestrian is reflected small in the display, a driver of the following vehicle is not aware of the pedestrian and cannot drive safely. Meanwhile, moving objects such as a ship or an aircraft as well as a vehicle may not be able to accurately ascertain a forward situation from a video. In addition, circumstances in which a forward situation is not able to be accurately ascertained from a video may occur even if the following vehicle is driven automatically.
One aspect of the embodiments is a device that displays a captured video on a display device installed toward the outer side of a moving object, and provides an image processing device capable of allowing other moving objects to accurately ascertain the situation of a subject within a video.
According to an embodiment, an image processing device provided in a moving object, including at least one processor and a memory holding a program which makes the processor function as a display control unit and a control unit. The display control unit is configured to be provided or positioned toward an outer side of the moving object to display a video acquired by image capturing performed by an imaging unit provided or positioned in the moving object on a display device. The control unit is configured to control superimposed display of a measurement result or an analysis result related to a subject included in the video on the video.
Further features of the disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
A vehicle-mounted display device 100 is an example of an image processing device provided, placed, positioned, located, or disposed in a vehicle 200 which is an example of a moving object. The vehicle-mounted display device 100 also functions as an imaging device. The vehicle 200 is a large-size vehicle such as, for example, a truck or a bus. Meanwhile, a moving object to which the disclosure can be applied is not limited to a vehicle. The disclosure can be applied to at least any of a vehicle, a ship, or an aircraft.
In the example shown in
The distance measuring unit 102 has a laser irradiation unit and a light receiving element, and measures a distance between the vehicle 200 and a subject through active distance measurement. The measured distance corresponds to the depth (depth information) of the subject in its depth direction. Specifically, the laser irradiation unit performs irradiation with laser, and light reflected from the subject is received by the light receiving element. The distance measuring unit 102 measures a distance from the vehicle 200 to the subject by measuring a time from laser irradiation to light reception. The subject is, for example, a vehicle preceding the vehicle 200, an oncoming vehicle, a point of intersection located in front, a traffic light, a person (for example, a pedestrian), or the like.
Examples of the above-described depth information include a map of the amount of image shift and a map of the amount of defocus. The map of the amount of image shift is calculated from a plurality of viewpoint images having different viewpoints, and the map of the amount of defocus is calculated by multiplying the amount of image shift by a predetermined conversion coefficient. Therefore, instead of the distance measuring unit 102, the arithmetic operation unit 103 may calculate a distance to the subject on the basis of the amount of parallax between a plurality of captured images having different viewpoints. For this, for example, the imaging unit 101 may be a stereo camera having a plurality of combinations of a lens and an imaging unit. The arithmetic operation unit 103 measures a distance based on the amount of parallax between images obtained by image capturing performed by a stereo camera, so that it is possible to omit the distance measuring unit 102 and to realize an inexpensive and simple configuration.
In addition, it is possible to calculate a distance to the subject by applying a self-position/posture estimation technique and estimating the three-dimensional position of the subject and the position of the camera (the imaging unit 101) using a monocular camera. For example, the arithmetic operation unit 103 uses two images having different image capturing times to calculate a distance to the subject on the basis of the amount of parallax and the amount of translation of the camera. In addition, if a subject of which the size is already known is reflected in an image, a distance to the subject may be calculated on the basis of the size of the image of the subject.
In addition, the vehicle speed measuring unit 105 measures the speed (vehicle speed) of the vehicle 200. For example, the vehicle speed measuring unit 105 measures the rotational speed of the wheel of the vehicle 200, and converts the measured rotational speed into the vehicle speed of the vehicle 200. Meanwhile, the vehicle speed of the vehicle 200 can also be calculated by calculating a change over time in the position of the camera (the imaging unit 101) obtained by the self-position/posture estimation technique.
The arithmetic operation unit 103 includes a central processing unit or a programmable processor and other devices such as a memory that holds or stores instructions. The instructions, when executed by the processor, cause the processor to perform as various function units described in the following. The arithmetic operation unit 103 controls the entirety of the vehicle-mounted display device 100, and performs image processing, a distance measuring operation, subject recognition, or the like. For example, the arithmetic operation unit 103 controls the display unit 106, and displays a video obtained by the imaging unit 101 on the display unit 106. In addition, the arithmetic operation unit 103 superimposes a measurement result or an analysis result, related to a subject included in the video displayed on the display unit 106, on the video, and displays it on the display unit 106 (superimposedly displays it on the video). The measurement result related to the subject is, for example, a distance between the vehicle 200 and the subject, the speed of the subject, or the like. In addition, the analysis result related to the subject is, for example, the subject being a specific subject determined in advance such as a pedestrian or a traffic light, or the like. In addition, the storage unit 104 is a secondary storage device, and stores data used for processing by the arithmetic operation unit 103 (for example, learned data obtained by machine learning). The storage unit 104 is a secondary storage device, and stores data used for processing by the arithmetic operation unit 103.
The display unit 106 is a display device provided, placed, positioned, located, or disposed in the vehicle 200. The display unit 106 displays a video obtained by the imaging unit 101 in accordance with control of the arithmetic operation unit 103. In addition, the display unit 106 superimposedly displays a measurement result or an analysis result, related to the subject included in the video, on the video in accordance with control of the arithmetic operation unit 103. In the present embodiment, the display unit 106 has a liquid crystal panel. The display unit 106 is installed toward the outer side of the vehicle 200 so that a video can be seen from another vehicle. For example, in order for a video to be able to be seen from a following vehicle, the display unit 106 may be installed behind the outside (for example, the back) of the vehicle 200, or the display unit 106 may be installed at the rear window inside the vehicle 200. In addition, for example, the display unit 106 may be installed as follows so that a video can be seen from another vehicle traveling toward the lateral side of the vehicle 200 on a road intersecting a road on which the vehicle 200 travels. For example, the display unit 106 may be installed on the outer lateral side of the vehicle 200, or may be installed at a window on the inner lateral side of the vehicle 200.
Next, in S4024, the arithmetic operation unit 103 determines whether the direction indicator of the following vehicle is blinking on the basis of the result of the detection process in S4023. If the direction indicator of the following vehicle is not blinking, the process ends. If the direction indicator of the following vehicle is blinking, it means that a driver of the following vehicle intends to pass the vehicle 200. Therefore, in this case, the process proceeds to S4025. Since S4025 and the subsequent steps are executed only when the following vehicle passes the vehicle 200, power consumption can be suppressed. Meanwhile, when it is not necessary to display a live video on the display unit 106, the arithmetic operation unit 103 may display information such as, for example, an advertisement on the display unit 106 during stop. In addition, display of a video may be started or ended in accordance with a traveling time, the weather, or the like. For example, by turning off the display unit 106 during driving at night when it is not necessary to display a video, the driver of the following vehicle can be allowed to travel safely without feeling the glare caused by screen light emission.
Next, in S4025, the imaging unit 101 captures an image of a region in front of the vehicle 200 to acquire a video. In S4026, the arithmetic operation unit 103 analyzes the video acquired in S4025, and executes a process of detecting a subject of attention within the video. In S4027, the arithmetic operation unit 103 determines whether the subject of attention has been detected. The subject of attention is a specific subject determined in advance, and is, for example, a vehicle preceding the vehicle 200, an oncoming vehicle, a pedestrian, a point of intersection, a traffic light, or the like. The preceding vehicle is an example of a moving object that moves the same direction as the traveling direction of the vehicle which is a moving object. In addition, the oncoming vehicle is an example of a moving object that moves in a direction opposite to the traveling direction of the vehicle which is a moving object. The arithmetic operation unit 103 detects a subject of attention using, for example, learned data stored in the storage unit 104. A method of detecting a subject of attention is not limited to the method of using learned data. The arithmetic operation unit 103 may detect a subject of attention using a known pattern matching technique.
If the subject of attention is not detected, the process ends. If the subject of attention has been detected, the process proceeds to S4028. In S4028, the arithmetic operation unit 103 controls the display unit 106, and starts to display a video obtained by image capturing performed by the imaging unit 101. According to the vehicle-mounted display device 100 of the present embodiment, it is possible to control the start or end of display of the video obtained by image capturing performed by the imaging unit 101 on the display unit 106 in accordance with the analysis result of a video obtained by image capturing performed by the imaging unit 107 or the imaging unit 101. Meanwhile, in another embodiment, S4023 and S4024 of
In S4032, the arithmetic operation unit 103 controls the display unit 106, and displays the video obtained by image capturing performed by the imaging unit 101 on the display unit 106. Thereby, the driver of the following vehicle of the vehicle 200 can know a situation in front of the vehicle 200. Subsequently, in S4033, the distance measuring unit 102 measures a distance from the vehicle 200 to the subject of attention. In S4034, the arithmetic operation unit 103 superimposedly displays information on the distance measured in S4033 on the video. Thereby, for example, the driver of the following vehicle of the vehicle 200 can ascertain whether there is sufficient space for the host vehicle to enter between the vehicle 200 and its preceding vehicle when passing the vehicle 200. In addition, even if the driver performs a lane change to an opposite lane for passing, he or she can know a distance to an oncoming vehicle.
Next, in S4035, the arithmetic operation unit 103 superimposedly displays the image and dimensional information (overall length) of the vehicle 200 stored in advance in the storage unit 104 on the video. Thereby, for example, the driver of the following vehicle of the vehicle 200 can ascertain a distance required to travel in the opposite lane when passing the vehicle 200. The image of the vehicle 200 may be, for example, an image obtained by capturing an image of a vehicle of the same type as the vehicle 200 in advance from the rear, or may be an image of a picture of the vehicle 200 drawn from the rear.
Next, in S4036, the arithmetic operation unit 103 determines whether the subject of attention is a preceding vehicle or an oncoming vehicle. If the subject of attention is neither a preceding vehicle nor an oncoming vehicle, the process ends. If the subject of attention is a preceding vehicle or an oncoming vehicle, the process proceeds to S4037. In S4037, the arithmetic operation unit 103 calculates the vehicle speed of the preceding vehicle or the oncoming vehicle. Specifically, the arithmetic operation unit 103 calculates the vehicle speed of the preceding vehicle or the oncoming vehicle by subtracting the vehicle speed measured by the vehicle speed measuring unit 105 from a change in a distance to the preceding vehicle or the oncoming vehicle acquired at a different time. In S4038, the arithmetic operation unit 103 superimposedly displays the vehicle speed of the preceding vehicle or the oncoming vehicle calculated in S412 on the video. Thereby, for example, a driver of the following vehicle of the vehicle 200 can ascertain whether the following vehicle can travel in the opposite lane with sufficient time until the oncoming vehicle approaches when passing the vehicle 200.
Next, in S4039, the arithmetic operation unit 103 determines whether the subject of attention is a traffic light. If the subject of attention is not a traffic light, the process ends. If the subject of attention is a traffic light, the process proceeds to S4040. In S4040, the arithmetic operation unit 103 highlights the traffic light to superimposedly display it on the video.
In a vehicle-mounted display device 100 of Example 2, a display unit 701 (
Next, in S9023, the arithmetic operation unit 103 determines whether the direction indicator of the vehicle 801 is blinking on the basis of the result of the detection process in S9022. If the direction indicator of the vehicle 801 is not blinking, the process ends. If the direction indicator of the vehicle 801 is blinking, it means that the driver of the vehicle 801 intends to cause the vehicle 200 to turn left. Therefore, in this case, the process proceeds to S9024.
Next, in S9024, the imaging unit 101 captures an image of a region in front of the vehicle 200 to acquire a video. In S9025, the arithmetic operation unit 103 starts to display the video on the display unit 701 provided, placed, positioned, located, or disposed on the side where the vehicle 801 is located. Meanwhile, similarly to Example 1, the display of the video on the display unit 701 may be started or ended depending on whether there is a subject of attention in the video acquired in S9024. The subject of attention is the same as that in Example 1, and is, for example, a vehicle preceding the vehicle 200, an oncoming vehicle, a pedestrian, a point of intersection, a traffic light, or the like.
Embodiment(s) of the disclosure can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like. The arithmetic operation unit 103, ay be implemented by such computer or processor.
While the disclosure has been described with reference to exemplary embodiments, it is to be understood that the disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
This application claims the benefit of Japanese Patent Application No. 2020-072257, filed Apr. 14, 2020, which is hereby incorporated by reference wherein in its entirety.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-072257 | Apr 2020 | JP | national |
