Patent Grant
11528413
This application is a U.S. National Phase of International Patent Application No. PCT/JP2017/026413 filed on Jul. 21, 2017, which claims priority benefit of Japanese Patent Application No. JP 2016-161826 filed in the Japan Patent Office on Aug. 22, 2016. Each of the above-referenced applications is hereby incorporated herein by reference in its entirety.
The present disclosure relates to an image processing apparatus, an image processing method, and a program. Specifically, the present disclosure relates to an image processing apparatus, an image processing method, and a program with which images photographed by a camera installed in a vehicle and composite images based on photographed images are displayed on a display unit (monitor) which a driver can observe, to thereby enable the driver to check vehicle surrounding conditions.
In a case where states in a travel direction of a car, for example, such as the direct front, sides, or rear of the car cannot be visually recognized directly from a driver seat, there is a risk that the car may hit a hidden obstacle present in the travel direction. One of systems to avoid such a risk is an image-presenting system to generate a virtual image showing a view from above the car on the basis of images photographed by a plurality of wide-angle cameras mounted on the front, rear, and sides of the car and displays them on a display unit of a driver seat.
It should be noted that the image showing a view observed from above the car is called “around-view (registered trademark) image”, “overhead image”, “bird's-eye view image”, or the like. Hereinafter, in this specification, the image showing a view observed from above the car will be described as the “overhead image”.
A configuration to further display an image showing a view in a vehicle travel direction in the image (overhead image) showing a view observed from above the car, for example, a system to display a photographed image of the rear next to the overhead image at the time of reversing (backing) of the car has also already been productized.
In North America, legal safety standards relating to obligation to present an image showing a view in a backing direction, which is information regarding the travel direction when a vehicle is backed, to a driver are defined. A configuration to present the image showing a view in the travel direction as described above is essential.
However, for example, in a case of backing while making a large curve in parking in a garage or the like, an obstacle on a left or right side of the vehicle is insufficiently checked only with an image showing a view from above the vehicle (overhead image) and an image showing a view in a rear direction which is the travel direction. Thus, there is a risk that the side of the vehicle may hit the obstacle.
The overhead image includes images of predetermined ranges around the vehicle, which show views in front, rear, left, and right directions of the vehicle. When taking an image of a stereoscopic object standing on a road surface in image conversion processing when generating an overhead image, the taken image is considered as being generally a pattern on the road surface without considering an original three-dimensional shape of the image, and the captured image is converted into the overhead image of the road surface virtually as the pattern on the road surface. Therefore, the image of the stereoscopic object, which is not present on the road surface, is distorted. It is thus difficult for the driver to accurately grasp a height of each display object (object) displayed in that overhead image showing a view from above and a perceived horizontal distance thereto.
The overhead image is generated as a composite image of images photographed by a plurality of cameras mounted in a periphery of the vehicle, for example, a composite image of images photographed by four cameras mounted on front, rear, left, and right parts of the vehicle.
The cameras mounted in the periphery of the vehicle each utilize a wide-angle lens such as a fisheye lens in order to photograph images of a wide viewing-angle region from a close distance. However, an image photographed by the wide-angle lens not using a central projection method causes a larger image distortion as compared to an image directly viewed when we generally view an object. In particular, in a case where a captured image is converted into an image at a different viewpoint and is viewed assuming that it is present on a particular spatial plane, an arbitrary target object whose projected image is expanded three-dimensionally in a world space causes a large distortion.
In conversion when generating an image called overhead image, assuming that a captured surrounding space extends in the surroundings of the vehicle body on a plane on which the wheels of the vehicle body are placed, overhead images photographed by the respective cameras at the positions of the cameras mounted on the vehicle body are generated and an overhead image to be displayed is generated by performing image correction processing such as composition of those respective images photographed by the plurality of cameras.
It should be noted that, for example, Patent Literature 1 (Japanese Patent Application Laid-open No. 2015-119225) has disclosed a conventional technology regarding the overhead image generation processing. Patent Literature 1 has disclosed a configuration to generate a composite image showing a view from above a car by performing overhead conversion on images photographed by image pickup apparatuses mounted in a periphery of a vehicle to combine them. Patent Literature 1 employs a configuration in which two corrected images are alternately displayed in a strip shape for a combined region of two overhead images which are targets to be combined.
However, unlike an image captured by a telecentric optical system truly from above the vehicle, which shows a stereoscopic object as it is actually viewed from above, an image of the shape of each object included is distorted in a case of a display image obtained by converting the images captured by the wide-angle cameras mounted on the sides of the vehicle into the overhead image showing a view virtually from above the vehicle. It is thus difficult to accurately judge a perceived distance from the vehicle body. In particular, it is difficult to accurately grasp a perceived distance from the vehicle body, which depends on a height of the object. In some cases, the driver cannot accurately judge a risk that the vehicle may hit each display object (object) displayed in the overhead image.
Patent Literature 1: Japanese Patent Application Laid-open No. 2015-119225
Patent Literature 2: Japanese Patent Application Laid-open No. 2005-339313
The present disclosure has been made of the above-mentioned problem, for example, and aims at providing an image processing apparatus, an image processing method, and a program with which images photographed by a camera installed in a vehicle and composite images based on photographed images are displayed on a display unit (monitor) which the driver can observe, to thereby enable the driver to more surely check vehicle surrounding conditions.
In accordance with a first aspect of the present disclosure,
receives an image photographed by a camera that photographs surroundings of a vehicle,
generates, on the basis of the received photographed image,
outputs the generated vehicle travel direction image and the generated vehicle lateral side image to the display unit.
In addition, in accordance with a second aspect of the present disclosure,
In addition, in accordance with a third aspect of the present disclosure,
It should be noted that the program of the present disclosure is, for example, a program that can be provided to an image processing apparatus or a computer system, which is capable of executing various program codes, through a storage medium provided in a computer readable manner or through a communication medium. By providing such a program in a computer readable manner, processing according to the program is performed in the information processing apparatus or the computer system.
Additional other objects, characteristics, and advantages of the present disclosure will be clear through a more detailed description based on an embodiment of the present invention to be described later and the accompanying drawings. It should be noted that the term “system” set forth herein is a logical complex configuration of a plurality of apparatuses and apparatuses with various configurations are not necessarily provided in the same housing.
In accordance with the configuration of the embodiment of the present disclosure, an image processing apparatus that achieves image display with improved visibility of information regarding the surroundings of a vehicle is realized.
In accordance with the configuration of the present invention, an image processing apparatus that achieves image display with improved visibility of information regarding the surroundings of a vehicle is realized.
It should be noted that effects set forth herein are merely exemplary and are not limitative and additional effects may be provided.
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
Hereinafter, an image processing apparatus, an image processing method, and a program of the present disclosure will be described in detail with reference to the drawings. It should be noted that descriptions thereof will be made in accordance with the following items.
[1. Regarding Example of Display Image of Display Unit of Vehicle]
First of all, an example of a display image of a display unit of a vehicle will be described.
The display unit 1 is set at a position such that the driver of the car can view it while driving.
As shown in
It should be noted that the display image example of the display unit 1 shown in
The overhead image 2 is a composite image generated by combining a plurality of overhead images generated by performing overhead conversion on each of images photographed by cameras mounted at a plurality of points such as the front, rear, left, and right of the car, and is a virtual image corresponding to a case of observing the car and the surroundings thereof from above the car.
It should be noted that the overhead image 2 includes an image spuriously generated, which shows the shape of its own vehicle, and the driver is enabled to check surrounding conditions of its own vehicle.
The travel direction image 3 (=the rear image) is an image generated by performing overhead conversion, cutting processing, and the like on the image photographed by the camera mounted on the back of the car, for example.
Such image processing on the photographed image is performed at an image processing unit inside the vehicle.
An example of mounting the cameras that photograph images to be required for generating the display image for the display unit 1 will be described with reference to
As shown in
The four cameras are mounted as follows:
a camera F21 is mounted on a front surface of the vehicle 10;
a camera L22 is mounted on a left surface of the vehicle 10;
a camera R23 is mounted on a right surface of the vehicle 10; and
a camera B24 is mounted on a back surface of the vehicle 10.
As shown in a camera configuration of
A distortion of an image excluding the optical axis is caused in an image photographed by using the wide-angle lens as compared to a projection method which is a central projection for photographs which is generally familiar. An example of a photographed image showing a view of a vertically lower region by using the wide-angle lens will be described with reference to
The camera 30 is a camera with a wide-angle lens such as a fisheye lens, for example.
A photographed image shown in
As shown in the photographed image of
That is a distortion of an image which is caused by photographing using a wide-angle fisheye lens.
In order to generate an image having a shape similar to the object to be photographed on the road surface by removing a distortion of the photographed image shown in
Such correction processing example will be described with reference to
A problem thereof will be briefly described with reference to
It should be noted that the image showing a view observed from above will be referred to as an “overhead image” and the image correction processing of performing conversion into the “overhead image” will be referred to as “overhead conversion”. The corrected image shown in
(
The overhead image 2 shown in
It should be noted that the travel direction image 3 (=the rear image) shown in
It should be noted that there may be employed a configuration in which besides the four cameras mounted at the respective positions of the front, rear, left, and right of the vehicle 10 shown in
[ 2. Regarding Problem in Observed Image (Overhead Image) Based on Virtual Viewpoint from above Vehicle]
As described above with reference to
However, as a result of such image correction, the shape of an object standing on a road surface in the display image, which is not present on a road surface of an assumed converted and projected plane, is different from the shape of the actual object in some cases. In particular, the shape of an object near an image composition region changes with high possibility.
Therefore, it is difficult for the driver to accurately grasp a specific shape of each object displayed in the overhead image displayed on the display unit, especially a height of the object and a relative distance from the vehicle body. In some cases, the driver cannot accurately judge a risk that the vehicle may hit a display object in the overhead image.
A specific example will be described with reference to
The vehicle 41 is to be backed to enter a parking space in accordance with the arrow shown in the figure by turning (rotating) a steering wheel to the right.
However, an obstacle (object) 42, i.e., a “trash can” is present on the left side of the vehicle 41 as shown in the figure.
The height of the obstacle (object) 42 cannot be directly observed from the driver because it is lower than a height of the vehicle.
At this time, an example of an image displayed on a display unit of the vehicle 41 is shown in
The display image of the display unit 1 shown in
As shown in
An obstacle (object) image 42a, i.e., the image of the “trash can” is displayed in the overhead image 2. However, deformation due to the various types of correction processing and the like in the generation process for the overhead image is large as described above, and thus it is not an image so clear that it can be judged to be the trash can and having a shape maintaining the shape of the trash can.
In particular, regarding a perceived distance from the vehicle body which depends on a height of an obstacle (object) image 42a, it cannot be judged at all. There is a possibility that the driver does not remark it because the shape of the obstacle (object) image 42a is displayed in a shape distorted and deformed from a shape intuitively viewed from the side. Further, even if the driver remarks something displayed, there is a possibility that the driver judges it to be a stain of the floor, a trash on the floor, or the like. Further, in a case of a flat wall surface or the like, it is difficult even to judge whether it is a wall or a road surface.
When the driver keeps moving the vehicle 41 rearward as it is on the basis of such a wrong judgement, the vehicle body is backed while drawing a sweeping track extended in a lateral-side direction in which the obstacle is present due to a difference between tracks followed by front and back outer wheels when turning (outer wheel difference). As a result, as shown in next
When the vehicle moves while making a curve by operating the steering wheel, the tracks of the front wheels are different from the tracks of the rear wheels. That is, at the time of backing, a difference between a rear-wheel track and a vehicle-body front portion track is caused due to the outer wheel difference.
As shown in
[3. Regarding Image Display Configuration Having Improved Safety when Vehicle Travels]
Next, the above-mentioned problem will be solved and a configuration of the present disclosure, i.e., an image display configuration having improved safety when the vehicle travels will be described.
A display image of the display unit 100 shown in
It should be noted that unless otherwise stated, the vehicle lateral side image may be an image according to a semi-central projection method using a vehicle body side surface as the viewing angle, which includes front wheels and a vehicle-body front portion from a camera-mounting position, an image according to the semi-central projection method using the vehicle body side surface as the viewing angle, which includes rear wheels and a vehicle-body rear portion from the camera-mounting position in a case where the vehicle travels forward, or a part of a camera image of a rear-view mirror replacement called camera monitor system described above.
The two images, (a) Overhead image 101 and (b) Travel direction image (rear image) 102B, are images similar to the images described above with reference to
It should be noted that as shown in the overhead image 101 at the center of
On the left side of the vehicle, a “trash can” which is an obstacle (object) similar to the one described with reference to
In the overhead image 101 at the center of
However, there is a fear that the driver cannot correctly recognize the shape and the height of that obstacle image (object image) 111a even when viewing it and judges that there is no crash possibility of the vehicle.
(C) Vehicle lateral side image (left side image) 103L shown in
The vehicle lateral side image (left side image) 103L is an image of a left side of the vehicle.
As described above, in a case where the vehicle backs the vehicle counter-clockwise (leftward) by turning the steering wheel to the right, the tracks of the front wheels are drawn outside the tracks of the corresponding rear wheels. That is, the vehicle front portion is rotated while the track of the vehicle front portion is expanded outward as compared to the track of the vehicle rear portion due to the outer wheel difference. As a result, there is a possibility that the left surface of the vehicle hits or crashes into the obstacle at the vehicle front portion, which it does not hit at the vehicle rear portion.
In this manner, a difference between the track of the vehicle front portion and the track of the rear portion due to the outer wheel difference is generated.
The vehicle lateral side image (left side image) 103L shown in
The vehicle lateral side image (left side image) 103L shown in
By checking that vehicle lateral side image (left side image) 103L of a central projection image less distorted, which is shown in
In the vehicle lateral side image (left side image) 103L shown in
Consequently, the driver can stop causing the vehicle to travel, perform an operation on the steering wheel and the like, change the vehicle travel direction to avoid the obstacle, and cause the vehicle to safely travel.
It should be noted that a guide line indicating the travel direction of the car is displayed also in the overhead image 101 or the travel direction image (rear image) 102B.
An arrow in the overhead image 101 and predicted tracks of left and right wheels or bumper corners of a vehicle body foremost portion, which are shown as the dotted lines in the travel direction image (rear image) 102B, are displayed as the guide line indicating the travel direction of the car.
It should be noted that the example shown in
That is, the vehicle front side moves in a left direction. An image of the vehicle lateral side image (left side image) 103L shown in
On the contrary, in a case where the vehicle is backed clockwise (rightward) by turning the steering wheel of the vehicle to the left, the front side of the vehicle is rotated while the track of the front side of the vehicle is expanded rightward as compared to the track of the rear side.
That is, the vehicle front side moves in a right direction. In this case, an image of a right side of the vehicle, not the vehicle lateral side image (left side image) 103L shown in
In the image display example shown in
It is an image layout for making it easy for the driver who views the display unit 100 to intuitively grasp that the vehicle lateral side image (left side image) 103L is an image of a front left side of the vehicle.
The following two images, which are the display images of the display unit 100 shown in
At the center of
The vehicle is set to be reversed (backed) by turning the steering wheel to the right.
In this setting, the vehicle is to move rearward as shown in the double-line arrow in (Reference diagram) rearward and substantially the front half of the vehicle is to move in the left direction as shown in the double-line arrow.
A rear image of the vehicle shown in the double-line arrow in (Reference diagram) shown in
Further, the image of the left side of the vehicle which is shown as the dotted-line arrow in (Reference diagram) is the vehicle lateral side image (left side) 103L shown on the left side of
A fan-shaped region B in (Reference diagram) corresponds to an image region of the travel direction image (rear image) 102B which is shown substantially on the right side of
In the vehicle lateral side image (left side) 103L shown on the left side of
That vehicle movement direction display line 112 is a line indicating a movement direction of the vehicle, which is determined in accordance with a steering-wheel operation of the vehicle.
The image processing unit inputs steering-wheel operation information of the vehicle, calculates a movement direction of the vehicle on the basis of the input steering-wheel operation information of the vehicle, determines a drawing line in accordance with the movement direction of the vehicle, which is determined in accordance with the result of calculation, superimposes the vehicle movement direction display line 112 on the image, and outputs them. Coordinates of the drawing line may form a line drawing a track at an outermost point on the road surface, which is swept by the vehicle body, or may form an expanded track line considering a margin of a certain safety margin (e.g., 10 cm). A setting like a VR virtual display 112a of a sweeping space of an outermost end portion of the vehicle body may be employed. Further, a drawing line depending on an amount of turning of the steering wheel or a track at the time of maximally turning the steering wheel may be additionally displayed or may be alternatively displayed. In addition, an image processing apparatus installed in the vehicle may be caused to perform a hollow-coordinate arithmetic processing and display target-object coordinates as hollow coordinates and perform color change processing of a hollow space at a hit site or perform display with emphasis by drawing a pattern as in an obstacle partial region image 111c in
Further, the vehicle lateral side image (left side) 103L is set to include an image of a side region of the vehicle, i.e., a vehicle lateral side image 113.
By providing the display image with the vehicle lateral side image 113 in this manner, it becomes possible to accurately grasp the distance from the vehicle to the obstacle and the size of the obstacle.
It should be noted that a setting to also provide the travel direction image (rear image) 102B shown in
By providing the display image with a vehicle bumper image 114 which is an end portion region image of the vehicle itself in this manner, it becomes possible to accurately grasp the distance from the vehicle to the obstacle and the size of the obstacle.
It should be noted that the vehicle lateral side image (left side) 103L and the travel direction image (rear image) 102B which are shown in
Alternatively, there may be employed a configuration in which dedicated cameras that photograph the vehicle lateral side image (left side) 103L and the travel direction image (rear image) 102B which are shown in
That is, there may be employed a configuration in which wide-angle cameras or normal cameras as dedicated cameras which are a camera that photographs a region from the lateral side to the front of the vehicle and a camera that photographs the rear of the vehicle are mounted on the vehicle and the image processing apparatus inside the vehicle inputs images photographed by those cameras and generates and displays the vehicle lateral side image (left side) 103L and the travel direction image (rear image) 102B which are shown in
The image display examples of the display unit 100, which have been described with reference to
Next, an example of the display image of the display unit 100 in a case where the vehicle is backed clockwise (rightward) by turning the steering wheel of the vehicle to the left will be described with reference to
As shown in
The two images, (a) Overhead image 101 and (b) Travel direction image (rear image) 102B, are images similar to the images described above with reference to
In the example shown in
However, the driver cannot recognize the shape and the height of that obstacle image (object image) 111a even when viewing it and judges that there is no crash possibility of the vehicle.
(C) Vehicle lateral side image (right side image) 103R shown in
In a case where the vehicle backs the vehicle clockwise (rightward) by turning the steering wheel to the left, the tracks of the front wheels are drawn outside the tracks of the corresponding rear wheels. That is, the vehicle front portion is rotated while the track of the vehicle front portion is expanded outward as compared to the track of the vehicle rear portion. As a result, there is a possibility that the right surface of the vehicle hits or crashes into the obstacle at the vehicle front portion, which it does not hit at the vehicle rear portion.
That is, a difference between the track of the vehicle front portion and the track of the rear portion due to the outer wheel difference is generated.
The vehicle lateral side image (right side image) 103R shown in
In the vehicle lateral side image (right side image) 103R shown in
By viewing the vehicle lateral side image (right side image) 103R shown in
In the vehicle lateral side image (right side image) 103R shown in
Consequently, the driver can stop causing the vehicle to travel, perform an operation on the steering wheel and the like, change the vehicle travel direction to avoid the obstacle, and cause the vehicle to safely travel.
It should be noted that the layout of the three images of the display unit 100 shown in
That is, in the image display example of the display unit 100 shown in
It is an image layout for making it easy for the driver who views the display unit 100 to intuitively grasp that the vehicle lateral side image (right side image) 103R is an image of a front right side of the vehicle.
The following two images, which are the display images of the display unit 100 shown in
At the center of
The vehicle is set to be reversed (backed) by turning the steering wheel to the left.
In this setting, the vehicle is to move rearward as shown in the double-line arrow in (Reference diagram) and substantially the front half of the vehicle is to move in the right direction as shown in the double-line arrow.
A rear image of the vehicle shown in the double-line arrow in (Reference diagram) shown in
Further, an image of the right side of the vehicle which is shown as the dotted-line arrow in (Reference diagram) is a vehicle lateral side image (right side) 103R shown on the right side of
A fan-shaped region B in (Reference diagram) corresponds to an image region of the travel direction image (rear image) 102B which is shown substantially on the left side of
In the vehicle lateral side image (right side) 103R shown on the left side of
That vehicle movement direction display line 112 is a line indicating a movement direction of the vehicle, which is determined in accordance with a steering-wheel operation of the vehicle.
The image processing unit inputs steering-wheel operation information of the vehicle, calculates a movement direction of the vehicle on the basis of the input steering-wheel operation information of the vehicle, determines a drawing line in accordance with the movement direction of the vehicle, which is determined in accordance with the result of calculation, superimposes the vehicle movement direction display line 112 on the image, and outputs them.
Further, the vehicle lateral side image (right side) 103R is set to include an image of a side region of the vehicle, i.e., a vehicle lateral side image 113.
By providing the display image with the vehicle lateral side image 113 in this manner, it becomes possible to accurately grasp the distance from the vehicle to the obstacle and the size of the obstacle.
It should be noted that a setting to also provide the travel direction image (rear image) 102B shown in
By providing the display image with the vehicle bumper image 114 in this manner, it becomes possible to accurately grasp the distance from the vehicle to the obstacle and the size of the obstacle.
The image display examples described with reference to
In such a case, an image of the left side or an image of the right side is additionally displayed as an image showing a view in the movement direction of the vehicle in addition to the rear image.
However, in a case of forward travel without turning the steering wheel, the vehicle only moves in the front and rear directions and does not move in the left and right directions. Therefore, display processing of the left side image and the right side image is not performed.
Only the overhead image 101 and the travel direction image (rear image) 102B are displayed on the display unit.
In this case, the vehicle does not move in the left and right directions, and even if an obstacle is present on the left or right sides of the vehicle, there is no possibility that the vehicle will hit or crash into the obstacle. Therefore, display processing of the side image and the right side image is not performed.
In this setting, only the overhead image 101 and the travel direction image (front image) 102F are displayed on the display unit.
Also in this case, the vehicle does not move in the left and right directions, and even if an obstacle is present on the left or right sides of the vehicle, there is no possibility that the vehicle will hit or crash into the obstacle. Therefore, display processing of the side image and the right side image is not performed.
The display unit 100 is provided in a region such that the driver of the car can view it while driving.
The display example of the display unit 100 shown in
The image to be displayed on the display unit 100 is switched in accordance with a movement direction of the vehicle. The image processing apparatus inside the vehicle inputs vehicle information, specifically, the movement direction of the vehicle, forward travel or rearward travel, the direction of rotation of the steering wheel, the angle-of-rotation information, and the like and performs switching processing of the display image of the display unit 100 in accordance with such vehicle information.
A shift example of the display image of the display unit 100 will be described with reference to
In the case of (S01) Forward travel in straight line, only the overhead image 101 and the travel direction image (front image) 102F are displayed on the display unit as described above with reference to
In this case, the vehicle does not move in the left and right directions, and even if an obstacle is present on the left or right sides of the vehicle, there is no possibility that the vehicle will hit or crash into the obstacle. Therefore, display processing of the side image and the right side image is not performed.
Also in the case of (S02) Rearward travel in straight line (backing in straight line), only the overhead image 101 and the travel direction image (rear image) 102B are displayed on the display unit as described above with reference to
Also in this case, the vehicle does not move in the left and right directions, and even if an obstacle is present on the left or right sides of the vehicle, there is no possibility that the vehicle will hit or crash into the obstacle. Therefore, display processing of the side image and the right side image is not performed.
In the case of (S03) Steering-wheel right-rotation rearward travel (counter-clockwise (left) backing), the three images, the overhead image 101, the travel direction image (rear image) 102B, and the vehicle lateral side image (left side image) 103L, are displayed on the display unit as described above with reference to
In this case, the vehicle front portion moves in the left direction. The vehicle lateral side image (left side image) 103L is an image showing a view in that movement direction. That image is displayed in the front left region of the overhead image 101. It is an image layout for causing the driver who views the display unit 100 to intuitively grasp that the vehicle lateral side image (left side image) 103L is the image of the front left side of the vehicle.
It should be noted that the vehicle lateral side image (left side image) 103L includes the left side image of the vehicle itself and a vehicle movement direction display line indicating the movement direction of the vehicle is further displayed.
In the case of (S04) Steering-wheel left-rotation rearward travel (clockwise (right) backing), the three images, the overhead image 101, the travel direction image (rear image) 102B, and the vehicle lateral side image (right side image) 103R, are displayed on the display unit as described above with reference to
In this case, the vehicle front portion moves in the right direction. The vehicle lateral side image (right side image) 103R is an image showing a view in that movement direction. That image is displayed in the front right region of the overhead image 101. It is an image layout for causing the driver who views the display unit 100 to intuitively grasp that the vehicle lateral side image (right side image) 103R is the image of the front right side of the vehicle.
It should be noted that the vehicle lateral side image (right side image) 103R includes the right side image of the vehicle itself and a vehicle movement direction display line indicating the movement direction of the vehicle is further displayed.
The four image display examples shown in
For example, the overhead image, the travel direction image (rear image), and the vehicle lateral side image (left/right side image) are displayed in the following vehicle states shown in
Further, the guide lines indicating the travel direction or movement direction of the car, which are shown in the respective images, are also sequentially updated in a manner that depends on the angle of rotation of the steering wheel.
The image processing apparatus inside the car sequentially inputs vehicle information such as an angle of rotation of the steering wheel and car speed information, calculates a travel direction or movement direction of the car on the basis of such input information, and performs drawing processing or updating processing of the guide line shown in each image.
[4. Regarding Other Image Display Examples on Display Unit]
Next, other image display examples for the display unit will be described.
[4-1. Other Image Display Example 1]
In the example described above with reference to
However, it is not essential to display those three types of images on the display unit. There may be employed a configuration in which any of those three types of images or any two combined images of them are displayed.
(S03) Steering-wheel right-rotation rearward travel (counter-clockwise (left) backing); and
(S04) Steering-wheel left-rotation rearward travel (clockwise (right) backing),
not (a) Overhead image,
but only the following two kinds of images:
(b) Travel direction image (rear image); and
(c) Vehicle lateral side image (left/right side image).
On the display unit, the vehicle lateral side image (left side image) is displayed at a position on the left side of the display unit, which corresponds to the vehicle's forward movement direction, and the travel direction image (rear image) is displayed on the right side.
On the display unit, the vehicle lateral side image (right side image) is displayed at a position on the right side of the display unit, which corresponds to the vehicle's forward movement direction, and the travel direction image (rear image) is displayed on the left side.
The setting to omit display of the overhead image and display only the travel direction image (rear image) and the vehicle lateral side image (left/right side image) as described above and a setting to display a combination of other images or only one image are also possible.
[4-2. Other Image Display Example 2]
The display unit 100 applied to the above-mentioned image display examples is set to be horizontally long.
The display unit installed in the vehicle is not limited to the setting to be horizontally long and may be set to be vertically long.
For example, as shown in
An image display example on the vertically long display unit 100 shown in
(
(
(
In the case of that vehicle state, the three images, the overhead image 101, the travel direction image (rear image) 102B, and the vehicle lateral side image (left side image) 103L, are displayed on the display unit 100 as shown in
The overhead image 101 and the vehicle lateral side image (left side image) 103L are displayed in an upper region of the display unit and the travel direction image (rear image) 102B is displayed in a lower region.
The vehicle lateral side image (left side image) 103L is displayed in the front left region of the overhead image 101. That display image layout is an image layout for causing the driver who views the display unit 100 to intuitively grasp that the vehicle lateral side image (left side image) 103L is the image of the front left side of the vehicle.
It should be noted that the vehicle lateral side image (left side image) 103L includes the left side image of the vehicle itself and a vehicle movement direction display line indicating the movement direction of the vehicle is further displayed.
(
In the case of that vehicle state, the three images, the overhead image 101, the travel direction image (rear image) 102B, and the vehicle lateral side image (right side image) 103R, are displayed on the display unit 100 as shown in
The overhead image 101 and the vehicle lateral side image (right side image) 103R are displayed in the upper region of the display unit and the travel direction image (rear image) 102B is displayed in the lower region.
The vehicle lateral side image (right side image) 103R is displayed in the front right region of the overhead image 101. That display image layout is an image layout for causing the driver who views the display unit 100 to intuitively grasp that the vehicle lateral side image (left side image) 103L is the image of the front left side of the vehicle.
It should be noted that the vehicle lateral side image (right side image) 103R includes the left side image of the vehicle itself and a vehicle movement direction display line indicating the movement direction of the vehicle is further displayed.
As in
(
(
In the case of that vehicle state, as shown in of
The overhead image 101 is displayed in the upper region of the display unit and the vehicle lateral side image (left side image) 103L is displayed overlapping in a front left region of a vehicle region of the overhead image 101. The travel direction image (rear image) 102B is displayed in the lower region.
The example shown in
With this display image, the driver can more surely check whether or not there is an obstacle in the movement direction of the vehicle.
This example superimposes and displays the vehicle lateral side image (left side image) 103L on a front left portion of the vehicle region of the overhead image 101 and is an image which makes it possible to inform the driver of the fact that it is necessary to pay attention to that position.
In the case of that vehicle state, the three images, the overhead image 101, the travel direction image (rear image) 102B, and the vehicle lateral side image (right side image) 103R, are displayed on the display unit 100 as shown in
The overhead image 101 is displayed in the upper region of the display unit and the vehicle lateral side image (right side image) 103R is displayed overlapping in a front right region of the vehicle region of the overhead image 101. The travel direction image (rear image) 102B is displayed in the lower region.
The example shown in (2) of
[5. Regarding Other Embodiments]
Next, a modified example of the above-mentioned embodiment will be described.
In the above-mentioned embodiment, a switching shift of the display mode described with reference to
Such display mode switching may be configured to perform cyclic switching according to an operator's button operation, switching according to an operation for an arrangement position desired to be displayed on the overhead image through a touch panel, and switching of the display mode according to a direction indication operation and the like through a gesture.
Further, the configuration example of displaying, on the display unit, the vehicle front lateral side image along with the image showing a view in the vehicle travel direction (in the rear of the vehicle) at the time of backing of the vehicle has been described in the embodiment above. However, the configuration of displaying the vehicle rear lateral side image is favorably employed also when the vehicle travels forward.
By performing such display processing, it is possible to cause the driver to check an obstacle in the left or right directions behind the vehicle in the case of traveling in the front direction by turning the steering wheel.
An example of the display image of the display unit 100 in a case of causing the vehicle to travel forward while curving the vehicle by turning the steering wheel when the vehicle travels forward will be described with reference to
As shown in
In the example shown in
However, there is a possibility that even when the driver views that obstacle image (object image), the driver cannot recognize the shape and the height thereof and judges that there is no crash possibility of the vehicle.
(C) Vehicle rear lateral side image (rear left side image) 103Lb shown in
In a case where the vehicle causes the vehicle to travel forward and counter-clockwise (leftward) by turning the steering wheel to the left, the tracks of the rear wheels are drawn inside the tracks of the corresponding front wheels. That is, the vehicle rear portion is rotated while passing inside the vehicle front portion. As a result, there is a possibility that the left surface of the vehicle hits or crashes into the obstacle at the vehicle rear portion, which it does not hit at the vehicle front portion.
That is, a difference between the track of the vehicle front portion and the track of the rear portion due to a difference between tracks followed by front and back inner wheels when turning (inner wheel difference) is generated.
The vehicle rear lateral side image (rear left side image) 103Lb shown in
The vehicle rear lateral side image (rear left side image) 103Lb shown in
By viewing the vehicle rear lateral side image (rear left side image) 103Lb shown in
In the vehicle rear lateral side image (rear left side image) 103Lb shown in
Consequently, the driver can stop causing the vehicle to travel, perform an operation on the steering wheel and the like, change the vehicle travel direction to avoid the obstacle, and cause the vehicle to safely travel.
It should be noted that in the layout of the three images of the display unit 100 shown in
It is an image layout for causing the driver who views the display unit 100 to intuitively grasp that the vehicle rear lateral side image (rear left side image) 103Lb is the image of the rear left side of the vehicle.
As shown in
In the example shown in
However, there is a possibility that even when the driver views that obstacle image (object image), the driver cannot recognize the shape and the height thereof and judges that there is no crash possibility of the vehicle.
(C) Vehicle rear lateral side image (rear right side image) 103Rb shown in
In a case where the vehicle causes the vehicle to travel forward and clockwise (rightward) by turning the steering wheel to the right, the tracks of the rear wheels are drawn inside the tracks of the corresponding front wheels. That is, the vehicle rear portion is rotated while passing inside the vehicle front portion. As a result, there is a possibility that the right surface of the vehicle hits or crashes into the obstacle at the vehicle rear portion, which it does not hit at the vehicle front portion.
That is, a difference between the track of the vehicle front portion and the track of the rear portion due to the inner wheel difference is generated.
The vehicle rear lateral side image (rear right side image) 103Rb shown in
The vehicle rear lateral side image (rear right side image) 103Rb shown in
By viewing the vehicle rear lateral side image (rear right side image) 103Rb shown in
In the vehicle rear lateral side image (rear right side image) 103Rb shown in
Consequently, the driver can stop causing the vehicle to travel, perform an operation on the steering wheel and the like, change the vehicle travel direction to avoid the obstacle, and cause the vehicle to safely travel.
It should be noted that in the layout of the three images of the display unit 100 shown in
It is an image layout for causing the driver who views the display unit 100 to intuitively grasp that the vehicle rear lateral side image (rear right side image) 103Lb is the image of the rear right side of the vehicle.
The image displayed on the display unit 100 is switched in accordance with a movement direction of the vehicle. The image processing apparatus inside the vehicle inputs vehicle information, specifically, the movement direction of the vehicle, forward travel or rearward travel, the direction of rotation of the steering wheel, the angle-of-rotation information, and the like and performs switching processing of the display image of the display unit 100 in accordance with such vehicle information.
A shift example of the display image of the display unit 100 will be described with reference to
In the case of (S01) Forward travel in straight line, only the overhead image and the travel direction image (front image) are displayed on the display unit.
In this case, the vehicle does not move in the left and right directions, and even if an obstacle is present on the left or right sides of the vehicle, there is no possibility that the vehicle will hit or crash into the obstacle. Therefore, display processing of the side image and the right side image is not performed.
Also in the case of (S02) Rearward travel in straight line (backing in straight line), only the overhead image and the travel direction image (rear image) are displayed on the display unit.
Also in this case, the vehicle does not move in the left and right directions, and even if an obstacle is present on the left or right sides of the vehicle, there is no possibility that the vehicle will hit or crash into the obstacle. Therefore, display processing of the side image and the right side image is not performed.
In the case of (S03) Steering-wheel right-rotation rearward travel (counter-clockwise (left) backing), the three images, the overhead image, the travel direction image (rear image), and the vehicle lateral side image (left side image), are displayed on the display unit.
In this case, the vehicle front portion moves in the left direction. The vehicle lateral side image (left side image) is an image showing a view in that movement direction. That image is displayed in the front left region of the overhead image. It is an image layout for causing the driver who views the display unit 100 to intuitively grasp that the vehicle lateral side image (left side image) is the image of the front left side of the vehicle.
It should be noted that the vehicle lateral side image (left side image) includes the left side image of the vehicle itself and a vehicle movement direction display line indicating the movement direction of the vehicle is further displayed.
In the case of (S04) Steering-wheel left-rotation rearward travel (clockwise (right) backing), the three images, the overhead image, the travel direction image (rear image), and the vehicle lateral side image (right side image), are displayed on the display unit.
In this case, the vehicle front portion moves in the right direction. The vehicle lateral side image (right side image) is an image showing a view in that movement direction. That image is displayed in the front right region of the overhead image. It is an image layout for causing the driver who views the display unit 100 to intuitively grasp that the vehicle lateral side image (right side image) is the image of the front right side of the vehicle.
It should be noted that the vehicle lateral side image (right side image) includes the right side image of the vehicle itself and a vehicle movement direction display line indicating the movement direction of the vehicle is further displayed.
In the case of (S05) Steering-wheel left-rotation forward travel (counter-clockwise forward travel), the three images, the overhead image, the travel direction image (front image), and the vehicle lateral side image (left side image), are displayed on the display unit.
In this case, the vehicle rear portion moves in the left direction. The vehicle lateral side image (left side image) is an image showing a view in that movement direction. That image is displayed in a rear left region of the overhead image. It is an image layout for causing the driver who views the display unit 100 to intuitively grasp that the vehicle lateral side image (left side image) is the image of the rear left side of the vehicle.
It should be noted that the vehicle lateral side image (left side image) includes the left side image of the vehicle itself and a vehicle movement direction display line indicating the movement direction of the vehicle is further displayed.
In the case of (S06) Steering-wheel right-rotation forward travel (clockwise forward travel), the three images, the overhead image, the travel direction image (front image), and the vehicle lateral side image (right side image), are displayed on the display unit.
In this case, the vehicle rear portion moves in the right direction. The vehicle lateral side image (right side image) is an image showing a view in that movement direction. That image is displayed in the rear right region of the overhead image. It is an image layout for causing the driver who views the display unit 100 to intuitively grasp that the vehicle lateral side image (right side image) is the image of the rear right side of the vehicle.
It should be noted that the vehicle lateral side image (right side image) includes the right side image of the vehicle itself and a vehicle movement direction display line indicating the movement direction of the vehicle is further displayed.
The six image display examples shown in
For example, the overhead image, the travel direction image (rear/front image), and the vehicle lateral side image (left/right side image) are displayed in the following vehicle states shown in
Further, the guide lines indicating the travel direction or movement direction of the car, which are shown in the respective images, are also sequentially updated in a manner that depends on the angle of rotation of the steering wheel.
The image processing apparatus inside the car sequentially inputs vehicle information such as an angle of rotation of the steering wheel and car speed information, calculates a travel direction or movement direction of the car on the basis of such input information, and performs drawing processing or updating processing of the guide line shown in each image.
It should be noted that the following setting may be made in the case where the vehicle is rotated while traveling forward as in (S05) or (S06) shown in
A table listing up the image display modes of (S1) to (S6) described with reference to
As shown in
In the case of (S01) Forward travel in straight line, the overhead image and the travel direction image (front image) are displayed.
In the case of (S02) Rearward travel in straight line (backing in straight line), the overhead image and the travel direction image (rear image) are displayed.
In the case of (S03) Steering-wheel right-rotation rearward travel (counter-clockwise (left) backing), the overhead image, the travel direction image (rear image), and the vehicle lateral side image (front left side image) are displayed.
In the case of (S04) Steering-wheel left-rotation rearward travel (clockwise (right) backing), the overhead image, the travel direction image (rear image), and the vehicle lateral side image (front right side image) are displayed.
In the case of (S05) Steering-wheel left-rotation forward travel (counter-clockwise forward travel), the overhead image, the travel direction image (front image), and the vehicle lateral side image (rear left side image) are displayed.
In the case of (S06) Steering-wheel right-rotation forward travel (counter-clockwise forward travel), the overhead image, the travel direction image (front image), and the vehicle lateral side image (rear right side image) are displayed.
It should be noted that as shown in the field “Display mode of lateral side image” in
Further, in a case where the vehicle travels rearward (is backed), the lateral side image is displayed by using a range within which there is an involving hit risk due to the outer wheel difference as a rearward image of the semi-central projection.
It should be noted that an image cut out from the image photographed by the wide-angle camera may be used as the lateral side image and it is favorable that it is converted into an image of a central projection method having less distortion like the one directly observed by eyes and is displayed.
[0] Further, the example in which the display position of the vehicle lateral side image is set to be arranged on the left- and right sides of the overhead image has been described in the embodiment above. However, various settings can be made for the display position of the image.
For example, although detailed arrangement of the screen on the vehicle lateral side may be uniquely determined by the image processing unit, the operator may be enabled to finely adjust such arrangement while the position of a sub-screen which is optimal for recognition and a situation where the adjustment can be intuitively understood are maintained in view of specific properties such as vehicle types and inner wheel difference and outer wheel difference characteristics and an operator's habit in virtual check and the like. On the other hand, for the arrangement with which a different operator who gets in the car can wrongly recognize the visual check position due to a check habit of a particular driver, warning may be provided or arrangement adjustment may be forbidden.
In addition, a configuration in which the vehicle includes execution units of surrounding-obstacle recognition processing and moving-object recognition processing may be employed. Further, there may be employed configurations in which in accordance with inputs from those processing execution units, a warning mark is displayed on the display image, a screen frame indicating an obstacle region or a body region is displayed flashing, and a gradation streaming pattern along a screen frame is displayed for calling for operator's attention, and sound alert is output.
[6. Regarding Configuration Example of Image Processing Apparatus]
Next, a configuration example of the image processing apparatus to be installed in the vehicle will be described with reference to
As shown in
The camera F, 202F corresponds to the camera F21 provided in the front surface of the vehicle 10 shown in
The camera L, 202L corresponds to the camera L22 provided in the left surface of the vehicle 10 shown in
The camera R, 202R corresponds to the camera R23 provided in the right surface of the vehicle 10 shown in
The camera B, 202B corresponds to the camera B24 provided in the back surface of the vehicle 10 shown in
Those cameras are all cameras including a wide-angle lens such as a fisheye lens, for example, and performs shooting of a moving image at a predetermined frame rate and outputs the shot image to the image processing unit 201.
It should be noted that this camera setting example is an example, and a configuration including a dedicated normal camera and a wide-angle camera for photographing the rear of the vehicle and the respective vehicle lateral sides which are the left and the right as described above, for example, may be employed.
Also in the case of including those cameras, images photographed by those cameras are all input into the image processing unit 201.
The image processing unit 201 inputs images photographed by the cameras mounted on the vehicle and generates and outputs images to be displayed on the display unit 204. The images to be generated are the following respective images described with reference to
The image processing unit 201 generates four overhead images by performing overhead conversion on the respective images photographed by the camera F, 202F to the camera B, 202B, for example, and further generates an overhead image to be displayed as a composite image in which those four images are combined, the overhead image to be displayed being an image showing a view observed from above the vehicle and having a schematic view of the vehicle which is arranged at the center.
It should be noted that the vehicle lateral side image (left/right side image) 103L/R and the travel direction image (rear image) 102B are generated by the image processing unit 201 performing the image correction processing such as the overhead conversion and the image cutting processing of the images photographed by the camera F, 202F to the camera B, 202B.
An image generated by the image processing unit 201 is displayed on the display unit 204 via the display control unit 203.
The display control unit 203 performs arrangement control and the like of the following respective images to be displayed on, for example, the display unit 204, i.e.,
For example, performed is display control processing in which the vehicle lateral side image (left side image) 103L is arranged and displayed in the upper left of the overhead image 101 when displaying the vehicle lateral side image (left side image) 103L, and the vehicle lateral side image (right side image) 103R is arranged and displayed in the upper right of the overhead image 101 when displaying the vehicle lateral side image (right side image) 103R.
The image processing unit 201 further performs processing of generating and drawing a guide line of an arrow or the like indicating the movement direction of the vehicle to be displayed superimposed on the following respective images, processing of updating it, and the like:
The vehicle information and other input information are input into the image processing unit 201 or the display control unit 203 via the control unit 210.
The input unit 205 includes, for example, a switch, various keys, a touch panel, and the like for inputting input information of a user, and is capable of performing processing such as movement and enlargement of the display image in accordance with an operation of the touch panel integrated with the display unit 204, for example.
The display control unit 203 performs display control of an image according to information input from the input unit 205. For example, enlargement of the image, reduction in size of the image, change of a display position of the image, and the like are performed. It should be noted that the input unit 205 may be configured as an input unit including an operator's gesture recognition unit. Further, a high-level input system such as head motion and hand gesture recognition, gesture analysis processing using another time-of-flight sensor (TOF sensor), a stereo camera, and an image, and the like may be used.
The vehicle information acquisition unit 206 acquires various types of vehicle information regarding the vehicle such as speed, travel direction, steering wheel setting information, and the like of the vehicle.
Such vehicle information is input into the image processing unit 201 or the display control unit 203 via the control unit 210.
The control unit 210 performs various types of data processing according to a program stored in the storage unit 207, for example.
Specifically, for example, the various types of vehicle information regarding the vehicle such as the speed, the travel direction, the steering wheel setting information, and the like of the vehicle, which is acquired by the vehicle information acquisition unit 206, are input. An image mode of the image to be generated by the image processing unit 201 is determined. Further, the display control unit 202 performs determination processing regarding an image layout mode to be set and the like.
In addition, calculation processing and the like for a line position, the shape, and the like for setting the guide line indicating the movement direction of the car to be drawn in the image generated by the image processing unit 201 are performed.
Such calculation processing is performed in accordance with an arithmetic program stored in the storage unit 207 in advance, for example.
The image processing unit 201 and the display control unit 203 input a result of calculation of the control unit 210 and perform generation and display control of the image in accordance with the input information.
The storage unit 207 stores programs, parameters, and the like regarding data processing to be performed at the control unit 210, the image processing unit 201, the display control unit 203, and the like. In addition, the storage unit 207 is utilized as a work area of each data processing unit. Specifically, the storage unit 207 includes a RAM, a ROM, and the like.
For example, as described above with reference to
(S05) Steering-wheel left-rotation forward travel (counter-clockwise forward travel); and
(S06) Steering-wheel right-rotation forward travel (counter-clockwise forward travel).
For example, in a case where the vehicle is in either (S01) Forward travel in straight line or (S02) Rearward travel in straight line (backing in straight line), only the overhead image and the travel direction image (rear image) are displayed on the display unit as described above.
In such a case, the vehicle does not move in the left and right directions, and even if an obstacle is present on the left or right sides of the vehicle, there is no possibility that the vehicle will hit or crash into the obstacle. Therefore, display processing of the side image and the right side image is not performed.
On the other hand, in a case where the vehicle state is either one of
In this case, the vehicle front portion moves in the left or right direction.
In a case where the vehicle front portion moves in the left direction, the vehicle lateral side image (front left side image) is displayed in the front left region of the overhead image.
In a case where the vehicle front portion moves in the right direction, the vehicle lateral side image (front right side image) is displayed in the front right region of the overhead image.
By performing such display processing, it becomes possible to suitably present, to the driver who views the display unit 100, a vehicle position to which the driver should pay attention and obstacle information of that position.
It should be noted that the vehicle lateral side image (left/right side image) includes the lateral side image of the vehicle itself and the vehicle movement direction display line indicating the movement direction of the vehicle is further displayed. With those images, the driver can accurately grasp the presence/absence of the obstacle and a distance to the obstacle and can cause the vehicle to safely travel.
On the other hand, in the case where the vehicle state is in either one of
In this case, the vehicle rear portion moves in the left or right direction.
In a case where the vehicle rear portion moves in the left direction, the vehicle lateral side image (rear left side image) is displayed in the rear left region of the overhead image.
In a case where the vehicle rear portion moves in the right direction, the vehicle lateral side image (rear right side image) is displayed in the rear right region of the overhead image.
By such display processing, it becomes possible to suitably present, to the driver who views the display unit 100, a vehicle position to which the driver should pay attention and obstacle information of that position.
It should be noted that the vehicle lateral side image (rear left/right side image) includes the lateral side image of the vehicle itself and the vehicle movement direction display line indicating the movement direction of the vehicle is further displayed. With those images, the driver can accurately grasp the presence/absence of the obstacle and a distance to the obstacle and can cause the vehicle to safely travel.
In this manner, the image processing apparatus of the present disclosure inputs an image photographed by a camera that photographs the surroundings of the vehicle, generates an overhead image, a vehicle travel direction image, and a vehicle lateral side image corresponding to a vehicle movement direction which is either left or right of the vehicle on the basis of the input image, and displays the generated image on a display unit. For example, a display image is generated or updated in accordance with a travel direction of the vehicle and direction-of-rotation information of a steering wheel. In a case where the vehicle is backed counter-clockwise, the overhead image, the vehicle rear image, and a vehicle left side image are generated and displayed. In a case where the vehicle is backed clockwise, the overhead image, the vehicle rear image, and a vehicle right side image are generated and displayed.
With such image display processing, the image display with the improved visibility of the information regarding the surroundings of the vehicle is achieved.
[7. Conclusion of Configuration of Present Disclosure]
The embodiment of the present disclosure has been described in detail with reference to the specific embodiment. However, it is obvious that those skilled in the art can make modifications and replacements of the embodiment without departing from the gist of the present disclosure. That is, the present disclosure is described herein for illustrative purposes only and should not be exclusively understood. In order to determine the gist of the present disclosure, appended claims should be taken into consideration.
It should be noted that the technology disclosed in this specification can take the following configurations.
(1) An image processing apparatus, including
an image processing unit that generates a display image for a display unit, which is visually recognizable by a vehicle operator, in which
the image processing unit
(2) The image processing apparatus according to (1), in which
the image processing unit
(3) The image processing apparatus according to (1) or (2), in which
the image processing unit
(4) The image processing apparatus according to (3), in which
the image processing unit
(5) The image processing apparatus according to (4), in which
the image processing unit
(6) The image processing apparatus according to (5), in which
the image processing unit
(7) The image processing apparatus according to (6), in which
(8) The image processing apparatus according to (5), in which
the image processing unit
(9) The image processing apparatus according to (8), in which
the image processing unit
(10) The image processing apparatus according to any of (1) to (9), in which
the image processing unit
(11) The image processing apparatus according to any of (1) to (10), in which
the image processing unit
(12) The image processing apparatus according to (2), in which
the image processing unit
(13) The image processing apparatus according to any of (1) to (12), in which
the vehicle travel direction image includes an end portion region image of the vehicle itself.
(14) The image processing apparatus according to any of (1) to (13), in which
the image processing unit
(15) An image processing method to be executed in an image processing apparatus including an image processing unit that generates a display image for a display unit, which is visually recognizable by a vehicle operator, the image processing method including:
by the image processing unit,
receiving an image photographed by a camera that photographs surroundings of a vehicle;
generating, on the basis of the received photographed image, a vehicle travel direction image and a vehicle lateral side image corresponding to a vehicle movement direction which is either left or right of the vehicle; and
(16) A program that causes image processing to be executed in an image processing apparatus including an image processing unit that generates a display image for a display unit, which is visually recognizable by a vehicle operator, the program causing the image processing unit to execute:
processing of receiving an image photographed by a camera that photographs surroundings of a vehicle;
processing of generating, on the basis of the received photographed image, a vehicle travel direction image and a vehicle lateral side image corresponding to a vehicle movement direction which is either left or right of the vehicle; and
processing of outputting the generated vehicle travel direction image and the generated vehicle lateral side image to the display unit.
Further, the series of processing described herein can be executed by hardware, software, or a combined configuration of both. When the processing is executed by software, the program with a processing sequence recorded therein can be installed in a memory within a computer embedded in dedicated hardware and executed, or a program can be installed in a general-purpose computer which can execute various types of processing and executed. The program can be recorded in advance in a recording medium, for example. The program can not only be installed from the recording medium to the computer but can also be received via a network such as a local area network (LAN) or the Internet and installed in a recording medium such as a built-in hard disk.
It should be noted that the various types of processing described herein may be executed in a time series in an order of the description or may be executed in parallel or individual manner in a manner that depends on a processing capability of a device which executes the processing or in a manner that depends on needs. Further, the term “system” set forth herein is a logical complex configuration of a plurality of apparatuses, and apparatuses with respective configurations are not necessarily provided in the same housing.
As described above, in accordance with the configuration of the embodiment of the present disclosure, an image processing apparatus that achieves image display with improved visibility of information regarding the surroundings of a vehicle is realized.
Specifically, for example, an image photographed by a camera that photographs the surroundings of the vehicle is input. An overhead image, a vehicle travel direction image, and a vehicle lateral side image corresponding to a vehicle movement direction which is either left or right of the vehicle is generated on the basis of the input image. The generated image is displayed on a display unit. For example, a display image is generated or updated in accordance with a travel direction of the vehicle and direction-of-rotation information of a steering wheel. In a case where the vehicle is backed counter-clockwise, the overhead image, the vehicle rear image, and a vehicle left side image are generated and displayed. In a case where the vehicle is backed clockwise, the overhead image, the vehicle rear image, and a vehicle right side image are generated and displayed.
With this configuration, the image processing apparatus that achieves the image display with the improved visibility of the information regarding the surroundings of the vehicle is realized.
| Number | Date | Country | Kind |
|---|---|---|---|
| JP2016-161826 | Aug 2016 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2017/026413 | 7/21/2017 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2018/037789 | 3/1/2018 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 7366595 | Shimizu et al. | Apr 2008 | B1 |
| 9219887 | Iga | Dec 2015 | B2 |
| 9569967 | Petridis | Feb 2017 | B2 |
| 10692372 | Nilsson | Jun 2020 | B2 |
| 20030137586 | Lewellen | Jul 2003 | A1 |
| 20060061008 | Karner | Mar 2006 | A1 |
| 20060164230 | DeWind | Jul 2006 | A1 |
| 20060202984 | Yang | Sep 2006 | A1 |
| 20060287825 | Shimizu et al. | Dec 2006 | A1 |
| 20060287826 | Shimizu | Dec 2006 | A1 |
| 20070008091 | Takenaga | Jan 2007 | A1 |
| 20110210868 | Yano | Sep 2011 | A1 |
| 20120296523 | Ikeda | Nov 2012 | A1 |
| 20120330541 | Sakugawa | Dec 2012 | A1 |
| 20130120578 | Iga | May 2013 | A1 |
| 20140152774 | Wakabayashi | Jun 2014 | A1 |
| 20140152827 | Yamamoto | Jun 2014 | A1 |
| 20140285666 | O'Connell | Sep 2014 | A1 |
| 20140313335 | Koravadi | Oct 2014 | A1 |
| 20140333770 | Baur | Nov 2014 | A1 |
| 20140354816 | Inanobe | Dec 2014 | A1 |
| 20150073664 | Petridis | Mar 2015 | A1 |
| 20150116495 | Kowatari | Apr 2015 | A1 |
| 20150161892 | Oremus | Jun 2015 | A1 |
| 20160200252 | Oota | Jul 2016 | A1 |
| 20160305094 | Chang | Oct 2016 | A1 |
| 20160368417 | Bassi | Dec 2016 | A1 |
| 20170343799 | Ito | Nov 2017 | A1 |
| 20170374287 | Lang | Dec 2017 | A1 |
| 20180056871 | Karner | Mar 2018 | A1 |
| 20180322655 | Nagai | Nov 2018 | A1 |
| 20180361933 | Ohta | Dec 2018 | A1 |
| 20190118860 | Gali | Apr 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 101311672 | Nov 2008 | CN |
| 101676149 | Mar 2010 | CN |
| 102416906 | Apr 2012 | CN |
| 102685382 | Sep 2012 | CN |
| 104442813 | Mar 2015 | CN |
| 105793496 | Jul 2016 | CN |
| 1065642 | Jan 2001 | EP |
| 1400410 | Mar 2004 | EP |
| 1870870 | Dec 2007 | EP |
| 2168815 | Mar 2010 | EP |
| 2620330 | Jul 2013 | EP |
| 2001-006097 | Jan 2001 | JP |
| 2002-109697 | Apr 2002 | JP |
| 2005-339313 | Dec 2005 | JP |
| 2006-165946 | Jun 2006 | JP |
| 2007-019804 | Jan 2007 | JP |
| 2007-124097 | May 2007 | JP |
| 2012-066616 | Apr 2012 | JP |
| 2012-124610 | Jun 2012 | JP |
| 2012-166689 | Sep 2012 | JP |
| 5696679 | Apr 2015 | JP |
| 2015-119225 | Jun 2015 | JP |
| 2012039256 | Mar 2012 | WO |
| Entry |
|---|
| Yi-Yuan Chen, Yuan-Yao Tu, Cheng-Hsiang Chiu and Y. Chen, “An embedded system for vehicle surrounding monitoring,” 2009 2nd International Conference on Power Electronics and Intelligent Transportation System (PEITS), Shenzhen, China, 2009, pp. 92-95, doi: 10.1109/PEITS.2009.5406797. (Year: 2009). |
| Y. Kuroki, et al., “Multimodal Cruising Assist to Enhance the Drivers' Abilities to Perceive Surrounding Contexts Using Panoramic Presentation with Dynamic Multiple Windows,” 5th Annual IEEE International Conference (PerComW07), White Plains, NY, USA, 2007, pp. 429-434, doi: 10.1109/PERCOMW.2007.83. (Year: 2007). |
| International Search Report and Written Opinion of PCT Application No. PCT/JP2017/026413, dated Oct. 17, 2017, 11 pages of ISRWO. |
| Office Action for CN Patent Application No. 201780050402.4, dated Feb. 10, 2021, 10 pages of Office Action and 20 pages of English Translation. |
| Office Action for CN Patent Application No. 201780050402.4, dated Aug. 9, 2021, 11 pages of Office Action and 17 pages of English Translation. |
| Office Action for CN Patent Application No. 201780050402.4 dated Jan. 13, 2022, 02 pages of English Translation and 01 pages of Office Action. |
| Miao, et al., “Real Time Mapping Method for Traffic Accident Scene Based on Image Guide”, Journal of Shanghai Jiao Tong University, vol. 7, No. 2, Feb. 16, 2013, 6 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20190174060 A1 | Jun 2019 | US |
