Field of the Invention
The invention relates to a technology that is used to process images showing surroundings of a vehicle.
Description of the Background Art
Conventionally, systems that combine captured images of surroundings of a vehicle and others and that display images showing the surroundings of the vehicle viewed from a driver seat are known. A user (typically a driver) can see the surroundings of the vehicle by using such a system, even in a cabin of the vehicle.
Recently, there is a known technology that superimposes a cabin image of a cabin viewed from a driver seat on an image showing surroundings of a vehicle and that displays the entire cabin image in a transparent or semi-transparent form and also even an obstacle hidden behind a body of the vehicle in visual contact. The user can see such an image and can recognize an object located in the surroundings of the vehicle, understanding a positional relationship between the vehicle and the surroundings of the vehicle.
However, if the entire cabin image is displayed in the transparent or semi-transparent form, various objects are displayed on the transparent or semi-transparent portion of the cabin. Therefore, the user cannot immediately determine an object requiring a closest attention in an image showing the surroundings. In this case, although an obstacle is displayed in the course of traveling, there has been a possibility that the user may miss the obstacle.
According to one aspect of the invention, an image processing apparatus configured to be used on a vehicle includes: (a) an image processor configured to (i) generate a surrounding image showing a surrounding region of the vehicle viewed from a virtual viewpoint located in the vehicle, by using an image captured by a camera mounted on the vehicle; (ii) obtain a vehicle image that is divided into plural portions showing the vehicle viewed from the virtual viewpoint; (iii) generate a combined image by combining the surrounding image and the vehicle image having the plural portions; and (iv) output the combined image for display on a display apparatus, and (b) a controller configured to determine a transparency percentage of each of the plural portions of the vehicle image. The image processor causes the plural portions to be semi-transparent or to be transparent at the determined transparency percentages such that the combined image includes the surrounding image and the vehicle image having the plural portions caused to be semi-transparent or to be transparent at the determined transparency percentages.
Since the image processing apparatus causes the plural portions into which the vehicle image is divided to be semi-transparent or to be transparent, the user can intuitively understand a positional relationship between the vehicle and a surrounding region of the vehicle.
According to another aspect of the invention, an image processing apparatus configured to be used on a vehicle includes: (a) an image processor configured to (i) generate a surrounding image showing a surrounding region of the vehicle viewed from a virtual viewpoint located in the vehicle, by using an image captured by a camera mounted on the vehicle; (ii) obtain a vehicle image that is divided into plural portions showing the vehicle viewed from the virtual viewpoint; (iii) generate a combined image by combining the surrounding image and the vehicle image having the plural portions; and (iv) output the combined image for display on a display apparatus, and (b) a controller configured to determine a transparency percentage of each of the plural portions of the vehicle image. The image processor causes the plural portions to be semi-transparent or to be transparent at the determined transparency percentages such that the combined image includes the surrounding image and the vehicle image having the plural portions caused to be semi-transparent or to be transparent at the determined transparency percentages, and the plural portions overlap each other when the surrounding region is viewed from the virtual viewpoint.
Since the image processing apparatus causes the plural portions into which the vehicle image is divided to be semi-transparent or to be transparent and the plural portions overlap each other when the surrounding region is viewed from the virtual viewpoint. Thus, the user can intuitively understand a positional relationship between the vehicle and the surrounding region of the vehicle.
Therefore, an object of the invention is to enable a user to intuitively understand a subject by displaying a surrounding image superimposed on a cabin image caused to be semi-transparent or to be transparent.
These and other objects, features, aspects and advantages of the invention will become more apparent from the following detailed description of the invention when taken in conjunction with the accompanying drawings.
An embodiment of the invention is hereinafter explained with reference to the drawings.
<1-1. Outline>
The cabin image is divided into plural portions. The image processing apparatus 3 determines the transparency percentage for each of the plural portions of the cabin image and causes each portion to be transparent or to be semi-transparent (hereinafter referred to collectively as transparent) at the determined transparency percentage.
The image processing apparatus 3 combines surrounding images AP obtained by the plural cameras 5 with the cabin image having the portions transparent at the determined transparency percentages, and generates the combined image.
In the embodiment, the plural “portions” into which the cabin image 200 is divided include “parts” that consist of the vehicle and that are physically independent of one another. Examples of the parts are a body and a door panel. Moreover, each of the “parts” is composed of separable “regions.” For example, the body can be separated into a roof, a pillar, a fender and other regions. Therefore, the roof, the pillar, the fender and the others regions of the body are also included in the “portions” as separate regions. The same holds true for the dashboard and parts other than the body that consist of the vehicle. Therefore, in this embodiment, the portions into which the cabin image 200 is divided may be referred to as “parts” or “regions.”
<1-2. Configuration>
The image processing system 1 includes the image processing apparatus 3 and the display apparatus 4. Moreover, the image processing apparatus 3 includes the plural cameras 5 that capture the images showing the surroundings of the vehicle 2.
The image processing apparatus 3 performs a variety of image processing, using the captured images and generates an image to be displayed on the display apparatus 4. The display apparatus 4 displays the image generated and output by the image processing apparatus 3.
Each of the plural cameras 5 (5F, 5B, 5L and 5R) includes a lens and an image sensor. The plural cameras 5 capture the images showing the surroundings of the vehicle 2 and obtain the captured images electronically. The plural cameras 5 include a front camera 5F, a rear camera 5B, a left side camera 5L and a right side camera 5R. The plural cameras 5 are disposed at positions different from one another on/in the vehicle 2 and capture the images from the vehicle 2 in directions different from one another.
A wide angle lens, such as a fish lens, is used for each of the plural cameras 5. The wide angle lens has an angle θ of 180 degrees or more. Thus, by using the four cameras 5 are used, the image showing 360-degree surroundings of the vehicle 2 can be captured.
With reference back to
The image processing apparatus 3 is an electronic control apparatus that is configured to perform a variety of image processing. The image processing apparatus 3 includes an image obtaining part 31, an image processor 32, a controller 33, a memory 34 and a signal receiver 35.
The image obtaining part 31 obtains the captured image captured by each of the four cameras 5. The image obtaining part 31 has an image processing function, such as A/D conversion that converts an analog captured image to a digital captured image. The image obtaining part 31 performs a predetermined image processing, using the obtained captured image and inputs the processed captured image into the image processor 32.
The image processor 32 is a hardware circuit that performs image processing to generate the combined image. The image processor 32 combines the plural captured images captured by the cameras 5 and generates the surrounding image AP showing the surroundings of the vehicle 2 viewed from a virtual viewpoint. The image processor 32 includes a surrounding image generator 32a, a combined image generator 32b and an image transparency adjustor 32c.
The surrounding image generator 32a combines the plural captured images captured by the four cameras 5 and generates the surrounding image AP showing the surroundings of the vehicle 2 from the virtual viewpoint. The virtual viewpoint includes a driver seat viewpoint to look at an outside of the vehicle 2 from the driver seat and an overhead viewpoint to look down at the vehicle 2 from a position of the outside of the vehicle 2.
The combined image generator 32b superimposes a vehicle body image 100 or the cabin image 200 of the vehicle 2 on the surrounding image AP generated by the surrounding image generator 32a.
The image transparency adjustor 32c changes the transparency percentage of the cabin image 200. In other words, the image transparency adjustor 32c performs the image processing such that the user can see a part of the surrounding image AP behind the cabin image 200 in a line of sight of the user, through the cabin image 200. In the processing, the image transparency adjustor 32c determines the transparency percentage for each of the plural portions of the cabin image 200 and causes the plural portions to be transparent at the determined transparency percentages individually. Here, “causing something to be transparent” means not only causing the cabin image 200 to be transparent on the surrounding image AP (i.e. possible to see the outside of the vehicle from the inside of the vehicle) but also causing the cabin image 200 to be transparent on a different cabin image 200 (i.e. possible to see the inside of the vehicle through an interior, such as a seat, from the vehicle).
The “transparency percentage” is a percentage at which a color of the surrounding image AP goes through a color of the cabin image 200 superimposed on the surrounding image AP, in the line of the sight of the user. Therefore, as the transparency percentage of an image is increased, lines and the color of the image become paler. Thus, the surrounding image AP goes through the cabin image 200 superimposed by the combined image generator 32b. For example, when the transparency percentage is set at 50%, the displayed cabin image 200 is pale in color, and the surrounding image AP is displayed through the cabin image 200 pale in color. In other words, the cabin image 200 becomes semi-transparent. When the transparency percentage of the cabin image 200 is set at 100%, the lines and the color of the cabin image 200 are not displayed, and only the surrounding image AP is displayed. On the other hand, when the transparency percentage is set at 0%, the cabin image 200 is displayed in normal color with lines, and a portion of the surrounding image AP overlapped with the cabin image 200 is not displayed.
The change of the transparency percentage is, concretely, a change of a percentage to mix elements of RGB color models of the cabin image 200 and the surrounding image AP. For example, in order to display the cabin image 200 at 50% of the transparency percentage, RGB elements of the cabin image 200 and the surrounding image AP are averaged. Moreover, in order to increase the transparency percentage of the cabin image 200 (i.e. to make the cabin image 200 “paler”), the RGB elements of the surrounding image AP are doubled, the doubled elements are added to the RGB elements of the cabin image 200, and then the summed RGB elements are divided by three. On the other hand, in order to decrease the transparency percentage of the cabin image 200 (i.e. to make the cabin image 200 “darker”), the RGB elements of the cabin image 200 are doubled, the doubled elements are added to the RGB elements of the surrounding image AP, and then the summed RGB elements are divided by three. Moreover, the transparency percentage of an image may be changed by using another well-know image processing method.
The controller 33 is a microcomputer, including a CPU, a RAM and a ROM, that controls the entire image processing apparatus 3. Each function of the controller 33 is implemented by the CPU performing arithmetic processing in accordance with a program stored beforehand. An operation performed by each function included in the controller 33 will be described later.
The memory 34 is a nonvolatile memory, such as a flash memory. The memory 34 stores vehicle image data 34a, a transparency model 34b, setting data 34c and a program 34d serving as firmware.
The vehicle image data 34a includes the vehicle body image data 100 and the cabin image data 200. The vehicle body image data 100 and the cabin image data 200 include external appearances of the vehicle 2 and images of the cabin of the vehicle 2 viewed from all angles.
The vehicle body image data 100 is an image showing the external appearance of the vehicle 2 viewed from an overhead viewpoint.
The cabin image 200 data is an image showing the cabin viewed from the inside of the vehicle 2, such as the driver seat. Moreover, the cabin image 200 is divided into the plural portions and the each of the plural portions is stored in the memory 34.
The plural portions of the cabin image 200 include a frame f showing a shape of the vehicle 2. Moreover, relationships between each viewing position and each view direction of the virtual viewpoints and positions of the plural portions of the cabin image 200 to be displayed may be defined and stored beforehand. Further, instead of the viewing position of the user looking rearward of the vehicle 2 in the driver seat, the viewing position looking rearward of the vehicle from a position of the rearview mirror may be used because when looking rearward of the vehicle, the user looks at an image of a rear side of the vehicle reflected on the rearview mirror.
In addition, in a case of the virtual viewpoint having the viewing position of the user looking rearward of the vehicle 2 in the driver seat, a seat is included in the view. Therefore, the combined image generator 32b may further retrieve data of an image of the seat (not illustrated) from the memory 34, may combine the image with the surrounding image AP and then may generate the combined image CP looking rearward of the vehicle where the seat image is placed.
With reference back to
Moreover, the transparency percentage of the vehicle image data 34a may be changed depending on brightness of the surroundings of the vehicle 2. In other words, in a case where illuminance of the surroundings of the vehicle 2 is low, for example at night or in a building without a light, the transparency percentage of the cabin image 200 may be increased to more than 50%. Thus, the user can see the surrounding image AP more clearly through the cabin image 200. Even when the illuminance of the surroundings of the vehicle 2 is low, the user easily understands the positional relationship of the vehicle 2 and the object located in the surroundings of the vehicle 2.
One of the transparency models 34b is selected by the user. The cabin image 200 of the selected transparency, model 34b is displayed on the display apparatus 4 at the transparency percentage of the selected transparency model 34b. Before one of the transparency models 34b is selected by the user (e.g. when being shipped from a factory), the transparency model 34b of the middle transparency percentage may be preset for the image processing apparatus 3. Thus, the surrounding image AP can be displayed through the cabin image 200 immediately after the image processing apparatus 3 is first activated.
The setting data 34c is data of the transparency percentage set by the user for each portion of the cabin image 200.
The program 34d is firmware that is read out and is executed by the controller 33 to control the image processing apparatus 3.
The signal receiver 35 obtains data relating to the vehicle 2 and sends it to the controller 33. The signal receiver 35 is connected to a shift sensor 35a, a steering wheel sensor 35b, a turn-signal switch 35c, a vehicle speed sensor 35d and a surrounding monitoring sensor 35e, via a LAN in the vehicle 2.
The shift sensor 35a detects a position of a shift lever, such as “DRive” and “Reverse.” The shift sensor 35a sends shift data representing a current position of the shift lever to the signal receiver 35.
The steering wheel sensor 35b detects an angle and a direction, either to the right or left, by/in which the user has rotated the steering wheel from a neutral position (a position of the steering wheel to drive the vehicle 2 straightforward). The steering wheel sensor 35b sends angle data of the detected angle to the signal receiver 35. In other words, the steering wheel sensor 35b is a rotated direction obtaining part that obtains a rotated direction of the steering wheel.
The turn-signal switch 35c detects the right or the left that a turn-signal operated by the user indicates. The turn-signal switch 35c sends direction data of the detected direction to the signal receiver 35. In other words, the turn-signal switch 35c is an operation obtaining part that obtains an operation status of the turn-signal of the vehicle 2.
The vehicle speed sensor 35d is a speed obtaining part that obtains a speed of the vehicle 2. The vehicle speed sensor 35d sends speed data of the obtained speed to the signal receiver 35.
The surrounding monitoring sensor 35e detects an object located in the surroundings of the vehicle 2 and sends object data showing a direction and a distance of the object from the vehicle 2, to the signal receiver 35. Examples of the surrounding monitoring sensor 35e are clearance sonar using a sound wave, radar using a radio wave or an infrared lay, and a combination of those devices.
Next, an operation of each part included in the controller 33 is explained. The controller 33 includes a viewpoint changer 33a, a transparency percentage setting part 33b and an image outputting part 33c.
The viewpoint changer 33a sets the viewing position and the view direction of the virtual viewpoint. The details are described later.
The transparency percentage setting part 33b sets the transparency percentage of the cabin image 200 in a range from 0% to 100%. Based on the transparency percentage set by the transparency percentage setting part 33b, the image transparency adjustor 32c, described earlier, determines the transparency percentages for the plural portions of the cabin image 200 and causes the portions to be transparent at the determined individual transparency percentages. In addition to the preset transparency percentages, an arbitrary transparency percentage is set by the user.
The image outputting part 33c outputs the combined image generated by the image processor 32 to the display apparatus 4. Thus, the combined image is displayed on the display apparatus 4.
<1-3. Image Generation>
Next described is a method used by the image processor 32 to generate the surrounding image AP showing the surroundings of the vehicle 2 and the combined image CP by superimposing the cabin image 200 on the surrounding image AP.
Once the front camera 5F, the rear camera 5B, the left side camera 5L and the right side camera 5R capture images of the surroundings of the vehicle 2, images AP (F), AP (B), AP (L) and AP (R) that show areas in front, behind, left and right of the vehicle 2, respectively, are obtained. The four captured images include data showing 360-degree surroundings of the vehicle 2.
The surrounding image generator 32a projects the data (value of each pixel) included in these four images of AP (F), AP (B), AP (L) and AP (R) onto a projection surface TS that is a three-dimensional (3D) curved surface in virtual 3D space. The projection surface TS is, for example, substantially hemispherical (bowl-shaped). The vehicle 2 is defined to be located in a center region of the projection surface TS (a bottom of the bowl). Each region of the projection surface TS other than the center region corresponds to one of the AP (F), AP (B), AP (L) and AP (R).
First, the surrounding image generator 32a projects the surrounding images AP (F), AP (B), AP (L) and AP (R) onto the regions other than the center region of the projection surface TS. The surrounding image generator 32a projects the image AP (F) captured by the front camera 5F onto a region of the projection surface TS corresponding to an area in front of the vehicle 2 and the image AP (B) captured by the rear camera 5B onto a region of the projection surface TS corresponding to an area behind the vehicle 2. Moreover, the surrounding image generator 32a projects the image AP (L) captured by the left camera 5L onto a region of the projection surface TS corresponding to an area left of the vehicle 2 and the image AP (R) captured by the right camera 5R onto a region of the projection surface TS corresponding to an area right of the vehicle 2.
Next, the surrounding image generator 32a sets a virtual viewpoint VP in the virtual 3D space. The surrounding image generator 32a is configured to set the virtual viewpoint VP at an arbitrary viewing position in an arbitrary view direction in the virtual 3D space. Then, the surrounding image generator 32a clips from the projection surface TS, regions viewed from the set virtual viewpoint VP within a view angle, as images, and then combines the clipped images. Thus, the surrounding image generator 32a generates the surrounding image AP showing the surroundings of the vehicle 2 viewed from the virtual viewpoint VP.
Next, the combined image generator 32b generates the combined image CP by combining the surrounding image AP generated by the surrounding image generator 32a, the cabin image 200 read out from the memory 34, depending on the virtual viewpoint VP, and an icon image PI used for the touch panel 4a.
For example, in a case of a virtual viewpoint VPa of which the viewing position is located at the driver seat of the vehicle 2 in the view direction looking ahead of the vehicle 2, the combined image generator 32b generates a combined image CPa showing the cabin and the area in front of the vehicle 2, overlooking the area in front of the vehicle 2 from the driver seat. In other words, as shown in
In a case of a virtual viewpoint VPb of which the viewing position is located at the driver seat of the vehicle 2 in the view direction looking rearward of the vehicle 2, the combined image generator 32b generates a combined image CPb showing a back area of the cabin of the vehicle 2 and the surrounding area behind the vehicle 2, using the cabin image 200 showing a rear gate, etc. and the surrounding image AP (B).
In a case of a virtual viewpoint VPc of which a viewing position is located directly above the vehicle 2 in a view direction looking down (virtual viewpoint two-dimensionally looking downward), the combined image generator 32b generates a combined image CPc looking down the vehicle 2 and the surrounding area of the vehicle 2, using the vehicle body image 100 and the surrounding images AP (F), AP (B), AP (L) and AP (R).
<1-4. Procedure>
Next explained is a procedure performed by the image processing apparatus 3 to generate the combined image CP.
First, each of the plural cameras 5 captures an image. The image obtaining part 31 obtains the four captured images from the plural cameras 5 (a step S11). The image obtaining part 31 sends the obtained captured images to the image processor 32.
Once the image obtaining part 31 sends the captured images to the image processor 32, the viewpoint changer 33a of the controller 33 determines the viewing position and the view direction of the virtual viewpoint VP (a step S12). It is recommended that the viewpoint changer 33a should set the viewing position at the driver seat in the view direction looking ahead of the vehicle 2, as an initial setting for a displayed image, because the viewing position and the view direction are most comfortable for the user in the driver seat.
However, when the steering wheel or the turn-signal has been operated, the viewpoint changer 33a changes the view direction to a direction to which the steering wheel or the turn-signal has been operated because the operated direction is a traveling direction of the vehicle. In this case, the viewpoint changer 33a sets the view direction based on the angle data sent by the steering wheel sensor 35b, the direction data sent by the turn-signal switch 35c, etc.
Moreover, when the view direction looking ahead of the vehicle 2 is selected, the view direction looking a left front area of the vehicle 2 may be set. The left front of the vehicle 2 is often a blind area of the user in a case of the vehicle 2 having the steering wheel on a right side. Similarly, in a case of the vehicle 2 having the steering wheel on a left side, the view direction looking a right front area of the vehicle 2 may be set.
Moreover, when the position of the shift lever is changed to the “Reverse,” the viewpoint changer 33a sets the view direction looking rearward of the vehicle 2 because the user intends to drive the vehicle 2 backwards. The viewpoint changer 33a determines the position of the shift lever based on the shift data sent from the shift sensor 35a.
Moreover, the viewing position and the view direction may be changed by an operation made by the user with the touch panel 4a. In this case, whenever the icon image PI displayed on the display apparatus 4 is operated, the virtual viewpoint VP is changed. In other words, images viewed from the three different virtual viewpoints VP are displayed in rotation. The three virtual viewpoints VP are: the virtual viewpoint VP having the viewing position located at the driver seat in the view direction looking ahead; the virtual viewpoint VP having the viewing position located at the driver seat in the view direction looking rearward; and the virtual viewpoint VP having the viewing position located at the overhead position in the view direction looking down straightly. Moreover, the image having the viewing position located at the driver seat and the image having the viewing position located at the overhead position may be simultaneously displayed side by side. In this case, the user can understand situations of the surroundings of the vehicle 2 viewed from plural positions, simultaneously. Therefore, the user can drive the vehicle 2 more safely.
Once the viewing position and the view direction of the virtual viewpoint VP are determined, the surrounding image generator 32a generates the surrounding image AP of the vehicle 2, using the method described above, based on the captured images captured by the image obtaining part 31 (a step S13).
Once the surrounding image AP is generated, the combined image generator 32b reads out the vehicle body image 100 or the cabin image 200, depending on the virtual viewpoint VP, from the memory 34 via the controller 33 (a step S14). In a case of the virtual viewpoint VP having the viewing position at the overhead position, the vehicle body image 100 is read out. In a case of the virtual viewpoint VP having the viewing position at the driver seat, the cabin image 200 is read out. A process of reading out the cabin image from the memory 34 performed by the combined image generator 32b is performed via the controller 33.
Next, the image transparency adjustor 32c performs a transparency process that changes the transparency percentage of the cabin image 200 read out in the method described above (a step S15). The transparency process will be described later.
Once the transparency percentage setting part 33b changes the transparency percentage of the cabin image 200, the combined image generator 32b generates the combined image CP based on the four captured images and the cabin image 200, in the method described above (a step S16).
Once the combined image generator 32b generates the combined image CP, the image outputting part 33c outputs the combined image CP to the display apparatus 4 (a step S17). The output combined image CP is displayed on the display apparatus 4 and the user can see the combined image CP.
Once the combined image CP is output, the transparency percentage setting part 33b of the controller 33 determines whether or not an instruction for setting the transparency percentage of the cabin image 200 has been given by the user via the touch panel 4a (a step S18).
Once determining that the instruction for setting the transparency percentage has been given (Yes in the step S18), the transparency percentage setting part 33b causes a screen used for setting the transparency percentage to be displayed on the display apparatus 4 and performs a setting process of the transparency percentage (a step S19). The setting process will be described later.
Once the setting process of the transparency percentage is performed or once the transparency percentage setting part 33b determines that the instruction for setting the transparency percentage has not been given (No in the step S18), the controller 33 determines whether or not an instruction for ending the display of the combined image CP has been given by the user (a step S20). The controller 33 determines whether or not the instruction has been given, based on presence or absence of an operation made by the user with a button (not illustrated) for ending the display of the image because there is a case where the user wants to end the display of the combined image CP for display of a navigation screen and the like.
Once determining that the instruction for ending the display of the combined image CP has been given (Yes in the step S20), the image outputting part 33c stops output of the combined image CP. Once the image outputting part 33c stops the output of the combined image CP, this process ends.
On the other hand, once the image outputting part 33c determines that the instruction for ending the display of the combined image CP has not been given (No in the step S20), the process returns to the step S11. Once the process returns to the step S11, the image obtaining part 31 obtains four captured images from the four cameras 5 again. Then, the process after the step S11 is repeated. In a case where, the user sets a different display mode in the step S19 or in a case where the user sets an arbitrary transparency percentage, the combined image CP is generated in the set display mode and/or at the set transparency percentage in the repeated process.
Next, the transparency process of the cabin image 200 performed in the step S15 is explained with reference to the drawing from
In a case where the controller 33 determines to cause the cabin image 200 to be transparent at the transparency percentage of the transparency model 34b (Yes in the step S51), the image transparency adjustor 32c causes the cabin image 200 to be transparent at the transparency percentage of the transparency model 34b selected beforehand by the user. The transparency process for the cabin image 200 is performed in the method described above (a step S52).
On the other hand, in the case where the controller 33 determines to cause the cabin image 200 to be transparent at the arbitrary transparency percentage set by the user (No in the step S51), the image transparency adjustor 32c causes the cabin image 200 to be transparent at the arbitrary transparency percentage set by the user (a step S53).
Next, the controller 33 determines whether or not a “setting of transparency percentage based on a vehicle state,” which is one display mode (a step S54). A vehicle state means a state of an apparatus included in a vehicle, such as an operation status of the steering wheel, and a state of the vehicle itself, such as a vehicle speed. In a case where the display mode of the “setting of transparency percentage based on a vehicle state” is on, the image transparency adjustor 32c determines the transparency percentage of the cabin image 200 based on the vehicle state.
When determining that the display mode of the “setting of transparency percentage based on a vehicle state” is on (Yes in the step S54), the controller 33 determines, based on a sensor signal sent from the steering wheel sensor 35b; whether or not the steering wheel has been operated by the user (a step S55).
When determining that the steering wheel has been operated (Yes in the step S55), the image transparency adjustor 32c changes the transparency percentage of a portion of the cabin image 200 showing an area in a direction in which the steering wheel has been operated (a step S56). The direction to which the steering wheel is operated refers to a direction to which the steering wheel is rotated. The viewpoint changer 33a sets the view direction of the virtual viewpoint in the direction in which the steering wheel has been operated. Moreover, the transparency percentage is changed. For example, the transparency percentage is increased by 50% as compared to the transparency percentage before the change. However, in a case of a low transparency percentage of less than 50% before the change, the image transparency adjustor 32c may set the transparency percentage approximately at 80% or 100%.
As mentioned above, since a traveling direction of the vehicle 2 is equivalent to the direction in which the steering wheel has been operated, by increasing the transparency percentage of the portion of the cabin image 200 in the direction, presence or absence of an obstacle in the traveling direction can be clearly shown to the user. Thus, when parking the vehicle 2, the user can intuitively understand a positional relationship between the vehicle 2 and another vehicle or equipment in the parking lot, and can avoid a contact to the obstacle, etc., easily.
Further, for example, when the transparency percentage of the left door panel is increased at a time of turning to the left at a traffic intersection, the user can more easily recognize a pedestrian, a motorcycle, etc. moving near the vehicle 2. Thus, it is helpful to prevent an accident involving the pedestrian, the motorcycle, etc. Further, when the transparency percentage of a portion of the cabin image 200 is increased as compared to other portions, more attention of the user can be drawn to the portion of which the transparency percentage is increased.
Next, when determining that the steering wheel has not been operated by the user (No in the step S55), the controller 33 determines, based on a control signal sent from the turn-signal switch 35c, whether or not the turn-signal is on (a step S57).
When determining that the turn-signal is on (Yes in the step S57), the image transparency adjustor 32c increases the transparency percentage of a portion of the cabin image 200 showing an area diagonally in front of the vehicle 2 on a side indicated by the turn-signal (a step S58). In other words, the image transparency adjustor 32c determines the transparency percentage of the cabin image 200 based on an operational status of the turn-signal. The image transparency adjustor 32c increases the transparency percentage of the portion of the cabin image 200 showing the area diagonally in front of the vehicle 2 on the side indicated by the turn-signal because the side indicated by the turn-signal is only a predicted traveling direction in which the vehicle 2 will travel and there is a case where the vehicle 2 has not moved or turned yet to the right or the left, being different from the case of the steering wheel. Therefore, when the turn-signal is on, it is recommended that the cabin image 200 having the portion showing the area diagonally in front of the vehicle 2 at an increased transparency percentage should be displayed, rather than a portion showing an area lateral to the vehicle 2 at an increased transparency percentage.
Next, the viewpoint changer 33a sets the view direction of the virtual viewpoint in the direction that the turn-signal indicates. However, the viewpoint changer 33a may set the view direction of the virtual viewpoint looking the area diagonally in front of or in front of the vehicle 2 on the side indicated by the turn-signal. In other words, as long as the surrounding image AP displayed on the display apparatus 4 includes the area diagonally in front of the vehicle 2 on the side indicated by the turn-signal, any direction may be set as the view direction. Moreover, a method of increasing the transparency percentage is a same as the method used in the step S56.
As mentioned above, since the side indicated by the turn-signal is the traveling direction in which the vehicle 2 will travel, it is recommended that the transparency percentage of a portion of the cabin image 200 showing an area diagonally in front of the vehicle 2 should be increased. Moreover, more attention of the user can be drawn to the portion of the cabin image 200.
In a case where the controller 33 determines that the vehicle speed is the high speed (“high speed” in the step S59), the image transparency adjustor 32c increases the transparency percentage of a higher portion of the cabin image 200 (a step S60). Moreover, the viewpoint changer 33a sets the view direction of the virtual viewpoint VP looking ahead or rearward of the vehicle 2, depending on the position of the shift lever. When the position of the shift lever is in “Drive,” the viewpoint changer 33a sets the view direction of the virtual viewpoint VP looking ahead of the vehicle 2. When the position of the shift lever is in “Reverse,” the viewpoint changer 33a sets the view direction of the virtual viewpoint VP looking rearward of the vehicle 2. The viewpoint changer 33a sets the view direction of the virtual viewpoint VP and the method for setting the view direction is also used in a step S61 and in a step 62, described later.
The image transparency adjustor 32c increases the transparency percentage of the higher portion of the cabin image 200 because the user generally looks far ahead or rearward, not near ahead or rearward, during driving at the high speed. Therefore, by increasing the transparency percentage of the higher portion of the cabin image 200 that is a portion of the surrounding image AP showing an area far ahead in the line of the sight of the user, an area that the user needs to see during the driving at the high speed can be displayed. The higher portion of the cabin image 200 is, for example, a portion higher than, approximately, one-half a height of the vehicle 2. Moreover, the higher portion of the cabin image 200 should include an actual view of the user during the driving at the high speed.
In a case where the controller 33 determines that the vehicle speed is the middle speed (“middle speed” in the step S59), the image transparency adjustor 32c increases the transparency percentage of a middle portion of the cabin image 200 (the step S61). Moreover, the viewpoint changer 33a sets the view direction of the virtual viewpoint VP looking an area in front of the vehicle 2 because the user generally looks slightly lower than the area far ahead, during driving at the middle speed. Therefore, by increasing the transparency percentage of the middle portion of the cabin image 200 that is a portion of the surrounding image AP showing an area slightly lower than the area far ahead in the line of the sight of the user, an area that the user needs to see during the driving at the middle speed can be displayed. The middle portion of the cabin image 200 is, for example, a middle of the vehicle 2 when the height of the vehicle 2 is divided into three. Moreover, the middle portion of the cabin image 200 should include the actual view of the user during the driving at the middle speed.
In a case where the controller 33 determines that the vehicle speed is the low speed (“low speed” in the step S59), the image transparency adjustor 32c increases the transparency percentage of a lower portion of the cabin image 200 (the step S62). Moreover, the viewpoint changer 33a sets the view direction of the virtual viewpoint VP looking an area in front of the vehicle 2 because user may pass by an obstacle during driving at the low speed so that the user generally looks things near the vehicle more often. Therefore, by increasing the transparency percentage of the lower portion of the cabin image 200 that is a portion of the surrounding image AP showing an area close to the vehicle in the line of the user, an area that the user needs to see during the driving at the low speed can be displayed. The lower portion of the cabin image 200 is, for example, a portion lower than, approximately, one-half the height of the vehicle 2. Moreover, the lower area of the cabin image 200 should include the actual view of the user during the driving at the low speed.
As described above, as the vehicle speed becomes higher, the image transparency adjustor 32c increases the transparency percentage of an area corresponding to a higher area of the vehicle, of the cabin image 200. Moreover, as the vehicle speed becomes lower, the image transparency adjustor 32c increases the transparency percentage of an area corresponding to a lower area of the vehicle, of the cabin image 200.
Next, the controller 33 determines whether or not a display mode of the “setting of transparency percentage based on a surrounding situation” is on (the step S63). Here, the surrounding situation refers to a situation in the surroundings of the vehicle that may have any influence on the vehicle, for example, presence or absence of an obstacle located adjacent to the vehicle.
When determining that the display mode of the “setting of transparency percentage based on a surrounding situation” is on (Yes in the step S63), the controller 33 determines whether or not there is an obstacle adjacent to the vehicle 2 (a step S64) based on the object data sent from the surrounding monitoring sensor 35e.
When the controller 33 determines that there is an obstacle (Yes in the step S64), the image transparency adjustor 32c increases the transparency percentage of a portion of the cabin image 200 showing an area in a direction where the obstacle is located (a step S65). In other words, the image transparency adjustor 32c determines the transparency percentage of the cabin image 200 based on a position of the obstacle located adjacent to the vehicle 2. Then, the viewpoint changer 33a sets the view direction of the virtual viewpoint looking in the direction where the obstacle is located. However, as long as the surrounding image AP includes the direction where the obstacle is located, any direction may be set as the view direction. A method of increasing the transparency percentage is a same as the method used in the step S56.
Once the procedure of increasing the transparency percentage of the cabin image 200 is performed, the procedure returns to the step S16 shown in
As described above, in the transparency process of the cabin image 200, after causing the cabin image 200 to be transparent based on the transparency model 34b or the setting data 34c set by the user, the image transparency adjustor 32c increases the transparency percentage of a part, depending on the vehicle state or the surrounding situation. Thus, the user can intuitively understand the positional relationship of the vehicle 2 and an object located in the surroundings of the vehicle 2.
Next, the setting process of the transparency percentage in the step S19 is explained with reference to
Next, the controller 33 receives an operation by the user with the touch panel 4a (a step S72).
The controller 33 determines whether or not the setting operation should be ended (a step S73) based on whether or not the user has touched a predetermined end button on the touch panel 4a.
When determining that the setting operation should be ended (Yes in the step S73), the controller 33 stores a set value input as the setting data 34c in the memory 34 (a step S74). The image transparency adjustor 32c determines the transparency percentage for each of plural portions of the cabin image 200, based on the setting data 34c set based on the operation made by the user (the step 53 in
Once the set value is stored in the memory 34, the process returns the procedure shown in
On the other hand, when determining that the setting operation should not be ended (No in the step S73), the controller 33 performs the step S72 again and receives the operation made by the user with the touch panel 4a. Then, until the end button for the setting operation is touched, the controller 33 repeats the procedure of receiving the operation.
Next, with reference to the drawings from
A list of parts of which the transparency percentages are settable is displayed on the transparency percentage setting screen S2. The user enters an arbitrary transparency percentage for each part. Examples of the parts of which the transparency percentages are settable are the left door panel, a right door panel, the rear gate, the tail lamps and the upper portion of the vehicle.
Due to limitations of space of the display apparatus 4, if it is not possible to display all the parts on one page, a button NB for moving to a next page may be provided and the parts may also be displayed on the next page. For example, the steering wheel, the dashboard, the tires, the wheel housings, the headlamps, the rearview mirror are listed on the next page. The transparency percentages of those parts are settable in a range from 0% to 100% on the transparency percentage setting screen S2.
The transparency percentage setting screen S2 shows the list of the parts. However, the parts may be displayed on the cabin image 200 and the user may touch and select one of the parts on the cabin image 200 to set the transparency percentage for the part.
The transparency percentage setting screen S3 via parts displayed on an image, shown in an upper drawing in
Then, based on the operation made by the user with the part displayed on the display apparatus 4, for which the transparency percentage is set, the image transparency adjustor 32c identifies the part and changes the transparency percentage thereof. Since the cabin image 200 including the part at the changed transparency percentage is superimposed on the surrounding image AP and is displayed, the user can immediately see a combined image displayed at the changed transparency percentage. Thus, the user can set the transparency percentage comfortable to a sense of each user. For example, as shown in a lower drawing in
As shown in
Next explained, with reference to the drawings from
In a case of the framed structure display of the vehicle external shape, the transparency percentage of the cabin image 200 is set at 100% and only the frame f of the vehicle 2 is displayed in lines as an outline of an external shape of the vehicle 2. Since the outline showing the external shape of the vehicle is not transparent on the cabin image 200, it is possible to see the surrounding image AP, except an overlap with the lines showing the external shape of the vehicle 2.
The external shape of the vehicle refers to an outline of the vehicle that is a most outer appearance of the vehicle viewed from an outside, i.e., an outer frame. The image transparency adjustor 32c displays the vehicle 2 in the framed structure by displaying the outer frame of the vehicle 2 in lines. By displaying the vehicle 2 in the framed structure, the user can recognize the surrounding image AP showing a broad area, understanding a position of the vehicle body displayed in the frame f. Therefore, the user does not have to see individual parts such as the tail lamps and tires so that the attention of the user is not distracted. Therefore, the user can concentrate on an obstacle and the like adjacent to the vehicle 2. The transparency percentage of the cabin image 200 is here set at 100%. However, a high transparency percentage, such as 90%, may be set instead of 100% of the transparency percentage. The transparency percentage may be any percentage as long as the user can clearly recognize the surrounding image AP showing the broad area.
When the tail lamp 207 is displayed in the mesh pattern, the user can see the surrounding image AP through the mesh pattern. The tail lamp 207 is usually displayed in red in the combined image CP. Thus, even if the transparency percentage of the tail lamp 207 is reduced; the user sees the surrounding image AP through the red tail lamp 207. In this case, since the user has to determine a color of an object located behind the tail lamp 207 through the red of the tail lamp 207, it is difficult for the user to determine the color of the object. Especially, if a lamp or another lighting system of a different vehicle or a traffic light is located behind the tail lamp 207 included in the surrounding image AP, the lamp or the light is overlapped with the red of the tail lamp 207 and is so unclear that the overlap may adversely affect a determination of the surrounding situation. Therefore, by the display of the tail lamp 207 in the mesh pattern, the user can understand the color of the surrounding image AP clearly and can determine the situation outside the vehicle accurately.
The tail lamp is displayed in the mesh pattern here. However, a part other than the tail lamp may be displayed in a mesh pattern. Moreover, it is recommended that a part having higher chroma, as compared to chroma of other parts, should be displayed in the mesh pattern because the part having higher chroma makes colors of the surrounding image AP difficult to be determined.
When no display of tail lamps is selected via the display mode setting screen S1, the combined image CP is displayed on the display apparatus 4 and then the tail lamps are gradually faded out (a tail lamp 207o and a tail lamp 202o) by a gradual increase of the transparency percentages of the right tail lamp 207 and the left tail lamp 202. When the transparency percentages of the tail lamps are increased to reach 100%, the tail lamps are not displayed completely (erased).
By the no display of the tail lamps, the user can see the surrounding image AP more clearly. In other words, when the back side of the vehicle is displayed, the tail lamps are displayed around a center area of the display apparatus 4. Therefore, even if the transparency percentages of the tail lamps are increased, the tail lamps are overlapped with the surrounding image AP displayed around the center area of the display apparatus 4 that the user desires to see. Thus, it may be difficult for the user to recognize an obstacle and the like. Therefore, by the gradual fade-out of the tail lamps, the user can clearly recognize the surrounding image AP. Moreover, since the tail lamps are gradually faded out, even after the tail lamps are erased, the user can remember originally displayed positions of the tail lamps. Therefore, it is easier for the user to understand the obstacle and the like adjacent to the vehicle 2.
As described above, the display of the tail lamps is faded out gradually. However, after the tail lamps are erased, the transparency percentages of the tail lamps may be gradually decreased to display the tail lamps again. Since positions of the tail lamps may serve as a reference to measure a height of the vehicle, by displaying the tail lamps again, it becomes easier for the user to understand a positional relationship between the vehicle 2 and an object located in the surroundings. In this case, it is recommended that a time interval between erasing and redisplaying the tail lamps should be set relatively long, for example, 10 seconds. If a relatively short interval, for example, two seconds or less, is set, the tail lamps displayed in the center area of the display apparatus 4 stand out, and it is more difficult for the user to see the surrounding image AP.
When the display mode of the no display of the upper portion of the vehicle is selected via the display mode setting screen S1, the image transparency adjustor 32c sets the transparency percentage of the portion of the cabin image 200 higher than the height h at 100%. Thus, the user can recognize the surroundings of the vehicle 2 more widely, understanding the position of the vehicle via a portion lower than the height h on the image.
The height h is, for example, as a same height as a waist of an upstanding person who may be a driver of the vehicle 2. When looking at the surroundings of the vehicle 2, by the erasing of the portion of the cabin image 200 higher than the waist, the user can widely recognize the surroundings of the vehicle 2 higher than the waist, understanding the position of the vehicle via the portion lower than the waist on the image.
As described above, the image processing apparatus in this embodiment determines the individual transparency percentages of the plural portions of the cabin image 200 and causes the plural portions to be displayed at the individual transparency percentages. Thus, the user can intuitively understand the positional relationship between an object in the surrounding area and the vehicle 2.
Moreover, since the image processing apparatus displays a predetermined portion of the cabin image 200 at an increased transparency percentage as compared to another portion, the attention of the user can be draw to the portion displayed at the increased transparency percentage. Thus, the user can drive more safely.
Further, since the user sees the surrounding image AP through the cabin image 200, as compared with a case where the surrounding image AP is only displayed, the user can immediately recognize a direction displayed on the display apparatus 4.
The embodiment of the invention is described above. However, the invention is not limited to the embodiment, and various modifications are possible. Examples of the modifications of the invention are described below. The embodiment described above and all forms including the modifications below may be arbitrarily combined.
In the embodiment described above, the combined image CP viewed from the driver seat viewpoint is displayed on the entire display apparatus 4. However, the display apparatus 4 may display the combined image CP viewed from the driver seat viewpoint and an overhead view image looked down from above the vehicle 2, side by side.
Moreover, in the embodiment described above, when an image is displayed at the transparency percentage set by the user, the transparency percentage is increased to the predetermined value, depending on the vehicle state or the surrounding situation. However, based on the transparency percentage set by the user, a new transparency percentage may be set. For example, the transparency percentage set by the user may be multiplied for a predetermined value.
Moreover, in the embodiment described above, the cabin image 200 is caused to be transparent. However, the vehicle body image 100 may be transparent. In this case, a virtual viewpoint should be set outside the vehicle.
In the embodiment described above, the virtual viewpoint is located at an arbitrary position in an arbitrary view direction in the virtual 3-D space. In a case of one camera, a position and a view direction of the camera may be the position and the view direction of a virtual viewpoint.
Moreover, in the embodiment described above, an example of an image including the vehicle image caused to be transparent, viewed from the driver seat viewpoint. However, during display of both an image viewed from the driver seat viewpoint and an overhead view image, the image viewed from the driver seat viewpoint may be changed over between an image having a transparent portion and an image having no transparent portion. In this case, during the display of both the image viewed from the driver seat viewpoint and the overhead view image, when the user presses a change-over button, the image having the transparent portion is changed to an image having no transparent portion, and vice versa. In other words, in a case where the driver seat viewpoint (VPa or VPb in
Contrarily, in a case where the overhead viewpoint has been selected, the driver seat viewpoint is selected as a new virtual viewpoint, and then the viewpoints may be changed in order based on a user instruction. In a case where a side-by-side image where the overhead view image and the image viewed from the driver seat viewpoint are displayed side by side is displayed, the image viewed from the driver seat viewpoint, the overhead view image and the side-by-side image may be displayed in order.
In the embodiment described above, the various functions are implemented by software using the CPU executing the arithmetic processing in accordance with the program. However, a part of the functions may be implemented by an electrical hardware circuit. Contrarily, a part of functions executed by hardware may be implemented by software.
Number | Date | Country | Kind |
---|---|---|---|
2013-073557 | Mar 2013 | JP | national |
Number | Name | Date | Kind |
---|---|---|---|
7212653 | Sato | May 2007 | B2 |
20020167589 | Schofield | Nov 2002 | A1 |
20050078325 | Momose | Apr 2005 | A1 |
20050231532 | Suzuki | Oct 2005 | A1 |
20080049150 | Herbin | Feb 2008 | A1 |
20080109751 | Fitzmaurice | May 2008 | A1 |
20080246843 | Nagata | Oct 2008 | A1 |
20100289634 | Ikeda | Nov 2010 | A1 |
20100315413 | Izadi | Dec 2010 | A1 |
20110001751 | Carlsson | Jan 2011 | A1 |
20110043632 | Satoh | Feb 2011 | A1 |
20110307176 | Sun | Dec 2011 | A1 |
20120113261 | Satoh | May 2012 | A1 |
20120242834 | Satoh | Sep 2012 | A1 |
20120249584 | Naruse | Oct 2012 | A1 |
20120249789 | Satoh | Oct 2012 | A1 |
20120327238 | Satoh | Dec 2012 | A1 |
20130135235 | Noutomi | May 2013 | A1 |
20130300872 | Park | Nov 2013 | A1 |
Number | Date | Country |
---|---|---|
H09-50541 | Feb 1997 | JP |
2003196645 | Jul 2003 | JP |
2010-114618 | May 2010 | JP |
A-2010-109684 | May 2010 | JP |
2011-023805 | Feb 2011 | JP |
2011188335 | Sep 2011 | JP |
2009-104675 | Aug 2009 | WO |
Entry |
---|
Bollinger, Susan A., et al. “Grinline identification using digital imaging and Adobe Photoshop.” Journal of forensic sciences 54.2 (2009): 422-427. |
Partial translation of Oct. 18, 2016 Office Action issued in Japanese patent application No. 2013-073557. |
Number | Date | Country | |
---|---|---|---|
20140292805 A1 | Oct 2014 | US |