The invention relates to a method for processing images of the exterior of a vehicle. In particular, the invention relates to a method for processing images of the exterior of a vehicle and displaying the processed image for viewing by the vehicle operator.
Apparatus for converting a camera image of a vehicle exterior into a bird's eye view image and displaying the bird's eye view image in the vehicle on which the camera is mounted are known. Such bird's eye view images however, can appear warped or distorted. Cameras in known bird's eye view systems are mounted to an exterior of the vehicle, such as near the license plate mount or in a side view mirror. Such cameras are oriented to the ground at an angle, such as about 45 degrees.
Processors in known bird's eye view systems assume facts about the environment viewed by the camera. For example, known processors assume that every object viewed is lying in the ground plane. Consequently, objects that are in, or very close to the ground plane, such as parking space markings or curbs, appear relatively without distortion in the processed bird's eye view image. In contrast, objects or portions of objects that are higher off the ground, such as the upper portion of a parked vehicle or the upper portion of the tires of the parked vehicle, are assumed to be further from the camera than objects closer to the ground. The known processors then attempt to compensate for the assumed distance of the relatively higher objects by adjusting the displayed image to enlarge the portions of objects assumed to be more distant from the camera. The known systems may thereby display a processed bird's eye view image wherein portions appear distorted. Such distortion makes it difficult for the vehicle driver to understand accurately the physical environment surrounding the vehicle.
For example, a representation of a known bird's eye view image is shown at 10 in
It is therefore desirable to provide a system that produces an improved bird's eye view image for viewing within a vehicle upon which a camera is mounted.
The present application describes various embodiments of a vehicle image processing method. One embodiment of the method for processing an image of the exterior of a vehicle includes transmitting successive camera images from a camera to a processor. Optical flow vectors from the multiple camera images are estimated. The optical flow vectors are compared and objects located on the ground are separated from objects located above the ground. Vehicle motion is estimated. Data from the successive camera images is processed to create an estimated three-dimensional (3D) bird's eye view image, and the bird's eye view image is displayed.
Other advantages of the vehicle image processing method will become apparent to those skilled in the art from the following detailed description, when read in light of the accompanying drawings.
As used in the description of the invention and the appended claims, the phrase “three dimensional” or “3D” is defined as the combination of the height, width, and distance from the vehicle of an object sensed or imaged by a vehicle mounted camera used in the method of the invention.
Referring now to the drawings, there is shown generally at 20 in
The vehicle 40 is equipped with at least one camera 42. In the illustrated embodiment, four cameras 42 are mounted to the rear, front, and sides, respectively, of the vehicle 40. In the illustrated embodiment, the side mounted cameras 42 are mounted on or within the side mirrors 46 of the vehicle 40. Alternatively, the side mounted cameras 42 may be mounted to any desired portion of the vehicle sides, such as the doors, front and rear quarter panels, or roof panel, such as the portion of the roof panel between the driver and passenger doors. In the illustrated embodiment, the front camera 42 is mounted to the grill and the rear camera 42 is mounted near the license plate mount. The cameras 42 may be mounted to any other desired locations in the front and rear of the vehicle. In another embodiment, the camera 42 may be mounted to the interior rear-view mirror.
The cameras 42 may be any desired digital camera, such as a charge coupled device (CCD) camera. Alternatively, any other type of camera may be used, such as a complementary metal-oxide-semiconductor (CMOS) camera. In the illustrated embodiment, the cameras 42 are CCD video cameras.
The processor 44 may be any type of image-processing unit suitable for carrying out image-processing. One example of a suitable image processor is the IMAPCAR® processor manufactured by NEC. Another example of a suitable image processor is the PowerPC® processor manufactured by Freescale Semiconductor. Alternatively, any image processor or computer that can recognize road markers such as white lines, stationary objects, and moving vehicles and pedestrians in real time may be used. The processor 42 may be located at any desired location in the vehicle. If desired, memory devices may be used with the processor 44. Examples of such memory devices include a hard disc drive, a DVD drive, and semi-conductor memory.
In a second step 24 of the method 20, optical flow vectors, such as illustrated by the vector arrows 54 and 62 in
The processor 44 may be programmed to assume that the largest portion of an image captured by the camera 42 is the ground. Accordingly, the largest area or portion of an image flowing in the same direction relative to the vehicle 40 may be assumed to be the ground. As shown in
As shown in
In one embodiment of the method, the ground flow rate of the largest area of optical flow may be estimated by identifying a peak value on one or more histograms of the flow rate and/or direction of pixel flow. In one embodiment of the histogram, the x-axis includes the value of the absolute velocity or magnitude of the pixel flow of the various portions of the image and the y-axis includes the frequency each value appears. In another embodiment of the histogram, the x-axis includes the direction of flow of each pixel in the image and the y-axis includes the frequency each pixel flow direction appears.
In a third step 26 of the method 20, objects located on the ground may be distinguished or separated from objects located above the ground, and further separated from objects moving on a trajectory different than the vehicle 40 upon which the video camera 42 is mounted.
In a fourth step 28 of the method 20, vehicle motion may be estimated. Vehicle motion may be estimated by any desired method. One embodiment of a method of estimating vehicle motion is shown in
If desired, the fourth step 28 of the method 20, may further include measuring vehicle motion, as shown at 30 in
In a fifth step 32 of the method 20, a 3D distortion-free bird's eye view image of the vehicle 40 and its immediate surroundings may be created.
An object or portion of the image that is flowing on a trajectory different than the vehicle 40 will be interpreted as being a moving obstacle. Examples of objects that may be sensed by the camera 42 and interpreted as being an obstacle include vehicles or other objects sensed by the camera 42 but moving on a trajectory different than the vehicle 40.
In the exemplary embodiment, a 3D image of the environment outside the vehicle may be estimated using one camera 42, as best shown in
Referring now to
The height h3 of the object 70 may then be calculated using the following formulas, wherein:
dmoved is the horizontal distance the vehicle V moved between positions V1 and V2.
d2 is the horizontal distance from the camera 42 in vehicle position V1 and the point y.
d1 is the sum of dmoved and d2.
θ1 is the measured angle from the camera in vehicle position V1 to the point y.
θ2 is the measured angle from the camera in vehicle position V2 to the point y.
h1 is the known height (vertical distance) of the camera above the ground.
h2 is the calculated height (vertical distance) from the point y to the camera.
h3 is the calculated height (vertical distance) of the object 70.
In a sixth step 34 of the method 20, processed 3D data may be displayed as a two-dimensional (2D) image, such as on an in-vehicle monitor. The in-vehicle monitor may be any desired monitor, such as a liquid crystal display (LCD) mounted in an instrument panel or dash board. One example of a representative corrected bird's eye view image that may be viewed on the monitor is shown at 66 in
Vehicle tires 17 are shown in the corrected image 66 in
If desired, the sixth step of the method 20 may further include generating and displaying a 3D version of the image on a 3D capable LCD screen, as shown at 36 in
The principle and mode of operation of the method and system for processing images of the exterior of a vehicle have been described in its preferred embodiment. However, it should be noted that the method described herein may be practiced otherwise than as specifically illustrated and described without departing from its scope.