The present invention relates to combining multiple images taken of an object from different angles. It finds particular application in conjunction with a bird's eye view system for a vehicle and will be described with particular reference thereto. It will be appreciated, however, that the invention is also amenable to other applications.
A display image of a bird's eye view system typically combines multiple (e.g., four (4) or more) camera sub-images into a single final image. In areas where the sub-images meet, some sort of stitching or blending is used to make the multiple sub-images appear as a single, cohesive final image. One issue with conventional stitching methods is that three-dimensional objects in combined areas are commonly not shown (e.g., the three-dimensional objects “disappear”) due to the geometric characteristics of the different sub-images. For example, only a lowest part (e.g., the shoes of a pedestrian) may be visible in the stitched area.
One process used to address the issue with 3D objects in combined sub-images is to make the blending of the sub-images more additive. However, one undesirable effect of additive blending is the appearance of duplicate ghost-like figures of objects in a final image, as the object is seen and displayed twice. Such ghosting makes it difficult for a user to clearly perceive the location of the object.
The present invention provides a new and improved apparatus and method for processing images taken of an object from cameras at different angles.
In one aspect of the present invention, a method for graphically indicating an object in a final image includes obtaining a plurality of sub-images including the object from respective image capturing devices at different angles, replacing a portion of a first of the sub-images with a corresponding portion of a second of the sub-images, replacing a portion of the second sub-image with a corresponding portion of the first sub-image, and generating the final image including a graphical representation of the object as a two-dimensional view not aligned with a common virtual viewpoint based on the first and second sub-images including the respective replaced portions.
In the accompanying drawings which are incorporated in and constitute a part of the specification, embodiments of the invention are illustrated, which, together with a general description of the invention given above, and the detailed description given below, serve to exemplify the embodiments of this invention.
With reference to
With reference to
With reference to
With reference to
Once the ECU 22 receives an instruction to begin obtaining preliminary images around the vehicle 10, the ECU 22 transmits signals to the cameras 12 to begin obtaining respective preliminary images. For example, the camera 121 begins obtaining first preliminary images (“first images”) (see, for example, an image 321), and the camera 122 begins obtaining second preliminary images (“second images”) (see, for example, 322). In the illustrated embodiment, both the first and second images 321,2 include images of the object 28. However, the first images from the first camera 121 view the object 28 from a first angle α1, and the second images from the second camera 122 view the object 28 from a second angle α2. The first image as recorded by the first camera 121 is represented as 321, and the second image as recorded by the second camera 122 is represented as 322. The images 321, 322 are received by and transmitted from the respective cameras 121, 122 to the ECU 22 in a step 112. In one embodiment, the images 321, 322 are transmitted from the respective cameras 121, 122 to the ECU 22 via the vehicle communication bus 26.
In a step 114, the ECU 22 identifies images 321 of the object 28 received from the first camera 121 as first sub-images and also identifies images 322 of the object 28 received from the second camera 122 as second sub-images. In one embodiment, each of the first and second images (e.g., sub-images) 321,2, respectively, includes a plurality of pixels 34. In a step 116, each of the pixels 34 is identified as having a respective particular color value (e.g., each pixel is identified as having a particular red-green-blue (RGB) numerical value) or gray-scale value (e.g. between zero (0) for black and 255 for white) or contrast level (e.g. between −255 and 255 for 8 bit images).
In a step 120, respective locations for each of the pixels 34 in the first and second sub-images 321,2 are identified. In one embodiment, images of in-the-ground plane markers 36, which may be captured by any of the cameras 12, are used by the ECU 22 to map pixel locations around the vehicle 10. The ground plane pixel locations mapped around the vehicle 10 are considered absolute locations and it is assumed that the cameras 121, 122 are calibrated to measure the same physical location for pixels in the ground plane, causing the cameras to agree on gray level and/or color and/or contrast level values there. Since the ground plane pixel locations mapped around the vehicle are absolute, pixels at a same physical ground plane location around the vehicle 10 are identified by the ECU 22 as having the same location in the step 120 even if the pixels appear in different sub-images obtained by different ones of the cameras 12. For example, since a pixel 341 in the first sub-image 321 is identified as being at the same physical ground plane location around the vehicle 10 as a pixel 342 in the second sub-image 322 (i.e., the 341 in the first sub-image 321 is at a corresponding location with the 342 in the second sub-image 322), the pixel 341 in the first sub-image 321 is identified in the step 120 as having the same location (e.g., same absolute ground plane location) as the pixel 342 in the second sub-image 322.
In a step 122, a determination is made for each of the pixels in the first sub-image 321 (i.e., the image from the first camera 121) whether the respective pixel substantially matches a color (or gray-scale or contrast) of the pixel at the corresponding location in the second sub-image 322 (i.e., the image from the second camera 122). In one embodiment, the respective numerical color (or gray-scale) value for each of the pixels in the first sub-image 321 is compared with the numerical color (or gray-scale or contrast) value of the pixel at the corresponding location in the second sub-image 322. If the numerical color (or gray-scale) value of the respective pixel in the first sub-image 321 is within a predetermined threshold range of the numerical color (or gray-scale) value of the respective pixel at the corresponding location in the second sub-image 322, it is determined in the step 122 that the respective pixel in the first sub-image 321 substantially matches the pixel at the corresponding location in the second sub-image 322 (thereby implying or signifying that the pixel seen there belongs to the ground plane). For example, if the first and second sub-images 321,2, respectively, are color images represented with RGB values, each of the respective R-value, G-value, and B-value of the pixel in the first sub-image 321 must be within the predetermined threshold range of the R-value, G-value, and B-value of the corresponding pixel in the second sub-image 322. This range can be captured with, for instance, the Euclidean distance between respective RGB values: square root ((R1−R2)*(R1−R2)+(G1−G2)*(G1−G2)+(B1−B2)*(B1−B2)). Other distance measures may be used, including Manhattan, component ratios, etc.
In one embodiment, it is contemplated that each of the predetermined threshold range for each of the respective R-value, G-value, and B-value is 10%. In this case, if each of the respective R-value, G-value, and B-value is a value of zero (0) to 255, the respective R-value, G-value, and B-value of the pixel in the first sub-image 321 must be within a range of twenty-six (26) along the zero (0) to 255 scale of the R-value, G-value, and B-value of the corresponding pixel in the second sub-image 322 to be considered with the predetermined threshold range. Alternatively, if the first and second sub-images are color images represented with gray-scale values, the gray-scale value of the pixel in the first sub-image 321 must be within the predetermined threshold range (e.g., within 10% along a range of zero (0) to 255 gray-scale values) of the gray-scale value of the corresponding pixel in the second sub-image 322.
Although the above examples disclose the predetermined threshold range as within 10% along a range of zero (0) to 255, it is to be understood that any other predetermined threshold range is also contemplated. For example, instead of a percentage, the predetermined threshold range may be defined as an absolute number.
In a step 124, any of the pixels in the first sub-image 321 (i.e., the image from the first camera 121) determined in the step 122 to not match respective pixels at corresponding locations in the second sub-image 322 (i.e., the image from the first camera 121) are replaced. The replacement value comes from the camera angularly nearer to the image location in question, for instance, the differing pixels in 321 are replaced by those from camera 122. In one embodiment, replacing a pixel 34 in the first sub-image 321 with the respective pixel 34 in the second sub-image 322 involves replacing the color value (or gray-scale value) of the pixel 34 in the first sub-image 321 with the color value (or gray-scale value) of the pixel 34 in the second sub-image 322. The effect of replacing the pixels in the step 124 is to replace a portion of the first sub-image 321 with a corresponding portion of the second sub-image 322. For example, the portion (e.g., pixels) of the first sub-image 321 that are inconsistent with a corresponding portion (e.g., pixels) of the second image 322 are replaced. The replacement “erases” the inconsistent views, using the background, such as the road surface, there instead. As both views are erased, the object is effectively removed.
In a step 126, a determination is made for each of the pixels in the second sub-image 322 (i.e., the image from the second camera 122) whether the respective pixel substantially matches a color (or gray-scale) of the pixel at the corresponding location in the original first sub-image 321 (i.e., the image from the first camera 121). As discussed above, if the numerical color (or gray-scale) value of the respective pixel in the second sub-image 322 is within a predetermined threshold range of the numerical color (or gray-scale) value of the respective pixel at the corresponding location in the first sub-image 321, it is determined in the step 126 that the respective pixel in the second sub-image 322 substantially matches the pixel at the corresponding location in the first sub-image 321. For example, if the second and first sub-images 322,1, respectively, are color images represented with RGB values, each of the respective R-value, G-value, and B-value of the pixel in the second sub-image 322 must be within the predetermined threshold range of the R-value, G-value, and B-value of the corresponding pixel in the first sub-image 321. Alternatively, if the second and first sub-images are color images represented with gray-scale values, the gray-scale value of the pixel in the second sub-image 322 must be within the predetermined threshold range of the gray-scale value of the corresponding pixel in the first sub-image 321.
In a step 130, any of the pixels in the second sub-image 322 (i.e., the image from the second camera 122) determined in the step 126 to not match respective pixels at corresponding locations in the first sub-image 321 (i.e., the image from the first camera 121) are replaced, in a manner similar to that previously described. In one embodiment, replacing a pixel 34 in the second sub-image 322 with the respective pixel 34 in the first sub-image 321 involves replacing the color value (or gray-scale value) of the pixel 34 in the second sub-image 322 with the color value (or gray-scale value) of the pixel 34 in the first sub-image 321. The effect of replacing the pixels in the step 130 is to replace a portion of the second sub-image 322 with a corresponding portion of the first sub-image 321. For example, the portion (e.g., pixels) of the second sub-image 322 that are inconsistent with a corresponding portion (e.g., pixels) of the first image 321 are replaced. In a step 132, leftover single or small groups of pixels are erased by a morphological erosion operation.
Since the first and second sub-images 321, 322 of the object 28 are taken from different angles α1, α2, respectively, the replacement of pixels in the steps 124 and 130 removes duplicated views (with differing aspects) of the object 28 from the original first and original second sub-images 321, 322. The first sub-image resulting from the step 124 is referred to as a modified first sub-image 321M (see
With reference to
In a step 136, an intersection point 40 between the modified first sub-image 321M and the modified second sub-image 322M that is a minimum distance to the first and second image capturing devices 121, 122, respectively, is identified. In one embodiment, the minimum distance from the intersection point 40 to the first and second image capturing devices 121, 122, respectively, is identified by determining, for each point at which the modified first sub-image 321M and the modified second sub-image 322M intersect, a total distance that is determined as a total of the respective distances to the first image capturing device 121 and the second image capturing device 122. The intersection point 40 having the smallest total distance is identified in the step 136 as minimum distance to the first and second image capturing devices 12022, respectively.
A base of the object 28 is identified at the intersection point 40 in a step 140. For example, an icon 42 (e.g., a triangle or circle) is placed in the final image 32F to represent the location of the base of the object 28. With reference to
With reference to
The height 44 of the object 28 is conveyed in the final image 32F in a step 144. In one embodiment, the height 44 of the object 28 may be conveyed by displaying the icon 42 in a particular color. For example, a red icon 42 may be used to identify an object 28 over 7 feet high (tall), a yellow icon 42 may be used to identify an object 28 between 4 feet and 7 feet high (tall), and a green icon 42 may be used to identify an object 28 less than 4 feet high (tall). Alternatively, or in addition to the colored icon 42, a number 46 may be displayed proximate the icon 42 indicating the height 44 of the object 28 in, for example, feet and/or a size of the icon 42 displayed in the final image 32F in may be based on the height 44 of the object 28 (e.g., an object less than 4 feet tall is represented by a relatively smaller icon 42 than an object greater than 4 feet tall, and an object less than 6 feet tall is represented by a relatively smaller icon 42 than an object greater than 6 feet tall).
The distance 50 between the base of the object 28 and the vehicle 10 is determined in the final image 32F in a step 146. In one embodiment, the distance 50 is determined to be the shortest distance between the base of the object 28 and the vehicle 10. It is to be understood that trigonometry is used by the ECU 22 to determine the shortest distance between the object 28 and the vehicle 10. The distance is conveyed in a step 150. The distance 50 may be conveyed by displaying the icon 42 in a particular color. For example, a red icon 42 may be used to identify an object 28 is less than 3 feet to the vehicle 10, a yellow icon 42 may be used to identify an object 28 that is between 3 feet and 6 feet to the vehicle 10, and a green icon 42 may be used to identify an object 28 more than 6 feet to the vehicle 10. The color of the object 28 may change as the distance between the object 28 and the vehicle 10 changes. For example, if the object 28 is initially more than 6 feet from the vehicle 10 but then quickly comes within 3 feet of the vehicle 10, the color of the object 28 would initially be green and then change to red. Optionally, an operator of the vehicle is notified if the object 28 is less than 6 feet from the vehicle 10. In addition, a time until the object 28 is expected to collide with the vehicle 10, based on a current rate of movement toward each other, could be indicated.
Alternatively, or in addition to the colored icon 42, a number 52 may be displayed proximate the icon 42 indicating the distance of the object 28 to the vehicle 10 in, for example, feet and/or a size of the icon 42 displayed in the final image 32F may be based on the distance 50 of the object 28 to the vehicle 10 (e.g., an object less than 3 feet to the vehicle 10 is represented by a relatively smaller icon 42 than an object greater than 3 feet to the vehicle 10, and an object less than 6 feet to the vehicle 10 is represented by a relatively smaller icon 42 than an object greater than 6 feet to the vehicle 10).
It is to be understood that different representations are used for conveying the height 44 of the object 28 and the distance 50 of the object 28 to the vehicle 10. For example, if color is used to convey the height 44 of the object 28, then some other representation (e.g., a size of the icon 42) is used to convey the distance 50 of the object 28 to the vehicle 10.
In one embodiment, it is contemplated that the object 28 and the vehicle 10 in the final image 32F are not aligned with a common virtual viewpoint. In other words, simple two-dimensional views of any of the objects 28 and the vehicle 10 are presented in the final image 32F without perspective on the display 24 (see
It is contemplated that the final image 32F is continuously generated from the first and second sub-images 321M, 322M and the first and second sub-images 321M, 322M are continuously generated from the first and second images 321, 322. Therefore, the final image 32F is continuously displayed in real-time (e.g., live) on the display device 24 (see
While the present invention has been illustrated by the description of embodiments thereof, and while the embodiments have been described in considerable detail, it is not the intention of the applicants to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and modifications will readily appear to those skilled in the art. Therefore, the invention, in its broader aspects, is not limited to the specific details, the representative apparatus, and illustrative examples shown and described. Accordingly, departures may be made from such details without departing from the spirit or scope of the applicant's general inventive concept.