The present disclosure relates to methods and systems for image processing.
The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.
It is common for surveillance systems such as, for example, security systems or navigation systems, to include multiple cameras that survey an area from different angles. The image data from the cameras is then combined and displayed as a single image. When image data from two or more cameras is combined, the resulting image can be difficult to understand, especially in the case where the image contains moving objects. For example, blind spots can exist where the object is not seen by one camera, and is only partially seen by the other camera. Additionally, if a moving object transitions from a single camera area to an area where both cameras are viewing, a dead spot may be displayed where the object disappears.
To compensate for these issues, conventional methods, for example, combine the images from both cameras by assigning a weight of 50% to each image and then displaying the combined image. However, this method results in strong visual artifacts which make the output difficult to understand. In addition, the object appears to “jump” or change angle unexpectedly as one camera loses sight of the object and the other camera picks it up.
Accordingly, a method of processing image data is provided. The method includes determining an overlap area based on image data from a first image sensor and image data from a second image sensor; computing a first weight and a second weight based on a relative position of the image data in the overlap area; and generating a final image by blending the image data from the first image sensor and the second image sensor based on the first weight and the second weight.
Further areas of applicability will become apparent from the description provided herein. It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.
The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.
The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses. It should be understood that throughout the drawings, corresponding reference numerals indicate like or corresponding parts and features. As used herein, the term module refers to an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that executes one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.
As can be appreciated, the image processing systems of the present disclosure can be applicable to any surveillance system that includes multiple image sensors. For exemplary purposes, the disclosure will be discussed in the context of a vehicle that includes at least two image sensors.
Referring now to
An image processing system (IPS) 28 processes the sensor image data from the first image sensor 12, the second image sensor 16, and/or the third image sensor 18 to generate an overall image of the vehicle surroundings for display. As shown in
Referring now to
In one embodiment, the image processing module 30 processes the image data pixel by pixel. In other embodiments, the image processing module 30 processes the image data frame by frame. In yet other embodiments, the image processing module 30 processes the image data according to a combination of pixel by pixel and frame by frame. For exemplary purposes, the image processing module 30 of
As shown in
The mapping module 30 generates a base image pixel map 52 and an overlap image pixel map 54. The pixel maps 52 and 54 associate the image data with the pixels 35 (
In one example, as shown in
Referring back to
In one example, as shown in
Referring back to
For example, the weight module 44 traverses the overlap area map 58. Once a first overlapping area is encountered, the X,Y location is recorded. A horizontal scale (HS) is calculated by finding a distance from a position of the first overlapping area to a maximum width (MaxW) of the total overlap area. A vertical scale (VS) is calculated by finding a distance from a position of the first overlapping area to a maximum height (MaxH) of the total overlap area. For example, the horizontal scale (HS) can be computed based on the following equation:
HS=MaxW−FirstOverlapXpos. (1)
The vertical scale (VS) can be computed based on the following equation:
VS=MaxH−FirstOverlapYpos. (2)
In various embodiments, the weight module 44 determines the maximum height (MaxH) and the maximum width (MaxW) when the maximum height or width of the total overlap area is not static, or not known at compile time. For example, to determine the maximum width (MaxW), a loop can ‘look ahead’ and test each following pixel until either the end of the row is reached, or a non-overlap area (zero) is found. To determine the maximum height (MaxH), a loop can ‘look ahead’ and test each following pixel until either the end of the column is reached, or a non-overlap area (zero) is found. Once the maximum height and width are found, the horizontal and vertical scales can be computed as described above.
Once the horizontal and vertical scales are determined, the weight module 44 continues to traverse the overlap area map 58. For each overlap area, the weight module 44 computes a relative position. For example, the relative position can be computed as the current position (XY coordinate) minus the first overlap position as shown as:
relative X position=CurrentXpos−FirstOverlapXpos; (3)
and
relative Y position=CurrentYpos−FirstOverlapYpos. (4)
The weight module 44 then computes a horizontal weight (HW) and a vertical weight (VW). For example, the horizontal weight (HW) can be computed as the relative horizontal position divided by the horizontal scale (HS) as shown as:
The vertical weight (VW) can be computed as the vertical position divided by the vertical scale (VS) as shown as:
In this example, the weights are computed as
This is to ensure that the top leftmost area of the total overlap area is given the highest weight. In various embodiments, the computation depends on the scanning direction and placement of the image sensor 16 or 18. If the overlap image data 50 is from the right-side image sensor 18, the weights are computed as
since the top leftmost overlap area should have equal weight between the right and rear image sensors 18 and 12 (
In various embodiments, the weight module 44 can limit the minimum weight of each axis. This prevents any one sensor image from disappearing from regions when the computed weight is too low. However, in this example, the weights are no longer fully linear. In various other embodiments, the weight module 44 computes the horizontal weight (HW) by changing the scale that is being used to compute the weights. For example, the weights are computed based on a desired minimum weight (DMW) as shown as:
For example, using 30% as the desired minimum weight, the equation is shown as:
The weight module 44 then computes a total overlap weight (OW) for the areal and a total base weight (BW) for the area. For example, the total overlap weight (OW) can be computed as an average of the horizontal weight (HW) and vertical weight (VW) as shown as:
The total base weight (BW) can be assigned the inverse weight as shown as:
BW=(1−OW) (10)
In various embodiments, the weight module uses other scanning directions and image sensor placements. The scale, weight, and relative position computations are adjusted so that even weight is given to any pixel which is equidistant from both image sensors. As can be appreciated, when processing image data from more than two image sensors, the weights are computed independently for each overlap image data. Some additional scaling may be needed to ensure that all weights sum to 100%.
In various embodiments, the weight module 44 can perform non-linear weight computations. For example, if an exponential weight decay is desired, each axis weight can be calculated using an exponential decay. In this case the axis weights are computed as shown as:
HW=(e−scalefactor)*HWprev+[(1e−scalefactor)]*MW. (11)
Where the scale factor determines how fast the decay occurs. A higher scale factor will increase the decay rate, and a lower scale factor will reduce the decay rate. When the first overlap area is found, the initial horizontal and vertical weights are set to one.
Once the weight module 44 computes the weights 60 and 62 for each pixel, the display module 46 receives as input the base image data (BIMdata) 48, the overlap image data (OIMdata) 50, the overlap weights (OW) 60, and the base weights (BW) 62. The display module 46 generates output pixel data (OP) 38 based on the received inputs. In one example, data for each output pixel 38 is computed based on the following equation:
OP=(OIMdata*OW)+(BIMdata*BW). (12)
Referring now to
In one example, the method may begin at 100. The base image data and the overlap image data are mapped at 102. Based on the mapping, image pixel data from the overlap image and corresponding image pixel data from the base image are obtained at 104. The image pixel data of the base image and the overlap image are evaluated at 106, 108, and 112. If image pixel data for the base image does not exist (e.g., null value) at 106 and image pixel data for the overlap image does exist (e.g., value other than null) at 108, the overlap weight is set to 100% and the base weight is set to 0% at 110. The output pixel is computed based on the overlap weight and the base weight at 126, for example, as shown in equation 12.
If, however, image pixel data for the base image does not exist at 106 and image pixel data for the overlap image does not exist at 108, the base weight is set to 0% and the overlap weight is set to 0% at 112. The output pixel is computed based on the overlap weight and the base weight at 126, for example, as shown in equation 12.
If, however, image pixel data for the base image exists at 106 and image pixel data for the overlap image does not exist at 114, the base weight is set to 100% and the overlap weight is set to 0% at 116. The output pixel is computed based on the overlap weight and the base weight at 126, for example, as shown in equation 12.
If, however, data for the base image pixel exists and data for the overlap pixel image exists at 106 and 114, the maximum width and height of the total overlap area are determined at 118 as discussed above and the horizontal and vertical scales are computed at 120, for example, as shown in equations 1 and 2. The horizontal and vertical weights are computed at 122, for example, as shown in equations 3-6. The base weight and the overlap weight are computed at 124, for example, as shown in equations 7 and 8. The output pixel is computed at 126, for example, as shown in equation 12.
The process continues for each pixel 35 (
In various embodiments, the exemplary method and/or systems can process the image data according to a pixel by pixel approach. In this case, none of the image maps would exist. The base image data 48 and the overlap images data 50 are both written to image buffers (not shown). Each pixel 35 of the overlap image data 50 is compared with the data of the base image buffer to determine the overlap. The weights are calculated on the spot, and the base image pixel and the overlap image pixel are combined into the resulting output pixel.
Those skilled in the art can now appreciate from the foregoing description that the broad teachings of the present disclosure can be implemented in a variety of forms. Therefore, while this disclosure has been described in connection with particular examples thereof, the true scope of the disclosure should not be so limited since other modifications will become apparent to the skilled practitioner upon a study of the drawings, specification, and the following claims.