The invention relates generally to video image quality measures. More specifically, the invention relates to an improved system and method for measuring stereo and single image quality.
Over the past, several measures have been taken to detect impairments of one or more stereo cameras of a stereo vision system. The forms of impairments can include camera obstruction of a partial or complete field of view blockage either by solid objects such as leaves on a windshield, environmental factors such as precipitation (water, snow or fog) or low-light conditions, problems with the optical system itself such as poor camera focus, poor camera calibration, poor stereo camera relative alignment or other unanticipated problems with the video. Additionally, low target contrast or texture (while not a camera impairment per se) can also cause poor system measurements when viewing the video images. For example one of these impairments could cause a critical error in stereo measurements by altering the relative orientation of the left and right camera, without benefit of a compensatory recalibration, which in turn would cause incorrect resulting depth computations, etc.
Collision detection systems are known in the art to compute stereo images to detect potential threats in order to avoid collision or to mitigate its damage. The impairments could easily cause the collision algorithms to misidentify these incorrect measurements as potential collision threats, thus creating a false alarm, the effects of which could be drastic. Thus the presence of such impairments, once identified, should cause the system to temporarily disable itself for the duration of the impairment, sometimes called a “failsafe” condition. This would be applicable also in less severe applications, which provide for much wider range of safety and convenience functions, for example, adaptive cruise control.
Stereo depth estimate accuracy can be computed precisely for a given stereo algorithm on a stereo image data with known position and/or ground-truth information. However, this ground-truth information may be unavailable or difficult to collect for real-world scenes, even in controlled settings, and are certainly not available in the uncontrolled settings of a deployed stereo or monocular imaging system. Moreover, such characterizations only measure the accuracy of a stereo algorithm under ideal conditions, and ignore the effects of the kinds of unanticipated impairments noted above. That is, a characterization of a stereo algorithm's accuracy under ideal conditions does not predict and is not able to measure its robustness to various impairments found in uncontrolled real-world conditions.
Some algorithms may attempt to characterize specific impairments such as rain or fog in an operating imaging system using specific characteristics of the impairment itself (such as expected particle size and density), but may not generalize to other impairments such as hail, sleet or a sandstorm and therefore would not be able to reliably invoke a needed failsafe condition. Thus the deployment of practical imaging systems, particularly stereo imaging systems, has a need for a general means to measure both monocular and stereo image quality.
The present invention provides a method for measuring image quality comprising receiving at least one input image in a region of interest and using an average adjacent pixel difference module for computing an adjacent pixel difference for each valid pixel in the region of interest, summing the at least some of the adjacent pixels' absolute differences and computing the average adjacent pixel difference within the region of interest.
In one embodiment of the present invention, the at least one input image is an original image received from an image sensor and every region in the pixel of interest is considered valid.
In another embodiment of the present invention, the at least one input image is a binarized disparity image formed from a stereo disparity image. The stereo disparity image is computed from a stereo algorithm which explicitly labels each output pixel in the region as valid or invalid.
The present invention describes various embodiments of image quality measures. Two such measures include frequency Content measure for monocular image quality and disparity measure for stereo image quality.
The images 104a and 104b are individually processed by an average adjacent pixel difference (AAPD) modules 106a and 106b respectively to provide the frequency content measure of the images as will be described in greater detail below. Based on the frequency content measure, each of the left 104a and the right images 104b from each of the cameras 102a and 102b are then processed jointly by a stereo disparity module 108 to produce a stereo disparity image as will be described in greater detail below. Alternatively, as illustrated in
The stereo disparity image 109 is then converted into a binarized stereo disparity image 112 using a binarized disparity module 110. The binarized stereo disparity image 112 is further processed by an AAPD module 106c to compute the disparity measure of the disparity image. Based on this disparity measure, the disparity image can either be forwarded for additional processing or ignored. If the disparity values are good, i.e. fall within a specific threshold, then the disparity image 109 is further processed for various purposes such as object detection, collision detection etc. However, if the disparity values are poor, i.e. fall outside the threshold value, then the disparity image 109 is ignored and a new left and right images are obtained from each of the cameras 102a and 102b to repeat the process as will be described in greater detail below.
Referring to
For example,
The frequency content measure for the AAPD measure described above applies to images obtained from a single or a monocular camera, although, the monocular measure can be computed and compared with images from two or more cameras. The frequency content measure would be quite similar for two or more cameras looking at the same scene or view unless there is an obstruction in one or more cameras as discussed above, which would increase the difference in the frequency content of the two images. If this difference were measured to be high, an obstruction or impairment of one of the cameras can be hypothesized.
In another embodiment of the present invention, the image quality measure for binocular images is stereo disparity quality measure. The stereo disparity quality measure includes producing a disparity image. A stereo disparity image is an image computed from images captured from at least two or more stereo cameras which together produce a single disparity image at a given resolution. Thus, unlike the frequency content measure, the stereo disparity quality measure is inherently applicable to at least two camera images, most commonly the left and the right camera images of a stereo image pair. Returning back to the flow chart in
As illustrated in table 1 above, the image quality measure for monocular image is return integer values between some lower bound and 100(%). The monocular image quality measure ranges between 0-100. So, for the image to be eligible for stereo disparity, the image frequency content values of each of the monocular images 104a and 104b must fall preferably within the range of 60% to 100% as provided in Table 1 above. The images 104a and 104b having the frequency content values falling within the range of 0% to 59% are preferably considered to be ineligible for stereo disparity.
Referring back to the flow chart in
Alternatively, as shown in
Then at step 212, an AAPD is computed by the disparity AAPD module 106c for the binarized disparity image 112. Then, using the same AAPD or average-adjacent-pixel-difference function described previously (having the notable advantage of requiring only a single pass through the disparity image pixels), the number of binary edge discontinuities between valid and invalid regions (both left-right and up-down) are summed, then this sum is subtracted from the total number of valid disparity image pixels, then this result is divided by the total number of valid disparity image pixels which yields the disparity quality values. As discussed above, the stereo disparity algorithm defines the valid and invalid disparity image pixels. A small number of large cohesive disparity regions will increase the disparity quality value, and a larger number of small, fragmented regions will decrease the disparity quality value. Thus, the obstructed/degraded image frames have lower disparity quality values due to their many small disparity regions and the unobstructed image frames will have higher disparity quality values due to their lesser number of large disparity regions. Note that the sharing of the basic AAPD computational functionality can have implementation advantages; for example both the monocular and stereo uses could share the same block of hardware such as a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASCI).
Referring back to
As illustrated in table 2 above, the image quality measure for stereo disparity image also returns integer values between some lower bound and 100(%). The stereo disparity quality measure actually ranges between −299 and 100, but any values below 0 may be treated as 0 since the quality is already so poor. So, for the stereo disparity image 109 to be considered eligible for additional processing, the stereo disparity measure of the binarized stereo disparity image 112 must fall preferably within the range of 75% to 100% as provided in Table 2 above. The binarized stereo disparity image 112 having the stereo disparity values falling within the range of any number below 0% to 74% are preferably considered to be ineligible for additional processing.
For most real-world scenes without any camera impairments, the stereo disparity image fragmentation measurement is expected to be low, that is, the stereo disparity quality is expected to be high. In other words, it is expected of the world to consist of mostly cohesive regions of disparity representing solid surfaces and objects. Conversely, cameras with impairments of the kind described previously would experience many disjoint regions of both valid disparity and invalid disparity pixels, and thus high disparity image fragmentation yielding a low stereo disparity quality, as the stereo algorithm struggles with the impairments.
As discussed above, stereo disparity value is measured as the number of valid disparity pixels minus the number of valid/invalid pixel transitions in the disparity image, divided by the number of valid disparity pixels. Thus a large roughly circular blob or region of disparity would have a low disparity image fragmentation, with a large number of valid disparity pixels and relatively small number of disparity region boundary valid/invalid pixel transitions. Conversely, multiple smaller regions of amorphous disparity would have a higher disparity image fragmentation, with a high ratio of disparity region boundary valid/invalid pixel transitions relative to its number of valid disparity pixels.
Referring back to the stereo image frame 1 of the
The two examples as described and illustrated in
Even though, the examples described above are related to the obstructive view received by one or more cameras and poor focus settings, the degradation may also occur due to camera misalignment. In that case, the stereo disparity measure will provide a poor score even though the individual camera AAPD scores will be relatively good. This is because as discussed above, the AAPD computes scores based on the images from an individual camera while the stereo measure compares the images based on the images from at least two cameras. So, the AAPD is a good monocular image quality measure, while the stereo disparity measure is focused on fragmentation of the computed disparity image.
Although, the present invention described above includes division of the field-of-view into three ROIs for both image and disparity-based quality metrics, corresponding to the left, center, and right regions, additional regions could be included. More generally, any ROI of interest, for example one potentially containing a single object, may be subjected to one or more of these quality metrics. Alternatively, a more general and dense image quality mask may be (generated by processing small regions around each pixel.
The techniques described are valuable because they provide a general measure of stereo and single image quality, which is applicable against unforeseen impairments as well as the examples described. Although various embodiments that incorporate the teachings of the present invention have been shown and described in detail herein, those skilled in the art can readily devise many other varied embodiments that still incorporate these teachings without departing from the spirit and the scope of the invention.
This application claims the benefit of U.S. Provisional Patent Application No. 61/020,554 filed Jan. 11, 2008, the entire disclosure of which is incorporated herein by reference.
This invention was made with U.S. government support under contract number 70NANB4H3044. The U.S. government has certain rights in this invention.
Number | Date | Country | |
---|---|---|---|
61020554 | Jan 2008 | US |