Patent Application
20070211154
The present invention relates generally to digital data processing of pictorial information derived from digital still cameras, digital video camcorders and other camera-like image-sensing instruments, and in particular relates to image enhancement by means of transformations for scaling the pixel intensity values of a captured image as a function of pixel position.
Vignetting is an optical phenomenon that occurs in imaging systems wherein the amount of light reaching off-center positions of a sensor (or film) is less than the amount of light reaching the center. This imaging defect causes the image intensity to decrease toward the edges of an image. The amount of vignetting depends upon the geometry of the lens and varies with focal length and f-stop. The effect is more apparent in lenses of lower f-stop (larger aperture), which are used especially in semi-pro/professional still cameras and video camcorders.
A lens vignetting correction algorithm may be applied to a digital image (or digitally-scanned photographic image), or to a series of such images, in order to compensate for the positionally unequal intensity effect of vignetting upon the image. This type of image enhancement or correction generally involves some kind of bitmap-to-bitmap transformation in which image intensity or “gray scale” data for the various picture elements (pixels) are appropriately scaled according to pixel position. Such an algorithm can be implemented in an image processor integrated circuit for use in digital still cameras, digital video camcorders, or any other camera-like image-sensing instrument affected by vignetting.
The particular algorithm and its implementation would preferably achieve optimal anti-vignetting without the need for much extra processing hardware (multipliers, look-up tables, etc.) and with adequate speed and efficiency (especially for real-time image processing). A reasonable set of parameters for the correction formula, and ready adaptability to a variety of lenses, is desirable.
U.S. Pat. No. 6,747,757 to Enomoto describes correction of several types of image errors, including decreasing brightness at the edge of the image field. It is geared especially to film scanners. The correction equation is log E(r)=a1r+a2r2+a3r3. It lacks a recursive approach suitable for real-time digital cameras. Accordingly, it relies upon look-up tables to implement the compensation upon the image data.
U.S. Pat. Nos. 6,577,378 and 6,670,988 to Gallagher et al. describe compensation of respective film and digital images for non-uniform illumination or light falloff at the focal plane, due to factors such as vignetting. A single light falloff correction parameter f2 is determined, based upon an assumed cos4 θ light falloff away from the optical axis. For a given camera type, lens, f-stop and focal length, and any flash device, the parameter f2 can be found by a least-squares fit using a representative sample of pixel elements pij of a captured image of a scene. Correction parameters specific to each type of camera lens may be stored in a meta-database. With the parameter f2 known, compensation values are applied to the individual pixels according to the light falloff compensation function for that parameter. Although only one parameter f2 needs to be determined, the correction calculation itself contains trigonometric functions (cos, tan−1), exponentiation (power of 4), logarithms and division, and is therefore quite complex and in need of extensive processor hardware.
U.S. Patent Application Publication No. 2004/0155970 of Johannesson et al. describes a vignetting compensation method for digital images. The method obtains a 5×5 matrix of coefficients kij (0≦i,j≦4) describing a polynomial surface function g(x,y) representing the relative brightness or vignetting gain at pixel positions (x,y). In particular,
g(x,y)=ΣiΣjkijxiyj.
The matrix coefficients kij are estimated by a least-squares fit to the measured intensity data from an image taken of a gray test chart. These coefficients are then stored in a nonvolatile memory. After being normalized, the function g is inverted to create a (look-up) gain table whose pixel-by-pixel multiplication with a raw image will produce a vignette-corrected image. Thus, the method involves both a complexity of computation and the need for look-up tables in order to implement the correction with any reasonable degree of efficiency.
The present invention is a lens vignetting correction algorithm implemented in image processing hardware and associated software. The correction applied to the image data is a 4th-order polynomial function in polar form with only the three even-order coefficients being non-zero. This function defines a radially symmetric curve that expresses the relative intensity correction to be applied to each pixel as a function of the radial distance of that pixel from a center. The center of the correction curve may vary with the optical alignment of each camera, or may be assumed to be very close to the center of the image. The variable coefficients, possibly camera-specific, may be found by fitting of the image data directly to the polynomial correction function, for example, by a least-squares fitting technique. The algorithm applies the polynomial correction function to the several pixels of the image using an iterative calculation that eliminates the need for extra hardware, such as multipliers or look-up tables.
With reference to
Next, a least-squares fit of the correction function (step 13) is performed. Least-squares (or minimum mean square error) is just one possible technique for fitting the parameters to the imaging characteristics of the optical system. Other fitting techniques could also be used to obtain a set of “best” parameters depending on the optimality criterion used.
For the present invention, the lens vignetting correction function is chosen to be a radially symmetric 4th-order polynomial,
F(x,y)=ar2+br4+c,
where (x,y) is the variable pixel position, r is the radial distance from a center (x0, y0) of correction,
r2=(x−x0)2+(y−y0)2
and a, b, and c are variable coefficients corresponding to the particular imaging system.
The center (x0, y0) of correction is usually assumed to coincide with the center (0,0) of the image, so that r2=x2+y2. In most cases, the alignment between image and lens is such that this assumption will be sufficiently accurate. However, if alignment between image and lens is not guaranteed, we can generalize to any center (x0, y0) of correction.
Applying this correction function F(x,y) to a raw image, we get a corrected image:
P(x,y)=P0(x,y)×F(x,y),
where P0(x,y) is the raw captured image and P(x,y) is the corrected image.
If a flat gray calibration object is imaged by the system, then the resulting raw calibration image can be used to find the coefficients, a, b and c, for that particular imaging system. That is, we can assume that applying a correction function F(x,y) with the appropriate coefficients, a, b and c, to the raw calibration image, should obtain a corrected image that is again a perfectly flat gray image, P(xi,yi)=constant.
Accordingly, we choose a fitting technique to obtain a set of coefficients, a, b, and c. Least-squares or minimum mean square error is one such fitting technique that could be used. Others may choose different criteria to obtain a set of “best” coefficients in relation to image data. Applying a least-squares fitting technique, we obtain a metric E2:
E2=(1/n)Σi[P0(xi,yi)−F(xi,yi)]2,
and find coefficients, a, b and c, of F, such that
{∂(E2)/∂a=0, ∂(E2)/∂b=0, ∂(E2)/∂c=0}.
The threefold criteria may be resolved into a matrix equation P=RA, where:
where the elements of the 1×N matrix P are the corresponding values P0 of the calibration pixels (xi,yi) over the entire calibration image (or a representative sample thereof), where the elements of the 1×3 matrix A are the correction coefficients a, b and c which are sought, and where the elements of the 3×N matrix R are the radial distances r of the calibration pixels (xi,yi) from the center of correction, raised to respective 2nd, 4th and 0th powers. The solution of this matrix equation is known to be:
A=R+P,
where R+=(RTR)−1RT is the Moore-Penrose matrix pseudoinverse of R, with RT being the matrix transpose of R. R+ can be found in advance for a given set of (xi,yi) and a given center of correction. Hence, least-squares fitting of the calibration image to find the correction coefficients (a,b,c) resolves into a relatively simple matrix multiplication.
If the alignment between imager and lens is not guaranteed, such that one cannot assume that the center of vignetting coincides with the center of the image, we can search for coefficients (a,b,c) for different “center” coordinates (x0h,y0k), where h and k are variable. The center coordinates that lead to a minimum mean square error (smallest E2 value) is then considered to be the center of vignetting, and the coefficients (a,b,c) are those found for that particular center.
After fitting the correction function F, we can use that function to correct subsequent raw images obtained by that imaging system. Accordingly, whenever a raw image is obtained (step 15 in
In order to eliminate the need for extra processing hardware, such as multipliers, and to operate efficiently in real-time, the algorithm that implements the image correction employs a recursive technique for updating r2. This technique may proceed as follows:
The vignetting correction of color images may use separate correction coefficients for each color. Consider the case of a Bayer pattern where each pixel is defined by one of three primary colors (e.g., red, green, and blue), and that uses a two-field interlaced format. Odd-numbered rows or lines 1, 3, 5, . . . may form a field 0 made up of alternating green and red pixels, while even-numbered rows or lines 2, 4, 6, . . . may form a field 1 made up of alternating blue and green pixels. This format may be indicated to the processor, for example, by means of one or more control register bits. Pixels in this particular format are effectively grouped into 2×2-color cells, made up of the primary colors is some defined pattern. For example:
