Patent Application
20070046787
The invention relates generally to digital image processing and more particularly to a method and apparatus for determining white balance correction in digital image processing.
Currently, there is a great demand for digital cameras. Commensurate with this demand is the demand for greater accuracy, i.e., “realism,” in images captured by digital cameras. Part of this realism is steeped in the desire to have the digital images reflective of how a person perceives the same image. However, reflecting this realism can be difficult.
After a digital camera captures a digital image, the circuitry within the camera performs image processing to make the image more realistic.
Color balance is the process of removing unrealistic color casts, so that the objects in a captured image are reflective of the perceived color. Color balance is achieved by balancing a captured image so that the image color is the same or close to the same as the image color perceived by the human visual system. For instance white balance is the process of removing unrealistic color casts due to the light source, so that white objects appear white in a captured image. Typically, when processing color in a digital image, the image may go through a series of processing steps, one of which is the white balance step.
An imaging processing system may apply a series of tests to determine the correct white balance for an image. One known approach to determining an automatic white balance uses the gray world model. The gray world model is premised on the entire image balancing out to gray, i.e., the average color of the image balancing out to gray, where gray is equivalent amounts of red, green, and blue signal values.
Based on this assumption, the signals received from pixels can be modified to be consistent with the gray world average. If the signals from the pixels are averaged together and the averaged color result is somewhat blueish, then compensation must be applied to make the reduce the blueish tint. This is commonly done when converting images that are taken under incandescent light (which are somewhat reddish) and compensating them by converting the reddish cast to gray.
However, the gray world model can be thrown off inherently by the subject matter of the image being taken. For example, if the image is a picture taken of a blue sky, the image really should average out to a predominance of blue, not gray. Additionally, if the picture is taken of grass, the image really should average out to a predominance of green, not gray. So there must be some other type of compensation when processing the digital signals. Furthermore, light sources are typically reddish or blueish, i.e., sunlight is blueish and incandescent light is often reddish.
Although mechanisms currently exist to compensate for color cast in a particular direction, the compensation often leads to detrimental side effects. For example, when processing a digital image containing green grass and a gray sidewalk, the compensation for the predominance of green grass would turn the sidewalk magenta. In general, it is known that green should generally be excluded because of the typical inherent colors of light sources (e.g., red and blue). This would address a great number of problems that occur for pictures taken outside. Magenta also should be partially excluded as complementary. So it would be desirable to have a filter that can be adjusted depending on the image being captured to exclude green and magenta colors when appropriate.
Therefore, it is desirable to have a white balance test that excludes certain pixels from the white balance correction computation for certain image conditions.
The present invention relates to an improved chrominance filter that selectively excludes pixels from the white balance correction. A YUV (i.e., luminance-chrominance) representation of a pixel is compared to an acceptable region that is defined by maximum and minimum values of U+V (i.e., blue chrominance+red chrominance) and U−V (i.e., blue chrominance−red chrominance) and if the pixel is within the acceptable region, the pixel is included in the white balance correction calculation. Because the filter requires a relatively simple calculation, only a few additional circuits are required to implement the filter and the size of the image processing system is not significantly increased.
The foregoing and other advantages and features of the invention will become more apparent from the detailed description of exemplary embodiments provided below with reference to the accompanying drawings in which:
In the following detailed description, reference is made to the accompanying drawings, which are a part of the specification, and in which is shown by way of illustration various embodiments whereby the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to make and use the invention. It is to be understood that other embodiments may be utilized, and that structural, logical, and electrical changes, as well as changes in the materials used, may be made without departing from the spirit and scope of the present invention.
The image sensor 190 is a sensor as is conventionally known and may be any type of solid state images including CMOS, CCD and others. One exemplary image sensor 190 is a CMOS image sensor with active pixel sensors (“APS”). As is conventionally known, the image sensor 190 receives image information in the form of photons, and converts that information to pixel analog electrical signals, which are subsequently provided to down stream processing circuits. In the imaging processing device 100, the image sensor 190 provides electrical signals to the image processing circuit 110.
The pre-processing circuit 120 performs an initial processing to the raw image data received from the image sensor 190. In a preferred embodiment, the pre-processing circuit 120 includes an analog-to-digital conversion circuit. The pre-processing circuit 120 may also include one (or a plurality of cascaded) gain stage circuit(s) and one (or a plurality of cascaded) amplifier circuit(s). The pre-processing circuit 120 outputs a digital signal corresponding to the analog signal received from the image sensor 190.
The first image correction circuit 130 performs a first level of processing to the digital signal received from the pre-processing circuit 120. The first image correction circuit 130 may include circuits that perform image defect correction, mosaic effects correction, image resizing, aperture correction, and correction for other effects or defects. In a preferred embodiment, the pre-processing circuit 120 outputs are RGB representational of the image data from red, blue and green pixels of sensor 190.
The color matrix circuit 135 receives the RGB signals from the first image correction circuit 130 and applies matrix modifications to the signals. The matrix modifications can include correction for color balance. The color matrix circuit 135 receives from the third image correction circuit 170 matrix modification coefficients. The color matrix circuit 135 provides modified RGB signals to the RGB-to-YUV conversion circuit 140.
The RGB-to-YUV conversion circuit 140 converts the RGB data structures into a second set of data structures in YUV format that are also representational of the RGB data. YUV, as is conventionally known, stores image data in Y-luminance (“luma”), U-blue chrominance (“blue chroma” or “Cb”) and V-red chrominance (“red chroma” or “Cr”). The RGB-to-YUV conversion circuit 140 provides the two sets of data structures, RGB and YUV data structures, to the AWB circuit 160. The RGB-to-YUV conversion circuit 140 receives white balance correction signals from the AWB circuit 160. The RGB-to-YUV conversion circuit 140 provides the two data structures, RGB and YUV, and the white balance correction signals received from the AWB circuit 160 to the second image correction circuit 150.
The second image correction circuit 150 performs a second level of processing to the digital signals received from the RGB-to-YUV conversion circuit 140. The second image correction circuit 150 may include circuits that perform white balance correction, gamma correction, pixel formatting and correction for other effects or defects. In a preferred embodiment, the second image correction circuit 150 outputs an RGB data structure representational of the image data of red, blue and green pixels. Alternatively, the second image correction circuit 150 outputs a data stream of digital information for further processing in down stream circuitry.
The AWB circuit 160 receives image data, in the form of the data structures RGB and YUV and computes the correction values, if required, to perform white balancing. The AWB circuit 160 computation of the appropriate white balance correction is explained in greater detail below. The AWB circuit 160 provides the results of its computation to the RGB-to-YUV conversion circuit 140 and to the third image correction circuit 170. The AWB circuit 160 uses either the RGB or the YUV data structures separately, or in combination, depending on the computation the AWB circuit 160 is conducting. The improved chrominance filter of the present invention resides in the AWB circuit 160.
It is preferable to perform the white balance computation in the AWB circuit 160 using the YUV data structure because of the inherent properties of the YUV data structure. As is known, the YUV data structure breaks down an image into the Y, the luminance values, and YUV, which is essentially a two-dimensional representation of color, where the two color components are U and V (i.e., Cb and Cr).
The third image correction circuit 170 receives information from the AWB circuit 160 and determines new settings for color matrix coefficients, which are provided to the color matrix circuit 135. The third image correction circuit 170 also calculates gain ratios, light compensation coefficients, and saturation/desaturation coefficients and provides those values to the sensor 190. In a preferred embodiment, the third image correction circuit 170 provides a single coefficient matrix which can reflect multiple corrections, where the matrix is generally a 3×3 (e.g., a three by three) matrix. The sensor 190 uses those values for control of capturing subsequent images. For example, the sensor 190 may use the light compensation coefficient to modify the image exposure time, i.e., the photo capture time period.
In an embodiment of the invention, circuits 120, 130, 135, 140, 150, 160, and 170 are implemented in hardware, preferably according to a system on a chip (“SOC”) configuration. Circuits 120, 130, 135, 140, and 150 are part of the “color pipe” of the imaging processing device 100. Circuits 160 and 170 are part of camera control of the imaging processing device 100.
As noted, above, one test used to determine the auto white balance correction of an image includes calculating pixels statistics in the image. For example, the number of red, blue, and green pixels are determined. The collected statistics are compared to expected values and the results of the comparison are used to correct the white balance. To avoid biasing the image (e.g., too much green), the improved chrominance filter excludes certain pixels from the automatic white balance computation.
Also seen in
An advantage of the improved chrominance filter of the present invention is that the amount of Green and/or Magenta that is included in the white balance computation, i.e., the first shaded region 290, can be easily adjusted by changing the MgG min and max values. Furthermore, in the present invention the inclusion/exclusion of Green and/or Magenta by modifying MgG min and max reduces the collateral effect on the inclusion/exclusion of U and/or V. For example, to reduce the number of Green pixels included in the automatic white balance in a conventional approach (i.e., shaded region 295) requires reducing U and/or V, which directly linearly reduces the U and/or V. Additionally, significant circuitry is required to implement a change in the max/min boundaries of the green/magenta in the shaded region 295 as it would require changing the shape of the second shaded region 295 from being rectangularly shaped to, at least, pentagonally or hexagonally shaped.
The chrominance filter circuit 300 includes an addition circuit 310, a subtraction circuit 312, comparison circuits 320, 322, 326, 328, and AND circuits 330, 332, 340. Programmable registers 366, 368, 362, 364 store MgG max, MgG min, BR max, and BR min values, respectively. In a preferred embodiment, MgG max, MgG min, BR max, and BR min are values that are determined empirically and stored in the chrominance filter circuit 300. In another aspect, sets of values for MgG max, MgG min, BR max, and BR min are determined empirically for different types of lighting environments. For example, there is a first set of values for MgG max, MgG min, BR max, and BR min for pictures taken outdoors in full sunlight and a second set of values for pictures taken inside under incandescent light. In another aspect, values for MgG max, MgG min, BR max, and BR min are established by an end user, i.e., the camera operator.
The inputs to the chrominance filter circuit 300 are assigned chrominance values, i.e., U, V, of a target pixel. The chrominance values U, V are calculated in an upstream circuit (not shown) and stored in registers 302, 304, respectively, preferably in the chrominance filter circuit 300. The blocks 310, 312 perform signed addition and subtraction, respectively, to the chrominance values U, V. For example, block 310 adds U to V and provides the results of the computation to comparison blocks 320, 322. Comparison blocks 320, 322 compare, respectively, the input they receive from block 310 to the MgG min, MgGmax values. The results of comparison blocks 320, 322 are compared (i.e., ANDed together) in block 330.
Block 312 subtracts V from U and provides the results of the subtraction to comparison blocks 326, 328. Comparison blocks 326, 328 compare, respectively, the input they receive from block 312 to the BRmax, BRmin values. The results of comparison blocks 326, 328 are compared (i.e., ANDed together) in block 332. The results of blocks 330, 332 are compared (i.e., ANDed together) in block 340 and the output is provided to a downstream circuit. In a preferred embodiment, if the target pixel is within the first shaded region 290 (
As the slope of U-V axis line is forty-five (45) degrees, this slope simplifies the computations of the boundaries of first shaded region 290 as, the trigonometric functions cancel out. Therefore, the boundaries can be implemented using simple ADDER circuit blocks (i.e., blocks 310, 312). If the slope of U-V axis line was not forty-five (45) degrees, additional circuitry would be required to compute which would increase the complexity and size of the circuit.
An advantage of the present invention is that certain colored pixels, e.g., green, can be excluded from the white balance computation without having a linear loss of collateral colored pixels. For example, in a conventional white balance filter (as seen as the second shaded region 295 in
Another advantage of the present invention is that the shaded region is not required to be centered at the intersection of the U and V axes because of the programmable values for registers 362, 364, 366, 368. Thus, the boundary of the shaded region 290 in the direction of Magenta (MgG max) does not have to be equivalent to the boundary of the shaded region 290 in the direction of Green (MgG min) (i.e., absolute value (MgG max) does not have to equal absolute value (MgG min)). Additionally, the boundary of the shaded region 290 in the direction of Blue (BR max) does not have to be equivalent to the boundary of the shaded region 290 in the direction of Red(BR min) (i.e., absolute value (BR max) does not have to equal absolute value (BR min)).
System 1100, for example a camera system, generally comprises a central processing unit (CPU) 1110, such as a microprocessor, that communicates with an input/output (I/O) device 1150 over a bus 1170. Imaging processing device 100 also communicates with the CPU 1110 over the bus 1170. The system 1100 also includes random access memory (RAM) 1160, and can include removable memory 1130, such as flash memory, which also communicate with the CPU 1110 over the bus 1170. The imaging processing device 100 may be combined with a processor, such as a CPU, digital signal processor, or microprocessor, with or without memory storage on a single integrated circuit or on a different chip than the processor.
It should be appreciated that other embodiments of the invention include a method of manufacturing the system 1100. For example, in one exemplary embodiment, a method of manufacturing an CMOS readout circuit includes the steps of providing, over a portion of a substrate corresponding to a single integrated circuit, at least an image sensor with an automatic white balance circuit (
While the invention has been described and illustrated with reference to specific exemplary embodiments, it should be understood that many modifications and substitutions can be made without departing from the spirit and scope of the invention. Accordingly, the invention is not to be considered as limited by the foregoing description but is only limited by the scope of the claims.
