Embodiments of the invention relate to a method, apparatus, and system for green-green imbalance compensation in imagers.
Imagers, such as, for example, charge coupled devices (CCD), complementary metal oxide semiconductor (CMOS) and others, are widely used in imaging applications including digital still and video cameras. A CMOS imager circuit includes a focal plane array of pixels, each one of the pixels including a photosensor, for example, a photogate, photoconductor or a photodiode for accumulating photo-generated charge in the specified portion of the substrate. Each pixel has a charge storage region, formed on or in the substrate, which is connected to the gate of an output transistor that is part of a readout circuit. The charge storage region may be constructed as a floating diffusion region. In some imager circuits, each pixel may include at least one electronic device such as a transistor for transferring charge from the photosensor to the storage region and one device, also typically a transistor, for resetting the storage region to a predetermined charge level.
The pixel output signals typically include a pixel reset signal, Vrst, taken off the floating diffusion region 30 when it is reset by reset transistor 40 and a pixel image signal, Vsig, which is taken off the floating diffusion region 30 after charges generated by photosensor 20 are transferred to it (thru transfer transistor 50). The Vrst and Vsig signals are sampled by the sample and hold circuit 265 and then subtracted by a differential amplifier 267 that produces a signal Vrst−Vsig for each pixel, which represents the amount of light impinging on the pixels 10. This difference signal is digitized by an analog-to-digital converter (ADC) 275. The digitized pixel signals are fed to an image processor 280, which performs various processing on the digital signals to form a digital image. The digitizing and image processing can be performed on or off the chip containing the pixel array 200.
In order for the imager circuit 208 to successfully capture color differential in a color image, the pixels 10 must be able to detect separate color components of a captured image. Separate color detection is often accomplished by placing a color filter array (CFA) above the pixel array 200 so that each pixel receives only the light of the color of its associated filter according to a specific pattern.
A Bayer color filter array 80, illustrated in
The main objective of a color filter array is to limit the spectral range of light detection of each pixel 10 to a single color in the form of a wavelength range designated by the pixel's associated filter. However, light sometimes passes through an individual color filter at such an angle that it strikes a neighboring pixel and affects the neighboring pixel's detection response. This undesirable occurrence, often referred to as “cross-talk,” is illustrated in
When cross-talk such as this occurs throughout a Bayer-filter patterned pixel array, green pixels neighboring red pixels are impinged by an unwanted amount of red-filtered light, while green pixels neighboring blue pixels are impinged by unwanted blue-filtered light. Due to the difference between the effects of the red and blue cross-talk, two distinctly different detection levels or “channels” of green exist where there should be uniformity. This difference, called “green-green imbalance,” can manifest as unwanted artifacts in a digitally captured photo, such as the appearance of a faint checkerboard pattern overlaying the image. Green-green imbalance can be particularly damaging when employing edge-sensitive demosaicing algorithms wherein the imbalance causes false edge defection, resulting in the appearance of distinct “labyrinth lines” of random short orthogonal edges. As pixel sizes grow smaller, the effects of cross-talk and green-green imbalance worsen. There is a need to reduce the effects of green-green imbalance.
In the following detailed description, reference is made to the accompanying drawings which form a part hereof, and in which are shown by way of illustration specific embodiments of the invention. These embodiments are described in sufficient detail to enable those skilled in the art to make and use them, and it is to be understood that structural, logical or procedural changes may be made. Particularly, in the description below, processing is described by way of flowchart. In some instances, steps which follow other steps may be in reverse or in a different sequence except where a following procedural step requires the presence of a prior procedural step. The processes illustrated in the flowcharts can be implemented as a pixel processing circuit provided in as part of an image processor 280 (
In the following description, the embodiments are described in relation to a CMOS image sensor for convenience purposes only. However, the described embodiments may have a wider applicability to any imager, including charge-coupled device (CCD) imagers and others.
According to embodiments described herein, green-green imbalance may be addressed by digitally correcting an image during image processing based on local green pixel image information in a kernel of pixels. A value representing the magnitude of the green-green imbalance is calculated to adjust the output value of at least one green pixel within the kernel and preferably all given pixels within the kernel, thereby compensating for the imbalance.
A kernel 120 or 130 limits a sampling of pixel outputs to a selection of local values to estimate green-green imbalance. The kernels 120, 130 may be used separately in embodiments of the invention, with kernel 120 containing X, e.g. 9, green 1 pixels G1 and Y, e.g. 4, green 2 pixels G2, and with kernel 130 containing an equal number, e.g. 6, of green 1 pixels G1 and green 2 pixels G2.
At step S4 the imbalance value GImb is compared to a predetermined threshold TRef to determine whether the imbalance value GImb should be applied to at least one pixel within the kernel 120 or 130. If GImb>TRef, then the imbalance value GImb is not applied. The comparison with TRef ensures that pixels G1, G2 within a kernel 120 or 130 which may include a significant amount of image edge pixels are not adjusted. When calculating GImb in a kernel 120 or 130 with high image edge content, it is difficult to distinguish contributions to the G1, G2 values inherent from the image edge content from actual green-green imbalance. Accordingly, for pixels G1, G2 within a kernel 120 or 130 which exhibits an unusually high imbalance value GImb, signified by exceeding the threshold TRef, the processing ends at step S5. The threshold TRef, which determines the cutoff level for excluding image edge pixels, can be set by experimenting to obtain the best perceptual image quality. TRef can be changed depending on the conditions of the camera operation on-the-fly. For example, in the low-light conditions it can be increased. It is highly desirable to correct all of the pixels in the kernel since down-stream processing algorithms such as demosaic, for example, use valves from several green pixels on each step.
For a kernels 120 or 130 in which step S4 evaluates as GImb<TRef, the process 140 continues with imbalance correction adjustment at step S6. G1Avg is compared to G2Avg to determine which green pixels G1 or G2 have the higher average output. If G1Avg>G2Avg, then corrected values G1c,G2c of at least one and preferably all green 1 and 2 pixels G1,G2 are calculated at step S7 as follows: G1c=G1−GImb, G2c=G2+GImb. Otherwise, if G1Avg<G2Avg, then corrected G1c,G2c for at least one and preferably all green 1 and 2 pixels G1,G2 within the kernel 120 or 130 are calculated at step S8 as follows: G1c=G1+GImb, G2c=G2−GImb. Accordingly, each pixel within a kernel 120 or 130 is adjusted by GImb to compensate for green-green imbalance. The process may then be repeated, successively moving the kernel 120 or 130 throughout a captured image until all green pixels in the array have been adjusted. Since the kernel size includes more than one pixel, each green pixel will be corrected several times.
The above described process 140 is appropriate for compensation of green-green imbalance in images where the noise magnitude is less than that of the green-green imbalance. However, if it is determined that the noise magnitude is greater than the green-green imbalance, the above process 140 may be compromised due to random contributions of the noise to ΣG1 and ΣG2 in a kernel 120 or 130. This problem may be addressed by increasing the size of the kernel 120 or 130. Calculating more accurate G1Avg, G2Avg values over a greater number of green pixels G1,G2 decreases the effect of the noise contribution on the GImb calculation and increases the accuracy and reliability of the value GImb. In the case where all the pixels in the kernel are corrected, the original data should remain unmodified. Thus, each corrected kernel is only used once and results of correction are discarded. If the results of correction are not discarded as described, GImb may be calculated incorrectly due to feedback into the original data from the previous correction steps.
In certain applications of the process 140, the processing hardware to expand the size of the kernel 120 or 130 may be limited to a maximum dimension A×B. If this maximum limit is reached and a more accurate GImb calculation is desired, the
The accuracy of the imbalance value GImb will increase as the processing moves through the array according to the
Embodiments may be implemented as part of a camera such as e.g., a digital still or video camera, or other image acquisition system.
The camera system 800 is one example of a system having digital circuits that could include an imager circuit 208. Without being limiting, such a system could also include a computer system, image, scanner, machine vision, vehicle navigation, video phone, surveillance system, auto focus system, star tracker system, motion detection system, image stabilization system, and other systems which acquire and process image data.
While embodiments have been described in detail, it should be readily understood that the invention is not limited to the disclosed embodiments. Rather the embodiments can be modified to incorporate any number of variations, alterations, substitutions or equivalent arrangements not heretofore described.
Number | Name | Date | Kind |
---|---|---|---|
6933971 | Bezryadin | Aug 2005 | B2 |
7119760 | Edge et al. | Oct 2006 | B2 |
7136103 | Oeda et al. | Nov 2006 | B2 |
20020122589 | Reiman et al. | Sep 2002 | A1 |
20020131495 | Prakash et al. | Sep 2002 | A1 |
20020154813 | Stevenson et al. | Oct 2002 | A1 |
20030184660 | Skow | Oct 2003 | A1 |
20040052414 | Schroder | Mar 2004 | A1 |
20050122408 | Park et al. | Jun 2005 | A1 |
20050179825 | Hicks | Aug 2005 | A1 |
20050213128 | Imai et al. | Sep 2005 | A1 |
20060013479 | Trimeche et al. | Jan 2006 | A1 |
20060098868 | Fainstain et al. | May 2006 | A1 |
20060164521 | Chikane et al. | Jul 2006 | A1 |
20060239549 | Kelly et al. | Oct 2006 | A1 |
20060245016 | Fukao et al. | Nov 2006 | A1 |
20060250623 | Newman et al. | Nov 2006 | A1 |
20060257022 | Hayaishi | Nov 2006 | A1 |
20060262198 | Une | Nov 2006 | A1 |
20060274160 | Cheng | Dec 2006 | A1 |
20060284991 | Ikeda | Dec 2006 | A1 |
20070002150 | Abe | Jan 2007 | A1 |
20070003136 | Shimbaru | Jan 2007 | A1 |
20070109316 | Fainstain | May 2007 | A1 |
Number | Date | Country |
---|---|---|
0961486 | Dec 1998 | EP |
WO 9205470 | Apr 1992 | WO |
WO 9939509 | Aug 1999 | WO |
WO 02075654 | Sep 2002 | WO |
WO 03085963 | Oct 2003 | WO |
WO 2007032824 | Mar 2007 | WO |
WO 2007054931 | May 2007 | WO |
Number | Date | Country | |
---|---|---|---|
20080252759 A1 | Oct 2008 | US |