Techniques for reducing color artifacts in digital images

Abstract

A technique for reducing artifacts in a digital image, in accordance with one embodiment, includes receiving a stream of raw filter pixel data representing the image. The raw filter pixel data is interpolating to produce red, green-on-red row, green-on-blue row and blue pixel data for each pixel. An artifact in one or more given pixels is reduced as a function of a difference between the green-on-red row and green-on-blue row pixel data of each of the given pixels to generate adjusted interpolated pixel data.

Description

BACKGROUND OF THE INVENTION

Computing devices have made significant contributions toward the advancement of modern society and are utilized in a number of applications to achieve advantageous results. Numerous devices, such as digital cameras, computers, game consoles, video equipment, hand-held computing devices, audio devices, and telephones, have facilitated increased productivity and reduced costs in communicating and analyzing data in most areas of entertainment, education, business and science. The digital camera, for example, has become popular for personal use and for use in business.

FIG. 1 shows an exemplary digital camera. The digital camera 100 typically includes one or more lenses 110, one or more filters 120, one or more image sensor arrays 130, an analog-to-digital converter (ADC) 140, a digital signal processor (DSP) 150 and one or more computing device readable media 150. The image sensor 130 includes a two-dimension array of hundreds, thousand, millions or more of imaging pixels, which each convert light (e.g. photons) into electrons. The image sensor 130 may be a charge coupled device (CCD), complementary metal oxide semiconductor (CMOS) device, or the like. The filter 120 may be a Bayer filter that generates a mosaic if monochrome pixels. The mosaic of monochrome pixels are typically arranged in a pattern of red, green and blue pixels.

In an exemplary implementation, the digital camera 100 may include a lens 110 to focus light to pass through the Bayer filter 120 and onto the image sensor 130. The photons passing through each monochrome pixel of the Bayer filter 120 are sensed by a corresponding pixel sensor in the image sensor 130. The analog-to-digital converter (ADC) 140 converts the intensity of photons sensed by the pixel sensor array into corresponding digital pixel data. The raw pixel data is processed by the DSP 150 using a demosaic algorithm to produce final interpolated pixel data. The final interpolated pixel data is typically stored in one or more of the computing device readable media 160. One or more of the computing device readable media 160 may also store the raw pixel data.

Referring now to FIG. 2, an exemplary Bayer filter is shown. The Bayer filter pattern alternates rows of red and green filters 210, 220 with rows of blue and green filters 230, 240. The Bayer filter interleaves the red, green and blue color filters so that (1) each pixel only senses one color, and (2) in any 2×2 pixel cluster on the sensory plane, there are always two pixels sensing green information, one pixel for red, and one for blue. A demosaic algorithm interpolates the other two color components for each pixel from the surrounding raw pixel data. For example, if a given pixel generates a red signal, the demosaic algorithm interpolates a green and a blue color signal from the surrounding raw pixel data.

The Bayer filter/image sensor is subject to color artifacts. A color artifact happens when a scene contains a high frequency pattern that is beyond the Bayer array's Nyquist rate. Accordingly, there is a continuing need for improved imaging processing techniques to reduce color artifacts.

SUMMARY OF THE INVENTION

Embodiments of the present invention are directed toward techniques for reducing artifacts in digital images. In one embodiment, a method of demosaicing digital image data includes receiving raw pixel data for a given image. The raw pixel data is low pass filtered using a first demosiac kernel size to determine first level interpolated pixel data for each pixel. The raw pixel data is also low pass filtered using a second demosiac kernel size to determine second level interpolated pixel data for each pixel. The presence or absence of an artifact in each pixel is determined from the first level interpolated pixel data. If an artifact is not present at a given pixel, the adjusted interpolated pixel data for the given pixel is equal to the first level interpolated pixel data for the given pixel. If an artifact is determined to be present at the given pixel, the adjusted interpolated pixel data for the given pixel is equal to a blend of the first level interpolated pixel data and the second level interpolated pixel data for the given pixel. The adjusted interpolated pixel data for the image may then be output and/or stored in memory.

In another embodiment, a method includes receiving a stream of Bayer filter pixel data for a given image. Horizontal first level and second level low pass filter values for each pixel are determined utilizing a first and second kernel size respectively. Vertical first level and second level low pass filter values for each pixel are also determined utilizing the first and second kernel sizes respectively. First level interpolated pixel data for each pixel is determined from the horizontal and vertical first level low pass filter values. Second level interpolated pixel data for each pixel is likewise determined from the horizontal and vertical second level low pass filter values. The color space of the first and second level interpolated pixel data are converted to separate the chroma and luma components for each level. The chroma component of the first level interpolated pixel data is reduced as a function of the difference between the green-on-red row and green-on-blue row chroma component of the pixel data of each of the given pixels to generate adjusted interpolated pixel data.

In yet another embodiment, the method includes receiving Bayer pixel data for a given image. The Bayer pixel data is low pass filtered using a first kernel size to determine first level interpolated pixel data. The Bayer pixel data is also low pass filtered using a second kernel size to determine second level interpolated pixel data. Adjusted interpolated data for a given pixel is equal to the first level interpolated pixel data for the given pixel, if a difference between chroma components in the first level interpolated pixel data is below a specified level. The adjusted interpolated pixel data for the given pixel is equal to a blend of the first level interpolated pixel data and the second level interpolated pixel data for the given pixel, if the difference between chroma components in the first level interpolated pixel data is above the specified level and a difference between chroma components in the second level interpolated pixel data is below the specified level. The method may further include generating additional levels of chroma components by low pass filtering the Bayer pixel data using increasingly larger kernel sizes until the difference between chroma components of a next level is below the specified level. In such case, the adjusted interpolated pixel data may be set equal to a blend of the interpolated pixel data for the level at which the difference between chroma components is below the specified level and one or more previous levels.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention are illustrated by way of example and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

FIG. 1 shows a block diagram of an exemplary digital camera.

FIG. 2 shows a block diagram of an exemplary Bayer filter.

FIG. 3 shows a flow diagram of a method of demosaicing digital image data, in accordance with one embodiment of the present invention.

FIGS. 4 and 5 show a flow diagram of a method of demosaicing digital image data, in accordance with another embodiment of the present invention.

FIG. 6 shows a block diagram of processing an exemplary array of pixels, in accordance with one embodiment of the present invention.

FIGS. 7 and 8 show a flow diagram of a method of demosaicing digital image data, in accordance with another embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to the embodiments of the invention, examples of which are illustrated in the accompanying drawings. While the invention will be described in conjunction with these embodiments, it will be understood that they are not intended to limit the invention to these embodiments. On the contrary, the invention is intended to cover alternatives, modifications and equivalents, which may be included within the scope of the invention as defined by the appended claims. Furthermore, in the following detailed description of the present invention, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it is understood that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the present invention.

Referring now to FIG. 3, a method of demosaicing digital image data, in accordance with one embodiment of the present invention. The method begins with receiving raw pixel data for a given image, at 310. The raw pixel data may include red, green and blue pixel data received as a stream upon which the following processing is substantially performed as the pixels are received. At 320, the received pixel data is low pass filtered using a first demosiac kernel size to determine first level interpolated pixel data P_L1 for each pixel. The first level interpolated pixel data P_L1 includes a red (R) color value, a green-on-red row (Gr) color value, green-on-blue row (Gb) color value and a blue (B) color value. At 330, the received pixel data is also low pass filtered using a second demosiac kernel size to determine second level interpolated pixel data P_L2 for each pixel. The second level interpolated pixel data includes a red (R) color value, green-on-red row (Gr) color value, a green-on-blue row (Gb) color value and a blue (B) color value. The kernel size of the second level is larger than the kernel size of the first level. The low pass filtering at the first and second levels may be performed by averaging the pixel data in the horizontal direction and recursive filtering the pixel data in the vertical direction.

At 340, it is determined if an artifact is present for each pixel. An artifact may be detected based upon the absolute difference between the Gr and Gb signals in the first level interpolated pixel data P_L1. If in a local region the Gr signal is substantially different from the Gb signal strength, it is likely that the scene frequency is beyond the Nyquist rate of blue and red. Therefore, a color artifact is detected when the Gr signal strength is substantially different from the Gb signal strength.

At 350, if a color artifact is not detected, the adjusted interpolated pixel data for the given pixel is set equal to the first level interpolated pixel data for the given pixel. In particular, the chroma component of the interpolated pixel data for the given pixel will be the first level chroma component. At 360, if a color artifact is detected, the adjusted interpolated pixel data for the given pixel is set equal to a blend of the first level interpolated pixel data and the second level interpolated pixel data for the given pixel. In particular, the chroma component of the interpolated pixel data for the given pixel will be generated by blending the first level chroma component with the second level chroma component. The blending ratio between the first and second level chroma components may be based upon the difference between the Gr and Gb signals. The adjusted interpolated pixel data may be further processed according to one or more other digital image processing techniques or it may be the final interpolated pixel data. At 370, the adjusted interpolated pixel is stored in a computing device readable medium.

Referring now to FIGS. 4 and 5, a method of demosaicing digital image data, in accordance with another embodiment of the present invention, is shown. The method of demosaicing digital image data will be further described with reference to FIG. 6, which illustrates the processing of an exemplary array of pixels. The method begins with receiving a stream of Bayer filter pixel data for a given image, at 410. At 415, a horizontal first and second level low pass filter value for each pixel is determined utilizing a first kernel size (W1) and a second kernel size (W2) respectively. If the incoming image width is W, the number of elements in the first level low pass filter will be W/W1 and the number of elements in the second level low pass filter will be W/W2. The kernel size W2 in the second level is greater than the kernel size W1 in the first level. In one implementation, W1 and W2 may be powers of 2. When the incoming pixel is received, the pixel value may be accumulated for level one (L1) and level two (L2) respectively, as illustrated in equations 1 and 2.Accum_L1+=Pin   (1)Accum_L2+=Pin   (2)When W1 pixels are accumulated, the sum is averaged and sent to the L1 row for vertical low pass filtering, as illustrated in equation 3. Similarly, when W2 pixels are accumulated, the sum is averaged and sent to the L2 row for low pass filtering, as illustrated in equation 4.If(Accum_L1=full) Accum_L1/=W1   (3)If(Accum_L2=full) Accum_L2/=W2   (4)If W1 and W2 are powers of 2, the averaging may be implemented by right shifting.

At 420, a vertical first and second level low pass filter value for each pixel is determined. In one implementation, the low pass filtering in the vertical direction can be done by recursive filtering, such as a one-tap infinite impulse response (1-tap IIR) filter. When horizontal averaging for a local W1 and W2 pixel group is done, the average value will be updated to the corresponding element in the L1 and L2 row, as illustrated in equations 5 and 6.L1[i]+=(Accum_L1−L1[1])/K_L1   (5)L2[j]+=(Accum_L2−L2[j])/K_L2   (6)K_L1 and K_L2, in equations 5 and 6, are IIR filter coefficients. If K_L1 and K_L2 are power of 2 the division can be implemented by right shifting.

At 425, first and second level interpolated pixel data for each pixel is generated. The low passed first and second level pixel values (P_L1, P_L2) of the demosaic output Pout may be generated by linear interpolation. For instance, suppose X is the horizontal coordinate of Pout, and m=floor(X/W1) and n=floor (X/W2) then P_L1 and P_L2 can be generated as illustrated in equations 7 and 8.P_L1=f1*L1[m]+(1−f1)*L1[m+1]  (7)P_L2=f2*L2[n]+(1−f2)*L2[n+1]  (8)Wherein f1=1−(X−m*W1)/W1 and f2=1−(X−n*W2)/W2.

At 430, the color space of the interpolated pixel data is converted to separate it into luma and chroma components. The objective of color artifact reduction is to replace the chroma components of Pout by P_L1, or P_L2 or the combination of the P_L1 and P_L2. For that a color conversion is performed to separate the luma and chroma components of the RGB color (Pout, P_L1 and P_L2), as illustrated in equations 9, 10, 11, 12, 13 and 14.Y=(R+(G*2)+B)/4   (9)U=B−Y   (10)V=R−Y   (11)The inverse transform is:R=Y+V   (12)G=Y−(U+V)/2   (13)B=Y+U   (14)Wherein Y is the luma component and U and V are the chroma components.

At 435, the color artifacts are reduced based upon how large the difference is between the green-on-red (Gr) and green-on-blue (Gb) components, as illustrated in equations 15, 16, 17 and 18.fL1=|Pout(Gr)−Pout(Gb)|/2^p   (15)fL2=|PL1(Gr)−PL1(Gb)|/2^p   (16)Uadjusted=(1−fL1)*Pout(U)+fL1*((1−fL2)*PL1(U)+fL2*PL2(U))   (17)Vadjusted=(1−fL1)*Pout(V)+fL1*((1−fL2)*PL1(V)+fL2*PL2(V))   (18)If Gr and Gb is large, the adjusted U and V will be close to the blending of L1 and L2 chroma components. The blending is also weighted by the Gr−Gb difference. In particular, if the Gr and Gb components at L1 is close, then the influence from L2 will be small. A few control parameters can also be used to adjust the weightings of the chroma blending, as illustrated in equations 19 and 20.f′=f−Coring   (19)If f′<0f′=0f″=f′*Weighting   (20)

fL1 and fL2 can have separate sets of {Coring, Weighting} parameters.

At 440, the adjusted interpolated pixel data is stored in one or more computing device readable media. The adjusted interpolated pixel data stored in the computing device readable media may be output for present to a user or may be further processed according to one or more other digital imaging techniques.

Referring now to FIGS. 7 and 8, a method of demosaicing digital image data, in accordance with another embodiment of the present invention, is shown. The method begins with receiving Bayer pixel data for a given image, at 710. The Bayer pixel data may be received as a stream or red, green and blue pixel data upon which the following processing is substantially performed on the fly as the pixels are received. At 720, the Bayer pixel data is low pass filtered using a first demosiac kernel size to determine first level interpolated pixel data P_L1 for each pixel. The first level interpolated pixel data include a red (R) color signal, a green-on-red row (Gr) color signal, a green-on-blue row (Gb) color signal and a blue (B) color signal. At 730, the received pixel data is also low pass filtered using a second demosiac kernel size to determine second level interpolated pixel data P_L2 for each pixel. The second level interpolated pixel data include a red (R) color signal, a green-on-red row (Gr) color signal, a green-on-blue row (Gb) color signal and a blue (B) color signal. The kernel size of the second level is larger than the kernel size of the first level.

At 740, if the difference between the Gr and Gb signals of the first level interpolated pixel data is below a specified level, the chroma component of the adjusted interpolated pixel data for the given pixel is set to the first level chroma component. At 750, if the difference between the Gr and Gb signals of the first level interpolated pixel data is above the specified level and the difference between the Gr and Gb signals of the second level is below the specified level, the chroma component of the adjusted interpolated pixel data for the given pixel is generated by blending the first level chroma component with the second level chroma component. The blending ratio between the first and second level chroma components may be based upon the difference between the Gr and Gb signals of the first level.

At 760, if the difference between the Gr and Gb signals of the first and second level interpolated pixel data are above the specified level, the received pixel data is low pass filtered using a third demosiac kernel size to determine a third level interpolated pixel data P_L2 for the given pixel. The kernel size of the third level is larger than the kernel size of the second level. At 770, if the difference between the Gr and Gb signals of the first and second level low pass filter values are above the specified level, the chroma component of the adjusted interpolated pixel data for the given pixel is generated by blending the second level chroma component with the third level chroma component. The blending ratio between the second and third level chroma components may be based upon the difference between the Gr and Gb signals in the second level. In another implementation, the chroma component of the final interpolated pixel data for the given pixel is generated by blending the first, second and third level chroma components, if the difference between the Gr and Gb signals of the first and second level interpolated pixel data are above the specified level. In yet another implementation, additional levels of chroma components can be generated by low pass filtering using increasingly larger kernel sizes until the difference between the Gr and Gb signals of a given level is above the specified level and the Gr and Gb signal of a next level is below the specified level. In such an implementation, the chroma component of the adjusted interpolated pixel data for a particular pixel may be generated by blending the chroma component from the next level with one or more of the previous levels. At 780, the adjusted interpolated pixel data is stored in a computing device readable medium.

The above described techniques for reducing artifacts in digital images may be implemented by the digital signal processor of a digital camera or by a separate computing device. The above described techniques may be embodied in computing device executable instructions (software), hardware and/or firmware. The techniques advantageously reduce false colors while reducing color bleaching. In addition, the above described techniques do not result in excessive line buffering.

The foregoing descriptions of specific embodiments of the present invention have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, and obviously many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the Claims appended hereto and their equivalents.

Claims

1. A method of demosaicing digital image data comprising: receiving raw pixel data for a given image;low pass filtering the raw pixel data using a first demosiac kernel size to determine first level interpolated pixel data for each pixel;low pass filtering the raw pixel data using a second demosiac kernel size to determine second level interpolated pixel data for each pixel;determining if an artifact is present for each pixel from the first level interpolated pixel data;if the artifact is not present at a given pixel, setting adjusted interpolated pixel data for the given pixel equal to the first level interpolated pixel data for the given pixel;if the artifact is present at the given pixel, setting the adjusted interpolated pixel data for the given pixel equal to a blend of the first level interpolated pixel data and the second level interpolated pixel data for the given pixel; andstoring the adjusted interpolated pixel data in a computing device readable medium.

2. The method according to claim 1, wherein the raw pixel data comprises Bayer pixel data.

3. The method according to claim 2, wherein: the first level interpolated pixel data for each pixel comprise a red (R) color value, a green-on-red row (Gr) color value, green-on-blue row (Gb) color value and a blue (B) color value; andthe second level interpolated pixel data for each pixel comprise a red (R) color value, a green-on-red row (Gr) color value, a green-on-blue row (Gb) color value and a blue (B) color value.

4. The method according to claim 3, wherein the demosaic kernel size of the second level is larger than the demosaic kernel size of the first level.

5. The method according to claim 4, wherein determining if a color artifact is present comprises determining the absolute difference between the green-on-red row (Gr) color value and green-on-blue row (Gb) color value in the first level interpolated pixel data.

6. The method according to claim 5, wherein a color artifact is present if the green-on-red row (Gr) color value is substantially different from the green-on-blue row (Gb) color value.

7. The method according to claim 6, wherein the blending ratio between the first and second level interpolated pixel data is a function of the difference between the green-on-red row (Gr) color value and green-on-blue row (Gb) color value.

8. One or more non-transitory computing device readable media containing a plurality of instructions which when executed cause a computing device to implement a method comprising: receiving a stream of Bayer filter pixel data for a given image;interpolating red, green-on-red row, green-on-blue row and blue pixel data from the Bayer filter pixel data for each pixel of the given image for two different kernel sizes;reducing an artifact in one or more given pixels as a function of a difference between the green-on-red row and green-on-blue row pixel data of each of the given pixels of the two different kernel sizes to generate adjusted interpolated pixel data equal to interpolated pixel data of one of the kernel sizes if the artifact is not present and equal to a blend of the interpolated pixel data for the two different kernel sizes if the artifact is present; andstoring the adjusted interpolated pixel data.

9. The one or more non-transitory computing device readable media containing a plurality of instruction which when executed cause a computing device to implement the method according to claim 8, wherein interpolating red, green-on-red row, green-on-blue row and blue pixel data further comprises: determining a horizontal first level low pass filter value for each pixel utilizing a first kernel size;determining a horizontal second level low pass filter value for each pixel utilizing a second kernel size;determining a vertical first level low pass filter value for each pixel utilizing the first kernel size;determining a vertical second level low pass filter value for each pixel utilizing the second kernel size;determining first level interpolated pixel data for each pixel from the horizontal first level low pass filter value and the vertical first level low pass filter value;determining second level interpolated pixel data for each pixel from the horizontal second level low pass filter value and the vertical second level low pass filter value;converting a color space of the first level interpolated pixel data to separate chroma and luma components; andconverting the color space of the second level interpolated pixel data to separate the chroma and luma components.

10. The one or more non-transitory computing device readable media containing a plurality of instruction which when executed cause a computing device to implement the method according to claim 9, wherein: determining the horizontal first level low pass filter value for each given pixel comprises averaging the pixels for each corresponding set of adjacent pixels of the first kernel size; anddetermining the horizontal second level low pass filter value for each given pixel comprises averaging the pixels for each corresponding set of adjacent pixels of the second kernel size.

11. The one or more non-transitory computing device readable media containing a plurality of instruction which when executed cause a computing device to implement the method according to claim 9, wherein: determining the vertical first level low pass filter value for each given pixel comprise regressive filtering the pixels for each corresponding set of adjacent pixels of the first kernel size; anddetermining the vertical second level low pass filter value for each given pixel comprises regressive filtering the pixels for each corresponding set of adjacent pixels of the second kernel size.

12. The one or more non-transitory computing device readable media containing a plurality of instruction which when executed cause a computing device to implement the method according to claim 9, wherein reducing the artifact in one or more given pixels further comprises: reducing the chroma component of the first level interpolated pixel data as the function of a difference between the green-on-red row and green-on-blue row chroma components of the pixel data of each of the given pixels to generate adjusted interpolated pixel data.

13. The one or more non-transitory computing device readable media containing a plurality of instruction which when executed cause a computing device to implement the method according to claim 12, further comprising adjusting a weighting of the chroma blending to generate the adjusted interpolated pixel data utilizing a coring parameter.

14. The one or more non-transitory computing device readable media containing a plurality of instruction which when executed cause a computing device to implement the method according to claim 12, further comprising: adjusting the weightings of a chroma blending to generate the adjusted interpolated pixel data utilizing a weightings parameter.

15. The one or more non-transitory computing device readable media containing a plurality of instruction which when executed cause a computing device to implement the method according to claim 10, wherein: the kernel size of the first level is a power of two; andthe kernel size of the second level is a power of two and greater then the kernel size of the first level.

16. One or more non-transitory computing device readable media containing a plurality of instructions which when executed cause a computing device to implement a method comprising: receiving Bayer pixel data for a given image;low pass filtering the Bayer pixel data using a first kernel size to determine first level interpolated pixel data;low pass filtering the Bayer pixel data using a second kernel size to determine second level interpolated pixel data;setting an adjusted interpolated pixel data equal to the first level interpolated pixel data for a given pixel, if a difference between chroma components in the first level interpolated pixel data is below a specified level;setting the adjusted interpolated pixel data to a blend of the first level interpolated pixel data and the second level interpolated pixel data for a given pixel, if the difference between chroma components in the first level interpolated pixel data is above the specified level and a difference between chroma components in the second level interpolated pixel data is below the specified level; andoutputting the adjusted interpolated pixel data.

17. The one or more non-transitory computing device readable media containing a plurality of instruction which when executed cause a computing device to implement the method according to claim 16, further comprising: low pass filtering the Bayer pixel data using a third kernel size to determine third level interpolated pixel data; andsetting the adjusted interpolated pixel data to a blend of the second level interpolated pixel data and the third level interpolated pixel data for a given pixel, if the difference between chroma components in the first level interpolated pixel data is above the specified level and a difference between chroma components in the second level interpolated pixel data is below the specified level.

18. The one or more non-transitory computing device readable media containing a plurality of instruction which when executed cause a computing device to implement the method according to claim 17, wherein: a blending ratio between the first and second level interpolated pixel data is a function of the difference between a green-on-red row (Gr) color value and a green-on-blue row (Gb) chroma value of the first level interpolated pixel data; anda blending ratio between the second and third level interpolated pixel data is a function of a difference between the green-on-red row (Gr) color value and green-on-blue row (Gb) chroma value of the second level interpolated pixel data.

19. The one or more non-transitory computing device readable media containing a plurality of instruction which when executed cause a computing device to implement the method according to claim 16, further comprising: generating additional levels of chroma components by low pass filtering the Bayer pixel data using increasingly larger kernel sizes until a difference between the green-on-red row (Gr) color value and green-on-blue row (Gb) chroma value of a given level is below the specified level; andgenerating the chroma component of the adjusted interpolated pixel data by blending the chroma component from the given level with one or more previous levels of the interpolated pixel data.