Patent Application
20060256215
Electronic image sensors are predominantly of two types: CCD (Charge Coupled Devices) and CMOS-APS (Complementary Metal Oxide Semiconductor—Active Pixel Sensors). Both types of sensors typically contain an array of photo-detectors, arranged in a pattern, that generate electrical charge in response to light. Each photo-detector corresponds to a pixel of an image and measures the intensity of light of the pixel within one or ranges of wavelengths, corresponding to one or more perceived colors.
However, despite advances in the manufacturing process, both CCD and CMOS image sensors often contain defects that produce undesirable noise in the image. For example, one significant source of noise in an image is known as “dark current noise.” Dark current noise is a fixed pattern noise that results from manufacturing defects in the photo-detectors. The defects cause the photo-detectors to accumulate charge even in the absence of light. Typically, the dark current in an image sensor creates an undesirable “dark” image that overlays the illuminated image.
Since the spatial pattern of dark current noise on a particular image sensor is mostly constant (based on the pattern of photo-detector defects), which generates a noise pattern with a fixed element and a smaller random element, image sensors have traditionally employed a dark current subtraction mechanism to remove the effects of dark current in an image. For example, in cameras that utilize a shutter, a dark image (dark frame) with the shutter closed can be obtained along with the illuminated image (image frame) with the shutter open. The dark frame is subtracted from the image frame to yield an image frame without most of the dark noise component.
Another example of a dark current subtraction process is described in U.S. Pat. No. 6,714,241 to Baer, entitled “Efficient Dark Current Subtraction in an Image Sensor,” (hereinafter referred to as the Baer patent). In the Baer patent, a dark noise image is stored on the image sensor and subtracted from each new image captured. However, the level of dark noise in a particular image is a function of both the image sensor temperature and the exposure time of the particular image. Therefore, to effectively remove the dark noise in a particular image, the stored dark noise image must be scaled for the temperature and exposure time. The Baer patent uses a few rows of “black pixels” on the image sensor to accomplish the dark noise scaling. Each black pixel is covered by a metal layer so that the sensor values of the black pixels represent only dark noise. By comparing the mean or clipped mean of the sensor values of the black pixels to the mean of the stored dark noise image, the appropriate scale factor for a particular image can be estimated.
The dark current subtraction process described in the Baer patent not only provides dark noise image correction in cameras without shutters, but also enhances the frame rate of cameras with shutters. However, the Baer dark current subtraction process does require several rows of pixels to be reserved as “black pixels,” which may not be desirable in many applications. As a result, if a camera without a shutter does not use an image sensor with black pixels, there is currently no dark current subtraction process available to remove the effects of dark current in the image.
Embodiments of the present invention provide an image processing system for subtracting dark noise out of images without requiring the use of “black pixels.” The image processing system includes an image sensor, a memory and a processor. The image sensor captures a current image and produces current image data representing the current image. The current image data includes both a current dark noise signal and an image signal. The memory stores reference image data representative of a reference dark noise signal produced by the image sensor. The processor estimates a dark noise scale factor from the current image data and the reference image data, scales the reference dark noise signal by the dark noise scale factor to produce a scaled dark noise signal and subtracts the scaled dark noise signal from the current image data to produce the image signal.
In one embodiment, the processor uses a linear regression algorithm to calculate the dark noise scale factor. The linear regression algorithm determines regression coefficients representing the linear relations between the current image data and the reference image data. The dark noise scale factor is calculated from the regression coefficients. For example, the regression coefficients can identify a slope value and an intercept value representing the linear relationship between the current image data and the reference image data.
In a further embodiment, the image sensor includes an array of pixels arranged in rows and columns. The processor uses respective selected raw sensor values of the reference image data and the current image data generated by corresponding selected pixels in the array to estimate the dark noise scale factor. In one aspect of the invention, the selected pixels include a portion of the pixels corresponding to a dark region of the current image. In another aspect of the invention, the selected pixels are randomly selected. In still another aspect of the invention, the selected pixels are selected to uniformly sample noise level pixel values of the reference dark noise signal. In a further aspect of the invention, the selected pixels include at least one row of pixels that produces a noise level distribution in the reference dark noise signal nearest a uniform noise level distribution in the reference dark noise signal.
The disclosed invention will be described with reference to the accompanying drawings, which show important sample embodiments of the invention and which are incorporated in the specification hereof by reference, wherein:
The image processing system 10 includes an image sensor 20, a processor 60 capable of executing a dark noise subtraction algorithm 65 and a memory 70 for storing dark noise reference image data 80. The image sensor 20 is part of a CMOS sensor chip or a CCD sensor chip, and includes a two-dimensional array of photo-detectors 40 arranged in rows and columns. Each photo-detector 40 generates a sensor value corresponding to a pixel of an image. The image sensor 20 provides raw image data 50 containing the raw sensor (pixel) values to the processor 60.
The processor 60 accesses the memory 70 to retrieve the reference image data 80, and uses the reference image data 80 while executing the dark noise subtraction algorithm 65 to remove the effects of dark current in the raw image data 50. The processor 60 can be a microprocessor, microcontroller, programmable logic device or any other processing device. In one embodiment, the processor 60 is built into the sensor chip incorporating the image sensor 20. In another embodiment, the processor 60 is part of a computing system that can be connected to the image sensor 20 via a wired or wireless interface. The memory 70 can be any type of memory device, such as a flash ROM, EEPROM, ROM, RAM or any other type of storage device. In another embodiment, the memory 70 further stores the dark noise subtraction algorithm 65 executable by the processor 60.
The dark noise subtraction algorithm 65 estimates a dark noise scale factor for the current image from the raw image data 50 and the stored reference image data 80 using any appropriate statistical process. The raw image data 50 includes both an image signal representing the image and a dark noise signal representing a dark noise component. The stored reference image data 80 includes a reference dark noise signal that represents a reference image captured by the image sensor 20 without illumination. The reference image data 80 can be stored at any time prior to generating the raw image data 50 for the current image. For example, the reference image data 80 can be generated and stored during manufacturing of the image sensor 20 or at any time subsequent to the manufacture of the image sensor 20.
The dark noise subtraction algorithm 65 scales the reference image data 80 by the estimated scale factor to produce scaled reference image data. The dark noise subtraction algorithm further subtracts the scaled reference image data from the raw image data 50 to subtract out the dark noise signal in the raw image data 50, thereby producing the desired image signal 90 without a significant part of the dark noise component.
Referring now to
The dark noise scale factor estimation logic 200 takes as input the raw image data 50 provided by the image sensor 20 and the reference image data 80, estimates the scale factor 220 for the current image from the raw image data 50 and the reference image data 80 and outputs the scale factor 220 for the current image. The dark noise scale factor estimation logic 200 uses any appropriate statistical process to estimate the scale factor 220.
For example, in one embodiment, the dark noise scale factor estimation logic 200 applies a linear regression process to the raw image data 50 and the reference image data 80 to determine the linear relationship between the raw image data 50 and the reference image data 80. The linear regression process determines linear regression coefficients of best-fitting linear functions that relate the raw image data 50 to the reference image data 80. More specifically, the linear regression coefficients represent the slope and intercept between the raw image data 50 and the reference image data 80. The dark noise scale factor estimation logic 200 calculates the estimated scale factor 220 from the slope using linear prediction.
As an example, if xj represents the stored reference image data 80, and yj represents the sensor value of a captured image at pixel location j, which has a signal level of sj if no dark noise were present, the expected sensor values E[yj] can be represented as:
E[yj]=sj+E[nj]+k*E[xj], (Equation 1)
where k is the dark noise scale factor for the particular captured image, nj represents noise components other than dark noise (e.g., read noise and shott noise), and E[x] represents the expected value of a random variable x. Since the dark noise is independent of the particular image being captured and other noise contributors, the image signal term sj and the non-dark noise term nj are independent of the dark noise term k*xj. As a result, the dark noise scale factor k can be estimated by a linear regression method, using the stored reference dark noise values E[xj] as the predictor and the image signal values yj as the data for all sampled j.
Using Equation (1) above, the dark noise scale factor k can be calculated by:
T=(XtX)−1XtY, (Equation 2)
where Y is a vector with sensor values yj at all sampled pixel locations, and X is a two-column matrix with the stored reference dark noise values xj at corresponding sampled pixel locations (corresponding to the sensor value yj pixel locations) in the first column and ones (1) in the second column. The column with ones is used to allow for an offset term in the regression solution, since the signal term sj has a positive expected value. The estimated scale factor k is the first element (i.e., the slope coefficient) in the solution T.
The dark noise scale factor estimation logic 200 provides the estimated scale factor 220 to the dark noise remover logic 210 for use in removing the dark noise in the raw image data 50. Thus, the dark noise remover logic 210 takes as input the raw image data 50, the estimated scale factor 220 and the reference image data 80. The dark noise remover logic 210 scales the reference image data 80 by multiplying the estimated scale factor 220 by each of the reference dark noise values in the reference image data 80. The resulting scaled reference image data is subtracted from the raw image data 50 on a pixel-by-pixel basis to produce the image signal 90.
For example, if the reference image data is represented by sensor values xj and the raw image data is represented by sensor values yj, where j represents a particular pixel location (photo-detector) in the image sensor, the scaled reference image data (SDj) can be represented as:
SDj=k*xj, (Equation 3)
and therefore, the image signal (Ij) is calculated as:
Ij=yj−(k*xj) (Equation 4)
In one embodiment, the dark noise scale factor estimation logic 200 and the dark noise remover logic 210 of the dark noise subtraction algorithm 65 are implemented on a computing system remote from the digital imaging device (e.g., camera) including the image sensor. For example, the dark noise subtraction algorithm 65 can be incorporated on a personal computer or computing device (e.g., photo-finisher station or photo printer). As another example, the dark noise subtraction algorithm 65 can be incorporated on a web server, and the raw image data 50 can be uploaded to the web server for dark noise removal. In another embodiment, the dark noise scale factor estimation logic 200 and dark noise remover logic 210 are implemented on a separate chip from the sensor chip in the imaging device. In still another embodiment, the dark noise scale factor estimation logic 200 and dark noise remover logic 210 are implemented on the sensor chip incorporating the image sensor.
Referring now to
The digital image signal containing the raw image data (i.e., raw sensor (pixel) values) is input to the processor 60, which accesses the memory 70 to retrieve the reference dark noise signal. The memory 70 can be included on the sensor chip 300 or on a separate chip. The processor 60 uses the stored reference dark noise signal to subtract out the dark noise in the digital image signal.
At block 430, a dark noise scale factor is estimated from the current image and the stored reference dark noise image, and at block 440, the reference dark noise image is scaled by the estimated scale factor to determine a scaled reference dark noise image representing the current dark noise in the current image. Since the level of dark noise in the current image is a function of the image sensor temperature at the time of image capture and the exposure time of the current image, the sensor values representing the current image can be used to determine the scale factor for scaling the stored reference dark noise image to the appropriate level for the temperature and exposure time of the current image. Once the dark noise in the current image is ascertained, at block 450, the dark noise is subtracted from the current image to produce an image without the fixed element of dark noise.
Although the sensor values for each pixel location in the current image and the reference image can be used to estimate the scale factor, the scale factor can also be estimated using a subset of the sensor values at sampled pixel locations, with minimal increase in estimation error. Reducing the number of pixels utilized to estimate the scale factor reduces the computational load on the image sensor and/or computing device. For example, the scale factor can be estimated with as few as one-hundred corresponding pixels from each image (current and reference). The sampled pixels can be randomly selected or selected based on the pixel values.
Dark noise distributions tend to concentrate on smaller noise levels, with few pixels having large noise values. Therefore, a random sample of pixel locations of the reference dark noise image will yield primarily low pixel values. Such a randomly sampled distribution is not desirable when using a linear regression method to estimate the slope. Therefore, by uniformly sampling pixel locations with an equal number of pixels at each noise level in the reference dark noise image, the scale factor estimates stabilize significantly faster as compared to random sampling.
In an exemplary embodiment, to minimize the storage space necessary for identifying the uniformly sampled pixels to be used in the scale factor estimation process, the pixel values in each row of the reference dark noise image can be searched to determine at least one row of pixels that producing a noise level distribution nearest the uniform noise level distribution in the reference dark noise signal. As a result, instead of storing the pixel locations of each uniformly sample pixel, only the row number(s) identifying the rows to be used in the scale factor estimation process need to be stored.
In another embodiment, the sampled pixels used to estimate the scale factor can be selected based on the pixel values in the current image. Darker regions of an image produce sensor values with lower signal levels and thus proportionally higher dark noise levels (i.e., lower signal-to-noise ratio). Thus, by using the sensor values in the dark regions of an image, the linear relationship between the dark noise in the current image and the stored dark noise can be more accurately determined. For example, the sampled pixel locations can be selected by doing an initial de-noising (e.g., median filtering) on the current image, and then selecting the pixel locations with pixel values (after de-noising) below a pre-set threshold (e.g., a digital value of 10 out of 255). As another example, the pixel locations can be divided into regions (or rows), and the selected pixels in the current image include the regions (or rows) with the most number of pixel locations having pixel values below the pre-set threshold or with an average pixel value below an average pre-set threshold.
At block 650, a dark noise scale factor is estimated from the sampled pixel values of the current and reference image, and at block 660, the reference dark noise image is scaled by the estimated scale factor to determine a scaled reference dark noise image representing the current dark noise in the current image. Thereafter, at block 670, the dark noise is subtracted from the current image to produce an image without the fixed element of dark noise.
As will be recognized by those skilled in the art, the innovative concepts described in the present application can be modified and varied over a wide rage of applications. Accordingly, the scope of patents subject matter should not be limited to any of the specific exemplary teachings discussed, but is instead defined by the following claims.
