The present invention relates generally to semiconductor imagers. More specifically, the present invention relates to a reduced-memory technique for fixed-pattern noise reduction in semiconductor imagers.
Various types of imagers or image sensors are currently used, including charge-coupled device (CCD) image sensors and complementary metal-oxide semiconductor (CMOS) image sensors. CMOS technology offers many benefits, such as lower cost, ease of manufacturing, and a higher degree of integration over the CCD image sensors. However, a drawback of the CMOS image sensor technology is the presence of fixed-pattern noise (FPN). Fixed pattern noise is generated from a mismatch of circuit structures due to variations in the manufacturing processes of integrated circuits. The effect of fixed pattern noise in a CMOS image sensor is that groups of pixels, typically each column in a sensor array, exhibit relatively different strengths in response to uniform input light.
Fixed pattern noise is not the only noise source existent in a CMOS image sensor. However, fixed pattern noise is largely independent of the other noise sources. Specifically, reducing the variance of column-wise fixed pattern noise with respect to the variance of other pixel-wise noise sources can reduce the appearance of the column-wise fixed pattern noise as an artifact in an image. In general, if the standard deviation, or square root of the variance, of the column-wise fixed pattern noise is 5 to 6 times smaller than other pixel-wise noise sources, then the human eye will not be able to detect the presence of the column-wise fixed pattern noise.
To remove the effect of fixed pattern noise, conventional calibration processes involve measuring an output based on a known optical input and comparing it against an expected value. In CMOS image sensors, a light of known intensity and frequency is typically shone onto the sensors and used as the input calibration signal. In some cases, output from inactive dark pixel sensors may also be used during calibration to compare the sensors' output to an expected dark output. In principle, if there is no mismatch in the sensor devices, the voltage signal output from every pixel cell should be identical. In reality, however, significant differences in signal output values are read out between the pixel columns of a sensor array, even when a uniform input light stimulus is applied to the array. The difference between the pixel signal output values and the expected pixel signal output values is typically referred to as a pixel offset. The pixel offsets are computed and then stored for use in calibrating the entire sensor array.
In a conventional sensor array calibration, or offset reduction procedure, the calculated offsets are applied to the output of each pixel during normal image sensor operation. A pixel whose un-calibrated output was higher than expected will have its signal output value reduced by the amount of its corresponding scaled offset value. Similarly, a pixel whose un-calibrated output was lower than expected will have its signal output value increased by the amount of its corresponding scaled offset value. Any change in operating mode that would affect the offset values will result in a necessary recalculation of offset values for each pixel.
Storing pixel offset values poses tremendous challenges. Conventional calibration techniques may require memory to store an offset value for each pixel, or, more commonly, an average offset value for each column of pixels in the sensor matrix. In the latter case, because each column may contain multiple channels (relating to multiple colors), multiple average offset values for each column may be required. CMOS image sensors that use a Bayer Color Filter Array (Bayer CFA) have two channels for each column. Pixels within a column are associated with and alternate between channels such that, for example, every pixel on an odd-numbered row outputs a signal corresponding to a first channel, and every pixel on an even-numbered row outputs a signal corresponding to a second channel. Columns stagger the channel outputs in a checkerboard pattern. The resulting number of offset values to be stored becomes 2N, where N is the number of columns in the sensor array. These potentially large storage requirements also translate into a similarly large memory footprint in or accompanying the CMOS imager device, which also means greater expense and longer calibration procedure times.
There is, therefore, a need or desire for a method and apparatus for both minimizing fixed pattern noise in a CMOS imager and reducing the memory used for such minimization.
The invention provides a method and apparatus that both minimizes fixed pattern noise in a CMOS imager and reduces the memory used for such minimization.
An offset value for each of a plurality of pixel columns in the imager is generated. A predetermined number of the plurality of pixel columns having corresponding offset values that are greater in value than all other generated offset values are selected. Noise is reduced in each of the selected pixel columns by applying the corresponding offset values to pixel outputs from each selected pixel column. The memory requirements of the imager are reduced by only storing offset values corresponding to the selected pixel columns. Memory is also conserved by reducing bit depth and only storing a single offset value for each of the plurality of pixel columns.
The foregoing and other advantages and features of the invention will become more apparent from the detailed description of exemplary embodiments provided below with reference to the accompanying drawings in which:
The present invention is now described in more detail herein in terms of various exemplary embodiments. This is for convenience only and is not intended to limit the application of the present invention. In fact, after reading the following description, it will be apparent to one skilled in the relevant art(s) how to implement the following invention in alternative embodiments. Moreover, the present invention can be implemented using software, hardware or any combination thereof as is known to one of ordinary skill in the art.
In an embodiment of the present invention, column-wise fixed pattern noise is reduced such that its standard deviation is at least five times lower than that of other pixel-wise noise sources. This is achieved by only calibrating a predetermined number of pixel columns that have offset values that are greater than all other offset values. Because not all pixel-columns are corrected, the total amount of memory required in a silicon implementation is reduced.
Referring now to
When the CMOS sensor array 210 is calibrated, a reference or dummy signal is applied to the CMOS sensor array 210. The applied reference or dummy signal may be either a uniform incident light of known frequency or a signal reflecting the absence of any incident light. Reference signal outputs are then recorded from the pixels 228. In an effort to reduce time and memory requirements for calibration, in accordance with the invention, reference signal outputs need not be read out and stored from every pixel 228. Instead, only a sufficient number of reference signal outputs need be read out and stored from each column 220. If it is deemed that R reference signal outputs for each channel for each column would be sufficient, then, in the example of
The process of applying, inputting and outputting reference or dummy signals may occur at chip startup, when the integration or exposure time may be set to a minimum to avoid measurement errors that result from pixel heating or dark current. The read out data may then be used to calibrate the CMOS sensor array 210 for any operation in the current operating mode of the imager. If the operating mode were to change, for example, to a binning mode, wherein the output from multiple adjacent pixels is combined to create a single output (in effect, creating a large pixel from several small pixels), the process of calculating and selecting average column offset values would need to be repeated to account for the effective change in offset values and number of columns. The signal measurements should be made at the highest analog gain setting to achieve the most accurate results possible. Future image acquisition procedures in the same operating mode, but at differing gain levels will utilize scaled-down results generated from the high gain settings. Generally, any scaling errors that may result during lower gain setting operation (e.g., 1× and 2× gains) will not be significant because the signal level is generally much higher than the noise at these gain levels.
Referring now to
In addition to calculating the average reference output signals for each channel, the average reference output signals for each channel per column need also be calculated (block 330). This may be achieved by summing a given channel output for a given column, and then dividing this sum by the number of occurrences of the given channel within the given column. Processing block 330 may be summarized by Equations 2.1 and 2.2.
where col=1,2,3, . . . n.
Afterwards, an average column offset for each column and channel may be calculated by determining the difference between μchannel and μcol, channel (block 340). Note that in Equations 2.1 and 2.2, only n average column offsets are determined at a time, where n represents the number of average column offsets that may be stored in a memory block at one time.
For every n average column offsets computed and stored, the locations of a predetermined number of the average column offsets that are greater in value than all other average column offsets are identified and stored (block 350). In an exemplary embodiment, the n2/N greatest value average column offsets are determined for every n columns. Once determined, the locations of the columns associated with the greatest value average column offsets are stored.
Once the columns with the greatest value average column offsets have been located for all N columns, the average column offsets for the specified columns may be re-measured and re-calculated (block 360). These re-measured average column offsets for the selected columns become the offset correction values that will be applied to calibrate any image acquired by the CMOS image array in the current operating mode, as demonstrated in
Referring now to
A significant advantage of this fixed pattern noise-reducing technique, besides the reduction of noise, is the overall reduction in memory used during the process. In the traditional method of correcting all columns with the same precision, assuming a four-channel sensor array, the memory requirement is shown below in Equation 3, where N represents the total number of columns, R is the necessary number of sampled pixels for a given channel in a given column, and B is the bit depth, or the number of bits used to store the average column offset values.
totalmemorytraditional=2N·(log2(R)+B) Eq. 3.
However, using this exemplary embodiment, the memory requirement reduces to:
totalmemoryreduced=2 n·(log2(R)+B)+N Eq. 4.
The reduced memory results from the fact that only n average column offset values of bit depth B need be stored, instead of N columns (which is all the columns in the array). The additional N bits shown in Equation 4 relate to a 1×N array necessary to store the location information of the columns selected for calibration.
In another exemplary embodiment, memory requirements may be further reduced by (1) changing the number of rows sampled to obtain average column offset values and (2) changing the bit depth of stored average column offset values. As such, the memory requirement of the additional embodiment becomes:
totalmemoryreduced=n·(log2(R)+b)+N·b Eq. 5.
The exemplary embodiment described by Equation 5 is used in the method illustrated in
Next, average column offset values may be calculated (block 530). This may be done in increments of n columns, where n is a subset of the total N columns, and where n represents the number of average column offset values that may be stored in a given memory block at one time. To determine the n average column offset values, only a sufficient number, R, of reference signal outputs need be read from each column. In this embodiment, no differentiation is made between the various channel outputs within a column at this stage of the process. Hence, only R reference signal outputs are read for each column (as opposed to 2 R in the previous embodiment). For each n columns, the average column offset value may be calculated according to equation 7.
For each of the n average column offset values, only the upper b bits of the value need be stored (block 540). In effect, this reduces the bit depth of the average column offset values from B to b. Although this results in less precision of the average column offset values, experiments have shown that full precision is not required to adequately remove the visual effects of fixed pattern noise in the image.
In this manner, the average column offset values may be determined and stored for each of the N/n subsets of columns, resulting in a total of N average column offset values, each with a bit depth of b bits. These N average column offset values remains constant until the imager is either reset or a mode change occurs.
This method of determining average column offset values also effectively reduces the time required to measure and determine the offset values, which both reduces memory requirements and improves performance. Referring now to
Additionally, although the bit depth of the average column offset values 720 has been reduced, a bit shifter 750 may be provided in order to achieve the desired correction range, thus accounting for either very large or very small average column offset values 720.
In an additional embodiment of the invention, a threshold offset value is utilized to add greater flexibility to the noise-reduction process. Instead of applying the calculated average column offset values to a predetermined number of pixel columns, a maximum offset value could be determined and then used as a threshold level; any pixel columns with average column offset values exceeding the threshold offset value will be modified in a noise-reducing process according to the process already described.
Similarly, a minimum offset value could be determined and used as a minimum threshold offset value. In an embodiment wherein a minimum threshold offset value is determined, pixel columns with average column offset values below the minimum threshold offset value are modified so as to increase the overall offset values of the selected pixel columns. This method, though less preferred, would still result in significant smoothing of the output image.
In a further embodiment, the applied average column offset values need not be precise. In other words, by applying a noise-reducing factor that is only similar in value to the calculated average column offset value, a noise-reducing effect will still be evident. Exact precision is not necessary.
Combined fixed pattern noise and memory reduction may be achieved in a CMOS imager through an appropriate implementation of the methods described above. As shown in
An imager as depicted in
A typical processor based system 900, which includes a CMOS imager device 930 according to the present invention is illustrated in
A processor system, such as a camera system, for example, generally comprises a central processing unit (CPU) 910, for example, a microprocessor, that communicates with an input/output (I/O) device 920 over a bus 990. The CMOS imager 930 also communicates with the system components over bus 990. The computer system 900 also includes random access memory (RAM) 940, and, in the case of a computer system may include peripheral devices such as a removable memory 950 which also communicates with CPU 910 over the bus 990. CMOS imager 930 is preferably constructed as an integrated circuit which includes pixels containing a photosensor, such as a photogate or photodiode. The CMOS imager 930 may be combined with a processor, such as a CPU, digital signal processor or microprocessor, with or without memory storage in a single integrated circuit, or may be on a different chip than the processor.
While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example, and not limitation. It will be apparent to persons skilled in the relevant art(s) that various changes in form and detail can be made therein without departing from the spirit and scope of the present invention. Thus, the present invention should not be limited by any of the above-described exemplary embodiments.