a illustrates a correction device in accordance with an embodiment described herein.
a illustrates a calibration device in accordance with an embodiment described herein.
One could use any type of image processor 10 to implement the various embodiments, including processors utilizing hardware including circuitry, software storable in a computer readable medium and executable by a microprocessor, or a combination of both. The embodiment may be implemented as part of an image capturing system, for example, a camera, or as a separate stand-alone image processing system which processes previously captured and stored images. Additionally, one could apply the embodiment to pixel arrays using any type of technology, such as arrays using charge coupled devices (CCD) or using complementary metal oxide semiconductor (CMOS) devices, or other types of pixel arrays.
As illustrated by
The overall process performed by image processor 10 is illustrated in
SV
corrected
=SV
acquired×Correction_factor.
The correction factor is determined using polynomial functions. The following polynomial of order n, referred to herein as the correction function, approximates the value of the correction factor:
Correction_factor=Qncoln+Qn-1coln−1+ . . . +Q1col1+Q0. (1)
Qn through Q0 are the coefficients of the correction function whose determination is described below. A different set of these Q coefficients is determined for each row of the array. The notation “col” refers to a variable which is the column value of the pixel determined with respect to an origin (0,0) located near the center of the array and scaled to a value between −1 and +1 depending on the pixel location in the array relative to the center (0,0). The letter “n” represents the order of the polynomial, so for embodiments using an order 5 correction factor, the correction factor would be represented as follows:
Correction_factor=Q5col5+Q4col4+Q3col3+Q2col2+Q1col1+Q0. (1a)
In equation (1), Q coefficients, Qn through Q0, are determined using polynomial functions. The following polynomials of order m approximate coefficients Qn through Q0:
Q
n
=P
(n,m)rowm+P(n,m-1)rowm−1+ . . . +P(n,1)row1+P(n,0) (2)
Q
n-1
=P
(n-1,m)rowm+P(n-1,m-1)rowm−1+ . . . +P(n-1,1)row1+P(n-1,0) (3)
. . .
Q
1
=P
(1,m)rowm+P(1,m-1)rowm−1+ . . . +P(1,1)row1+P(1,0) (4)
Q
0
=P
(0,m)rowm+P(0,m-1)rowm−1+ . . . +P(0,1)row1+P(0,0), where (5)
P(n,m) through P(0,0) are coefficients determined and stored during a calibration process discussed below. The notation “row” refers to a variable which is the row value of the pixel determined with respect to an origin (0,0) located near the center of the array and scaled to a value between −1 and +1, depending on the pixel location in the array relative to the center (0,0). The letter “m” represents the order of the polynomial.
As equations (1) through (5) illustrate, the polynomial approximating the correction factor has n+1 Q coefficients. Each Q coefficient is approximated by a polynomial having m+1 P coefficients. In the above equations, the first coefficients, Qn, P(n,m), P(n-1,m), . . . , P(1,m), and P(0,m), are referred to as leading coefficients.
After image processor 10 has determined the correction function for the row, it can then determine the correction factor for the pixel in the row. To do this, image processor 10 first determines the column number of the pixel in the row and scales the column number to a value between −1 and +1 (step 42) depending on the pixel location in the array relative to the center (0,0). Next, image processor 10 inputs the scaled column number into the correction function and evaluates the correction function (step 43) to determine the correction factor for the pixel.
a illustrates an embodiment of a correction device 44. Correction device 44 includes elements 45-51. Element 45 determines and scales a row number of the array. Element 46 retrieves stored coefficients from a memory. Element 48 generates a correction function for the row based on the scaled row position and the retrieved coefficients. Element 47 determines and scales a column position for a pixel in the row. Element 50 determines a correction factor for the pixel based on the pixel's scaled column position and the correction function generated by element 48. Element 49 determines a pixel value associated with the pixel, and element 51 determines and outputs a correct pixel value by multiplying the pixel value determined by element 49 by the correction factor determined by element 50. It should be appreciated that the elements 45-51 could be individual circuits or logic, one circuit, a combination of circuits or logic, etc.
The embodiment shown in
First, at step 52, a pixel array 2 is exposed to an evenly illuminated calibration image field. The calibration image should have characteristics that would cause every pixel in an ideal camera's pixel array to generate the same pixel value. For example, such a calibration image could be an evenly illuminated uniform field like the gray field of
As explained above, in a typical camera, capturing such a calibration image causes the pixels to generate pixel values that differ from each other and from what is expected. During a calibration phase, the calibration processor designates one of these pixel values as a reference value. The calibration processor then determines correction factors for each of the other pixels based on this reference value. These correction factors are proportioned to the reciprocal of the attenuation of the pixels; in other words, the amount that each pixel value must be multiplied by so that the pixel value equals the reference value.
For example, for 10-bit digital pixel values, each pixel generates a signal representing a number between 1 and 1024. One could generate a calibration field that when photographed by a camera causes a reference pixel to generate a pixel value equal to 512, the pixel's 50% saturation point. If exposing the camera to the same field causes a different pixel to output a pixel value equal to, for example, 450, then the calibration processor would correct that pixel value by multiplying it by
which corresponds to that pixel value's correction factor.
Steps 53, 54, and 56 of
After the calibration processor calculates correction factors for every pixel in the row, the system calculates a polynomial function approximating the row of correction factors (step 58). Procedures for finding the best-fitting curve to a given set of points are well known and include, but are not limited to, least squares fitting. As illustrated above in equation (1), the letter Q refers to the coefficients of this polynomial. At steps 60 through 66 the calibration processor repeats for each row of the array the process of acquiring the pixel values in the row (step 60), calculating correction factors for each pixel (step 62), and generating a polynomial function approximating the correction factors of the row (step 64).
When the processor completes these steps, it will have generated one polynomial for every row of the pixel array. For example, if the pixel array has 1024 rows, then the calibration processor generates 1024 polynomial functions. If each polynomial is of order five, then each polynomial will have six Q coefficients as in the example equation 1(a) above. In practice, lower order polynomial functions, for example, of order three may be used.
Each of the polynomials generated for each row will have a leading coefficient. At steps 68 and 70, the processor generates a polynomial that approximates these leading coefficients. This is done by fitting the leading coefficients to a curve using any procedure for finding the best-fitting curve to a given set of points, such as, for example, least squares fitting. The polynomial generated corresponds to equation (2) described above.
After generating a polynomial approximating the leading coefficient as a function, the calibration processor then repeats this process to generate polynomials approximating the other coefficients of the row polynomials (steps 72, 74, and 76). These polynomials would correspond to equations (3) through (5) set forth above.
For example, if order three polynomials were chosen to approximate the correction factors for each row, then each polynomial generated for each row would have four coefficients. In this case, the calibration processor would generate four more polynomials: a first polynomial approximating the leading coefficient, a second polynomial approximating the second coefficient, and two other polynomials approximating the third and fourth coefficients. This application uses the letter P to represent the coefficients of these polynomials, as illustrated above in equations (2) to (5). After the processor generates these polynomials approximating the coefficients of all the correction factor polynomials, the processor then stores the P coefficients in a memory (step 78) for use in subsequent pixel value correction procedures, such as the one illustrated in
a illustrates a calibration device 80 that includes device elements 82, 84, 86, 88, and 90. Element 82 acquires pixel values from a pixel array. Element 84 determines a reference pixel value. Element 86 determines correction factors for each pixel in the array based on pixels values acquired by element 82 and the reference pixel value determined by element 84. Element 88 determines for each row of the array a correction function approximating the correction factors for the pixels in the row. Element 90 determines a polynomial approximating the leading coefficients of each correction function as well as polynomials approximating the other coefficients of each correction function. Element 90 could also store the coefficients of the polynomials it determines. It should be appreciated that the elements 82, 84, 86, 88, and 90 could be individual circuits or logic, one circuit, a combination of circuits or logic, etc
Instead of generating and storing a single set of P coefficients, embodiments could generate and store multiple sets of P coefficients. Each set of P coefficients could be specific to a certain type of pixel, for example, a pixel of a particular color. Having multiple sets of P coefficients where each set regenerates correction functions customized to certain types of pixels can provide better pixel value correction. This could help to correct anomalies related to differences in color type or other anomalies related to differences in pixel position.
For example, to capture color images, digital cameras often use color filters with a pixel array. This causes certain pixels to only receive certain colors of light. One popular type of filtering arrangement is known as a Bayer color filter array.
During the calibration process, instead of calculating a single correction function approximating correction factors for every pixel in a row, the calibration processor could calculate two correction functions for the row. A first correction function could approximate the correction factors for one type of color pixel in the row; a second correction function could approximate the correction factors for a second color type of pixel in the row. For example, systems having Bayer color filters have four unique types of pixels. As such, the calibration processor could calculate four unique correction functions for every two rows, thus generating four sets of correction functions. From each of these four sets of correction functions, the calibration processor could then calculate and store four separate sets of P coefficients. During subsequent correction procedures, image processor 10 would be able to regenerate four different calibration functions for every two rows instead of one calibration function for every row.
Processor 100 operates with a pixel array using a Bayer color filter. As such, this embodiment uses four different sets of P coefficients to regenerate four different correction functions. Each one of these four correction functions corrects one of the four color types of pixels. For example, one set of P coefficients regenerates a correction function approximating correction factors for blue pixels; another set of P coefficients regenerates a correction function approximating correction factors for green pixels located in a row with blue pixels; another set of P coefficients regenerate a correction function approximating a correction factor for red pixels; another set of P coefficients regenerate a correction function approximating correction factors for green pixels located in a row with red pixels.
Each of the four sets of P coefficients contains four subsets of coefficients dedicated to approximating one of the four Q coefficients. For example, the set of P coefficients used to regenerate correction functions for the blue pixels has a subset of coefficients for regenerating the leading Q coefficient of the correction function, a subset of coefficients for regenerating the second Q coefficient of the correction function, and so on. In
Parts p3r, p2r, p1r, and p0r are registers that temporarily hold coefficients from RAMs P3 through P0. Register “0” represents a register that would be used with an additional register file RAM if one were to approximate each Q coefficient using a polynomial of order 5 instead of order four. Parts q4e, q4o, q3e, q3o, q2e, q2o, q1e, q1o, q0e, and q0o are registers that temporarily store Q coefficients calculated using the P coefficients.
Convert element 102 receives integer values of column and row numbers, converts them to floating point values, and scales them to values between −1 and +1 depending on location of a pixel in an array. Convert element 104 receives integer values of pixel values and converts them to floating point values. Poly4 evaluates a polynomial according to inputted coefficients co4, co3, co2, co1, and co0 and an inputted variable value from convert element 102. Element 110, labeled with an asterisk, performs multiplication; and element 112, labeled Control, provides various control signals. Convert element 106 converts floating point values to integer values. Methods of implementing these elements are well known, and one could implement any of these elements using all hardware, all software, or a combination of software and hardware.
The following describes a way of operating processor 100 to process pixel values in a row having red pixels in even numbered columns and green pixels in odd numbered columns. First, after processing a previous row of pixel values and before reading pixel values in a next row, the system 100 reads from registers P0, P1, P2, and P3 the P coefficients of the polynomial approximating the leading coefficient of the correction function for either the pixels in the even numbered columns of the next row or the pixels in the odd numbered columns of the next row. Next, system 100 receives value Y, which is the integer of the next row of a pixel array. Element 102 scales value Y to a value between −1 and +1, which corresponds to “row” variable of the equations described above. Based on the scaled Y value and the read P coefficients, poly4 calculates the value of the leading coefficient Qn of the correction function. The processor 100 then stores the leading coefficient in either register q4e or q4o depending on whether the leading coefficient corrects pixels in the even numbered columns or pixels in the odd number columns. Next, the processor 100 repeats this process of retrieving stored P coefficients and calculating Q coefficients until all of registers q4e through q0o contain their corresponding Q coefficients. At this point processor 100 will have calculated coefficients for a correction function associated with pixels in the even numbered columns and a correction function associated with pixels in the odd numbered columns.
Once registers q4e through q0o contain their appropriate coefficients, the processor 100 begins reading and processing the pixel values generated by the pixels in the next row. For each pixel in the next row, processor 100 first determines its column value (X), converts the column value to a floating point value, then scales it to a value between −1 and +1. This scaled column value corresponds to the “col” variable of the equations describe above. For even values of X, poly4 calculates a correction factor from the scaled value of X and from coefficients q4e, q3e, q2e, q1e, and q0e. Then the processor 100 multiplies this correction factor by the pixel value acquired from the pixel array to generate a corrected pixel value. In the illustrated embodiment, the pixel value is converted from an integer value to a floating point value by component 104 before the multiplication at component 106.
The embodiment of
In some embodiments, processor 10 could calculate the Q coefficients during a blanking period that corresponds to a period after reading and processing a previous row of pixels and before reading a next row of pixels. However, other embodiments could perform the various calculations at other points.
The pixel output signals typically include a reset signal Vrst taken off of a floating diffusion region (via a source follower transistor) when it is reset and a pixel image signal Vsig, which is taken off the floating diffusion region (via the source follower transistor) after charges generated by an image are transferred to it. The Vrst and Vsig signals for each pixel are read by a sample and hold circuit 261 and are subtracted by a differential amplifier 262, which produces a difference signal (Vrst−Vsig) for each pixel 230, which represents the amount of light impinging on the pixel 230. This signal difference is digitized by an analog-to-digital converter (ADC) 275. The digitized cell signals are then fed to an image processor 280 to form a digital image output. Image processor 280 could be implemented using various combinations of processing capabilities. Additionally, image processor 280 could perform the polynomial generation and correction functions described above with respect to
Although
System 300, for example a camera system, generally comprises a central processing unit (CPU) 302, such as a microprocessor for controlling camera operations, that communicates with one or more input/output (I/O) devices 306 over a bus 304. The imager device 208 can communicate with CPU 302 over bus 304. Processing system 300 may also include random access memory (RAM) 310, and removable memory 314, such as flash memory, which also communicates with CPU 302 over bus 304.
As mentioned above, embodiments may include various types of imaging devices, for example, charge coupled devices (CCD) and complementary metal oxide semiconductor (CMOS) devices, as well as others.
The above description and drawings illustrate various embodiments. Although certain embodiments have been described above, those skilled in the art will recognize that substitutions, additions, deletions, modifications and/or other changes may be made. Accordingly, the invention is not limited by the foregoing description of example embodiments.
Number | Date | Country | Kind |
---|---|---|---|
0617001.3 | Aug 2006 | GB | national |