Patent Grant
8031262
This invention is generally related to solid state cameras, and more particularly to techniques for determining exposure parameters in such cameras.
Solid state cameras, just like the conventional film camera, are limited in their ability to take pictures which faithfully replicate the full range of colors and brightness in a scene. This is because natural scenes exhibit a wide dynamic range, i.e., some regions of a scene are very bright while others are very dark. As a result, conventional solid state cameras, and particularly consumer products such as digital cameras and video cameras, have a number of adjustable exposure parameters that control the sensitivity of a camera imager. The best pictures are usually taken after the camera's exposure parameters have been adjusted according to the amount of light in the scene. For instance, if the scene is relatively bright, then the exposure, e.g., the period of time the camera imager is allowed to “sense” the incident light, is accordingly reduced so as to better capture the brightness variations in the scene. In conventional solid state cameras, a separate light meter sensor and associated circuitry are used to quickly give an immediate luminance reading of the scene prior to adjusting the exposure parameters and then taking the picture. However, both the light metering circuitry and the camera imager must be calibrated, at the time of manufacturing the camera, to a reference light source. Otherwise, the technique may not yield the proper exposure parameters.
There is a limited conventional technique for determining the optimal exposure that does not use a separate light metering circuit. In that technique, the camera is equipped with a means for providing a histogram of the captured scene at a given exposure setting. The histogram shows a distribution of pixel values obtained by the imager at the selected exposure setting. A person can then manually change the exposure setting and then visually evaluate another histogram of the scene using the new exposure setting. The exposure setting is repeatedly adjusted in this way until the optimal distribution of pixels has been obtained, and then the picture is taken using this optimal exposure setting. This technique suffers, however, when implemented in commercial solid state cameras, because it is too slow and is not automatic for the average consumer who likes the point and shoot convenience of automatic cameras.
According to an embodiment of the invention, a method is disclosed for automatically generating a final set of exposure parameters for a solid state camera having a camera imager, without using a light metering circuit separate from the camera imager. An iterative automated search methodology is used to arrive at the final set of exposure parameters from an initial exposure setting, and sample captures of the scene are evaluated at each trial exposure setting.
In a particular embodiment, the method of generating the final exposure setting includes selecting one of a number of predetermined exposure settings as a current exposure setting for the solid state camera. A captured scene is then generated by the camera imager using the current exposure setting. In response to the captured scene being underexposed or overexposed, another one of the exposure settings is selected to be the current setting according to the automated search methodology. The two latter steps are repeated until the captured scene is neither underexposed or overexposed. The search methodology performs a coarse granularity search so long as the captured scene is either grossly overexposed or grossly underexposed, and a fine granularity search otherwise.
Other features and advantages of the invention will be apparent from the accompanying drawings and from the detailed description that follows below.
The invention is illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements and in which:
An embodiment of the invention is directed to a method for automatically determining an optimal exposure setting prior to taking each picture using a solid state camera. The method may be used to determine the proper exposure setting when initiated by a person pressing a camera's shutter button down at least half-way. The final setting is selected from a number of previously determined exposure parameters using a multi-tiered automated search methodology. These parameters may be general and not specific to any particular camera, or they may be customized for the particular camera. The individual exposure parameters can be determined by those of ordinary skill in the art by conventional exposure calculations. Several tests are given here to evaluate the effectiveness of each selected exposure setting, based on a statistical analysis of a sample capture of the scene using that setting. A technique is given that computes the expected noise level for various exposure settings, based on relationships between noise and exposure parameters determined at each power-up of the camera. Use of this expected noise level helps achieve speedy and accurate determination of the optimal exposure.
In these embodiments, the invention presents a further advantage in that it reduces manufacturing cost of the camera by eliminating a light metering circuit separate from the camera imager. Also, use of the camera imager for both capturing the final picture and determining the optimal exposure eliminates calibration steps that would normally be required to characterize the light metering circuit and the camera imager at the time of manufacturing the camera. This is in part because the sample captured scenes used to determine the exposure setting are obtained through the same imaging data path for the final picture. Thus, the signal level in the final picture will be the same as the signal level in the sample captured scene and is, therefore, properly accounted for when determining the optimal exposure setting. An additional advantage of certain embodiments of the invention is that they enable the signal to noise ratio of the final picture to be optimized for ambient temperature conditions, so that the full capability of the camera is realized.
The imager 108 and the optics 104 are under the control of automatic exposure control block 128. The automatic exposure control block 128 evaluates digitized pixel values for one or more sample captured scenes, and in response determines the appropriate exposure setting. Each exposure setting is defined by a number of exposure parameters. These include aperture size for the optics 104, one or more integration times applied to the photocells 112, a gain value (normally an analog gain value provided to the analog processing 116), and a flash signal to a strobe 132 used to further illuminate the scene. The exposure setting may be a combination of one or more of these parameters and perhaps additional parameters that may need to be controlled to provide the best quality picture, as recognized by one of ordinary skill in the art.
The integration time defines the amount of time that a photocell 112 is allowed to detect incident light. Depending on the particular technology used for the photocell, the integration time may be implemented in various different ways. For instance, in a photocell 112 implemented as part of a CMOS active pixel sensor, the integration time is the interval between the moment at which a photodiode is isolated, such that its voltage is allowed to decay in response to the incident light, and the point at which the photodiode voltage is read by external circuitry.
The gain parameter sets the analog voltage and/or current gain to be applied to the pixel values prior to their being digitized by the A/D converter 120. In addition or as an alternative to analog gain, a digital gain applied by the digital processing 124 may be controlled by the gain parameter.
The aperture parameter controls the amount of incident light that enters the optics 104. A wide range of different automated aperture mechanisms may be used to provide the desired range of F-stops. Alternatively, the aperture may be fixed as in certain low-cost consumer cameras.
The analog processing 116 may include correlated double sampling circuitry, as well as any gain and filtering needed to translate the analog pixel values into the proper input values required by the A/D converter 120. The output range of the A/D converter 120 is typically fixed, such as 0-255 for an 8-bit converter, and the entire range of analog pixel values are mapped into digitized pixel values in this range. The digital processing 124 may be used to format the digitized pixel values into a form accepted by the automatic exposure control block 128. Normally, the exposure control techniques of the invention, and in particular those in which an expected noise value is computed, are applied to raw pixel values, which are not yet subjected to dark current noise reduction or any image processing algorithms. However, it is also possible to use pixel data that has been converted to luminance or other color filter array interpolated formats.
The automatic exposure control block 128 may be implemented as processor instructions on a machine-readable medium such as a semiconductor memory, as dedicated hardwired logic for greater speed of execution, or as a combination of the two. In a particular embodiment of the invention, the imager and the optics form an electronic camera, such as a digital camera, while the exposure control block 128 is implemented by software loaded into a separate data processing device which is not exclusively a stand-alone camera, such as a personal computer. Alternatively, the exposure control block may be integrated in the electronic camera.
A particular methodology to be followed by the exposure control block 128 is shown as a flow chart in
When the user has aimed the camera at the desired scene and starts to depress the shutter button in step 303, operation continues with step 304 in which a sample window of pixels having a camera system-defined position and size is defined. The sample window will initially encompass all of the scene. Operation then continues with step 308 in which the initial capture is made with exposure parameters associated with a default exposure setting/illumination level. The image from this captured sample window is then histogrammed. Operation continues thereafter with any one of steps 312, 320 or 336. For this example only, operation continues with step 312, in which the test is whether the histogram data is “centered out.” An example of what is meant by centered out is shown in
If the test in step 312 is true, then this means that the imager's dynamic range is far too small to capture the whole scene's dynamic range. Thus, the current sample window may not be the best window to determine the optimal exposure setting for this particular scene. In this case, operation will proceed with step 316 in which the sample window is reduced to concentrate effort on determining a final exposure for the main subject, which is likely positioned in the center of the scene. This change may be beneficial in that it might exclude certain peripheral elements in the scene which may be less important to render with detail, such as, for instance, the sun in the sky. For instance, making the sample window smaller and centered over the captured scene is generally appropriate for most consumer applications, as consumers tend to center the subject of interest when taking a picture. The reduction in window size is allowed until the size reaches a predetermined minimum. After changing the window, operation then loops back to step 308 where a histogram is performed of the new sample window and the test in step 312 is repeated.
If the test in step 312 is not true, then the next test may be step 320 to determine whether the histogram data is “centered in,” as shown in
Returning to step 324, if the capture has few, if any, mid or dark tones, such as when the minimum histogram value is greater than the aim mean, then the image is overexposed. In a particular embodiment of the invention, this aim mean is 18% of the maximum digitized signal range, i.e., noise value+(2N−1−noise value)*0.18. This is based on the assumption that the optimal exposure setting for the scene is the same as that needed to properly expose an equivalent 18% gray card, under the same lighting as the scene being captured. If overexposed, then operation proceeds with step 332, such as by performing a binary chop of the current index range (between top and bottom) as in table 404a, and then resetting the range, as shown in table 404b (
Returning now to decision step 320, if the centered in test is true, then the current exposure setting resulted in a capture that is neither grossly overexposed or grossly underexposed. In addition, if preceded by test 312 for “centered out,” this will also mean that any clipping at the outer limits of the histogram is either not an issue or has been addressed to the best of the system's ability. As a result, the search methodology for the next exposure setting changes from coarse granularity to fine granularity, beginning with step 336.
In step 336, the histogram is further tested to determine whether the histogram mean is within the allowable tolerance of the aim mean. If this test is true, then the optimal exposure setting has been found, such that the final picture may be taken using this setting. If, however, the test in step 336 is not true, then operation continues with step 340 in which the mean of the current histogram and the mean of a histogram of a previous capture are compared to the aim mean. If the aim mean is straddled by the mean values of the current and previous captures, then operation continues with step 344 in which the exposure setting that yielded the histogram mean being closest to the aim mean is chosen to be the optimal setting. The straddling of the aim mean may occur if the captured scene, and in particular the sample window, has exceedingly high contrast. The tolerance around an 18% aim mean value may be selected by one of ordinary skill in the art following an error budget analysis to determine the best exposure tolerance bands. For digital cameras having imagers built using metal oxide semiconductor (MOS) fabrication processes and 8 bit digitized pixel values, an aiming for less than ¼ exposure value (EV) error and given a black level (noise floor) of 54 A/D units, an aim mean of 90 units may be selected with a tolerance of +/−6 A/D units.
If the outcome of the straddle test in step 340 is false, then the histogram mean using the current exposure setting is compared to the aim mean of the histogram. If the histogram mean is greater than the aim mean, i.e., overexposed, then the exposure setting index is incremented to the next adjacent (higher) inferred illumination level and its corresponding exposure setting. If the histogram mean is less than the aim mean, i.e., underexposed, then the index is decremented to the next adjacent (lower) inferred illumination level and its corresponding exposure setting (see
Thus, with the current setting at index 10 yielding an overexposed scene, a binary chop is performed to select the next setting at index 30, which is halfway between a current setting and the bottom of the table. The scene is then captured again using the new current setting at index 30. Note that the top boundary of the range of exposure settings in Table 404(b) is now at index 10. If the current exposure setting at index 30 results in an underexposed scene, then the binary chop will pick the point halfway between index 10 and index 30, which is index 20. In Table 404(c), the current setting is now at index 20. Assume now that the current setting at index 20 results in a capture of the scene having a centered in characteristic, as determined by its histogram. This means that the captured scene is only mildly over- or underexposed, such that the binary chop algorithm should be abandoned in favor of an incremental step to the next adjacent exposure setting. Thus, if the histogram data indicates that the current captured scene is still underexposed, then the exposure setting index is decremented from 20 to 19, as shown by the pointers in Table 404(d). Once again, if the captured scene using the current setting at index 19 is still underexposed, then the setting index is decremented to 18. Finally, if the histogram mean of the captured scene obtained using the current setting at index 18 is within the tolerances surrounding the aim mean, then the optimal exposure setting is found, as indicated in Table 404(e).
The inventors have discovered that it is useful to switch from coarse granularity to fine granularity, while searching for the final exposure among a number of predetermined exposure settings, when the current capture of the scene becomes under- or overexposed at a relatively mild level. Otherwise, continued use of coarse granularity for selecting the next exposure setting may not converge to the final setting. The failure of a course granularity search such as the binary chop may occur because the mean of the pixel values in each capture shifts with different exposure levels. When the scene dynamic range exceeds what a camera may capture, pixels will be clipped at the maximum and minimum A/D boundaries. As different exposures change the amount of incoming light, some clipped pixels at one A/D extreme will no longer be clipped. This, in turn, affects the histogram's mean value which is an input parameter of the binary chop technique. Because this input parameter changes in an inexact, unpredictable manner with different exposures, the binary chop will fail to converge for some scenes.
The embodiments of the invention described above utilize the concept of an aim mean that is compared to a histogram mean to help determine when the search methodology should switch from coarse granularity to fine granularity and also when the captured scene is over or underexposed. In a particular embodiment of the invention, the aim mean is replaced with a “dynamic” aim mean that is computed as a function of each exposure setting. A dynamic aim mean as defined here is a noise-dependent variable that is computed for each captured scene, based on the current exposure setting. The possible sources of noise that may be taken into account when determining the dynamic aim mean are illustrated in
where the Dynamic Mean Noise is defined as the expected mean of the noise floor (a function of integration time and gain, see below). In the example given in this disclosure, N=8 bits.
The inventors have determined that the final exposure setting for taking a picture using a digital camera may be found relatively quickly using the above-described automated methodology when the histogram mean is compared to a dynamic aim mean computed for each given exposure setting. In a further embodiment of the invention that uses the dynamic aim mean, mathematical relationships that define a noise variable as a function of different exposure parameter variables are determined. These relationships may be linear (y=ax+b, where y is noise and x is the exposure parameter) or may alternatively be of higher order if needed to more accurately describe the relationship. They can be determined by fitting a curve or line to the measured test data, as shown in the example of
dynamic mean noise∝a3*T_integration+a5*gain+b5
where b5=b1+b2 and where proportionality constants have been omitted. By using such predetermined mathematical formulas to determine the noise floor, the dynamic mean noise, and the dynamic aim mean as a function of each trial exposure setting, a more accurate determination of the exposure setting may be obtained. Use of such a technique also allows the dynamic range of the scene to be mapped onto the camera's available dynamic range.
To summarize, various embodiments of the invention as a method for determining a final exposure setting automatically for a solid state camera without a separate light metering circuit have been described. In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. For instance, the exposure determination techniques described above may be applied to a wide range of solid state cameras, including video cameras. Also, the invention is not limited to the centered in and centered out tests described above. One of ordinary skill in the art after having read this disclosure may be able to develop alternative tests for determining when a series of captures change from being grossly overexposed or grossly underexposed to being only mildly so. The specification and drawings are, accordingly, to be regarded in an illustrative rather that a restrictive sense.
This application is a continuation of Ser. No. 12/146,218, filed on Jun. 25, 2008, entitled “Determining a Final Exposure Setting Automatically for a Solid State Camera Without a Separate Light Metering Circuit”, which is a continuation of Ser. No. 10/304,838, filed on Nov. 25, 2002, now U.S. Pat. No. 7,403,222, which is a continuation application of Ser. No. 09/294,851, filed on Apr. 20, 1999, now U.S. Pat. No. 6,486,915.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4329029 | Haskell | May 1982 | A |
| 4635126 | Kinoshita | Jan 1987 | A |
| 4748506 | Hieda | May 1988 | A |
| 5053888 | Nomura | Oct 1991 | A |
| 5146258 | Bell et al. | Sep 1992 | A |
| 5194960 | Ota | Mar 1993 | A |
| 5309243 | Tsai | May 1994 | A |
| 5376964 | Soga et al. | Dec 1994 | A |
| 5450502 | Eschbach et al. | Sep 1995 | A |
| 5544258 | Levien | Aug 1996 | A |
| 5745808 | Tintera | Apr 1998 | A |
| 5751354 | Suzuki et al. | May 1998 | A |
| 6486915 | Bell et al. | Nov 2002 | B2 |
| 6630960 | Takahashi et al. | Oct 2003 | B2 |
| 6952233 | Nagano | Oct 2005 | B2 |
| 7053954 | Canini | May 2006 | B1 |
| 20010013903 | Suzuki et al. | Aug 2001 | A1 |
| Number | Date | Country |
|---|---|---|
| 0528433 | Feb 1993 | EP |
| 60204181 | Oct 1985 | JP |
| 5056341 | Mar 1993 | JP |
| 6098167 | Apr 1994 | JP |
| 7038801 | Feb 1995 | JP |
| 7303205 | Nov 1995 | JP |
| 9065074 | Mar 1997 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20100315531 A1 | Dec 2010 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 12146218 | Jun 2008 | US |
| Child | 12851235 | US | |
| Parent | 10304838 | Nov 2002 | US |
| Child | 12146218 | US | |
| Parent | 09294851 | Apr 1999 | US |
| Child | 10304838 | US |
