Patent Application
20070287218
The present invention may now be described, by way of example only, with reference to the accompanying drawings, in which:
As mentioned above, noise caused by variation of pixel response across the array of pixels may be a drawback of known image sensors. Because the NFV is narrowband, a first possible approach would be to deploy a fixed notch-filter at the frequency of variation of response, which in most cases may be the nyquist frequency. It can also be set at a chosen other frequency for special case image sensors.
While this approach may be the preferred option for some implementations, there may be some problems associated with it. In the world of video and stills imaging, an effective notch-filter is expensive, and loss of high-frequency data (blurring) is the inevitable result of a practical filter design. For color image processing, if notch-filtering is performed before the demosaic step, the Nyquist region is relatively smaller and blurring is intensified. If performed after demosaic, any prior noise reduction is compromised as described earlier.
A further possible option that may mitigate the blurring effect is the use of an adaptive filter. Again, this approach may be preferred for some applications but in general presents some difficulties: it is difficult to separate noise and data in regions of high luminance activity. It is difficult to develop a model on which adaptation can occur, other than in regions of low textural activity where the notch-filter would bring good results anyway. Thus, the NFV tends to survive around object edges in the scene, and cancellation is only partially effective.
A further weakness of the adaptive filtering approach may be that the notch-filter may be minimum-phase, to allow fade-in and fade-out without phase-shifting. A minimum-phase notch-filter at Nyquist is difficult to achieve (the fixed-response filter does not suffer from this restriction, as a phase-shift by a fixed distance is perfectly acceptable over an entire image).
A more advanced option that avoids these problems is the use of coring. Coring is a technique, which in itself is well known. It is applied to signal differences to drive differences by a certain amount towards, but not beyond, zero. The operation of a coring function is shown in
Coring does not involve neighboring pixel data, although neighboring data may have contributed to the formation of either or both of the two signals whose difference is subject to coring. Thus coring may be only applicable to minimum phase (co-sited) signal pairs. The function of coring is to drive differences by a certain amount towards, but not beyond, zero.
A common example of coring can be found in the peaking, or sharpening, of images. This signal processing step may be usually performed late in the image reconstruction chain giving a bandpass or highpass boost to image data, compensating for upstream lowpass mechanisms. As residual noise is also boosted by this process, coring can act as an effective late-stage noise reduction process, while still passing the larger signal swings which serve to sharpen image detail. Level settings may be subtle if visible image degradation is to be avoided, thus coring is not usually deployed in normal (low-noise) operation.
Cored peaking is only employed part of the time during operation of existing image sensors, and under normal operating conditions is only employed with subtle coring levels. Furthermore, peaking comes after noise-reduction and demosaic in the reconstruction chain, by which time the damage is done.
The use of peaked coring for NFV suppression has not previously been considered because the actual problem of how to remove noise caused by variation of pixel response across the array of pixels has not previously been considered. Furthermore, the coring levels for NFV suppression are much higher than that of cored peaking and may always be deployed. So the use of cored peaking for NFV suppression would be seen as too little, too late.
However, Applicant has realized that a coring method can in fact be applied to the NFV problem. This may hereafter be referred to as “nyquist coring”. In this case, the model of NFV is a gain error, and the coring level is made level-dependent, i.e. it is expressed as a percentage of the signal value. We may refer to this percentage value as the “adaptive coring level”. It can be varied within an image with no loss of generality. The adaptive coring level can further be modulated by the co-sited signal estimate to produce a “weighted coring level” used in the coring process.
This coring process may now be discussed with reference to
The estimate and the actual value from the central pixel site are input to a nyquist coring module 66. The difference between this estimate and the actual value at the site then represents the NFV, and is cored-out if below the set coring level.
The scope of the invention is not limited to any particular cositing filter design. One possible option is to simply calculate the average of the neighbors. However, an improved version can make use of an adaptive gradient analysis, which is illustrated in
A “stripe” in the north east direction is then calculated, defined as the absolute value of {(2*central pixel value)+(sum of north east pixel 58 and south west pixel 60)−(sum of north west pixel 56 and south east pixel 60)} and a “stripe” in the south east direction is calculated, defined as the absolute value of {(2*central pixel value)+(sum of north west pixel 56 and south east pixel 60)−(sum of north east pixel 58 and south west pixel 60)).
It can be seen that if the sum of the north-east and south-west pixels 58, 60 is greater than the sum of the north-west and south-east pixels 56,62, then the value of the north-east stripe may be greater than the value of the south-east stripe. Alternatively, if the sum of the north-east and south-west pixels 58,60 is less than the sum of the north-west and south-east pixels 56, 62, then the value of the north-east stripe may be less than the value of the south-east stripe.
Furthermore, a null stripe value is also defined based on a predetermined stripe threshold. This determines the sensitivity of the gradient analysis by setting an arbitrary level for the stripe detectors, above which it is considered a stripe to be detected. The orthogonal stripe, north-east stripe, south-east stripe and the stripe threshold are then sorted by magnitude. If the maximum value is the north-east stripe, the gradient analysis means 68 outputs a “NE” angle signal, while if the maximum value is the south-east stripe, the gradient analysis means 68 outputs a “SE” angle signal. Otherwise, the angle signal is output as a null value, indicating that no discernable gradient is present.
The output angle signal is then input to an adaptive cositing filter means or a cositing filter (70 in
It is to be appreciated that the above is only an example, and that the gradient analysis is not limited to comparison of only two diagonals. The stripe value in any determined direction can be found by comparison with any other direction, termed as a comparison direction. The comparison direction may usually be normal to the determined direction, but it does not need to be so. Also, as well as diagonal stripes that include the central pixel value, the stripes could be off-diagonal angles that include the central pixel value, other directions that do not include the central pixel value, and horizontal and/or vertical directions.
Next, the scale of the problem (how great is the maximum expected NFV at a given point) may be estimated throughout the image. As stated earlier, this is a complicated function of spatial index and of scene data. In a simple configuration a coring level 72, representing the envisaged worst-case NFV, can be set externally. However, this may inevitably result in over-attenuation of scene detail at the Nyquist frequency. Another approach is to maintain an internal estimate of NFV which maintains a running estimate of NFV magnitude and adjusts the coring level in response to different NFV conditions, as might be encountered in different regions of an image as described earlier. This allows an acceptable margin to be maintained between transmission of NFV and attenuation of scene detail.
An alternative and possibly simpler means to the same end is to use a “roughness” measure, derived from neighboring data (e.g. the difference between the brightest and darkest pixel) to inhibit coring in highly-textured regions. Here NFV is less noticeable, for example, “out in the open” (blue sky), where suppression is mandatory. A roughness measurement means or a roughness measurer 74 is illustrated in
The nyquist coring is carried out by coring means 66, on the basis of the cosited pixel data compared with the central pixel value 54. As mentioned above, the coring level 72 can further be modulated by the co-sited signal estimate to produce a “weighted coring level” used in the coring process.
An example coring process is illustrated in
The filter of this invention can be implemented in an image sensor and in any device incorporating the image sensor, for example, a digital still image camera, mobile communications device, optical pointing device such as an optical mouse, or a webcam, among others. Various improvements and modifications may be made to the above without departing from the scope of the invention. For example, the various “means” for performing the functions of the invention may typically comprise electrical circuits that are either programmed or programmable to implement digital filtering and signal processing algorithms. Any combination of hardware, firmware or software may be chosen as convenient. However, it may also be appreciated that the physical mechanisms may comprise other types of circuit or other types of transmission, in well known manners.
| Number | Date | Country | Kind |
|---|---|---|---|
| 06270057.0 | Jun 2006 | EP | regional |
