Patent Application
20090202167
The present invention relates in general to computing an output image on the basis of an input image, wherein the input image may be, for example, a still image or part of a video sequence.
A wide variety of image enhancement techniques is available for improving the image quality as perceived by observers of both still images and video sequences.
An important class of enhancement techniques is formed by contrast enhancement functions. An example of an algorithm, referred to as the Burt-pyramid algorithm, implementing a contrast enhancement function is disclosed in U.S. Pat. No. 5,717,789. This algorithm permits the synthesis of an original high-resolution image from component sub-spectra images without the introduction of spurious spatial frequencies due to aliasing. In accordance with the Burt-pyramid algorithm, an original image is split into a plurality of sub-images, with a hierarchy of separate component images. Each sub-image may be a Laplacian image comprising of different spatial frequency ranges of the original image plus a remnant Gaussian image.
Image enhancement typically results in changing the pixel values within the corresponding frequency ranges. An original image G0 can be enhanced into an enhanced image G0′:
G
0
′=g
0
D
0
+g
1
D
1
+g
2
D
2
+ . . . +g
n-1
D
n-1
+G
n, (1)
with gn≧1, g the pixel enhancement factor, and Dn the sub-images or derived sub-images in corresponding spectral frequency ranges of the original image. Dn can be understood as a set of sub-images derived from “primary” sub-images Gn. In the context of this disclosure, both Dn and Gn can be understood as sub-images, wherein Dn is derived from Gn, through down-sampling. Enhancing pixel values in all frequencies ranges, except for the lowest frequency range, can enhance both sharpness and contrast in smooth image areas, with little contrast.
However, changing the pixel values regardless of image content may result in overshoots, enhancing pixel values in areas with edges higher than desired and necessitating clipping or other measures to prevent image distortion, which generally results in poor image quality.
It is an object of the present invention to provide a solution for contrast enhancement of an input image that alleviates the above problem and prevents or limits the amount of overshoot and thereby the amount of clipping.
This object is realized by a method as defined in claim 1 and an apparatus as defined in claim 13.
The present invention proposes using the pixel values from the input image in a multi-scale approach for modifying pixels in the input image. In a method according to the present invention, the input image is split into multiple sub-images, each sub-image corresponding to a frequency band of the input image. Although typically all sub-images will be processed, the present invention is not limited thereto.
A pixel headroom signal is calculated by quantifying the margin available for enhancement of pixel values of pixels in the input image. Although the pixel headroom signal can be based solely on the input image, other variants that include the use of further information will be described below.
The pixel headroom signal is used to calculate pixel enhancement factors for pixels within respective sub-images of the multiple sub-images to be modified. Typically, such pixel enhancement factors will be computed for every pixel in the respective sub-image. The pixel enhancement factors are calculated in such a way that the margin as quantified by the pixel headroom signal is spread across the respective sub-images to be modified. In their turn, the pixel enhancement factors are used to modify the respective sub-images to be modified by enhancing pixel values in the sub-images, using the corresponding pixel enhancement factors. The modified sub-images are subsequently used to generate the output image.
By using a multi-scale approach and explicitly taking into account the pixel headroom available for pixels in the input image by spreading the available pixel headroom across the sub-images through the use of the pixel headroom signal in the pixel enhancement factor calculation, it is possible to enhance the sub-images as well as the output image, without or with only a limited amount of overshoot, thereby requiring no or only a limited amount of clipping. The present invention therefore provides accurate clipping control.
By enhancement using a multi-scale approach, the available headroom can be used to enhance features within particular sub-images. Individual sub-images can be favored by allocating a proportionally large amount of headroom to them, thereby allowing the use of larger pixel enhancement factors within the favored sub-images while maintaining the same level of control.
In a preferred embodiment, the sub-images are modified consecutively and a pixel headroom signal is calculated for each sub-image to be modified. Moreover, the pixel headroom signal calculated for a respective sub-image being modified takes into account pixel headroom (margin) consumed as a result of previously modified sub-images. Consequently, the margin consumed in generating a modified sub-image will no longer be available for modifying further sub-images. In this manner, pixel headroom can be distributed by allocation of pixel headroom over sub-images as well as through prioritization of sub-image processing. Meanwhile, the ability to prevent or limit the amount of overshoot is preserved. In addition, headroom not consumed in enhancing previously modified sub-images is available for enhancement of further sub-images.
In another preferred embodiment, the order of sub-image processing is established by correlating the calculated pixel headroom signal with pixel values in respective sub-images. In this manner, the potential for enhancement in the respective sub-images is established in order to decide on the order of sub-image processing. Based on the pixel headroom signal, a variety of processing orders can be selected, such as a greedy enhancement algorithm wherein sub-images with the largest potential for enhancement are enhanced first. Alternatively, a more subtle enhancement algorithm can be envisaged wherein the sub-images in which only minor enhancements are possible are enhanced first, thereby safeguarding this enhancement and resulting in a more balanced enhancement over the sub-images as compared to the greedy enhancement algorithm.
In a further embodiment, the order of sub-image processing is fixed, starting with the sub-image comprising information in the highest frequency range of the input image to the sub-image comprising information in the lowest frequency range of the input image. As a result, the high-frequency bands are favored, which results in enhanced image sharpness.
In a further embodiment, two pixel headroom signals are calculated, a pixel headroom signal quantifying the margin of the pixel values of respective pixels of the input image to a maximum pixel value and a further pixel headroom signal quantifying the margin of the pixel values of respective pixels of the input image to a minimum pixel value. Subsequently, the pixel enhancement factor is based on the pixel headroom signal when the pixel value of the pixel of the sub-image is positive, and is based on the further pixel headroom signal when the pixel value of the pixel of the sub-image is negative. This particular feature allows enhancement of bright details, such as highlights, in predominantly dark regions or, alternatively, enhancement of dark details in predominantly bright regions.
When changing pixel values in individual sub-images without precautionary measures, so-called halo artifacts may appear either as false-light emissions or as false shadows around high-contrast object edges.
In a preferred embodiment, the pixel headroom signal takes into account the spatial profile of the pixel headroom in a region around respective pixels by calculating a pixel headroom signal that quantifies the smallest pixel headroom in an area surrounding the respective pixels. As a result, the pixel headroom signal may be used to determine a regional maximum linear gain that will not result in clipping within a region. Consequently, enhancements within a region will be more continuous.
In a further preferred embodiment, the pixel headroom signal is filtered by using a different filter for each respective sub-image to be modified in order to calculate pixel enhancement factors that comprise frequency components within the same frequency range as the sub-image. In combination with the previous embodiment, which takes the pixel headroom within a region of a pixel of a sub-image into account, a substantially smoother frequency spectrum of the pixel enhancement factors can be obtained. The probability that halos are introduced within a sub-image is thus reduced.
In a further embodiment, the pixel enhancement factor is upper-bounded by a single maximum pixel enhancement factor value for all sub-images. When sufficient bandwidth is available, all modified sub-images will be enhanced by using this maximum pixel enhancement factor, thereby resulting in an enhancement resembling a global linear histogram stretch. As a result, “light-emission” artifacts that result from over-enhancing regions with an already substantial contrast are prevented.
In a further embodiment, the pixel enhancement factor is based on the maximum linear gain of a pixel in a sub-image without clipping, attenuated by a scaling factor α from within the range of 0 to 1. Consequently, only a fraction α of the available headroom will be used for the current sub-image, and a fraction of (1−α) of the available headroom will be available for enhancing the remaining sub-images. The “light-emission” artifacts would thus be attenuated as well.
In a preferred embodiment, the pixel enhancement factor for a pixel of a sub-image is set to attenuate the pixel value of the pixel, when the difference between the maximum and minimum pixel value of pixels of the sub-image within a pre-defined region around the pixel is below a threshold value. In doing so, noise resulting from e.g. DCT quantization can be reduced.
In a further preferred embodiment, the present method is used to enhance a color image. In order to enhance a color image, the luminance component of the color image is used as input image and is split into sub-images. After processing, the output image is an enhanced luminance component of the color image. Subsequently, an enhanced color image may be formed by scaling the RGB components of the color image, using a scaling factor determined by the ratio of the enhanced luminance component over the luminance component. Use of this particular method renders it possible to substantially preserve color saturation of the original color image.
The present invention further relates to a computer program product tangibly embodied in a record carrier, the computer program product comprising instructions which, when executed, cause a processor to perform the steps of any one of the methods of claims 1 to 12.
The present invention further relates to an image signal comprising pixel values forming an input image and a set of pixel enhancement factors allowing processing of the input image in accordance with any one of the methods of claims 1 to 12.
These and other aspects will be further elucidated and described with reference to the drawing in which:
Throughout the drawings, identical reference numerals refer to the same elements, or to elements that perform the same function.
Three steps are required to generate a modified sub-image:
a step 110 of calculating a pixel headroom signal,
a step 115 of calculating pixel enhancement factors, and
a step 120 of modifying the respective sub-image on the basis of the pixel enhancement factor.
The present invention involves a step 110 of calculating a pixel headroom signal. The pixel headroom signal quantifies the margin available for enhancement of pixel values of pixels in the image. The available margin is spread across sub-images in such a way that, when pixels in the respective sub-images are modified within their respective fraction of the pixel headroom, the combined pixel values of the modified sub-images will not overshoot. Consequently, there is no need for clipping or saturation in order to prevent image distortion in the output image. Alternatively, a small amount of clipping may be allowed to improve the perceived enhancement. In a first variant, a single pixel headroom signal is calculated for the image and the available headroom is subsequently distributed across the various sub-images, for example, by allocating a predefined fraction of the entire pixel headroom for enhancing the respective sub-images. This work mode has the disadvantage that headroom which is not consumed in enhancing a particular sub-image is lost.
In a second variant, which is particularly useful when sub-images are processed sequentially, a pixel headroom signal is calculated before processing each sub-image to be modified, and the respective pixel headroom signals account for pixel headroom consumed in previously modified sub-images. As a result, headroom that is not consumed in enhancing one or more sub-images can be used for enhancing further subsequent sub-images.
In a third variant, the pixel headroom signal calculated before processing is subsequently used as a basis for establishing the potential for enhancing the respective sub-images. As a result, the pixel headroom signal can be used to determine the preferred order of sub-image modification in an image-dependent manner. Although the third variant is more computationally intensive, it allows prioritization of the enhancement of individual sub-images.
The embodiment shown in
The pixel headroom signal in this variant quantifies the margin available for enhancing pixel values of pixels in each respective sub-image. The modification of pixel values in the image as shown in
The present invention has for its object to prevent or limit the amount of overshoot, and thereby reduce or minimize the necessity of clipping. The amount of pixel headroom consumed in modifying the first sub-image will affect the amount of headroom available for enhancing further sub-images. It should be noted that the flow chart in
The next step 115 comprises calculating the pixel enhancement factors which, for pixels within the sub-image, are based on the pixel headroom signal calculated for the sub-image. In this embodiment, the pixel enhancement factors are dependent on the pixel headroom signal calculated for the image as well as on pixel headroom used in previous enhancements of other sub-images. Although all pixels are enhanced in this embodiment, it is possible to use masks and enhance only certain areas or segments of a sub-image without departing from the scope of the present invention.
In this embodiment, the pixel headroom signal is used to establish the maximum linear gain for a pixel of a sub-image that does not result in clipping. Based on this information, it is possible to calculate a pixel enhancement factor for a particular pixel of the sub-image. The pixel enhancement factor can be calculated by using the pixel headroom signal for the pixel and the respective pixel value of the sub-image. If a particular amount of overshoot is tolerated, it is possible to choose a higher pixel enhancement factor than is allowed on the basis of the pixel headroom signal. It will be evident to the skilled person that this will require further measures such as (soft-)clipping in order to limit visual image distortion.
The third step 120 comprises modifying the sub-image of the multiple sub-images by using the calculated pixel enhancement factor. In practice, this involves the use of pixel enhancement factors of respective pixels in the sub-image and enhancement of pixel values of the respective pixels, using the corresponding pixel enhancement factors by multiplying them.
Once a modified sub-image is generated, the process of modifying sub-images can be repeated in a step 125, until all sub-images are processed and/or modified in step 125. Finally, the output image 145 is generated by combining the modified sub-images. In one embodiment, this can be accomplished through addition of all modified sub-images.
Although the flow chart in
When sub-images are processed iteratively and a pixel headroom signal is calculated every iteration, the order of sub-image processing is relevant. Pixel headroom consumed during modification of a sub-image will no longer be available for modification of further sub-images. It may therefore be important to prioritize sub-image processing.
In a first variant, prioritization is image-independent. Sub-images are processed in an order ranging from high-frequency sub-images to low-frequency sub-images. As a result, the maximum amount of headroom is available for enhancing high-frequency sub-images, and the remaining headroom is available for low-frequency sub-images. Consequently, the high-frequency sub-image transitions can be enhanced more, resulting in a sharpening effect.
In a further variant of the embodiment shown in
To maintain color saturation of the color image, it is preferable to enhance color images by determining a scaling factor based on the luminance information in the image, and scaling the respective RGB components of the image accordingly. As a result, color saturation is substantially preserved.
A further preferred embodiment of the present invention has for its object to provide contrast enhancement and meanwhile prevent clipping and also to reduce “light-emission” artifacts that would result from enhancing regions with edges of high contrast. To this end, this embodiment sets out to enhance all frequency bands by the same amount, provided that sufficient headroom is available. Consequently when a region has sufficient headroom, this will result in mean-preserving global linear histogram stretching, an operation that will not cause “light-emission” artifacts.
In regions where there is insufficient headroom, this embodiment proposes iteration over all spatial scales from the finest scale (highest spatial frequency) to the coarsest scale (lowest spatial frequency), the latter scale representing the average luminance (the DC component) of the image, thereby favoring the high-frequency sub-images. Not all sub-images need to be included in the processing operation and, in one variant, the DC component is preserved in an unchanged form. Moreover, as described below in more detail, the pixel enhancement factors are generated in such a way that the variations in pixel enhancement factors show a smooth behavior within a sub-image, thereby effectively reducing “light-emission” artifacts within sub-images.
First, the input image 100, indicated by V in the equations, is split in a step 105 into n sub-images Bi. In this particular embodiment, the individual sub-images are obtained through consecutive low-pass filtering, using a spatial low-pass filter FLPi for scale i.
VLPi=FLPiV (2)
wherein denotes a spatial convolution. After low-pass filtering, the frequency band Bi is extracted by subtracting subsequent low-pass filtered versions of the video.
B
i
=V
LP
i
−V
LP
i-1 (3)
In this particular embodiment, the processing order of the sub-images is predetermined from high to low and the first pixel headroom signal is computed first for the highest frequency sub-image. In this particular embodiment, two pixel headroom signals will be computed Ĥupi and Ĥdoi for each sub-image. In order to establish these two pixel headroom signals, two intermediate signals are computed first: Hupi and Hdoi, see equations (4) and (5). Hupi(x,y) signifies the headroom for a pixel at position (x,y) in the image towards the maximum pixel value Pmax and Hdoi(x,y) signifies the headroom for a pixel at position (x,y) towards the minimum pixel value Pmin. The respective headroom signals are used to compute different pixel enhancement factors for positive and negative pixel values of the present sub-image. This embodiment can thus emphasize dark details in substantially bright areas and emphasize bright details in substantially dark areas. If a single overall pixel headroom signal were used, this would not be possible due to lack of headroom on one side.
In order to calculate the pixel headroom signal, the present embodiment uses an intermediate signal VENHi-1 that represents the intermediate result of the contrast enhancement up to scale i. VENHi-1 comprises previously modified sub-images as well as still unmodified sub-images and can be seen as an intermediate output image after processing i sub-images. V0ENH is initialized as the original image V.
H
up
i(x,y)=ABS(Pmax−VENHi-1(x,y)) (4)
H
do
i(x,y)=ABS(Pmin−VENHi-1(x,y)) (5)
Subsequently, the pixel headroom signals Ĥupi and Ĥdoi are calculated. In this particular embodiment, the pixel headroom signals are regional pixel headroom signals that take a spatial region S around the respective pixels (x,y) into account. In order to generate Ĥupi and Ĥdoi for a pixel (x,y), the regional minimum of Hupi(x,y) and the regional maximum of Hdoi(x,y) are established for the region S. Subsequently, the pixel headroom signals are filtered to ensure that the pixel enhancement factors based upon the respective pixel headroom filters fall within the frequency range of the pixel values in the respective sub-image.
Ĥ
up
i(x,y)=FLPi(MIN(Hupi(S))) (6)
Ĥ
do
i(x,y)=FLPi(MIN(Hdoi(S))) (7)
For best results, the aperture S should be equal to or larger than the filter aperture of FLPi. In this particular embodiment, the spatial low-pass filter used to split the image into sub-images is also used to form the pixel headroom signals.
The calculated pixel headroom signals are subsequently used to calculate the pixel enhancement factor gi(x,y) for respective pixels within the sub-image. In this embodiment, the pixel enhancement factors are calculated by using either Ĥupi or Ĥdoi depending on the pixel value of the respective pixel in the sub-image.
g
i(x,y)=CLIP(0,α(Ĥdoi(x,y)/Bi(x,y),gmax) for Bi(x,y)<0 (8)
g
i(x,y)=CLIP(0,α(Ĥupi(x,y)/Bi(x,y)),gmax) for Bi(x,y)≧0 (9)
As can be seen from equations (8) and (9), the pixel enhancement factors are bounded by a maximum overall pixel gain gmax that is the same for all bands. Provided that sufficient headroom is available, the gains will be gmax for all pixels in all sub-images resulting in a mean-preserving global histogram stretch. However, if the headroom is insufficient, the pixel enhancement factor will be based on either Ĥdoi(p,l)/Bi(p,l) or Ĥupi(p,l)/Bi(p,l) which quantifies the maximum linear gain in a region that does not result in clipping.
The parameter α with a user-defined value between 0 and 1 allows distribution of the available pixel headroom across the various sub-images. Only a fraction α of the available headroom will be used for enhancements in each sub-image to be modified. By using a fractional pixel headroom allocation, the headroom is gradually distributed across all sub-images. In this particular embodiment, a fraction α of the maximum linear gain can be used for enhancing B1, a fraction of α (1−α) can be used for B2, i.e. provided that the fraction α was used for enhancing B1. If not all headroom allocated to B1 is used, a fraction α of the remaining maximum linear gain after enhancement of B1 will be used for enhancing B2.
The above mechanism limits the pixel enhancement factors within a sub-image. Moreover, through the use of a regional pixel headroom signal, which is additionally filtered to match the sub-image frequency band, the pixel enhancement factors show a smooth behavior in the frequency domain. This in turn prevents the creation of halo artifacts that may appear as false-light emissions and false shadows around high-contrast edges. Although this does not reduce halo artifacts that are already present in the input image, it does reduce the probability of introducing additional halo artifacts.
Once the pixel enhancement factors are calculated, the sub-image can be modified, resulting in modified sub-image {circumflex over (B)}i.
{circumflex over (B)}i=giBi (10)
When the modified sub-image is available, the enhanced image VENHi can be computed for use in processing subsequent sub-images, or, alternatively, when all sub-images have been processed to obtain an enhanced luminance image.
Although it is possible to use the above embodiment for processing the luminance component of a YUV color image and preserving the U and V components as is, this may result in unwanted de-saturation in bright areas, and over-saturation in dark areas. To preserve saturation, it is preferable to use the ratio of the enhanced luminance component over the original luminance component so as to scale the image in a constant saturation direction, for example, by converting the YUV representation into an RGB representation and scaling the individual RGB components.
R
ENH=(VENH/V)R (12a)
G
ENH=(VENH/V)G (12b)
B
ENH=(VENH/V)B (12c)
As the above embodiment favors sub-images comprising high-frequency components, it may also sharpen noise. This noise may be introduced in part by DCT quantization. DCT quantization artifacts often result in small differences between the minimum and maximum pixel values within a region of a sub-image.
In order to prevent enhancement of such artifacts, it is possible to detect the occurrence of such quantization noise and subsequently overrule the pixel enhancement factor calculation. In order to detect DCT quantization noise, one can determine the minimum and maximum pixel value within the region S around a pixel and determine whether the absolute difference between the minimum and maximum is below a threshold value. If the value is below a threshold, the pixel enhancement factor is set to a value lower than 1 in order to attenuate the effect, and if the value is above the threshold, the pixel enhancement factor is based on the available pixel headroom signal instead.
It will be clear to the skilled person that, instead of using a single threshold, a more gradual scheme can be implemented that uses multiple thresholds, or, alternatively, the difference of the minimum and maximum so as to implement a more gradual transition from enhancement to attenuation.
The above-described embodiment has for its object to realize similar results as those presented in
Although an input image may be split into sub-images of equally sized frequency bands, the present invention is not limited thereto. It will be evident that changing the size of the frequency band will affect the ratio of available headroom over frequency range, and may therefore require adaptation of the fraction of the available margin for enhancement over the various sub-images. Moreover, it is within the scope of the present invention to separate the DC-component of the image, treat this as a separate sub-image and optionally force the pixel enhancement factor of the DC-component to 1.
As mentioned before, the order of processing sub-images also affects the margin available for enhancement of individual sub-images. When the first sub-image is to be enhanced, all pixel headroom is still available and the entire fraction of pixel headroom available for enhancing the first sub-image may be used. By lowering the fraction of headroom allocated to the first sub-image, the “greedy” nature of the algorithm can be moderated.
As illustrated above, the present invention may be used to modify pixel values within both still images and/or video sequences. It can be embodied in a stand-alone image enhancement device, or integrated in other video-processing devices. These devices include those arranged to process video, and are not limited to devices for rendering, but may also include devices for capturing, storing and/or distributing images and video sequences.
The present invention may be readily integrated in a wide range of devices such as displays, video displays, digital still cameras, video cameras, video recorders, hard-disk recorders, television sets, video cards for personal computers, LCD-displays, plasma-displays, electrophoretic displays, mobile phones, personal digital assistants (PDAs), hand-held video-rendering devices, or hand-held game consoles.
The apparatus 600 comprises a splitting means 610 arranged to split an image into multiple sub-images, each one of the multiple sub-images comprising information from a respective frequency band of the input image 100.
The apparatus 600 further comprises a first calculating means 620 arranged to calculate a pixel headroom signal quantifying the margin available for enhancing pixel values of pixels in the input image 100, the pixel headroom signal being based at least in part on the input image.
The apparatus 600 further comprises a second calculating means 630 arranged to calculate pixel enhancement factors for pixels within respective sub-images to be modified in dependence upon the pixel headroom signal, such that the margin as quantified by the pixel headroom signal is spread across the respective sub-images to be modified, and a modifying means 640 arranged to modify the respective sub-images to be modified by using the pixel enhancement factors and the corresponding pixel values from the respective sub-images. In addition, the apparatus 600 further comprises a generating means 650 arranged to generate an output image 145 by using the modified sub-images. The output image 145 is sent to both a further database 665 and to a display device 695 for displaying the output image 145.
Optionally, an apparatus 600 may provide an input for an image signal 685 according to the present invention, in which the image signal comprises pixel values forming an image and a set of pixel enhancement factors allowing enhancement of the image, and the generated pixel enhancement factors use a method according to the present invention. Using this signal, the apparatus 600 may subsequently enhance the image comprised in the signal by directly using the pixel enhancement factors. Such pixel enhancement factors may be acquired by using e.g. equations (12a), (12b), and (12c).
The image signal 685 in
It will be evident to the skilled person that a method of modifying pixel values according to the present invention can be implemented on a dedicated hardware platform such as an Application Specific Integrated Circuit (ASIC), or on a general-purpose micro-processor in a personal computer (PC). In fact, the method may be implemented partially in dedicated hardware and partially in software running on a programmable processor.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims.
In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb “comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. Use of the article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements.
The invention can be implemented by means of hardware comprising several distinct elements and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
| Number | Date | Country | Kind |
|---|---|---|---|
| 06119613.5 | Aug 2006 | EP | regional |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/IB2007/053418 | 8/27/2007 | WO | 00 | 2/24/2009 |
