The invention relates to the decompression of image data and to the color management of image data.
For the last fifteen years, image compression algorithms have been using “vision pyramid” models to reduce visual quality loss during quantization. A typical image compression flowchart is illustrated on
A color gamut defines the complete set of colors an imaging display device is able to reproduce, or an image capturing device is able to capture, or that is contained within an item of visual content such as an image. Because of hardware limitations, many color devices are unlikely to provide a good reproduction of colors of images which have been processed in order to be correctly reproduced on other displays, for example with wider color gamut. A common case would be the comparison between colors of images being rendered on a CRT monitor and the same images coming out of a low quality color printer.
In order to ensure the best reproduction quality of images on different color reproduction devices having different color gamuts, several techniques have been proposed throughout the years, as those using so-called “Gamut Mapping” of colors. Gamut Mapping of colors ensures that the colors of a source device are transformed to be located inside the gamut of a destination device. Such an operation can be performed using a series of criteria that will minimize the impact of the transformation that inherently introduces changes in the colorimetry of images. Several algorithms have been developed to map source colors into destination colors, such as lightness mapping or chroma mapping, which represents a first direction towards Human Visual System (“HVS”) criteria, since the human eyes do process visual data through lightness and chroma considerations.
The CIE standardized a visual perceptually-uniform color space denominated CIE 1976 (L* u* v), also known as CIELUV, accounts for the psychophysics of human vision by constructing a three-dimensional color space that is substantially perceptually uniform. The coordinates of a color in CIELUV requires a color in well known CIE1931 XYZ to be normalized with respect to the XYZ coordinates of a “white point”, as defined in the art. One axis of CIELUV color space is denominated the CIE 1976 Lightness, noted L*, and is defined using a cube-root function in XYZ with a straight-line segment at the origin. The position of a color along the L* axis contains only information derived from its luminance as related to the reference white point. The other two axes of CIELUV are derived from the chromaticity of the image point related to the chromaticity of the reference white point. Multiplying the difference between the chromaticities of the image point and the reference white point by a value proportional to L* mimics the psycho-physical effect that causes darker colors to appear less chromatic, thus leading to perceptual uniformity in the CIELUV space. The cylindrical representation of the CIELUV color space is also useful. The L* axis remains unchanged while the plane defined by the two Cartesian chrominance axes is expressed in terms of a phase angle corresponding to the so-called hue H and a radius associated with the so-called chroma C. In 1976, the CIE also standardized another visual perceptually-uniform color space with characteristics similar to the CIELUV color space, denominated the CIE 1976 (L* a* b), or CIELAB, which also can be expressed in terms of a Cartesian coordinate system (L by a by b) or a cylindrical coordinate system (L by hue H by chroma C). In a perceptually-uniform color space, the three Cartesian bases are denominated luminance (L) and chrominance (C1, C2). The Cartesian chrominance plane may be also described in terms of two polar chrominance bases denominated chroma (C) and hue angle (H). Thus, chroma (C) is defined as the square root of the sum of the squares of the two chrominance values (C1 and C2) and hue angle (H) is defined as the arctangent of the ratio of the two chrominance dimensions (C2/C1).
Many color Gamut Mapping techniques tend to focus on 2D or 3D geometric solutions. Such techniques consist in reshaping/mapping parts of a source color gamut with either hard or soft color compression/clipping or extension algorithms so these reshaped/mapped parts can fit in a destination color gamut. Some algorithms use criteria in order to limit the visual impact of the mapping. Known criteria are for instance hue conservation, spatial homogeneity of mapping in color space and contrast preservation.
A typical 2D geometric mapping method would perform clipping in the CIEL*a*b* color space, more precisely in a L*C* subspace, where L is the lightness of a color and where C* represents the chroma of a color:
(a high C* value represents a high saturation level). Examples of hard and soft color clipping methods can be observed on
Some color gamut mapping algorithms have less visual impact than others depending on the criterion to be minimized. Generally, the criterion to be minimized is global and all pixels of the image to be color mapped are processed usually with the same algorithm. Such methods do not rely on local variations in the image, and are likely to generate inappropriate contrasts in certain cases (e.g. increasing quantization or acquisition noise in homogeneous areas).
On the opposite, different parts of an image can be color mapped differently depending on their local characteristics. The color gamut mapping methods using such information are referred to as “content-dependent”, notably “spatial frequency-dependent”. Human vision studies teach that a human observer tends to hierarchically process visual data in a scene, from coarse to fine elements. This behavior is generally modeled after a Gaussian pyramid, or any kind of signal processing tool that can distinguish and organize the content of an image from low to high spatial frequencies of distribution of color values within said image.
The document U.S. Pat. No. 6,961,477 discloses applying different color gamut mapping algorithms to different regions of an image according to the spatial frequency content of color values within this image. According to this document, the regions of the original image are spatially segregated into “busy” regions and “smooth” region and each region is associated with one or plural color gamut mapping algorithms. An important issue with this solution is that the color mapping of adjacent pixels having the same color value may be very different at the boundary of two regions, then introducing unacceptable artifacts. EP1098511, US2003/012427 and U.S. Pat. No. 6,516,089 illustrate other examples of color gamut mapping algorithms that depend on spatial frequency content of the images to map. In U.S. Pat. No. 5,450,216, such a spatial filtering exploits the differing spatial frequency regions of insensitive human visual response to both lightness and chrominance changes, according to the so-called Contrast Sensitivity Function (CSF), which concerns the lightness and is achromatic, and according to Chromatic Contrast Sensitivity Functions (CCSF), which concerns the chrominance according first to a red-green chroma, and secondly to a blue-yellow chroma, then allowing the mapping of colors in a manner that minimizes the human visual response both to the luminance and to the chrominance changes. For instance, as stated at column 8, lines 51-57, of this document, source colors are mapped in the direction of luminance only at low spatial frequencies to which human eyes are relatively insensitive to luminance variations but more sensitive to chrominance variations, then allowing a minimum change of chrominance.
The above-mentioned CSF and CCSF characterizing the Human Visual System (HVS) are very well known in the art.
An object of the invention is to improve such a method in view of saving computation resources.
For this purpose, the subject of the invention is a method of processing a compressed image into a color gamut mapped decompressed image, comprising the steps of:
The use of frequency transforms to organize image data shows an interesting relationship between image compression and content-dependent gamut mapping. As a matter of fact, using a common transform for both decompression and color gamut mapping advantageously save significant computation resources in a decoder. Moreover, as in many cases image data are converted into lightness-chrominance color spaces (typically YUV) before compression, color gamut mapping can also take advantage of such a conversion.
According to the invention, the spatial frequency analysis contained in the compressed image is used not only for decompression of images but also for the selection of an appropriate color gamut mapping algorithm. Computation resources are then saved because no specific spatial frequency analysis is required for the selection of an appropriate color gamut mapping algorithm.
Preferably, if the color values of each pixel are represented by at least a lightness or luma value and a chroma value, then, in said selecting step:
In color gamut mapping, lightness-chroma conversion is usually performed using the CIELab colorspace, but using the more compression-compliant so-called YUV color space is very advantageous here, as it fit well with compressed JPEG/MPEG image data streams. The YUV color space corresponds to the YCbCr color space, which was developed as part of the ITU-R BT 601 standard.
Preferably, if the color values of each pixel are represented by at least a lightness or luma value and a chroma value, if, within each block area, a spatial bandwidth of the spatial frequency distribution of lightness or luma values within said block area is defined and a spatial bandwidth of the spatial frequency distribution of chroma values within said block area is defined, said selecting step is performed for said block area according to the difference between the value of the contrast sensitivity function (CSF) for said lightness or luma spatial bandwidth within said block area and the value of the chromatic contrast sensitivity function (CCSF) for said chroma spatial bandwidth within the same block area.
Preferably, each compression block area comprises 8×8 pixels.
Preferably, the compressed image is compliant with JPEG or MPEG compression standard.
The method of processing is also applicable to series of images, as video sequences.
The invention will be more clearly understood on reading the description which follows, given by way of non-limiting example and with reference to the appended figures in which:
Although the illustrative embodiment is described herein with reference to the accompanying drawings, it is to be understood that the present principles is not limited to those precise embodiments, and that various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the scope or spirit of the present principles. All such changes and modifications are intended to be included within the scope of the present principles as set forth in the appended claims.
The present description illustrates the principles of the invention. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the present principles and are included within its spirit and scope. It will be appreciated by those skilled in the art that the block diagrams and flow charts presented herein represent conceptual views of illustrative circuitry embodying the present principles. The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. The appropriate software may be implemented as an application program tangibly embodied on a program storage unit.
The method according to the invention is adapted to process images that are compressed as illustrated on
The image to compress is divided into adjacent block areas and decomposed into coefficients showing the spatial frequency distribution of color values attached to the pixels of this image; a spatial part of the image around each pixel have to be remapped from the 2D image space to a 2D frequency space. A very well known way to accomplish this operation is to process a Discrete Cosines Transform (DCT) of the neighborhood of each pixel within each block area of pixels (DCT is notably used for Jpeg and Mpeg compression). The DCT transforms a matrix of color values to a matrix of coefficients representing the spatial frequency decomposition of the image.
The following well-known expression is used to perform this DCT transformation, where N represents the size of the matrix:
The N×N coefficient matrix obtained by this transformation represents the contribution of the spatial frequencies represented in
The DCT transform is an example of spatial to frequency transformation but other techniques of analysis in 2D frequency space can be applied such as Fourier transform, wavelets, other sub-bands defined by a filter bank, or others.
After quantization, a usual entropy coding is applied to reduce the size of quantized spatial frequency distribution coefficients such as to obtain the compressed image.
An embodiment of the method of processing this compressed image into at least one gamut mapped image according to the invention will now be illustrated in reference to
A plurality of color gamut mapping algorithms and a selection rule for a gamut mapping algorithm according to a spatial frequency distribution of color values within any given block area should be preliminarily defined.
The plurality of color gamut mapping algorithms that is proposed here is based on color clipping methods. As already explained, color clipping methods map colors along mapping trajectories that are anchored to an anchor point, in the mapping color space. These color mapping trajectories are generally straight lines. The anchor points generally belong to the so-called achromatic axis or L* axis (or “grey line”) of the color space. These anchor points can be chosen according to the shape of the source and destination gamuts such as described, for instance, by Montag and Fairchild in their paper entitled “Gamut mapping: Evaluation of chroma clipping techniques for three destination gamuts”, published in 1998 at the IS&T/SID Sixth Color Imaging Conference in Scottsdale. The way how the color is moved along such color mapping trajectories can be various. Simple mapping algorithms clip all colors that are outside the target color gamut onto the target color gamut surface along the color mapping trajectories. More complex color mapping algorithms compress linearly all colors along the color mapping trajectories by linear scaling such that they keep on the same mapping trajectories but are all inside the destination color gamuts. Still more complex, non-linear parametric functions apply specific, sometimes piece-wise defined scaling operations to the colors to move them along the defined lines inside the target color gamut. For example, Braun and Fairchild suggest in their publication entitled “Image Lightness Rescaling Using Sigmoidal Contrast Enhancement Functions” published in 1999 in the Proceedings of SPIE Electronic Imaging Conference (Eli99) a non-linear, S-shaped function. When mapping colors from a source gamut into a destination gamut, the above cited papers use straight lines as color mapping trajectories, variable anchor points and mapping functions that define how far a color on a mapping straight line needs to be moved. The mapping trajectories, the anchor points and the mapping functions can be fixed or adapted to the source and target gamut boundaries in a given mapping color space.
In the color gamut mapping algorithms that are proposed here in reference to
The selection rule of the color gamut mapping algorithm will use the CSF and CCSF. More details will now be given on these Sensitivity Functions in reference to
The main grounds of the selection rule of the color gamut mapping algorithm is adapted to preserve as much as possible the local contrast of the image to map: it allows determining whether lightness preservation should be a priority over chroma preservation or vice-versa. By analyzing the spatial frequencies of distribution of color values around each pixel of an image to map, i.e. locally, we can determine if some high frequencies are present or not and set a value of Fmax. This spatial frequency is preferably expressed in cycle per degree, considering the spatial frequencies of the image and also the viewing conditions as explained below. Then the difference between the CSF and the CCSF for this frequency Fmax is computed, and, from this difference, a range ΔL of variation of the position of the anchor point is calculated as explained below. In the homogenous parts of the image to map where the spatial frequency decomposition indicates very low frequencies for both lightness and chroma, the CCSF is higher than the CSF. In this case the HVS is more sensible to the chroma contrast than to the lightness contrast. The value of ΔL is then 0, meaning the anchor point is fixed and that a Node color Clipping algorithm (see
These two cases above correspond to the extreme values of the CSF and CCSF, respectively for the very low and very high spatial frequencies. The selection rule that is proposed in this embodiment sets the value of ΔL in a continuous way between these two opposite cases. It means that the selection rule is adapted to manage the low and high frequencies to preserve the local contrast but also all the frequencies between these opposite cases by a continuous process. Advantageously, all values of the spatial frequencies are managed according to the HVS, without discontinuous steps in the treatment, and without boundaries that introduce artifacts. It means that a continuous variation of ΔL from ΔL=Lmax−L0 to ΔL=0 allows to continuously pass from the Straight Clipping method to the Node Clipping method by using the variable anchor point concept.
The detailed computation of the range ΔL of variation of the position of the anchor point will be now detailed. In order to have both luminance and chrominance spatial frequencies influence contrast preservation term ΔL=0, it is proposed to define this term as a function ΔL(.) having the luma spatial bandwidth FDCTLum
In the case of a Discrete Cosine Transform F(x,y) of the image Im(i,j), generally speaking, a spatial bandwidth Wmax can be defined as the spatial frequency value for which the image signal has reached 95% (or a lower percentage) of its energy as follows:
ts corresponding value in cycle per degree (cpd) is defined as
where α is the angle of view of a pixel of the image in degree, provides information about a “locally representative” spatial frequency. Similarly, the values of a luma spatial bandwidth WDCTLum
Contrast Sensitivity Function and Chromatic Contrast Sensitivity Functions may be expressed in function of the frequency F by using the algebraic formula given by J. A. Movshon and L. Kiorpes in the article entitled “Analysis of the development of spatial contrast sensitivity in monkey and human infants”, published in the Journal of the Optical Society of America, A 5, pages 2166-2172 (1988) and by G. M. Johnson and M. D. Fairchild, in the article entitled “Darwinism of Color Image Difference Models”, published in IS&T/SID 9th Color Imaging Conference, Scottsdale, 24-30 (2001):
CSF(F)=a.Fc.e−b.F
CCSF(F)=a1.Fc
where the parameters a, b, c, a1, b1, c1, a2, b2, c2 are defined to fit human vision, e.g. in the already quoted Johnson & Fairchild's article and as copied in following tables 1 and 2.
Here, the two CCSF functions are merged into an only-one function
whereas previous description (see
used in gamut mapping algorithms.
Since, as already stated above, the limit conditions to meet are: ΔL=0 for CCSF>>CSF in the low spatial frequency range and ΔL=MIN{Lmax−L0;L0} for CCSF<<CSF in a higher spatial frequency range, with an intermediate situation where 0<ΔL<MIN{Lmax−L0; L0} in-between and a smooth transition between all cases, the following function ΔL(.) can be used to compute the range ΔL of variation of the position of the anchor point according to both the luma spatial bandwidth FDCTLum
where x=CSF(FDCTLum
Using this ΔL(.) function to define the range ΔL of variation of the position of the anchor point, a plurality of gamut mapping algorithms is now defined as clipping methods using each three regions: a central region having a variable anchor point varying in a range ΔL around a central value L0, located between two regions of fixed anchor points, one of these positioned at L0+ΔL, the other positioned at L0−ΔL. Moreover, a selection rule is now defined that associates any spatial frequency distribution of color values within a given block area of an image to map to one color gamut mapping algorithm that has to be used to map the colors of this given block.
The different steps of the embodiment of a method of processing a compressed image into a gamut mapped image can now be processed as follows, in reference to
In a preliminary step, the values of L0 and Lmax are determined. Lmax is the maximum lightness value, which is 100 in case of CIELAB space. A reasonable value for L0 is 50. Lmax and L0 can be kept constant for all blocks of the images. Another possibility is to calculate L0 from the mean lightness of the block of the spatial neighborhood of the block. L0 should not be larger that the lightness of the display white or should not be smaller than the lightness of display black.
The first step is entropy decoding of the compressed image into a plurality of adjacent compression block areas, which is known per se and is not detailed.
The second step is dequantizing the decoded image so as to provide spatial frequency distribution of color values of pixels of this image within each block area of this image. Each block area correspond to a block in which the image has been compressed, comprising generally 8×8 pixels of the image. As the compressed image has been compressed according to DCT transform, all coefficients that are obtained are DCT coefficients. The DCT coefficients that are obtained in the x direction for a given block area represent the vertical spatial frequency distribution of colors values of this block area and the DCT coefficients that are obtained in the y direction represent the horizontal spatial frequency vertical distribution of colors of this block area. For each block area, a matrix of DCT coefficients is obtained for luma Y, and a matrix of DCT coefficients is obtained for each of the two chrominance components U, V. The YUV color space corresponds approximately to the YCbCr color space, which was developed as part of the ITU-R BT 601 standard with Y standing for the luma, Cb representing the color difference between blue and Y, and Cr denoting the color difference between red and Y. YUV traditionally represents an unscaled value being from 0-255, whereas YCbCr scales and clamps the luma to a range between 16-235 and the chroma to a range of 216-240.
At the end of this second step, for each block area of the image, three matrices are then available. Within each block area, U and V matrices are then merged into one chromatic matrix in order to provide the algorithm a scalar value for chromatic frequency distribution analysis. Based on the scalar CCSF computation previously detailed, we use once again a coarse approximation using an Euclidian distance value. Therefore, for each DCT coefficient, we compute:
In reference to
The third step is, for each block, the inverse DCT transformation of these three matrices into color values for each pixel of this block area, i.e. luma values and chrominance values. Such a step corresponds to the inverse transform of the transformation that is illustrated on
The fourth step is the association of each block area with one value of ΔL allowing the definition of a specific color gamut mapping algorithm for this block area. The purpose of this fourth step is basically to compare the spatial frequency distribution (DCT coefficients) with the Contrast Sensitivity Function and with the merged Chromatic Contrast Sensitivity Function, in order to decide to favor luminance preservation or to favor chrominance preservation for the subsequent color mapping step. An additional smoothing operation may be added on the 8×8 block boundaries to avoid abrupt changes during the subsequent mapping step.
Using the algebraic formula (see above) representing the CSF and the CCSF in function of the spatial frequency F, a value of CSF(F) is calculated for F=FDCTLum
The fifth step and last step is, for each block area of the image, the gamut mapping of color values that have been obtained by the third step, by using the specific clipping method selected for this block in the fourth step, having three regions as defined above, so as to provide finally the decompressed color gamut mapped image.
Advantageously, using the same DCT matrices both for decompression and for color gamut mapping allows to save significant computation resources. Globally, by providing a joint decompression and content-based gamut mapping system, computation resources can be saved, since both operations share common image analysis tools, namely luminance-chrominance conversions and space-frequency transforms.
Moreover, as image data are available into luminance-chrominance color spaces (typically YUV), such color spaces are also advantageous for color gamut mapping. As a matter of fact, in color gamut mapping, luminance-chrominance conversion is usually performed using the CIELab color space, but using the more compression-compliant color space YUV can be a reasonable approximation. Also, the local Discrete Cosine Transforms (DCT) is advantageously used to map images. This directly fits well with JPEG/MPEG data streams, since these standards also use local (8×8 blocks) DCT transforms. Other compression standards, such as JPEG2000, based on Wavelet Transforms, could also be used, since a Wavelet Transform keeps track of both spatial and spectral content in an image.
It will be understood that the present invention has been described purely by way of example, and modifications of detail can be made without departing from the scope of the invention.
When the compressed image comprises a description of an image-specific color gamut as disclosed for instance in U.S. Pat. No. 6,633,668 (see col.4, lines 20-24), the method of processing of the invention may comprise an additional step of extraction of this description (see col.4, line 55 up to col.5, line 9) which is then used for the color gamut mapping step of the invention, in combination, for instance, with a description of the color gamut of a color reproduction device.
While the present invention is described with respect to preferred embodiments, it is understood that the present invention is not limited to these embodiments. The present invention as claimed therefore includes variations from the particular examples and preferred embodiments described herein, as will be apparent to one of skill in the art. While some of the specific embodiments may be described and claimed separately, it is understood that the various features of embodiments described and claimed herein may be used in combination.
Number | Date | Country | Kind |
---|---|---|---|
08305169.8 | May 2008 | EP | regional |