Patent Application
20090003723
Several applications for image editing allow the user to associate an enhancement or an image filter, such as a color change, with a color range. These techniques can be found in such programs as Nik Software's Color Efex Pro™ and Nikon's Capture NX2™. For instance, these products contain tools allowing the user to darken all blue pixels in the image. Example tools are Channel Mixer, Select Color Range, or Hue/Saturation in Adobe® Photoshop®, Contrast Color Range in Nik Color Efex Pro™, or LCH Editor in Nikon Capture NX2™.
A common feature of these tools is that the strength of an effect is determined by the color of a pixel. The disadvantage of this approach is that small local variations among pixel values, resulting from noise and compression artifacts, can lead to larger variations in effect strength, because the small local variations are typically amplified during image editing routines.
An example filter expression is g(x)=T[f(x)], where f(x) is a function of x, g(x) is the filtered function, and T is the filter function. The function f(x) is considered a one-dimensional image because it depends on one variable. More commonly, image-editing functions filter two-dimensional images. An example two-dimensional image filter expression is
g(x, y)=T[f(x, y)]
where f(x, y) is a two-dimensional original image to be filtered, g(x, y) is the filtered image, and T is the filter function. Here, filter inputs and outputs, such as f or g, typically represent an image consisting of a plurality of pixels. Any specific individual pixel may be referred to by specifying the underlying coordinates, as in f(x0, y0) is the pixel at coordinates (x0, y0) in the image f.
An example filter g=T[f] may perform a darkening operation. There are many ways to express a brightness change filter based on a color in a simple form, but here, a general formulation convenient for explaining the invention is used. The filtering operation may darken colors in a color range near a user-selected color, c. As a result of filtering, pixels with colors similar to the color c receive a strong darkening effect, pixels with colors dissimilar to c do not receive a darkening effect, and a smooth transition between darkened areas and areas with no darkening is desired.
A “color distance” is an inverse measure of color similarity between two pixels, with more similarly colored pixels having a smaller color distance. More precisely, a color distance, Dc, is a function that assigns a non-negative value to the difference in the color of two pixels. Many such distance, or difference, functions are known in the art, and have the characteristics that for any three pixels p, q, and r:
D
c(p, q)≧0 with Dc(p, q)=0 if and only if pixels p and q are the same color,
D
c(p, q)=Dc(q, p), and
D
c(p, q)+Dc(p, r)≧Dc(p, r).
One suitable function measuring color distance between the color of a pixel f(x0, y0), denoted as “color[f(x0, y0)]”, and the color c is:
D
c=∥color[f(x0, y0)]−c∥2, (1)
where color[f(x0, y0)] and c are color values in an appropriate color reference system and ∥r∥2 denotes the l2 norm of r, as explained further below. Appropriate color reference systems include grey-scale and color reference systems such as RGB, YCrCb, or Lab color systems, where a color c is expressed as a three-coordinate vector of values. In an RGB system, for example, c is expressed as a three-coordinate vector, c=(c[r], c[g], c[b]), where c[r] is the red coordinate, c[g] is the green coordinate, and c[b] is the blue coordinate. If f(x0, y0)=(f[r], f[g], f[b]), for example, then a suitable norm is
D
c
[f(x0, y0), c]={(c[r]−f[r])2+(c[g]−f[g])2+(c[b]−f[b])2}0.5.
The norm above is known in the art as an l2 norm. Alternatively, color distance may be expressed as an l1 norm, given by
D
c
[f(x0, y0), c]=∥color[f(x0, y0)]−c∥1=|c[r]−f[r]|+|c[g]−f[g]|+|c[b]−f[b]|.
Let the function s[f(x0, y0), c] denote a measure of similarity of color of the pixel f(x0, y0) and the color c, such as
s[f(x0, y0), c]=1/(1+Dc[f(x0, y0), c]). (2)
In Equation 2, when s equals 1 there is a maximum similarity of color, and s equals zero represents a lower limit for color similarity.
A filtering effect based on similarity of color at any pixel (x, y) can be written using a filtering function
g(x, y)=f(x, y)+Δh s[f(x, y), c], (3)
with Δh representing a maximum change in pixel color value. The value Δh can be a scalar value if monochromatic brightness changes are desired, or it can be a color triple if a color modifying filter is desired. Note that, in typical reference color systems for digital images, darker colors have lower pixel color values, and, because the example filter produces a darkening effect, a scalar valued Ah is less than zero for darkening.
When Δh=(Δh[r], Δh[g], Δh[b]) is an RGB color, the example darkening filter becomes an example color modifying filter, modifying colors in the image. Even colorizing operations can be performed in the image, such as when Δh[r]>Δh[g]>Δh[b], for example, a filter warming up the area will be performed. Other usages of Δh are discussed below. For purposes of the explanation that follows, the terms “warming filter,” “darkening filter,” and “brightness changing filter,” may be used as species of “color changing filters.”
Also note that, in a typical digital image, pixel color values are scaled between a value of zero and a maximum value, MAX, dependent on the number of bits representing a pixel color. Typically, when colors are represented by n-bit non-negative numbers, the maximum color value, MAX, is given by MAX=2n−1. Note that a value lower than zero or higher than MAX can occur in Equation 3 and needs to be compensated for in actual code, but can be trivially corrected by saturating the formula at a lower value of zero and an upper value of MAX, as known to those skilled in the art.
An alternative change that a user may wish to perform at any pixel (x, y) is to increase or decrease the contrast in a certain area, using
g(x, y)=f(x, y)+Δk s[f(x, y), c][f(x, y)−
where Δk denotes the desired contrast change, and
The filter used to produce
The color-changing and contrast-changing effects of Equation 3 and Equation 4 may be combined using the formula
g(x, y)=f(x, y)+{Δh+Δk[f(x, y)−
The examples above relate to a filter that changes regions in an image that are represented by a single color. A simple version of a polarization filter, darkening and adding contrast to sky-blue areas, or a typical skin-brightening filter may be implemented using Equation 5.
However, there is an inherent disadvantage in the formulations of Equations 3 through 5, as this method is not robust in the presence of noise. Consider, for instance, an image with a large blue-sky area whose mean color value is the color c. If the image is shot with a high ISO number and with strong JPG compression, not all pixels in the sky will be equally blue. In this situation, the similarity expression s[f(x, y), c] reflects the noise present in a blue-sky pixel to a certain extent. In the worst case, where variations in the color are due only to noise but the underlying colors are very similar, the function s modifies the noise amplitude to the extent of
similarity_noise_amplitude=original_noise_amplitude*max|s|.
In example Equations 3-5, the change is added to the original image. The resulting filtered image g has amplified noise, with {1+max |s|} times the amplitude of what it was in the original image. The noise increase is also based upon the values of Δh and Δk, which is characteristic to most image editing filters. The increase in noise level will frequently be visible to the observer of the image, and visibly degrades the image.
Conventional approaches to the noise enhancement problem are (1) to first reduce the noise in the original image and then perform the filtering function, or (2) to first perform the filtering function and then reduce the noise in the resulting image. These prior art approaches have the disadvantage that actual desired features in the image are being reduced.
An alternative prior art approach is to apply noise reduction selectively to the affected areas, but this also produces unwanted side effects, such as loss of detail, unwanted artifacts, or an uneven distribution of image structure.
The invention meets this need by providing with an improved method of filtering an image reduces the noise enhancement of prior art image-enhancing filters. In prior art filtering, the magnitude of a filtering effect is based on structure in the image. The improved method processes the image while preserving the edges of the image, but reduces noise in non-edge areas. In one embodiment a structure-adaptive filter processes the image to determine the intensity of one or more image-enhancing changes, and the one or more image-enhancing changes are performed on the underlying image as a function of the determined intensity.
In an illustrative embodiment of a filter using the method, filter parameters include a selected reference color and a selected change of the reference color. The image is filtered with an edge-preserving detail-reducing filter (EPRDF). At each pixel to be modified, the selected change of color, scaled by a measure of color similarity to the selected reference color in the corresponding pixel in the EPRDF image, is added to the unmodified corresponding pixel value. In a second illustrative embodiment, the user selects filter parameters including a reference color and a change of contrast for the reference color. The image f is EPDRF-filtered to produce an image {circumflex over (f)}, and then highpass-filtered. At each pixel to be modified, the desired change of contrast, scaled by the measure of color similarity to the selected reference color in the corresponding pixel in the EPDRF image and multiplied by the corresponding pixel in the highpass image, is added to the unmodified corresponding pixel value in the original image. In further embodiments, the methods of the first two embodiments are combined and generalized to support an arbitrary number of reference colors, and corresponding brightness and contrast changes. Alternative embodiments demonstrate a variety of uses of noise-reduced similarity functions in alternative formulations of noise-reduced image processing filters.
The color similarity function is one type of generalized structure-adaptive filter. Other embodiments of the invention utilize more complex structure-adaptive filters, and the invention includes means to modify the intensity of an enhancement effect as a function of other user selected parameters, such as pixel location, relative spatial distance, local image frequency content, or local image pattern matching.
In a preferred embodiment, a particular edge-preserving detail-reducing filter (EPRDF) is used. In addition, for some types of EPRDF, the image processing chain of the method may be differently arranged and/or differently partitioned in an equivalent structure. Example equivalent structure is described.
A method for applying an enhancement filter to a digital image with a noise level, and comprising a plurality of pixels, with minimal amplification of the noise level, is disclosed, the method comprising filtering the digital image with an edge-preserving detail-reducing filter, determining a matrix from the filtered digital image, wherein the matrix is the result of one or more structure adaptive functions applied to the filtered digital image, and wherein the matrix comprises one or more values, each of the one or more values determining a magnitude of the enhancement filter to apply at a corresponding pixel in the digital image, and modifying the digital image using the enhancement filter, adjusted by the one or more matrix values, to produce an enhanced digital image.
Alternatively, the method can proceed by first determining a matrix from the digital image, wherein the matrix is the result of one or more structure adaptive functions applied to the digital image, then filtering the matrix with an edge-preserving detail-reducing filter to generate a filtered matrix, and then performing the modifying step.
The enhancement filter can be, for example, selected from the group consisting of a color change filter, a contrast change filter, or both a color change and a contrast change filter.
The edge-preserving detail-reducing filter can be, for example, selected from the group consisting of an arctan Gaussian filter, an arctan blurring filter, a wavelet-based detail-reduction filter, a bilateral filter, a Beltrami flow filter, an adaptive Gaussian blurring filter, and an edge-preserving smoothing filter using geodesic distances.
The matrix in the determining step can further be the result of a function of one or more color similarities to a reference color, the color similarity being determined from a color distance from a corresponding pixel in the digital image to the reference color. Optionally, the matrix is the result of a function of one or more geometric distances, wherein the geometric distance is determined from the location of a corresponding pixel in the digital image in relation to a reference pixel or a reference region.
A method for enhancing a digital image with a noise level, and comprising a plurality of pixels, with minimal amplification of the noise level, is also disclosed, the method comprising filtering the digital image with an EPDR edge-preserving detail-reducing filter; generating a first image by filtering the EPDR filtered digital image with a structure-adaptive filter; generating a second image by filtering the EPDR filtered digital image with a highpass filter; and adding the first generated image and second generated image to produce an enhanced image.
The structure-adaptive filter could be, for example, a user defineable image reference point or region. Further, it could comprise a function of one or more color similarities, wherein the color similarity is determined from the similarity of a corresponding pixel in the EPDR filtered digital image with a reference color. Still further, the structure-adaptive filter further comprises a function of one or more geometric distances, wherein the geometric distance is determined from the location of the corresponding pixel in relation to a reference pixel or a reference region.
A method for editing a digital image comprising a plurality of pixels, to modify similarly colored pixels and the contrast of similarly colored pixels, id disclosed, the method comprising generating lowpass and highpass versions of the digital image; receiving a set of one or more than one reference color, and for each of the one of more than one reference color, a corresponding desired color change and a corresponding desired contrast change; and modifying one or more pixels in the digital image by determining the location of each pixel to be modified in the digital image, and the location of the corresponding pixels in the lowpass version of the digital image and the highpass version of the digital image; and for each reference color in the set of one or more than one reference color, measuring the similarity of the corresponding pixels in the lowpass digital image to the reference color using a similarity function, scaling the corresponding desired color change using the measured lowpass similarities to generate a scaled color change, scaling the corresponding desired contrast change using the measured lowpass similarities to generate a scaled contrast change, determining a partial modification of the pixel values of the pixels in the digital image using the scaled color change, determining a partial modification of the pixel values of the pixels in the digital image by adding the product of the scaled contrast change and the corresponding pixel in the highpass digital image, and adding the determined partial modifications to the unmodified pixel values to produce modified pixel values.
A method of filtering a digital image comprising pixels with an edge-preserving detail-reducing filter, is disclosed, the method comprising filtering the digital image with a lowpass filter; applying a constraining function to limit the differences between the digital image and the lowpass filtered image; and adding the constrained differences to the digital image to produce a edge-preserved detail-reduced digital image. The constraining function could be, for example, an arctan function.
An apparatus for image editing is described, the apparatus comprising a memory, a processor, an input/output interface, and a set of processor instructions, stored in the memory, wherein, when the set of processor instructions are accessed by the processor and executed, the set of executed processor instructions are operable to receive one or more user-selectable image editing parameters; access a digital image in memory to be modified, wherein the digital image comprises a plurality of pixels; filter the digital image with an edge-preserving detail-reducing filter; determine a matrix from the filtered digital image, wherein the matrix is the result of one or more structure adaptive functions applied to the filtered digital image, and wherein the matrix comprises one or more values, each of the one or more values determining a magnitude of an enhancement effect to apply at a pixel in the digital image; and modifying the digital image using an enhancement filter, adjusted by the one or more matrix values, to produce an enhanced image.
These and other features, aspects, and advantages of the present invention will become better understood with reference to the following description, illustrations, equations, appended claims, and accompanying drawings where:
The invention is useable as a plug-in supplemental program similar to Nik Color Efex Pro™, as an independent module that may be integrated into any commercially available image processing program such as Adobe® Photoshop®, Apple® Aperture®, or Nikon Capture NX2™, into any image processing device that is capable of modifying and displaying an image, such as a color copier or a self service photo print kiosk, or even a camera, as a dynamic library file or similar module that may be implemented into other software programs whereby image measurement and modification may be useful, or as a stand alone software program. These are all examples, without limitation, of image processing of a digital image. Although embodiments of the invention adjusting color, contrast, noise reduction, and sharpening are described, the present invention is useful for altering any attribute or feature of the digital image.
Furthermore, it will become clear with regard to the current invention that the user interface for the current invention may have various embodiments, which will become clear later in this disclosure.
As noted above, there are many ways to express an image-processing filter. A simple color-based brightness change filter is used to illustrate the method of the current invention. The method is usable with a wide variety of other structure-adaptive filters, as described further below. As noted above, a suitable similarity function at any point (x, y) is given by:
s[f(x, y), c]=1/(1+Dc[f(x, y), c]).
Alternate suitable embodiments of a similarity function include
s[f(x, y), c]=1/(1+w2Dc[f(x, y), c]),
s[f(x, y), c]=1/(1+|w|Dc[f(x, y), c]),
s[f(x, y), c]=exponential(−Dc[f(x, y), c]/w2), and
s[f(x, y), c]=exponential(−Dc[f(x, y), c]/|w|),
where w is a spreading parameter affecting the breadth of similar colors.
Equation 3 may be rewritten for any point (x, y) as:
g(x, y)=f(x, y)+Δh s[f(x, y), c],
g(x, y)=f(x, y)+ChangeIntensity[Δh, c, f(x, y)], where (6)
ChangeIntensity[Δh, c, f(x, y)]=Δh s[f(x, y), c]. (7)
In Equation (7), the desired color change, Δh, is scaled by a measure of similarity in the image f to the reference color c at the coordinates (x, y) to determine the amount of change in the pixels of f at the corresponding coordinates. (A pixel in one image corresponds to a pixel in another image if they have the same coordinates.)
An improved form of example Equation 7 provides a significant advantage in the treatment of noise. Instead, an improved filter uses the function
ChangeIntensity[Δh, c, {circumflex over (f)}(x, y)]=Δh s[{circumflex over (f)}(x, y), c], where (8)
{circumflex over (f)}(x, y)=E[f(x, y)] is a filtered version of f(x, y).
In the improved filter, the desired color change, Δh, is scaled by a measure of similarity in the image {circumflex over (f)} to the reference color c at the coordinates (x, y) to determine the amount of change at the corresponding coordinates. The filter E is an edge-preserving detail-reducing filter (EPDRF), as discussed in more detail below.
A generalization of Equation 6 is provided by:
g(x, y)=ColorChange f(x, y), ChangeIntensity[Δh, c, {circumflex over (f)}(x, y)]},
g(x, y)=ColorChange f(x, y), u}, (9)
where ColorChange{f(x, y), u} represents a function applying a desired color change intensity u to the color values of the unmodified pixels of f(x, y). Example ColorChange functions are
Colorchange{f(x, y), u}=f(x, y)+u, (10)
Colorchange{f(x, y), u}=f(x, y) u, or (11)
Colorchange{f(x, y), u}=f(x, y)exponential[u], (12)
The quantity u, output from the ChangeIntensity function, may be thought of as a scalar quantity, or alternatively, the change in color applied in Equations 10 to 12 may be expressed, for example, as a three-component RGB color u=(u[r], u[g], u[b]).
Note that, in Equation 7, f is a direct input of the function ChangeIntensity. Therefore, the ChangeIntensity function is susceptible to noise in f. In the improved Equation 8, the ChangeIntensity function reduces noise by modifying the image f to create an image {circumflex over (f)} to measure similarity, while applying the filtering result to the actual image, as in Equation 9 and example Equations 10 to 12. In other words, in this embodiment the color change filter is a function of both the original image, and a change in intensity that is itself a function of a version of the original image filtered with an EPDRF filter. The increase in noise level, if any, should be minimal, and may not be visible to the average observer of a printed or displayed version of the enhanced image.
In Equations 8 and 9, a filtered image {circumflex over (f)} is being used for the detection of the desired change, while the color changes are rendered based on the unmodified values of f. If {circumflex over (f)} is a noise-reduced version of f, the resulting filtered image g will contain all the desired color changes, but the increase in noise is less than that using the original image f as input to the similarity function as in Equation 7. If, in addition, {circumflex over (f)} preserves the edges of f, the resulting filtered image g will contain all the original detail of f.
Various example EPDRF filtering techniques and their relative processing requirements are shown in Table 1. Those skilled in the art will know that implementing these advanced noise-reduction techniques can be very computationally intensive. For instance, the technique known as wavelet-based noise reduction requires a lot of processing time and memory if the results are to be visually appealing. However, the requirements for a filter to produce the image {circumflex over (f)} for the current convention can be substantially less strict than those for the advanced EPDRF techniques of Table 1 and still produce visually appealing results.
Any edge-preserving detail-reducing filter (EPDRF) filter can be used for the creation of {circumflex over (f)} from f. An EPDRF, as defined hereinafter, is a filter that reduces the weak high-frequency detail in the image while leaving the strong edges in the image unaffected. EPDRF filtering techniques are known by various names in the art, such as “edge-preserving smoothing filters,” “edge-based adaptive smoothing,” “edge-preserving MAP estimation,” “nonlinear mean filters,” “detail-preserving smoothing,” “edge preserving filters using geodesic distances,” “feature-preserving de-noising,” “edge-sensitive smoothing,” “nonlinear smoothing,” “adaptive Gaussian blurring” and “bilateral filters.” The EPDRF filtering problem and a bilateral filtering solution are discussed in C. Tomasi & R. Manducci, Bilateral Filtering for Gray and Color Images, in Proceedings of the IEEE Conference on Computer Vision, pp. 839-846, January 1998.
As discussed above, the resulting image g is computed based on the original image f, containing the original noise and artifacts, while the change intensity value is computed based on {circumflex over (f)}, containing less noise and artifacts. Hence, the resulting image g still contains all of the original noise, artifacts and detail, but the contrast and color changing routine does not increase this detail, resulting in a natural looking image with visual appeal.
The Arctan Gaussian filter is an example EPRDF suitable to practice the invention. The Arctan Gaussian is further generalized here to provide an Arctan Blurring filter. The Arctan Blurring filter provides a very fast yet effective implementation of an EPDRF filter for the invention.
Let f be the unfiltered image and let
{circumflex over (f)}(x, y)=f(x, y)+[
where “∥ ∥” denotes a color norm. In an alternate embodiment for a color system which can be linearly scaled, each component of the color of [
{circumflex over (f)}
r(x, y)=fr(x, y)+arctan(
{circumflex over (f)}
g(x, y)=fg(x, y)+arctan(
{circumflex over (f)}
b(x, y)=fb(x, y)+arctan(
Alternatively, instead of the arctan function, any other continuous, monotonic function H, with a rate of change denoted H′, that satisfies the following four conditions may be used:
(i) H′(0)=1,
(ii) H(0)=0,
(iii) H′(x)=0 for |x|>>0, and
(iv) |H(x)|<ε for all x, (14)
where ε is a small quantity. For example, with H(x) defined by
if (x<ε), then H(x)=x; else H(x)=ε,
the generalized resulting EPDRF version {circumflex over (f)} of f can be described in one embodiment as
{circumflex over (f)}=f+(
Equation 15 and the conditions on the function H in Equation 14 can be interpreted as follows. In featureless areas, the difference between
This means that in plain areas the image {circumflex over (f)} is close to the blurred version of f, in edgy, high frequency areas the image {circumflex over (f)} equals the image f with an addition or subtraction not exceeding ε. In even other words, the image is expressed as a sum of the image and the difference between the blurred image and the original image, while said difference is damped in its amplitude if it is high.
H(x)˜exp [−x2/2r2],
whereas, for a two-dimensional Gaussian blur,
H(x, y)˜exp [−(x2+y2)/2r2],
where r is referred to as the radius of the distribution. Note that, in waveform 503, the one-dimensional image {circumflex over (f)}(x) shown suppression of noise in featureless areas, while preserving structure near edges.
Similarly, an embodiment of a contrast-changing filter using the method of the current invention uses a variant of Equation 4, repeated here:
g(x, y)=f(x, y)+Δk s[f(x, y), c][f(x, y)−
where Δk denotes the desired contrast change, and
g(x, y)=ContrastChange{f(x, y), ChangeIntensity[Δk, c, {circumflex over (f)}(x, y)]}, where (16)
ChangeIntensity[Δk, c, {circumflex over (f)}(x, y)]=Δk s[{circumflex over (f)}(x, y), c], and (17)
ContrastChange{f(x, y), v}=f(x, y)+v[f(x, y)−
In this example embodiment, Equation 17, like Equation 8, uses a similarity function operating on the EPDRF filtered image {circumflex over (f)}(x, y) to scale the desired effect.
Equations 9 and 16 may be combined and generalized to combine a desired color and contrast change. In this case, a general form for the filter is given by
g(x, y)=CombinedChange{f(x, y), DualChangeIntensity[{circumflex over (f)}(x, y), Δh, Δk, c]}, where (19)
DualChangeIntensity[{circumflex over (f)}(x, y), Δh, Δk, c]=(uxy, vxy) (20)
CombinedChange {u, v}xy=f(x, y)+uxy+vxy[f(x, y)−
where, as in Equations 8 and 17,
u
xy=ChangeIntensity[Δh, c, {circumflex over (f)}(x, y)], and
v
xy=ChangeIntensity[Δk, c, {circumflex over (f)}(x, y)].
Equation 19 may also be generalized for an arbitrary number of reference colors. This can be described by introducing a series of N three-component controls
CCT={(c[1], Δh[1], Δk[1]), . . . , (c[N], Δh[N], Δk[N])}, (22)
hereinafter named the Color Change Table (CCT), where, when index n ranges from 1 to N, each c[n] represents a reference color, each Δh[n] represents a corresponding desired color change, and each Δk[n] represents a corresponding desired contrast change. Equation 19 may be re-expressed as:
CombinedChange(u, v)xy=f(x, y)+Modification(u, v)xy, where
Modification(u, v)xy=uxy+vxy[f(x, y)−
For an arbitrary number of reference colors, the modifications are combined in the image editing function by defining
CombinedModsxy=Σ Modification(u[n], v[n])xy, (24)
where the summation ranges from n equals one to N. The nth pair of scaled changes is given by
(u[n], v[n])xy=DualChangeIntensity{{circumflex over (f)}(x, y), Δh[n], Δk[n], c[n])}, (25)
as in Equation 20 where c[n] is the nth reference color, Δh[n] is the corresponding nth desired color change, and Δk[n] is the corresponding nth desired contrast change. The output of the filter is given by
g(x, y)=f(x, y)+CombinedModsxy. (26)
In the filters described above, the method applies an EPDRF to determine the channel used to calculate the effect intensity. It does not apply the EPDRF to the original image component of the resultant image, and does not apply the EPRDF to the resultant image. Several alternate embodiments similar to example Equation 8 can be used to achieve the desired effect. Two embodiments are:
M=Colorchange{f(x, y), EPRDF[ChangeIntensity(f(x, y), c)]}, and (27)
M=Colorchange{{circumflex over (f)}(x, y), ChangeIntensity({circumflex over (f)}(x, y), c)}+(f(x, y)−{circumflex over (f)}(x, y)). (28)
where {circumflex over (f)}(x, y)=EPRDF(f(x, y)).
In Equation 8, the effect intensity is calculated based upon an EPDRF version of f and then applied to f. In Equation 27, an EPDRF function is applied to a matrix representing a plurality of effect intensities, the result of which is then applied to f. In Equation 28, the details are being removed from f with an EPDRF, to which the effect is then being applied, and then the detail, (f−{circumflex over (f)}), is added to the image again.
In a further generalization, consider any filter of the form
g(x, y)=T{f(x, y), axy(f)}, where (29)
f(x, y) is a two-dimensional original image to be filtered;
axy(f) is a scalar-valued function that defines the intensity of an image change at the coordinates (x, y) as a function of the image f;
T is an image filter, modifying an image based on provided scalar intensity; and
g(x, y) is the two-dimensional filtered image.
The reader may note that this is a generalization of Equations 6 to 7, where
a
xy
=Δh s(f(x, y), c).
In this example, the scalar-valued intensity function is a function of two arguments, the similarity of a pixels to a reference color, and a user-selected color change, Δh. As used hereinafter, the function that generates the scalar-valued intensity function is defined as a structure-adaptive filter if each output of the filter comprises a function of the color of an associated pixel. Functions of the color of the pixels may include quantities derived from the color, such as brightness, luminosity, saturation, hue, and color temperature. A structure-adaptive filter can also functionally depend on one or more additional parameters, including the location of the associated pixel, the neighboring pixels' values, pattern matching in a neighborhood of the associated pixel, frequency content in a neighborhood of the associated pixel, directional orientation of present structure, entropy of the pixels and neighboring pixels, or one or more other user-selected parameters.
Let {circumflex over (f)} be an EPDRF filtered version of f. Using this notation, embodiments of filters of the present invention may be expressed as:
g=T{f, a({circumflex over (f)})}, (30)
g=T[f, EPDRF{a(f)}], and (31)
g=T{{circumflex over (f)}, a({circumflex over (f)})}+(f−{circumflex over (f)}). (32)
In Equation 30, the EPDRF filter is applied to the image f to create {circumflex over (f)}, while the structure-adaptive filter is applied to the image {circumflex over (f)}. In Equation 31, the structure-adaptive filter is applied to the original image, but an EPDRF is applied to the result. In Equation 32, the entire image filtering function, consisting of the filter function T as well as the scalar-valued function a, is applied on an EPDRF version of f, but to avoid loss of detail, the image details in (f−{circumflex over (f)}) are restored by adding the last term. Note that, for some editing functions T, the formulas turn out to be equivalent. For instance, when T is a ColorChange function given by Equation 10, Equations 30-32 are equivalent. Also note that, in the alternate embodiment of Equation 31, the edge-data used by the EPDRF can be derived from the original image f instead of ChangeIntensity(f, c), providing results with better precision in some implementations. Other embodiments of the structure-adaptive filter are discussed below.
In
In
J1. Polarization
A simple digital approach to creating a polarization-like effect, mainly intended to darken the sky, is to darken all blue areas in the image. Note that commercial polarization filters need to do more refined operations to affect contrast, colors, and color brilliance. For clarity of explanation purposes, this example will focus on merely darkening the blue colors. A polarization filter can be implemented using Equation 8 with
g=Colorchange{f, ChangeIntensity[{circumflex over (f)}, c, Δh] }, where
c equals the blue color value,
Δh is negative, and
{circumflex over (f)} is the result of applying an EPDRF to f.
In
J2. Brightening Skin
An example filter intended to brighten up skin colors can be implemented using Equation 8 with
M=Colorchange{f, ChangeIntensity[{circumflex over (f)}, c, Δh]}, where
c equals a skin tone color value,
Δh is positive, and
{circumflex over (f)} is the result of applying an EPDRF to f.
J3. Conversion of Color Image to Black and White
This example filter would simulate a real-world red-orange color filter used to capture an image in an analog black and white camera. The example simulation filter operates on sampled color values using Equations 22-26 with
g=BW{f(x, y)+CombinedModsxy}, where
N=5,
c[1]=a red color value,
c[2]=an orange color value,
c[3]=a yellow color value,
c[4]=a green color value,
c[5]=a blue color value,
Δh[1]=Δh[2]=1,
Δh[3]=0.5,
Δh[4]=Δh[5]=−1,
Δk[n]=0 for all n, and
the function BW calculates a grey scale value as a function of a color value.
K. Structure-Adaptive Filtering including Location and Color Dependent Enhancements.
The invention may be used to improve noise immunity in all image-editing functions where the color of a pixel influences the strength of an applied effect. Improved noise immunity is achieved in simple filters, like the one improved in Equations 7-8, where effect intensity depends only on color similarity, and more complex filters where effect intensity depends not only on similarity to a reference color, but also on one or more factors based on the spatial location or other structural information within the image. For instance, a more complex filter darkens blue pixels in the upper half of the image and brightens green pixels in the lower half of the image.
The inventor has provide other examples of structure-adaptive filters in various United States patent applications, the content of each of which is incorporated by reference in this disclosure in its entirety, as follows: U.S. Pat. No. 7,257,269, titled “Digital Polarization Filter,” filed Jun. 27, 2003; U.S. patent application Ser. No. 11/674080, titled “Self-Adaptive Brush for Digital Images,” filed Feb. 12, 2007, now Publication No. 2007/0188510; U.S. Pat. No. 6,865,300, titled “User Defineable Image Reference Points,” filed Apr. 13, 2004; U.S. Pat. No. 6,728,421, titled “User Defineable Image Reference Points,” filed Apr. 13, 2004; U.S. Pat. No. 7,031,547, titled “User Defineable Image Reference Points,” filed Mar. 3, 2005; and U.S. patent application Ser. No. 11/832,599, titled “User Definable Image Reference Regions,” filed Aug. 1, 2007, now Publication No. 2008/0137952.
These example filters include not only functionality to receive user input to identify one or more reference colors and a means of measuring a color difference from the one or more reference colors, but also functionality to receive user input related to spatial intensity of the filter. For instance, user input related to spatial intensity can specify an area of an image defined by a single pixel or reference point, bounding coordinates, an arbitrarily defined region, or a brush stroke. Image-processing software then uses this spatial intensity input in combination with a pixel's color dependency to determine the strength of a desired effect for each pixel.
All of the example image-editing systems above have in common that the intensity of an effect applied at a pixels is dependent, among other user-input parameters, on the color attributes of a pixel. The examples above include systems where a structure-adaptive system is implemented that receives input from a user in the form of regions of interest, of locations of interest, or of brush strokes that are subject to be refined by software.
In one embodiment, a function s(f, x, y <, . . . , . . . >) for processing a two-dimensional image f is considered structure-adaptive when
|∂s/∂x|<ε,
|∂s/∂y|<ε, and
|∂s/∂f|<ε,
for a small quantity δ; and there exist images f1, f2 and location coordinates (x, y) such that
s(f1,x,y<, . . . , . . . >)≠s(f2,x,y<, . . . , . . . >).
The expression <, . . . , . . . > indicates that s can have one or more optional parameters, such as user-provided region boundaries, coordinates, brush stroke data, global filtering strengths, pattern matching parameters, and the like.
A structure-adaptive function s(f, x, y, <, . . . , . . . >) that computes a scalar value or weight for every pixel in f may be used to produce a structure-adaptive enhancement filter. An enhancement filter that processes an image is considered structure-adaptive when the filter comprises a structure-adaptive function s used to determine to what extent a pixel or an attribute of a pixel in the image is changed. The attributes may include any stored attribute of a pixel or a quantity determined as a function of stored attributes of a pixel, such as brightness, sharpness, saturation, warmth, hue, opacity, and the like.
The matrix of values provided by s calculated for a certain image f, collectively associated with an image, would normally have some edges and, typically, all the edges found in an image corresponding to s would coincide with the edges found in f. In some degenerate cases, however, an image f can be found where a corresponding image may not have edges. For instance, s can be a function that detects a color similarity to a reference blue color, but an image f may not contain any blue colors. In some cases, depending on its implementation, an image corresponding to s may not show any edges or corners. In addition, the orientation of the edges in an image corresponding to s do not have to be identical to the orientation of the edges in f, where orientation is understood as the sign of the color difference from one side of the edge to the other side, and not the spatial, directional orientation.
In another embodiment, a filter is considered structure-adaptive to an image f if the image contains details (twigs, eyelashes, grain, noise, JPG-artifacts, skin pores etc.) and if the strength of the effect applied by the filter varies in the vicinity of some of said details as a function of said details.
In the example image-editing systems above, one value of effect intensity is calculated for every pixel by a function s. The examples can benefit from the invention using, for example, the three embodiments of Equations 30-32:
In further embodiments, two or more instances of this invention may be implemented, so that in one software application two or more effect intensity values can be processed per pixel, for instance to empower the user to selectively modify brightness and color temperature within one software module, or red, green, blue component, or sharpness, contrast and hue.
In one embodiment, operations that combine a color range and a spatial range can be approximated locally, at least in very small areas of the image, as a purely color-based contrast change. The invention described herein can also be modified to apply in these cases. In order to localize effects, each effect is associated with a location. The CCT is modified to produce a color-location change table (CLCT), where each entry is a four-component control consisting of a color, a location-dependent function, a corresponding desired color change, and a corresponding desired contrast change. A CLCT with N entries is given by
CLCT={(c[1], L[1], Δh[1], Δk[1]), . . . , (c[N], L[N], Δh[N], Δk[N])}.
The modified filtering equation is
g=LocalChanges{f, LocationIntensities({circumflex over (f)}, CLCT)}, where (33)
{circumflex over (f)} is an EPDRF version of f.
The new location-dependent functions L[1], . . . , L[N] are determined from locations in the image. For example, the functions {L[1], . . . , L[N]} can be a measure of spatial distance to a reference pixel or to a reference group of pixels. In other words, L[n] is a function that scales the desired color and contrast change performed at various pixel locations. The nth location-dependent function, L[n], has a particular value at the coordinates (x, y), denoted L[n]xy. L[n]xy indicates how much the nth image-enhancing function affects the image pixel at the location with coordinates (x, y).
In one embodiment of a location-dependent structure-adaptive filter, the intensity of an effect at a pixel is dependent on the Euclidean distance of the pixel from a reference object as well as the color distance from a reference color. In one example embodiment, the user centers the nth color and contrast change at pixel coordinates (x[n], y[n]), and the structure-adaptive filter function at a pixel with coordinates (x, y) is
a
xy=exp[−{(x−x[n])2+(y−y[n])2}/2r2·s({circumflex over (f)}(x, y), c[n]).
The matrix L[n] for the reference color c[n] may contain the values of
exp[−{(x−x[n])2+(y−y[n])2}12r2],
for example. The radius r may be assigned a default value, or, alternatively, r is a user-selectable parameter.
Since {L[1], . . . , L[N]} typically represent continuous underlying functions, they change only gradually from pixel to pixel, and therefore abrupt changes in the filtered image are only generated by a contrast change when the difference in similarity to a reference color between adjacent pixels is large, that is, when |s(f[p], c[n])−s(f[q], c[n])| is large with p and q adjacent pixels in the image. Using the described method to measure the difference in similarity based on {circumflex over (f)}, large differences in adjacent pixels will be ameliorated, with improved visual quality in the resulting filtered image.
L. Alternate Embodiments of Image Processing Path
In the specific embodiment of
g(x, y)=f(x, y)+Δh s({circumflex over (f)}(x, y), c).
Note that
As a concrete example, the filter 801 may use a 23 by 23 mask to filter the original image f, where the filtering operation is represented by
{circumflex over (f)}(x, y)=1/b ΣΣ kij f[(x−12+i), (y−12+j)], where
b=ΣΣ kij,
and where the double summations run from i equals 1 to 23 and j equals 1 to 23. The ColorDistance filter T2802 determines the distance from an output pixel of T1 to the reference color c, and may, for example, be
D
c(x, y)=∥{circumflex over (f)}(x, y)−c∥2,
The cascade of filters T1 and T2 is represented by a single filter G1901 in
D
c(x, y)=1/b∥ΣΣkij{f[(x−12+i), (y−12+j)]−c}∥2. (34)
Note that, in
M. Additional Comments
Note also that various user-editing operations can be combined in a similar manner. The result of the function ChangeIntensity can be interpreted as a selection-strength instead of a color change, used to determine how much a particular editing operation contributes to the filtering result. The resulting combined user-editing operations will be less subject to noise in the filtered image.
Advantageously, the invention can be embodied on a computer readable medium having contents for causing a computer-based information handling system to perform the steps described herein.
The invention is not limited to a particular computer hardware implementation, and any hardware implementation presently existing or to be developed that permits processing of digital images using the described method can be used.
The term memory block or data block refers to any possible computer-related image storage structure known to those skilled in the art, including but not limited to RAM, processor cache, hard drive, or combinations of those, including dynamic memory structures. The described methods can be embodied in a computer program that operates as a stand-alone computer application, or can be provided as a plug-in application augmenting the capabilities of an existing image-processing program. Further, the methods described herein can be embodied in supplemental tools for an existing program, in independent modules that can be integrated into any commercially available image-processing program, or integrated into any image-processing device that is capable of modifying and displaying an image, such as a color copier or a self service photo print kiosk. The methods can be implemented in dynamic library files or similar modules that can be combined with other software programs facilitating image modification. The methods can also be used in an image-processing program accessed over a network.
Any currently existing or future developed computer readable medium suitable for storing data can be used to store the programs embodying the afore-described interface, methods and algorithms, including, but not limited to hard drives, floppy disks, digital tape, flash cards, compact discs, and DVDs. The computer readable medium can comprise more than one device, such as two linked hard drives.
The drawings and the associated descriptions are provided to illustrate embodiments of the invention and not to limit the scope of the invention. Reference in the specification to “one embodiment” or “an embodiment” is intended to indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least an embodiment of the invention. The appearances of the phrase “in one embodiment” or “an embodiment” in various places in the specification are not necessarily all referring to the same embodiment.
As used in this disclosure, except where the context requires otherwise, the term “comprise” and variations of the term, such as “comprising”, “comprises” and “comprised” are not intended to exclude other additives, components, integers or steps.
The term “storage medium” can represent one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic disk storage mediums, optical storage mediums, flash memory devices, electrical storage mediums or other mediums for storing information in a form readable by a machine such as, for example, a computer. The term “data element” refers to any quantum of data packaged as a single item. The term “data unit” refers to a collection of data elements or data units that comprise a logical section. The term “image block” refers to a complete copy or partial copy of a digital image that is stored in a separate storage location and can be altered without affecting the original stored digital image.
In the foregoing specification, the invention has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes can be made thereto without departing from the broader spirit and scope of the invention. The specification and drawing are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It should be appreciated that the present invention should not be construed as limited by such embodiments.
The present Application claims the benefit of U.S. Provisional Patent Application No. 60/946,362 titled “Method for Noise-Robust Color Changes” filed Jun. 26, 2007, the content of which is incorporated by reference in this disclosure in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 60946362 | Jun 2007 | US |
