It is appreciated that certain techniques can be used to improve the quality of digital images. For example, “unsharp mask” algorithms may be implemented to enhance the perceived texture of digital images. However, such unsharp mask algorithms are based on non-edge preserving Gaussian smoothing filters. That is, unsharp mask algorithms essentially separate low frequency image components from high frequency image components via a Gaussian smoothing filter. This allows the respective components to be modulated by a constant to adjust their relative contributions. Generally, the high frequency components comprise the textures and some contributions from the edges within the images, while the low frequency components comprise large smooth regions plus the remaining edge contributions. Thus, the high frequency components can be modulated (e.g., increased) to accentuate textures as desired. However, since the high frequency components also comprise some portion of the edges within the image (when separated out with Gaussian smoothing filters), the edges within the image are also enhanced when textures are accentuated by modulating high frequency components. While this is generally not an issue for small amounts of enhancement, it becomes problematic when more substantial adjustments are made. For example, when the high frequency components are increased beyond a certain threshold, ringing artifacts or halos may be introduced around sharp edges in a scene. Such ringing artifacts or halos are undesirable, at least, because they can distract the viewer by introducing erroneous edges. Accordingly, there is room for improvement in digital image enhancement.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key factors or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
As provided herein, the quality of a digital image can be enhanced while mitigating the formation of ringing artifacts or halos within the image. That is, bilateral filtering is implemented to allow textures and edges of an image to be adjusted separately. More particularly, bilateral filtering is used to decompose an image into two component images: a detail feature image and a large-scale feature image, where the image's textures are primarily comprised within the detail image, and the image's edges are primarily comprised within the large-scale feature image. By decomposing the image into these two sub-images and then globally scaling their respective magnitudes, it is possible to adjust the textures within the image substantially independent of the edges in the image and vice versa. It can be appreciated that this allows the apparent amount of texture in the scene to be enhanced while mitigating the formation of ringing artifacts or halos around edges in the image.
To the accomplishment of the foregoing and related ends, the following description and annexed drawings set forth certain illustrative aspects and implementations. These are indicative of but a few of the various ways in which one or more aspects may be employed. Other aspects, advantages, and novel features of the disclosure will become apparent from the following detailed description when considered in conjunction with the annexed drawings.
The claimed subject matter is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the claimed subject matter. It may be evident, however, that the claimed subject matter may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing the claimed subject matter.
Turning initially to
With the image decomposed into a detail (e.g., texture) image and a large-scale (e.g., edge) image, the respective magnitudes of the different images are adjusted independently at 106 to achieve a desired result. Once the detail and large-scale image are adjusted (independently) as desired, these images are recombined at 108 to render the original, but adjusted, base image, and the methodology ends thereafter. It will be appreciated that two arrows are illustrated between 104 and 106, and between 106 and 108. This is to illustrate that the detail and large-scale images are separate images and that they can be independently adjusted or otherwise acted upon as desired before being recombined into a single image at 108. It will also be appreciated that it may be advantageous to perform bilateral filtering operations logarithmically. Accordingly, an optional first logarithmic operation may be performed on the subject image just after it is acquired at 102 and a second logarithmic (e.g., exponent or inverse logarithmic) operation may be performed just after the recombination at 108.
By way of example,
It will also be appreciated that while enhancements are generally illustrated herein as features having a heavier line weight or that are darker, enhancements or adjustments as mentioned herein are not intended to be so limited. Rather, the more salient point is that the textures and edges can be adjusted independently of one another, regardless of whether they are darkened, lightened, shaded, hatched, colored, etc. Accordingly, unlike the situation in
After the detail feature image (e.g., textures) and the large-scale feature image (e.g., edges) within the image are adjusted independently as desired (
Turning to
The decomposition component 704 outputs the detail and large-scale feature images to the adjustment component 706 which is configured to adjust, respectively, the textures and edges of the image (independently of one another). With the textures and edges adjusted as desired, the (adjusted) detail and large-scale feature images are forwarded from the adjustment component 706 to the re-composition component 708. The re-composition component 708 renders the adjusted original image from the adjusted detail and large-scale feature images.
It will be appreciated that the bilateral filter is an edge-preserving smoothing filter in both domain and range. In the domain (spatial), it acts as a typical Gaussian smoothing filter. In the domain (radiometric differences), it merely combines pixel values together that are close to the value at the center of the kernel, based upon a Gaussian distribution. This serves to mitigate smoothing across edges. Note that it is assumed that the scaling of the image components is a linear function in scale space (resulting in a gamma remapping in linear space). For consistency within results when dealing with ln(x) space, image values may be scaled between 0 and 1. Also, prior to scaling, the minimum possible image value should be made larger than 0, as 0 is undefined in ln(x) space.
The bilateral filter comprises:
Where some of the nomenclature is defined as follows:
BO Bilateral Filter
gO Gaussian or Pseudo-Gaussian Function (normalization is unnecessary)
I Original Input Image
s Pixel Being Solved
p A Pixel within the Kernel Ω Surrounding s
Ω The Kernel surrounding s (Typically a Square Region)
σh The Spatial Falloff of the Bilateral Filter (e.g., 4 pixels for 1 Megapixel resolution images). This may vary depending on how far the subject was from the camera when the image was acquired and what the image comprises. For example, if the subject is far away from the camera then the magnitude of this coefficient may be lower. Similarly, if the subject is detected to be a face (e.g., through facial recognition software) then certain settings deemed appropriate for facial/portrait images may be used.
σi The Radiometric Difference Falloff of the Bilateral Filter (e.g., 0.3)
k Large-Scale Feature Scalar
m Detail Feature Scalar
Accordingly, the large-scale features 808 of the image are output from the bilateral filter 806. The large-scale features of the image are at times also referred to as the large-scale feature image. In establishing the large-scale feature image 808, pixels which are similar to one another are combined together to substantially remove texture, leaving regions of similar intensity that have sharp edges but little to nothing else. This promotes edge retention and is accomplished in a manner that is much faster than other edge preservation techniques, such as anisotropic diffusion, for example. It will be appreciated that large-scale features generally comprise the information (e.g., edges) that most humans utilize to recognize objects.
The large-scale feature image 808 is applied to a differencing operation 810, as is the original image 802. Since the large-scale feature image 808 substantially comprises the edges of the image, the difference between this image 808 and the original image 802 corresponds to the textures of the image, which is referred to as the detail feature image. The detail feature image 812 generally comprises the subtle variations differentiating pixels whose values are near, but not necessarily similar, to one another. The detail feature image 812 is output from the differencing block 810 and is fed to a multiplier block 814 to selectively adjust the magnitude thereof. In the illustrated example, the detail feature image 812 is multiplied by a constant m in the multiplier block 814 to increase (e.g., m>1) or decrease (e.g., m<1) the magnitude of the detail feature image, and thus the relative amount of texture presented therein.
Similarly, the large-scale feature image 808 is applied to a multiplier block 816 to selectively adjust the magnitude thereof. In the illustrated example, the large-scale feature image 808 is multiplied by a constant k in the multiplier block 816 to increase (e.g., k>1) or decrease (e.g., k<1) the magnitude of the large-scale feature image, and thus the intensity of the edges presented therein. The adjusted large-scale feature image 808a and the adjusted detail feature image 812a are applied to an addition block 818 and a second optional logarithmic operation (e.g., expo) 820 is performed to recombine the images and render the original, but adjusted, image 802a back in the linear domain. Nevertheless, the first 804 and second 820 logarithmic operations are said to be optional as the foregoing calculations can also be performed in linear space.
It will be appreciated that the resultant image R 802a (e.g., the original adjusted image) can be obtained in the linear domain according to R=k(B(I,σh,σi))+m(I−B(I,σh,σi)).
The large-scale feature image 808 L is determined according to L=B(I,σh,σiwhere B(Input,σh,σi)s is the bilateral filter.
The detail feature image D 812 is thus determined according to D=I−L, where I is the input image 802 and L is the large-scale feature image 808.
The resultant image R 802a can also be thought of as R=(kL+mD), where m 814 is the detail image scalar, and k 816 is the large-scale image scalar
The adjusted large-scale feature image 808a can thus be thought of as Lσ=kL and the adjusted detail feature image 812a can be thought of as Dθ=mD such that the resultant image R 802a corresponds to R=Lσ+Dθ.
In the logarithmic domain, the resultant image R 802a can be determined according to R=exp(k(B(ln(I),σh,σi))+m(ln(I)−B(ln(I),σh,σi))).
The large-scale feature image 808 L is determined according to L=B(ln(I),σh,σi) and the detail feature image D 812 is determined according to D=ln(I)−L, where I is the input image 802 and L is the large-scale feature image 808. Nevertheless, it will be appreciated that L and D in the logarithmic domain are different than L and D in the linear domain (above).
The resultant image R 802a can also be thought of as R=exp(kL+mD), where m 814 is the detail image scalar, and k 816 is the large-scale image scalar.
The adjusted large-scale feature image 808a can thus be thought of as Lσ=kL and the adjusted detail feature image 812a can be thought of as Dθ=mD such that the resultant image R 802a corresponds to R=exp(Lθ+Dσ).
It will be appreciated that as an edge preserving filter, the bilateral filter provides a more perceptually-correct adjustment to the texture of digital images than traditional “unsharp mask” algorithms which are based on non-edge-preserving Gaussian filters. It will also be appreciated that a graphics processing unit or GPU of a computer can be utilized for the numerical processing necessary to implement the provisions set forth herein.
Turning to
Still another embodiment involves a computer-readable medium comprising processor-executable instructions configured to apply one or more of the techniques presented herein. An exemplary computer-readable medium that may be devised in these ways is illustrated in
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
As used in this application, the terms “component,” “module,” “system”, “interface”, and the like are generally intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a controller and the controller can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.
Furthermore, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. The term “article of manufacture” as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or media. Of course, those skilled in the art will recognize many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.
Although not required, embodiments are described in the general context of “computer readable instructions” being executed by one or more computing devices. Computer readable instructions may be distributed via computer readable media (discussed below). Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), data structures, and the like, that perform particular tasks or implement particular abstract data types. Typically, the functionality of the computer readable instructions may be combined or distributed as desired in various environments.
In other embodiments, device 1212 may include additional features and/or functionality. For example, device 1212 may also include additional storage (e.g., removable and/or non-removable) including, but not limited to, magnetic storage, optical storage, and the like. Such additional storage is illustrated in
The term “computer readable media” as used herein includes computer storage media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions or other data. Memory 1218 and storage 1220 are examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by device 1212. Any such computer storage media may be part of device 1212.
Device 1212 may also include communication connection(s) 1226 that allows device 1212 to communicate with other devices. Communication connection(s) 1226 may include, but is not limited to, a modem, a Network Interface Card (NIC), an integrated network interface, a radio frequency transmitter/receiver, an infrared port, a USB connection, or other interfaces for connecting computing device 1212 to other computing devices. Communication connection(s) 1226 may include a wired connection or a wireless connection. Communication connection(s) 1226 may transmit and/or receive communication media.
The term “computer readable media” may include communication media. Communication media typically embodies computer readable instructions or other data in a “modulated data signal” such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” may include a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
Device 1212 may include input device(s) 1224 such as keyboard, mouse, pen, voice input device, touch input device, infrared cameras, video input devices, and/or any other input device. Output device(s) 1222 such as one or more displays, speakers, printers, and/or any other output device may also be included in device 1212. Input device(s) 1224 and output device(s) 1222 may be connected to device 1212 via a wired connection, wireless connection, or any combination thereof. In one embodiment, an input device or an output device from another computing device may be used as input device(s) 1224 or output device(s) 1222 for computing device 1212.
Components of computing device 1212 may be connected by various interconnects, such as a bus. Such interconnects may include a Peripheral Component Interconnect (PCI), such as PCI Express, a Universal Serial Bus (USB), firewire (IEEE 1394), an optical bus structure, and the like. In another embodiment, components of computing device 1212 may be interconnected by a network. For example, memory 1218 may be comprised of multiple physical memory units located in different physical locations interconnected by a network.
Those skilled in the art will realize that storage devices utilized to store computer readable instructions may be distributed across a network. For example, a computing device 1230 accessible via network 1228 may store computer readable instructions to implement one or more embodiments provided herein. Computing device 1212 may access computing device 1230 and download a part or all of the computer readable instructions for execution. Alternatively, computing device 1212 may download pieces of the computer readable instructions, as needed, or some instructions may be executed at computing device 1212 and some at computing device 1230.
Various operations of embodiments are provided herein. In one embodiment, one or more of the operations described may constitute computer readable instructions stored on one or more computer readable media, which if executed by a computing device, will cause the computing device to perform the operations described. The order in which some or all of the operations are described should not be construed as to imply that these operations are necessarily order dependent. Alternative ordering will be appreciated by one skilled in the art having the benefit of this description. Further, it will be understood that not all operations are necessarily present in each embodiment provided herein.
Moreover, the word “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous over other aspects or designs. Rather, use of the word exemplary is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise, or clear from context, “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims may generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form.
Also, although the disclosure has been shown and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based upon a reading and understanding of this specification and the annexed drawings. The disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular regard to the various functions performed by the above described components (e.g., elements, resources, etc.), the terms used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure which performs the function in the herein illustrated exemplary implementations of the disclosure. In addition, while a particular feature of the disclosure may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms “includes”, “having”, “has”, “with”, or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising.”