Patent Application
20140369554
This application is directed, in general, to electronic image retouching and, more specifically, to real-time face beautification.
Digital photography has made it possible for even the most modest photographer to retouch images. Retouching digital portraits, sometimes referred to as face beautification, has become particularly popular among digital camera users. With the advent of smart phones and ever more powerful mobile computing devices, particularly those with embedded digital cameras, face beautification reaches far beyond traditional desktop and laptop personal computers (PCs).
Face beautification has generally been limited to basic imperfections, color, and texture. The proliferation of digital cameras in mobile devices has fostered a fresh demand in more robust techniques that seek to improve the overall appeal of a human face. While improving the apparent appeal of a face may seem unconventional, and sometimes superficial, it has many practical uses, including: special effects, advertising, social media, and dating services, in addition to traditional applications such as professional photography and art.
Objective standards for face beautification remain elusive; however, statistical models have proven helpful in removing at least some of the subjectivity. Many variables contribute to the complexity of the problem of objectively characterizing face beautification, including: gender, ethnicity, culture, and age, among others. For these reasons, automated solutions are generally limited to specialized software designed to detect and remedy objectively identifiable imperfections in still images. To the extent a particular enhancement remains subjective, automated face beautification typically requires an affirmative action by the user. Additionally, manual steps are often available to remedy structural imperfections in the skin, mouth, nose, eyes, or other facial feature.
One aspect provides a face beautification system. In one embodiment, the system includes: (1) a coarse feature detector configured to generate an approximation of facial features in an image, (2) an edge-preserving filter configured to reduce distortions in the approximation, and (3) a feature enhancer operable to selectively filter a facial feature from said approximation and carry out an enhancement.
Another aspect provides a method of face beautification for an image. In one embodiment, the method includes: (1) applying a coarse feature mask to the image, (2) low-pass filtering the coarse feature mask, (3) detecting precise facial features in the coarse feature mask, and (4) employing a feature enhancement on the precise facial features.
Yet another aspect provides a computing device. In one embodiment, the device includes: (1) a camera operable to capture an image, and (2) a graphics processing unit (GPU) having: (2a) coarse feature masks operable to generate an approximation of facial features in the image, (2b) an edge-preserving filter configured to reduce distortions in the approximation, (2c) a precise feature mask operable to extract a facial feature from the approximation, and (2d) an enhancement processor configured to apply an enhancement to the facial feature.
Reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:
It is realized herein that real-time face beautification is possible given the appropriate implementation. Real-time face beautification enables many video applications, including video conferencing and computational camera. It is further realized herein that real-time implementations of face beautification techniques, also called “feature enhancements,” allow for a more comprehensive, or “holistic,” face beautification on facial features, including but not limited to: skin, eyes, hair, nose, mouth, ears, chin, and cheeks.
It is realized herein that real-time face beautification can be embodied on a system-on-a-chip (SOC) with optimized features, such as filtering and sampling. For example, one embodiment of real-time face beautification is a graphics processing unit (GPU) configured for the various feature enhancements. It is realized herein that a coarse approximation of facial features and an edge-preserving filter may be used to localize facial features to a certain region of interest (ROI) within the image. A coarse approximation of facial features identifies the boundaries of the face and roughly detects major features, such as eyes, nose, and mouth. The edge-preserving filter refines the coarse approximation. It is realized herein the ROI should then be processed for precise feature detection. Precise feature detection labels each pixel in the ROA as belonging to the particular feature detected or not, identifying specific features to enhance, such as eye color, teeth, hairlines, and others. Precise feature detection offers increased accuracy and confidence in the location of the detected feature. A variety of conventional or later-developed face beautification techniques, or feature enhancements can then use the resulting precise feature mask. Additionally, post-processing can follow the enhancements to achieve additional effects, such as face glare and specular highlights of hair, among others.
An original image 140 is captured and passed to face beautification system 100 as an input. Original image 140 includes at least one human face. Original image 140 is first processed by coarse feature detector 110. Coarse feature detector 110 generates a coarse approximation of the facial features in original image 140 by localizing features to a certain region of interest within the image. Certain embodiments of coarse feature detector 110 include coarse feature masks formed by sampling original image 140 and deciding whether each sampled pixel is part of a facial feature or not. Use of a coarse approximation over something more precise yields significant time savings via lower sampling rates and lower decision thresholds. Although less computationally intense, coarse feature detector 110 can introduce distortions to the finished product, which can be mitigated by further processing.
Edge-preserving filter 120 receives the coarse feature representations detected by coarse feature detector 110. Edge-preserving filter 120, in certain embodiments, is simply a low-pass filter. In alternate embodiments, edge-preserving filter 120 is a domain transform filter. Edge-preserving filter 120 reduces the distortions introduced by coarse feature detector 110. Implementation of edge-preserving filter 120 on a GPU or SOC allows for its inclusion in face beautification system 100 without introducing unreasonable amounts of latency.
Enhancement module 130 operates on the filtered approximations of facial features yielded by a combination of coarse feature detector 110 and edge-preserving filter 120. Enhancement module, in certain embodiments, includes a precise feature detector and an enhancement processor. For example, one embodiment of an enhancement module could include a precise feature detector for detecting teeth and an enhancement processor for applying a whitening effect to the teeth. Certain embodiments of precise feature detection employ a precise feature mask that represents a particular facial feature. Construction of the feature mask would label each pixel within the ROI as belonging to the particular feature or not.
A wide variety of feature enhancements, such as teeth whitening, are possible, including: skin smoothing, re-colorization of skin, removal of color and structural imperfections on skin, increasing contrast and brightness of eyes (sclera brightening), increasing color saturation of iris, and modification of facial geometry, among many others. An enhanced image 150 results from processing at least one feature enhancement.
Multiple feature enhancements can be processed based on the filtered approximations. These multiple feature enhancements provide for a holistic face beautification system.
GPU 220 includes a coarse feature detector 230, an edge-preserving filter 240, a precise feature detector 250, and an enhancement processor 260. In certain embodiments, the components of GPU 220 are implemented as configurable hardware modules within the architecture of GPU 220 and having a software interface. In other embodiments, the components of GPU 220 are implemented as software modules. Selecting a hardware implementation versus a software implementation is a subjection matter, largely depending on the application and tolerance for latency. Certain hardware implementations offer faster processing at the expense of dynamic configuration. Certain software implementations are significantly more flexible, but cannot carry out face beautification processing as fast as the hardware alternatives.
Coarse feature detector 230 is configured to generate a coarse approximation of facial features present in the original image captured by camera 210. The coarse approximation is computationally efficient, requiring fewer samples and computations, and less time than more precise alternatives.
The coarse approximation can also introduce distortions to the enhanced image. Edge-preserving filter 240 mitigates the distortions by smoothing the coarse approximations while preserving the boundaries of facial features. Precise feature detector 250 detects particular facial features, such as eyes, ears, mouth, nose, hair, and others. Once the particular facial feature is detected, enhancement processor 260 carries out the actual enhancement, which typically includes some amount of filtering, gain, attenuation, or some other manipulation of the original image.
In a precise detection step 340, precise facial features are detected for later processing of feature enhancements. In an enhancement step 350, a feature enhancement is employed on the precise facial features.
In alternate embodiments, the method further includes a post-processing step. Post-processing occurs after employing feature enhancements and can includes post-processing effects such as bloom, or face glare, and increased specular highlights of hair, among others. These additional effects operate to improve the overall appeal of faces in the image. The method then ends in an end step 360.
Those skilled in the art to which this application relates will appreciate that other and further additions, deletions, substitutions and modifications may be made to the described embodiments.
This application claims the benefit of U.S. Provisional Application Ser. No. 61/863,487, filed by Albuz, et al., on Aug. 8, 2013, entitled “Method for Automatic Face Beautification in Real-Time Video Applications,” commonly assigned with this application and incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61836487 | Jun 2013 | US |
