Patent Grant
9101320
The present application is related to the U.S. patent application 13/859,384 and entitled “Skin Diagnostic and Image Processing Systems, Apparatus and Articles,” which is filed concurrently herewith, and the disclosure of which is incorporated by reference in its entirety herein.
Embodiments of the invention generally relate to skin diagnostic techniques, and more particularly, to skin diagnostic techniques employed in conjunction with image processing techniques.
Skincare or cosmetic visualizations aim to predict and illustrate to a consumer how the consumer's appearance may change in connection with the use of a skincare product or cosmetic treatment. However, the speculative nature of such exercises presents challenges in existing approaches with respect to accuracy and consistency of the visualized consumer results.
That is, a visualization is only as accurate as the data from which the visualization is derived. If the consumer results represented in the visualization are superficially determined based on mere speculation, then such results will not be accurate, and the consumer may become disillusioned with the skincare product or cosmetic treatment.
But even if the projected consumer results represented in the visualization happen to be close to actual results, how accurately the results are visualized can also have a significant effect on whether or not the consumer decides to purchase the skincare product or cosmetic treatment.
Embodiments of the invention provide skin diagnostic techniques employed in conjunction with image processing techniques.
In one embodiment, a method comprises the following steps. One or more diagnostic operations are performed on at least one portion of a user skin image to generate user skin image data, wherein the one or more diagnostic operations are associated with an identified skin-related application. The user skin image data is processed in accordance with the identified skin-related application. The processing comprises identifying one or more sets of skin image data in a database that correspond to the user skin image data based on one or more parameters specified by the skin-related application, and determining at least one image processing filter based on the one or more sets of identified skin image data from the database. The method further includes applying the at least one image processing filter to the at least one portion of the user skin image to generate a simulated user skin image.
In another embodiment, a system comprises a user information module, a graphical user interface, a skin image database, a processor and an output display. The user information module captures a user skin image. The graphical user interface enables selection of a skin-related application from a plurality of skin-related applications. The processor is coupled to the user information module, the graphical user interface, and the skin image database. Additionally, the processor is configured to determine user skin image data from the user skin image, and identify one or more sets of skin image data in the skin image database that correspond to the user skin image data based on one or more parameters specified by the skin-related application. The processor is also configured to apply at least one image processing filter that corresponds to the one or more identified sets of skin image data from the skin image database to the user skin image to generate a simulated user skin image. The output display, coupled to the processor, displays the simulated user skin image.
Embodiments of the invention can also be implemented in the form of an article of manufacture tangibly embodying computer readable instructions which, when implemented, cause one or more computing devices to carry out method steps, as described herein. Furthermore, other embodiments can be implemented in the form of an apparatus including a memory and at least one processor device that is coupled to the memory and operative to perform method steps.
Other embodiments of the invention can be implemented in the form of means for carrying out method steps described herein, or elements thereof. The means can, for example, include hardware module(s) or a combination of hardware and software modules, wherein the software modules are stored in a tangible computer-readable storage medium (or multiple such media).
Advantageously, illustrative embodiments of the invention provide techniques that leverage detailed skin and product information against image processing capabilities to generate accurate visual estimations for consumers.
These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.
Embodiments of the invention will be described herein with reference to exemplary computing and imaging system architectures. It is to be understood, however, that embodiments of the invention are not intended to be limited to these exemplary architectures but are rather more generally applicable to any system architectures wherein skin diagnostic techniques can be improved with the use of image compositing techniques such that accurate visual estimations are generated for the skin of a given subject.
As used herein, the term “image” is intended to refer to a rendered image (e.g., an image displayed on a screen), a data set representing an image (e.g., a data set stored or storable in memory), or some combination thereof. Thus, for example, the phrase “user skin image” comprises a rendered image of a portion of user skin, corresponding stored data representing the portion of the user skin, or some combination thereof. In the detailed description to follow, whether an image is being stored or rendered at a given time instance will be evident from the context of the particular illustrative embodiment being described.
As used herein, the phrase “skin-related application” is intended to refer to a diagnostic function or other process associated with the skin of a given subject. By way of example only, such skin-related applications that are embodied by a skin diagnostic and image compositing system as will be described herein may include, but are not limited to, a foundation matching application, a line and wrinkle application, a skin lightening application, a skin evenness application, a skin de-yellowing application, and a de-aging application. The particular application being performed by an application module of the system may be selectable by a user or automatically determined by the system from contextual information obtained and/or derived by the system.
As used herein, the term “module,” is intended to generally refer to hardware, software, or some combination thereof, that is configured to perform one or more particular functions in the system. If a module is intended to be implemented specifically as hardware or software, it will be referred to herein as a hardware module or a software module, respectively.
As will be described in illustrative detail below in the context of the figures, embodiments of the invention provide skin diagnostic and image compositing techniques which include, inter alia, obtaining a user skin image to generate corresponding user skin image data, processing the user skin image data against a database in accordance with a skin-related application, and generating a simulated user skin image based on the application of an identified image processing filter(s). Additionally, one or more embodiments of the invention may also include displaying the updated or simulated user skin image in conjunction with a recommendation of one or more relevant skin care products and/or treatment methods.
Referring initially to
In at least one embodiment of the invention, the personal information capture module 106 enables the acquisition of one or more user skin images and other user information. The module 106 may include one or more image capture devices for acquiring an image. The one or more capture devices may include image capture devices capable of capturing images in accordance with different ranges of the electromagnetic spectrum.
By way of example only, the captured images may include, but are not limited to, visible images, infrared (IR) images, and ultraviolet (UV) images. The phrase “visible image” refers to an image captured by a device configured to capture energy in the visible wavelength range of the electromagnetic spectrum. Similarly, an “infrared or IR image” and an “ultraviolet or UV image” respectively refer to images captured by devices configured to respectively capture energy in the IR wavelength range and the UV wavelength range of the electromagnetic spectrum. It is to be understood that the phrase UV images also may include “near UV” images. Further, the phrase “spectral image” refers to images in multiple wavelength ranges including, but not limited to, visible, IR and UV ranges.
Still further, the phrases “RGB image” and “Lab image” are used herein. RGB images are images generated based on the RGB color space model, which is an additive color model in which red, green, and blue light components are added together in different specified proportions to reproduce a broad array of colors. Lab images are images generated based on a color space model with a dimension L for lightness and a and b components representing color-opponent dimensions. The Lab color space model is based on nonlinearly compressed CIE (International Commission on Illumination) XYZ color space coordinates. RGB images and Lab images may be considered visible images. In one or more embodiments, as will be further explained below, RGB values are converted to Lab values, and vice versa, in a known manner.
As is known, ordinary white (visible) light is made up of waves that can travel at all possible angles. Light is considered to be “linearly polarized” when it is composed of waves that only travel in one specific plane. Thus, light waves that travel in a plane parallel to a reference plane of travel are considered parallel light waves, while light waves that travel in a plane perpendicular to the reference plane of travel are considered perpendicular light waves. Thus, as used herein, the phrase “polarized image” refers to an image that is separated into constituent linear polarization components including a “perpendicular light image component” and a “parallel light image component.” In contrast, the phrase “non-polarized image” refers to an image that is not separated into such constituent linear polarization components.
As further used herein, the phrase “cross polarization” refers to a polarization condition whereby an image is separated into two components: a “specular component” and an “undertone component.” The specular component represents light that reflects off of the surface of the skin, and the undertone component represents light that traverses the surface of the skin and reflects off of a subsurface. In one embodiment, the parallel light image component is comprised of a specular component and half of an undertone component, while the perpendicular light image component is comprised of the other half of the undertone component.
The capture module 106 may also enable the user 102 to enter other information, as well as select one or more previously captured images (viewable via the GUI 104) for processing by the system 100. Additionally, the user 102 can be queried by the system (for example, via the GUI 104) to respond to a series of questions to guide a subsequent analysis of the data corresponding to the captured skin image. Such analysis is carried out in accordance with a selected application via the application module 108. The application can be selected by the user 102 via the GUI 104 or can be automatically determined based on the user responses to the noted queries.
Based on the selected or determined application, one or more relevant portions of the databases 110 are accessed to aid in carrying out the analysis. As further described in connection with
Also, as described further herein, the application module 108 includes a processor module (not expressly shown in
The output of the analysis is generated for presentation on the output display 112 and includes an updated/simulated image and/or a changing series of simulated images. Such an output can include a visualization of the initial user skin image (e.g., user skin image prior to skin diagnostic operations being performed by the system 100) as well as a visualization representing how that image would change over a selected period of time, based on the severity of the queried variables in the user image contrasted against the severity of those variables in the relevant databases in relation to corresponding parameters such as age, race, gender, etc. It is noted that because different variables may change or evolve at different rates depending on initial severity and one or more corresponding parameters, such an analysis may not present a linear process. As such, an embodiment of the invention includes generating and leveraging relevant non-linear curves in connection with processing user skin images with one or more databases.
Accordingly, the system 100 is generally configured to acquire or select a skin image, process the skin image to obtain relevant skin image data, process the skin image data against one or more relevant databases to determine pertinent skin image data corresponding to a selected diagnostic application, and output a resulting set of simulated skin image data to a display.
Details of how the system 100 is able to perform these and other steps and operations are described below in connection with
By way of example,
It is to be appreciated that at least a portion of the data in the above databases 110 is compiled from images (for example, but not limited to, facial images) captured from a large number of human test subjects. The images include images covering a wide range of varying human demographics such as, for example, age, race, ethnicity, gender, geographic origin, etc. Data compiled from these images can include skin color data, skin texture data, etc., as will be further explained below. Thus, the data in the databases 110 includes images, information derived from such images, and information used to derive other information usable by the system 100.
The data in databases 110 is to be distinguished from data captured or otherwise obtained from a user of the system 100. That is, the data in databases 110 is predominantly data previously obtained from test subjects and other sources that is compiled for use in performing the selected diagnostic operations on a skin image provided by a current user 102 of the system 100. However, it is to be understood that data from the current user 102, subject to their approval, may become part of the data in databases 110 and used for a subsequent user 102.
Additionally, the images in the databases 110 may include images that are linked to specific skin conditions, as well as images that are linked to specific skincare products and/or treatments. In at least one embodiment of the invention, a skin-related application is identified via specification of a skin product and/or a skin product category (data associated therewith which is stored in one or more of the databases). Skincare product data can include, for example, age-related skin image data and skincare product efficacy data. Additionally, parameters specified by a skin-related application can include, by way of example, a severity-time parameter based on information (e.g., stored in product clinical performance database 212).
Furthermore, the databases 110 include spectral imaging data (e.g., stored in spectral imaging database 227). Spectral imaging data includes, but is not limited to, a plurality of two-dimensional digital spectral images of human skin that are captured from a variety of human subjects and stored (and categorized) in the database. A spectral image as mentioned above refers to image data captured at different wavelength ranges across the electromagnetic spectrum. Such spectral images can include visible images, as mentioned above, as well as images captured at wavelengths that allow extraction of additional information that the human eye fails to capture with its receptors for the red, green and blue (RGB) light components, e.g., infrared images, ultraviolet images, etc. Each spectral image stored in the database defines a target area of skin. By way of example only, such digital spectral images may be captured and stored in a manner described in International Publication No. WO2011/112422, entitled “System for Skin Treatment Analysis Using Spectral Image Data to Generate 3D RGB Model,” filed on Mar. 3, 2011, and commonly owned by the assignee of the present application, the disclosure of which is incorporated by reference herein in its entirety.
Thus, a corresponding plurality of two-dimensional digital RGB (red, green, blue) color model images are captured and stored in the databases 110 (e.g., image analysis database 229). Each of the RGB images corresponds at least in part to at least one of the spectral images defining a target area of skin. During processing for user 102, as will be further explained below, a portion (or all) of the plurality of spectral images are analyzed to identify within the respective spectral image one or more spectral image datasets. As used herein, a spectral image dataset refers to the minimum amount of spectral image digital data required to uniquely define a condition of the skin, as, for example, associated with a particular variable or parameter such as skin type, blood or melanin level, oxygen saturation, percent hemoglobin, percent water or moisture content, etc.
As discussed herein in connection with one or more embodiments of the invention, the selected or defined skin condition may be a skin condition not needing treatment or correction, or the skin condition may be a treatable or correctable skin condition such as, for example, dry, oily, cracked, and other treatable, correctable skin conditions. In any case, the spectral image datasets define one or more such skin conditions.
As noted, each element within each image is recorded and indexed based, for example, on pixel coordinates on the image, RGB values of the pixel and/or spectral content of the pixel, and type of skin condition at that pixel. Accordingly, each skin condition is mapped to one or more pixels in the respective image. More specifically, each spectral image dataset is mapped to a location within the respective spectral image (referred to herein as the spectral location). That is, a spectral location includes the pixel coordinate location within a spectral image for a spectral image dataset. In an RGB image corresponding to a respective spectral image, a location is mapped that corresponds to each spectral location. The location in the RGB image is referred to herein as the RGB location; that is, the pixel coordinate location within an RGB image that corresponds to a spectral location in a respective spectral image.
Additionally, as used herein, an RGB dataset refers to the minimum amount of digital RGB data required to uniquely identify an RGB color profile associated with that respective location. Accordingly, in at least one embodiment of the invention, the spectral image dataset is effectively correlated to an RGB dataset that corresponds to at least one known skin condition defined by said spectral image dataset. Also, an RGB dataset is created, pixel-by-pixel, from each spectral image dataset by passing the spectral image data through a conversion function with the area under each resulting curve being summed to provide the RGB dataset. The spectral curve for each pixel in the spectral image dataset for a specific subject is fit, using known curve fitting methods, to reveal the details of the skin biology and chemistry. One parameter, melanin concentration, is uniquely tied to the whitening behavior of certain products. In order to simulate such whitening effects of a product by the alteration of melanin concentrations in skin, the spectral image dataset at each pixel is first divided by a function Rmel(λ) which describes the reflectance of melanin at the particular concentration of melanin x for that subject. This results in a “melaninless” spectral curve, which is multipled by the new melanin curve, which is found by a function RNmel(λ). Before RNmel(λ) can be calculated, the change in melanin concentration is found by using a data chart (e.g., such as the one in
The function which describes Rmel(λ) and RNmel(λ) is EXP^(0.012x*MUAmelλ*(4.51−Log10(x*MUAmel%) where x is the average melanin concentration at a specific timepoint of interest and MUAmei represents the known absorbtion curve of melanin. This new curve is then multiplied by the “melaninless” curve to create the new spectral curve at the new melanin concentration. This process yields a spectral image dataset with the altered melanin concentration. The spectral image dataset is then converted to an RGB image dataset.
The conversion function for transforming the spectral image dataset to an RGB dataset involves multiplying the spectral image dataset by individual R, G, and B spectral response functions, and subsequently summing the area below the curve for each and then dividing each by the respective area below the curve of each corresponding spectral response curve. This results in values for R, G, and B that yields the color image in RGB color space. The spectral response function is obtained by spectral imaging of standard R, G and B color reference targets with a spectral camera. In this manner, a series of images are created which simulates the effects of whitening from product usage over time related to melanin concentration. The general RGB conversions for whitening at each timepoint are then found in a straightforward manner by dividing the average RGB values of an average area of the starting image by the corresponding average area in each simulated image, i.e., using the specific x calculated from the data in a melanin percentage change chart (e.g.,
The conversion function is optimized from the minimization of the differences between the measured RGB values in RGB space and those values calculated from the transformation RGB of the spectral dataset. Accordingly, in at least one example embodiment of the invention, a virtual look-up table (LUT) between the RGB dataset and the spectral image dataset is established that is representative across all spectral image datasets. Such mappings and LUTs are stored in the databases 110 (e.g., stored in the spectral imaging database 227, the image analysis database 229, or a combination thereof).
Advantageously, different skin conditions are catalogued in spectral datasets and correspond to determinable reference RGB datasets. The captured spectral images and corresponding captured RGB images are compiled and stored along with the spectral image datasets representing skin conditions, the spectral locations, the RGB locations and the reference RGB datasets.
Still further, in one or more embodiments, the RGB datasets are converted to Lab datasets such that the different skin conditions are catalogued in spectral datasets that correspond to determinable reference Lab datasets.
Still referring to
Additionally, the databases 110 includes data that indicates how physical properties such as wrinkles, pores, fine lines, dark circles, reddening in the cheeks, elasticity of the skin, etc. change and vary in different demographic groups (e.g., stored in the textural science database 207).
As also noted above, the databases 110 include data pertaining to product clinical performance (e.g., stored in databases 213 and 215). Apparent age data (e.g., stored in database 213) contains data and models that are used to assign an apparent age, as compared to a chronological age, to a person. The phrase “chronological age” or actual age refers to the age of a person in terms of the person's actual life span. The phrase “apparent age” refers to the age that a person is visually estimated or perceived to be, based on their physical appearance, particularly the overall appearance of the face. Chronological age and apparent age are generally measured in years and parts thereof. One goal of anti-aging skincare products is to reduce apparent age relative to chronological age, preferably reducing apparent age below chronological age, so that a person appears younger than their actual age. Products that achieve this goal are able to prevent skin damage and/or remove damage induced by age-promoting factors. By way of example only, such apparent age data and models may be generated and stored in a manner described in International Publication No. WO2010/028247, entitled “An Objective Model of Apparent Age, Methods and Use,” filed on Sep. 4, 2009, and commonly owned by the assignee of the present application, the disclosure of which is incorporated by reference herein in its entirety.
Product efficacy data (e.g., stored in database 215) includes data that indicates how certain skincare products and treatments behaved and/or reacted in connection with various types of human skin over varying periods of time and treatment regimens. More specifically, skincare products and treatments are composed and/or arranged in certain manners and with certain sets of ingredients or components so as to target and/or treat one or more particular skin conditions (for example, reduce or remove wrinkles, lighten skin tone, even-out skin tone, etc.). Such information is included in the product efficacy database 215, along with data pertaining to the corresponding targeted objectives of the product or treatment.
Data that describes the uniformity, radiance, or dullness of the skin, or the location and size of different types of spots, including age spots, freckles, etc. (below the skin) is stored in the databases 110 (e.g., cross polarization database 223). Data describing the location, size, severity, and length of wrinkles, and the location, size, severity and diameter of pores is also stored in the databases 110 (e.g., photography database 225).
In at least one embodiment of the invention, the image capture module 302 includes, for example, one or more image capture devices for acquiring an image. For example, the one or more capture devices may include image capture devices capable of capturing images in accordance with different ranges of the electromagnetic spectrum, e.g., visible images, infrared images, and ultraviolet images. That is, module 302 includes one or more digital cameras capable of capturing visible images, and one or more cameras, devices and sensors capable of capturing images in other electromagnetic spectrum wavelength regions (e.g., infrared, ultraviolet, etc.). In one embodiment, the camera is a polarization-enabled camera which is configured to capture three image components: parallel, perpendicular, and non-polarized. One or more of the image capture devices are also preferably configured to capture specular image components and undertone image components, as described herein with regard to cross polarization embodiments.
Additionally, the text capture module 304 can include, for example, a keyboard or keypad for manual text input, and/or a device configured for automatic speech recognition (ASR) such as a speech-to-text (STT) module.
The captured and/or compiled information is used to analyze skin conditions of an individual subject or user by comparing datasets derived from the images to reference datasets in the databases 110 depicted in
As shown, the GUI 104-1 includes touch screen-enabled selection features 402, 404 and 406. Such features enable the user to direct the system to capture his/her own image (or “photo”) via feature 402, or connect to a system database and upload a pre-existing image from either a set of models 405 (via feature 404) or other kiosk users (via feature 406).
Accordingly, in the example implementation of a retail location, a user or customer has his or her photograph taken at a kiosk, the photograph is analyzed in accordance with the system 100, and an advisor or other enterprise personnel subsequently provides diagnostic results and/or recommendations generated by the system 100 away from the kiosk via a tablet or other device configured according to enterprise preference or specifications. Of course, the results and/or recommendations may be presented directly to the user or customer without the need for an advisor or other personnel.
Additionally, another GUI feature includes a zoom-in and/or zoom-out feature for shrinking or enlarging a portion of the user skin image, and localized inspection of images. That is, the user is able to point to a specific facial area in the image and have that location enlarged (and then shrunk again) within a window such as the circular window labeled 504.
Further, GUI features may also include a contrast feature, as well as a lighting simulation feature so as to, for example, simulate daylight or incandescent lighting. Still further, GUI features may include a foundation finder “wand” or selection feature to redefine a diagnostic sampling area for determining foundation shades. It is to be appreciated that the GUI 104 may provide the user 102 with any known image manipulation features (not expressly shown) that would aid in the diagnostic operations of the system, as well as aid in increasing the positive experience the user has with the system.
Such GUI features can, for example, be implemented in the form of active buttons on the user interface, via a pop-up tool bar on the user interface, etc. Further, in at least one embodiment of the invention, additional options on the GUI include links to external sites and sources such as various e-commerce enterprises, global positioning systems, social networks, etc.
In connection with the depiction in
Accordingly, the diagnostic module 602, in conjunction with the sub-application selection module 604, is configured to determine one or more conditions that need correcting on the user's skin from the one or more images captured of the user. Then, based on the diagnosed problem, the appropriate sub-application is selected. The user can specify a skin region that he/she wishes to be diagnosed by the system. Alternatively, the system can automatically find the problem region(s). Still further, the user can directly specify what sub-application he/she wishes to engage. In any event, a diagnostic region is chosen, and a sub-application is selected in accordance with modules 602 and 604.
Once the sub-application is chosen, the sub-application operates in conjunction with data in the database environment 110, as described above, to generate an image (or set of images) via the image processing module 606 that represents results of the particular diagnostic operations performed in accordance with the chosen sub-application. The image (simulated appearance change image) is displayed via module 608 (through GUI 104 and output display 112 in
As shown in
The image processing module 606 applies: (i) the non-polarized image filter 708 to the non-polarized image component 702 to generate a proscenium image component; (ii) the parallel light image filter 710 to the parallel light image component 704 to generate a simulated parallel light image component; and (iii) the perpendicular light image filter 712 to the perpendicular light image component 706 to generate a simulated perpendicular light image component. The simulated parallel light image component and the simulated perpendicular light image component are combined in a first combination module 714, for example, using the equation (Para+Perp)/2, to generate a base simulated user image for the skin-related application. The base simulated user image is combined with the proscenium image component in a second combination module 716 to generate the simulated user skin image. The combination operations are referred to herein as “image compositing,” a visual example of which will be described below in the context of
Recall that databases 110 contain data describing a large range of facial features (e.g., pore size, wrinkle lengths and widths, age spots, skin color, skin whitening/yellowing, skin uniformity, under eye dark circles, etc.) as a function of natural aging and specific product effects. The data includes average values as a function of age and average values of the time effects of products. Hence, these numerical sequences represent a record of how the average skin changes for that specific feature, either as a direct function of aging or as a result of the specific product application time. This data has been compiled over time by research and clinical scientists, using physical measurements (e.g., photographic, etc.) and expert panel assessment of photographic imagery.
As such, image processing module 606 obtains the polarized image components (parallel light image component 704 and perpendicular light image component 706) for the subject user and the corresponding filters (710 and 712) then transform the image components on a pixel by pixel basis such that the resultant combined non-polarized image visually matches the expected overall average time behavior of the particular product. The image transforming filters 710 and 712 are created using photographic reference and physical measurement information from the databases 110. These filters perform mathematical transformations on each pixel such that the resultant transformed polarized images, when combined into the non-polarized image, give the realistic rendering of a product's average behavior at a particular time.
It is to be undrstood that the image filter 708 is driven by facial reconignition where the face is automatically located, and the eyes, nose, lips, hair, and the edge of the face are then located. All other parts of the image component that are not a part of these located areas are made transparent and used to create the proscenium image, which allows the background to remain constant as well as the eyes, nostrils, and lips which do not change during a skin treatment application. In one embodiment, the filtered parallel and perpendicular image components are combined by use of the equation (Para+Perp)/2 to create the displayable facial image.
As an example, the function that describes the time varying behavior of a whitening skincare product relies on physical measurements that determine the change in skin color over time. Expert panel assessments of photographic images have been acquired for these products, which yield qualitatively similar trends to the physical measurements. However, exact color measurements can be used. For these whitening products, the average values of L, a, and b show a modulation over time yielding a function that describes the average change for a given skin parameter, see
In accordance with an alternative embodiment of the invention, a methodology is provided to create a simulation of the continuous change of facial appearance over time (de-aging or a product effect) as a sequence of images, similar to a scrolling or of a playing movie. The methodology, in one embodiment, incorporates five timepoint changes, however, this could be any number of timepoints. In this alternative method, the initial captured polarized image components (“Para initial” and “Perp initial”) are mixed with polarized image components (“Para final” and “Perp final”) and subsequently combined to form the non-polrized image for any given time point. “Para final” and “Perp final” are created from the initial polarized image components, by changing the image components to reflect an overall product endpoint, or by zeroing out the specific facial features to bring a person back to their pre-aging youthful state. To a first approximation, a linear mixing of the images is used. “Para Initial” is combined with “Para Final,” following the equation ParaInitial(1−T)+ParaFinal(T)=ParaTransformed at timepoint T. “Perp Initial” is combined with “Perp Final,” following the equation PerpInitial(1−T)+PerpFinal(T)=PerpTransformed at timepoint T. T corresponds to normalized time and lies between zero and one, T=1 is final time. This linear mixing function could also be given a nonlinear functional form as described in the apparent age or the product time functional behavior stored in the databases 110. However, visually realistic simulations of product behavior and de-aging are achieved using the linear relationship, which can be subsequently adjusted to exactly match visual changes in appearance due to products or age.
As noted above, three image components are captured and processed as inputs, and a single image is created and displayed as the simulated user skin image. As depicted in
Combined simulated user skin image 904 represents all three layers (902-1, 902-2, and 902-3) composited to form the final image. Further, by way of illustration, the bottom image in
As detailed above in the context of
It is assumed that at least one user skin image is obtained. Via the GUI 104, the user 102 chooses a product type and also selects an area in the user skin image that he/she wishes to have diagnosed or otherwise processed by the system, referred to as “choose average method” (
Thus, in this specific example, module 1002 obtains the user skin image and determines average color values for the given area of the image selected by the user, i.e., generates skin image data from the skin image. Module 1004 identifies one or more sets of skin image data in the databases 110 that match or correspond to the user skin image data generated by module 1002. Module 1006 processes the image to determine the appropriate image processing filters (e.g., 708, 710, and 712 in
Additionally, in conjunction with module 1008, the user 102 (via GUI 104) may select a match mode (“choose match mode”) and an application mode (“choose application mode”), as well as one or more particular shades or tones (“choose shades”), if applicable. The application mode allows a user to apply a specific shade onto the skin, adjust how much is applied and allows the user to see half of the face (or some other percentage) with the product on the face while the other half (or remaining percentage) is his/her original image. The choose shades option allows the user to choose other shades other than the natural match shade to account for consumer preferences. The application can show shades that are lighter, darker, more yellow, or more red, as compared to the natural shade, but that would still be appropriate for the user. The match mode selection allows for choosing parameters used by the sub-application to find the closest matches.
As described herein, it is to be understood that diagnostic operations of the sub-application include determining user RGB color space values for one or more areas of the selected or identified portion of the user skin image. Additionally, the sub-application includes calculating average RGB color space values of the user RGB color space values for the areas of the selected portion of the user skin image, and converting the average RGB color space values to user L, a, b color space values. Further, one or more sets of skin image data are identified in the database that correspond to the user skin image data via identifying one or more L, a, b color space values in the database that approximately match the user L, a, b color space values. The appropriate image processing filters are determined and/or set based on the one or more identified L, a, b color space values from the database. Further, as described herein, the sub-application includes accessing a look-up table (LUT) for identifying one or more L, a, b color space values from one or more spectral feature values.
Thus, advantageously in the foundation matching example shown in
More particularly, in one embodiment, the user touches and/or selects an area of the image (for example, a cheek portion of the face). RGB values are averaged over a region (for example, a 50×50 pixel region) in the selected or touched region of the image. Ravg, Gavg, Bavg values are converted to Lavg, Aavg, Bavg color space values using conventional color model conversion techniques, and the deviation of Lavg, Aavg, Bavg values from product colors stored in the databases 110 is calculated using the expression E=sqrt((L−Lavg)2+((A−Aavg)2((B−Bavg)2). A pre-determined number (for example, five) of the closest matches from the databases are returned and the RGB values for the relevant shades are returned and used to set the appropriate filters for image processing, i.e., generate and apply the filters for the appropriate foundation shades.
Such techniques and such an example application are useful, for example, for simulating the application of powder foundations and can be adjusted to clinically determined behavior. Further, as with other applications, the foundation matching application enables the user to redefine the sampling region using a GUI selection feature.
While
By way of further example, a skin lightening application (i.e., sub-application 616 in
A lines and wrinkles application (i.e., sub-application 614) includes displaying timepoint product behavior of line and wrinkle-related skincare products. The image processing filters are set to match particular product behaviors obtained through clinical product testing. More specifically, in accordance with a lines and wrinkles application, an image is chosen from a database or library, or a user image is captured. The user touches and/or selects an area of the image (for example, a cheek portion of the face). A box (by way of example, a 3″×3″ box) blur is applied to the parallel image within the relevant image processing filter and the result is combined with the original parallel image. The opacity of the blur image is controlled by a calibration matrix and can, in general, vary from approximately 0.1 opacity at early product usage times to approximately 0.6 opacity at subsequent product usage times.
Advantageously, with the given processing and filtering framework provided herein, one of ordinary skill in the art will realize many additional applications that can be implemented by the skin diagnostic and image compositing system in a straightforward manner. Other examples include, but are not limited to, a pore application, skin non-uniformity application, and dark under eye circles application.
By way of illustration,
As used herein, the term “processor” refers to one or more individual processing devices including, for example, a central processing unit (CPU), a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other type of processing circuitry, as well as portions or combinations of such circuitry elements.
Additionally, the term “memory” refers to memory associated with a processor, such as, for example, random access memory (RAM), read only memory (ROM), a removable memory device, a fixed memory device, and/or a flash memory. Media 1118 may be an example of removable memory, while the other types of memory mentioned may be examples of memory 1104. Furthermore, the terms “memory” and “media” may be viewed as examples of what are more generally referred to as a “computer program product.” A computer program product is configured to store computer program code (i.e., software, microcode, program instructions, etc.). For example, computer program code when loaded from memory 1104 and/or media 118 and executed by processor 1102 causes the device to perform functions associated with one or more of the components and techniques of system 100. One skilled in the art would be readily able to implement such computer program code given the teachings provided herein. Similarly, the components and techniques described herein may be implemented via a computer program product that includes computer program code stored in a “computer readable storage medium.” Other examples of computer program products embodying embodiments of the invention may include, for example, optical or magnetic disks. Further, computer program code may be downloaded from a network (e.g., through network interface 1114) and executed by the system.
Still further, the I/O interface formed by devices 1106 and 1108 is used for inputting data to the processor 1102 and for providing initial, intermediate and/or final results associated with the processor 1102.
It is to be appreciated that one, more than one, or all of the computing devices 1204 in
As described herein, the computing devices 1204 may represent a large variety of devices. For example, the computing devices 1204 can include a portable device such as a mobile telephone, a smart phone, personal digital assistant (PDA), tablet, computer, a client device, etc. The computing devices 1204 may alternatively include a desktop or laptop personal computer (PC), a server, a microcomputer, a workstation, a kiosk, a mainframe computer, or any other information processing device which can implement any or all of the techniques detailed in accordance with one or more embodiments of the invention.
One or more of the computing devices 1204 may also be considered a “user.” The term “user,” as used in this context, should be understood to encompass, by way of example and without limitation, a user device, a person utilizing or otherwise associated with the device, or a combination of both. An operation described herein as being performed by a user may therefore, for example, be performed by a user device, a person utilizing or otherwise associated with the device, or by a combination of both the person and the device, the context of which is apparent from the description.
Additionally, as noted herein, one or more modules, elements or components described in connection with embodiments of the invention can be located geographically-remote from one or more other modules, elements or components. That is, for example, the modules, elements or components shown and described in the context of
By way of example, in an Internet-based and/or telephony-based environment, the system is configured to enable a user to capture (or select) an image via a smart phone or mobile device (one of the computing devices 1204 in
Additionally, for example, in a kiosk-based environment, a device (one of the computing devices 1204 in
In a LAN-based environment, all image capture, processing and analysis can be performed by one or more computing devices (1204 in
It is to be appreciated that combinations of the different implementation environments are contemplated as being within the scope of embodiments of the invention. One of ordinary skill in the art will realize alternative implementations given the illustrative teachings provided herein.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Additionally, the terms “comprises” and/or “comprising,” as used herein, specify the presence of stated values, features, steps, operations, modules, elements, and/or components, but do not preclude the presence or addition of another value, feature, step, operation, module, element, component, and/or group thereof.
The descriptions of the various embodiments of the invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5478238 | Gourtou et al. | Dec 1995 | A |
| 5537211 | Dial | Jul 1996 | A |
| 5562109 | Tobiason | Oct 1996 | A |
| 5622692 | Rigg et al. | Apr 1997 | A |
| 5636637 | Guiolet et al. | Jun 1997 | A |
| 5785960 | Rigg et al. | Jul 1998 | A |
| 5797750 | Gouriou et al. | Aug 1998 | A |
| 5944667 | Leveque et al. | Aug 1999 | A |
| 5945112 | Flynn et al. | Aug 1999 | A |
| 5987351 | Chance | Nov 1999 | A |
| 6091836 | Takano et al. | Jul 2000 | A |
| 6097838 | Klassen et al. | Aug 2000 | A |
| 6111973 | Holt et al. | Aug 2000 | A |
| 6292575 | Bortolussi et al. | Sep 2001 | B1 |
| 6293284 | Rigg | Sep 2001 | B1 |
| 6297420 | Heincke | Oct 2001 | B1 |
| 6571003 | Hillebrand et al. | May 2003 | B1 |
| 6603550 | Flynn et al. | Aug 2003 | B1 |
| 6681032 | Bortolussi et al. | Jan 2004 | B2 |
| 6807297 | Tankovich et al. | Oct 2004 | B1 |
| 6944491 | Leveque | Sep 2005 | B2 |
| 7035456 | Lestideau | Apr 2006 | B2 |
| 7061617 | Querleux et al. | Jun 2006 | B2 |
| 7104800 | Ortiz-Valero et al. | Sep 2006 | B2 |
| 7130454 | Berube et al. | Oct 2006 | B1 |
| 7218759 | Ho et al. | May 2007 | B1 |
| 7233693 | Momma | Jun 2007 | B2 |
| 7336810 | Fujii et al. | Feb 2008 | B2 |
| 7392204 | Matsumoto et al. | Jun 2008 | B1 |
| 7402135 | Leveque et al. | Jul 2008 | B2 |
| 7437344 | Peyrelevade | Oct 2008 | B2 |
| 7454046 | Chhibber et al. | Nov 2008 | B2 |
| 7477767 | Chhibber et al. | Jan 2009 | B2 |
| 7489960 | Leveque | Feb 2009 | B2 |
| 7522769 | Harville et al. | Apr 2009 | B2 |
| 7532743 | Morisada | May 2009 | B2 |
| 7558416 | Payonk et al. | Jul 2009 | B2 |
| 7634538 | Mori et al. | Dec 2009 | B2 |
| 7764303 | Pote et al. | Jul 2010 | B2 |
| 7819311 | Rowe et al. | Oct 2010 | B2 |
| 7820972 | Miyamae et al. | Oct 2010 | B2 |
| 7831072 | Rowe | Nov 2010 | B2 |
| 7835554 | Rowe | Nov 2010 | B2 |
| 7840064 | Chhibber et al. | Nov 2010 | B2 |
| 7894651 | Gutkowicz-Krusin et al. | Feb 2011 | B2 |
| 7901355 | Querleux et al. | Mar 2011 | B2 |
| 7916905 | Yen et al. | Mar 2011 | B2 |
| 7916910 | Cotton et al. | Mar 2011 | B2 |
| 7944689 | Nelson | May 2011 | B2 |
| 7970456 | Preece et al. | Jun 2011 | B2 |
| 7986987 | Bazin et al. | Jul 2011 | B2 |
| 8026942 | Payonk et al. | Sep 2011 | B2 |
| 8027505 | Edgar et al. | Sep 2011 | B2 |
| 8043227 | Van Gogh et al. | Oct 2011 | B2 |
| 8055066 | Xiong et al. | Nov 2011 | B2 |
| 8094186 | Fukuoka et al. | Jan 2012 | B2 |
| 8109875 | Gizewski | Feb 2012 | B2 |
| 8121430 | Corcoran et al. | Feb 2012 | B2 |
| 8131029 | Chhibber et al. | Mar 2012 | B2 |
| 8139854 | Oh et al. | Mar 2012 | B2 |
| 8150192 | Niemeyer et al. | Apr 2012 | B2 |
| 8155413 | Chhibber et al. | Apr 2012 | B2 |
| 8290257 | Demirli et al. | Oct 2012 | B2 |
| 20030045799 | Bazin et al. | Mar 2003 | A1 |
| 20030050561 | Bazin et al. | Mar 2003 | A1 |
| 20030067545 | Giron et al. | Apr 2003 | A1 |
| 20040125996 | Eddowes et al. | Jul 2004 | A1 |
| 20040218810 | Momma | Nov 2004 | A1 |
| 20050053637 | Ma'Or et al. | Mar 2005 | A1 |
| 20050106103 | Dussaud et al. | May 2005 | A1 |
| 20050111729 | Caisey | May 2005 | A1 |
| 20050119539 | Bazin | Jun 2005 | A1 |
| 20060029265 | Kim et al. | Feb 2006 | A1 |
| 20060092315 | Payonk et al. | May 2006 | A1 |
| 20060129411 | Bhatti et al. | Jun 2006 | A1 |
| 20060229912 | Negishi et al. | Oct 2006 | A1 |
| 20060239547 | Robinson et al. | Oct 2006 | A1 |
| 20070058858 | Harville et al. | Mar 2007 | A1 |
| 20070076013 | Campbell et al. | Apr 2007 | A1 |
| 20070086651 | Stephan et al. | Apr 2007 | A1 |
| 20070092160 | Fujii et al. | Apr 2007 | A1 |
| 20080080766 | Payonk et al. | Apr 2008 | A1 |
| 20080130970 | Niemeyer et al. | Jun 2008 | A1 |
| 20080161661 | Gizewski | Jul 2008 | A1 |
| 20080194928 | Bandic et al. | Aug 2008 | A1 |
| 20080212894 | Demirli et al. | Sep 2008 | A1 |
| 20080214907 | Gutkowicz-Krusin et al. | Sep 2008 | A1 |
| 20080240606 | Yamaguchi | Oct 2008 | A1 |
| 20080260218 | Smith et al. | Oct 2008 | A1 |
| 20090028380 | Hillebrand et al. | Jan 2009 | A1 |
| 20090054744 | Kitamura et al. | Feb 2009 | A1 |
| 20090076639 | Pak | Mar 2009 | A1 |
| 20090141956 | Chhibber et al. | Jun 2009 | A1 |
| 20090196466 | Capata et al. | Aug 2009 | A1 |
| 20090201365 | Fukuoka et al. | Aug 2009 | A1 |
| 20090213379 | Carroll et al. | Aug 2009 | A1 |
| 20090253162 | Windsor et al. | Oct 2009 | A1 |
| 20090327890 | Mertz et al. | Dec 2009 | A1 |
| 20100027002 | Claps | Feb 2010 | A1 |
| 20100105102 | Hanes et al. | Apr 2010 | A1 |
| 20100185064 | Bandic et al. | Jul 2010 | A1 |
| 20100195911 | Yen et al. | Aug 2010 | A1 |
| 20100202673 | Zhang et al. | Aug 2010 | A1 |
| 20100226531 | Goto | Sep 2010 | A1 |
| 20100302358 | Chen et al. | Dec 2010 | A1 |
| 20110016001 | Schieffelin | Jan 2011 | A1 |
| 20110020857 | Honkonen et al. | Jan 2011 | A1 |
| 20110116691 | Chung et al. | May 2011 | A1 |
| 20110123703 | Mohammadi et al. | May 2011 | A1 |
| 20110134275 | Nguyen | Jun 2011 | A1 |
| 20110164787 | Legagneur et al. | Jul 2011 | A1 |
| 20110206254 | Patwardhan | Aug 2011 | A1 |
| 20110213253 | Kruglick | Sep 2011 | A1 |
| 20110286643 | Kislal | Nov 2011 | A1 |
| 20110300196 | Mohammadi et al. | Dec 2011 | A1 |
| 20110304705 | Kantor et al. | Dec 2011 | A1 |
| 20110317917 | Free | Dec 2011 | A1 |
| 20120008838 | Guyon et al. | Jan 2012 | A1 |
| 20120038787 | Petrescu et al. | Feb 2012 | A1 |
| 20120041282 | Nichol et al. | Feb 2012 | A1 |
| 20120075432 | Bilbrey et al. | Mar 2012 | A1 |
| 20130058524 | Cohen et al. | Mar 2013 | A1 |
| Number | Date | Country |
|---|---|---|
| 2010028247 | Mar 2010 | WO |
| 2011112422 | Sep 2011 | WO |
| PCTUS2014014886 | Apr 2014 | WO |
| PCTUS2014014891 | May 2014 | WO |
| Entry |
|---|
| N. Bhatti et al., “Mobile Cosmetics Advisor: An Imaging Based Mobile Service,” Multimedia on Mobile Devices, Proceedings of SPIE—IS&T Electronic Imaging, Jan. 2010, 11 pages, vol. 7542. |
| Number | Date | Country | |
|---|---|---|---|
| 20140323873 A1 | Oct 2014 | US |
