The invention relates to generating photorealistic renderings of cosmetic products, in particular for an augmented reality application.
Augmented reality denotes artificially superimposing elements computed by a computer system onto a real scene.
Augmented reality may be used to obtain an overview of a result of the element computed by the computer system being present in the real scene.
One possible application of augmented reality is that of a computer system superimposing a virtual cosmetic product on a person. Such superimposition makes it possible to have an overview of the application of the cosmetic product to the person. It is thus possible for a person to virtually try out cosmetic products.
For example, the cosmetic product may be a make-up product (blusher, foundation, mascara, lipstick, gloss, nail varnish, eyeliner, etc.). The cosmetic product may also be a haircare product such as a dye.
It is difficult to realistically generate and apply a virtual cosmetic product to a person.
In order to simplify the application of virtual cosmetic products to a person, virtual cosmetic products are generally limited to predefined products, in particular to products having predefined shades and colours. Such a solution therefore has the drawback of not allowing a user to virtually try out cosmetic products that are not provided among the predefined products.
To resolve this drawback, some methods have been proposed for virtually applying a cosmetic product to a person based on a reference image of another person to whom the cosmetic product is actually applied. It has in particular been proposed to use complex artificial neural networks configured to extract the appearance of the cosmetic product from the reference image and then to virtually apply the extracted cosmetic product to the image of another person.
Such artificial neural networks require a significant amount of computing resources to be implemented. Furthermore, the execution time of these neural networks is relatively lengthy. This solution thus cannot generally be implemented on portable appliances, for example smartphones, to obtain rendering of the virtual cosmetic product in real time.
In addition, the quality of the renderings able to be achieved using such artificial neural networks is strongly dependent on the data used to train these artificial neural networks. For example, the renderings may fail when the cosmetic product to be generated virtually has unusual colours.
Moreover, it is possible to use rendering engines based on physical principles to generate realistic renderings of cosmetic products. These rendering engines are referred to using the expression “realistic physically based rendering engine”. It is difficult to correctly configure such rendering engines such that they generate a given cosmetic product realistically. Indeed, it is then necessary to define characterizing parameters of the cosmetic product in order to provide them to the rendering engine. Defining these characterizing parameters is difficult and tedious.
There is therefore a need to propose a simple solution for generating photorealistic renderings of cosmetic products from an image of a real cosmetic product applied to a person, in particular in real time with limited computing resources.
According to one aspect, what is proposed is a method for generating a photorealistic rendering of a cosmetic product, comprising:
The method may be used to generate photorealistic renderings of cosmetic products of various types on a person. For example, the method may be used to generate photorealistic renderings of make-up products or haircare products on a person. In particular, the make-up products may be blusher, foundation, mascara, lipstick, gloss, nail varnish or eyeliner. The haircare product may be a dye.
The rendering engine is configured to generate a photorealistic rendering of a cosmetic product on a person based on certain parameters of the cosmetic product. Such a rendering engine is conventional and well known to those skilled in the art.
The method thus implements an artificial neural network trained to automatically extract parameters of the cosmetic product from the reference image. These parameters make it possible to configure the rendering engine so as to generate a photorealistic rendering of the cosmetic product on the person.
The method therefore makes it possible to obtain a photorealistic rendering of a cosmetic product for any cosmetic product forming part of the cosmetic product type for which the rendering engine is designed. The virtual cosmetic products able to be generated are therefore not limited to a few predefined cosmetic products.
The artificial neural network that is used has the advantage of needing few computing resources to be executed, in comparison in particular with conventional methods using inverse graphics networks in which a gradient descent is performed.
The proposed method may therefore be implemented by appliances with low computational capabilities, in particular by portable appliances, for example smartphones, tablets, etc.
Advantageously, said at least one source image is obtained using a photography device. Furthermore, the transformed image is displayed on a screen.
In one advantageous mode of implementation, the cosmetic product is a make-up product for the lips. The parameters of the vector may then be an opacity, a colour, a reflection intensity, an amount and a texture of the make-up product for the lips.
According to another aspect, what is proposed is a method for training an encoding artificial neural network as described above, comprising:
In one advantageous mode of implementation, the method for training the encoding artificial neural network furthermore comprises:
According to another aspect, what is proposed is a computer program product comprising instructions that, when the program is executed by a computer, prompt said computer to:
According to another aspect, what is proposed is a system comprising:
Further advantages and features of the invention will become apparent upon examining the detailed description of modes of implementation and embodiments, which are in no way limiting, and the appended drawings, in which:
[
[
[
[
The cosmetic product may be a make-up product for the face or a haircare product. In particular, the make-up products may be blusher, foundation, mascara, lipstick, gloss or eyeliner. The haircare product may be a dye.
The rendering generation method comprises obtaining 10 a reference image Xref. The reference image Xref is an image of a real cosmetic product PC applied to a person P1. For example, when the cosmetic product is a make-up product for the face or a haircare product, the reference image Xref may be a portrait of a person to whom the make-up product or the haircare product is applied. In this case, the cosmetic product is a lipstick.
The method may comprise defining 11 an area of interest ZI of the reference image Xref, the area of interest ZI corresponding to the area of the reference image in which the cosmetic product PC is located, in this case the mouth of the person P1 to whom the lipstick is applied.
The method also comprises obtaining 12 at least one source image Xsourcej. Said at least one source image Xsourcej is at least one image of a person P2 to whom the cosmetic product from the reference image is not applied. It is for example possible to obtain a single source image or else a sequence of images of a video.
The rendering generation method makes it possible to virtually apply the cosmetic product PC from the reference image to the person P2 from said at least one source image Xsourcej.
In particular, the rendering generation method comprises implementing 13 an encoding artificial neural network E. This encoding artificial neural network E is trained to extract characterizing parameters of a cosmetic products from an image received at input of this encoding artificial neural network.
The characterizing parameters make it possible to define the features of the cosmetic product present in the image received at input. For example, when the cosmetic product is a make-up product for the lips, in particular a lipstick or a lip gloss, the characterizing parameters may be chosen from among an opacity, a colour, a reflection intensity, an amount and a texture (roughness) of the make-up product for the lips.
The encoding artificial neural network E is for example a convolutional neural network.
Various modes of implementation of a method for training the encoding artificial neural network E will be described hereinafter with reference to
More particularly, in the rendering generation method, the encoding artificial neural network is implemented by taking the reference image or the defined area of interest of the reference image at input. The encoding artificial neural network E then makes it possible to determine characterizing parameters associated with the cosmetic product PC shown in the reference image Xref or in the area of interest of the reference image. These characterizing parameters are shown in a vector E(Xref).
The rendering generation method then comprises implementing 14 a rendering engine R.
The rendering engine R is configured to receive at least one source image and characterizing parameters of a cosmetic product at input. The rendering engine R is also configured to generate a photorealistic rendering of a cosmetic product on the person shown in said at least one source image at output based on the characterizing parameters provided at input.
The rendering engine R is therefore configured to generate at least one transformed image, carrying over said at least source image, and in which the generated cosmetic product is applied virtually to the person shown in said at least one source image.
The generated cosmetic product then has the features defined by the characterizing parameters provided at input of the rendering engine R.
In particular, in the rendering generation method, the rendering engine R is implemented by receiving said at least one source image Xsourcej and the vector E(Xref) of characterizing parameters determined by the encoding artificial neural network at input. The rendering engine then makes it possible to generate at least one transformed image R(Xsourcej, E(Xref)) depicting the person P2 from said at least one source image Xsourcej to whom the cosmetic product PC shown in the reference image Xref is virtually applied.
Such a rendering method may be used by a user to virtually try out a cosmetic product. The provided source image Xsourcej is then an image of this user, for example their portrait. The user may provide, as reference image, an image of a person to whom the cosmetic product that the user wants to virtually try out is applied.
The rendering generation method may also be used to simply generate vectors of characterizing parameters of a cosmetic product from a reference image Xref. These vectors of characterizing parameters may then be used at input of a rendering engine when a user wishes to virtually try out the cosmetic product.
In this embodiment, the rendering engine R receives, in step 20, characterizing parameters relating to the colour COL of the make-up product for the lips PM, to the reflection REF of this product PM and to the texture TXT of this product PM.
The rendering engine R also receives, in step 21, a source image XS of the lips of the person P3. The rendering engine R may then subsequently compute, in step 22, a three-dimensional model M3D of the lips from the source image XS. The rendering engine R may also estimate, in step 23, the reflection of light from the lips from the source image XS.
The rendering engine R then determines, in step 24, corrected colour parameters based on the colour parameters of the make-up product PM that are provided at input. These corrected colour parameters take into account brightness on the lips from the source image.
Next, in step 25, the rendering engine R generates a first intermediate image IM1 in which the rendering engine applies the corrected colour parameters to the lips from the source image, taking into account the previously computed three-dimensional model of the lips.
The rendering engine R then generates, in step 26, a second intermediate image IM2 in which the reflection parameters provided at input are applied to the lips from the first intermediate image IM1, also taking into account the reflection of light from the lips from the source image.
The rendering engine R then generates at output, in step 27, a transformed image IMT in which the texture parameters provided at input are applied to the lips from the second intermediate image.
The method comprises obtaining 30 at least one predefined vector gi of characterizing parameters representative of the features of a cosmetic product.
The method also comprises obtaining 31 an image Xi. This image Xi shows a person P4 to whom the cosmetic product associated with the predefined vector gi of characterizing parameters is not applied.
The method comprises implementing 32 the rendering engine R described above with reference to
Implementing 32 the rendering engine R makes it possible to obtain a transformed image R(Xi, gi) in which the cosmetic product PC associated with the vector of characterizing parameters is applied virtually to the person P4 shown in the image Xi.
The method may comprise defining 33 an area of interest ZI of the transformed image R(Xi, gi). The area of interest ZI corresponds to the area of the transformed image R(Xi, gi) in which the cosmetic product is located.
Next, the training method comprises implementing 34 the encoding artificial neural network E described above. The encoding artificial neural network E is implemented by taking the transformed image R(Xi, gi) or the area of interest ZI of this transformed image at input. The encoding artificial neural network E thereby makes it possible to determine characterizing parameters associated with the cosmetic product shown in the transformed image R (Xi, gi) or in the area of interest ZI of the transformed image. These characterizing parameters are shown in a vector E(R(Xi, gi).
The method then comprises computing 35 a cost Lgraphics (or “loss”) in order to compare the vector E(R(Xi, gi) with the expected vector gi. This comparison makes it possible to determine a difference between the expected values of the vector gi and the values of the vector E(R(Xi, gi) that are determined by the encoding artificial neural network E. In particular, the cost Lgraphics may be computed using the following formula:
n being equal to the number of vectors gi used.
The training method then comprises adapting 36 the weights of the encoding artificial neural network E, in particular through gradient backpropagation, based on the computed cost Lgraphics, in order to reduce the difference between the expected values of the vector gi and the values of the vector E(R(Xi, gi) that are determined by the encoding artificial neural network, and therefore to minimize the cost Lgraphics.
The method may be repeated for various predefined vectors gi and/or for multiple different images Xi in order to improve the performance of the encoding artificial neural network.
The method comprises obtaining 40 at least one predefined vector gi of characterizing parameters representative of the features of a cosmetic product.
The method also comprises obtaining 41 an image Xi. This image Xi shows a person P5 to whom the cosmetic product associated with the predefined vector gi of characterizing parameters is not applied.
The method comprises implementing 42 the rendering engine R described above with reference to
Implementing 42 the rendering engine R makes it possible to obtain a transformed image R(Xi, gi) in which the cosmetic product PC associated with the vector of characterizing parameters is applied to the person P5 shown in the image Xi.
The method may comprise defining 43 an area of interest ZI of the transformed image R(Xi, gi), the area of interest ZI corresponding to the area of the transformed image in which the cosmetic product PC is located.
Next, the training method comprises implementing 44 the encoding artificial neural network E described above. The encoding artificial neural network E is implemented by taking the transformed image R(Xi, gi) or the area of interest ZI of this transformed image at input. The encoding artificial neural network E thereby makes it possible to determine characterizing parameters associated with the cosmetic product PC shown in the transformed image R(Xi, gi) or in the area of interest ZI of the transformed image. These characterizing parameters are shown in a vector E(R(Xi, gi).
Next, the training method comprises implementing 45 a rendering artificial neural network I. This rendering artificial neural network I is trained to imitate the rendering engine R. The rendering artificial neural network I is nevertheless distinguishable from the rendering engine R. The rendering artificial neural network I may be a convolutional neural network.
In particular, the rendering artificial neural network I is configured to receive a source image and characterizing parameters of a cosmetic product at input. The rendering artificial neural network is also configured to generate a photorealistic rendering of a cosmetic product on the person shown in the source image at output based on the characterizing parameters provided at input.
The rendering artificial neural network I is therefore configured to generate at least one transformed image, carrying over the source image, and in which the generated cosmetic product is applied virtually to the person shown in said at least one source image.
The generated cosmetic product then has the features defined by the characterizing parameters provided at input of the rendering artificial neural network. The transformed image generated by the rendering artificial neural network is however distinguishable from a transformed image generated by the rendering engine from one and the same vector of characterizing parameters associated with a cosmetic product.
In particular, in the training method, the rendering artificial neural network I is implemented by receiving the source image Xi and the vector E(R(Xi, gi) of characterizing parameters determined by the encoding artificial neural network at input. The rendering artificial neural network then makes it possible to obtain a transformed image I (E(R(Xi, gi))). This transformed image I (E(R(Xi, gi))) then depicts the person from said at least one source image Xi to whom the cosmetic product associated with the vector E(R (Xi, gi) is virtually applied.
The method then comprises computing 46 a first cost Lgraphics in order to compare the vector E(R(Xi, gi) with the expected vector gi. This comparison makes it possible to determine a difference between the expected values of the vector gi and the values of the vector E(R(Xi, gi) that are determined by the encoding artificial neural network. In particular, the cost Lgraphics may be computed using the following formula:
n being equal to the number of vectors gi used.
The method also comprises computing 47 a second cost Lperceptual in order to compare the transformed image I (E(R(Xi, gi))) generated by the rendering artificial neural network with the transformed image R(Xi, gi) generated by the rendering engine. This comparison makes it possible to determine a difference between the expected transformed image R(Xi, gi) generated from the vector gi and the transformed image I (E(R(Xi, gi))) generated from the vector E(R(Xi, gi) determined by the encoding artificial neural network. In particular, the cost Lperceptual may be computed using the following formula:
n being equal to the number of vectors gi used.
The training method then comprises adapting 48 the weights of the encoding artificial neural network, in particular through gradient backpropagation, based on the computed first cost Lgraphics and the computed second cost Lperceptual.
In particular, the encoding artificial neural network is adapted based on a total cost Ltotalf able to be computed using the following formula:
L
total
f=λ3Lgraphics+Δ4Lperceptual, where A3 and A4 are predefined weighting coefficients.
Such a training method makes it possible to take into account an impact of the characterizing parameters of the vector E(R(Xi, gi) on a transformed image generated from this vector.
It is thereby possible to improve the adaptation of the encoding artificial neural network by taking into account the characterizing parameters having the greatest impact on the transformed image I (E(R(Xi, gi))).
The method may be repeated for various predefined vectors gi and/or for multiple different images Xi in order to improve the performance of the encoding artificial neural network.
The method may be repeated for various predefined vectors gi and/or for multiple different images Xi in order to improve the performance of the encoding artificial neural network.
The method comprises obtaining 50 at least one predefined vector gi of characterizing parameters representative of the features of a cosmetic product.
The method also comprises obtaining 51 an image Xi. This image Xi shows a person P6 to whom the cosmetic product associated with the predefined vector gi of characterizing parameters is not applied.
The method comprises implementing 52 the rendering engine R described above with reference to
Implementing 52 the rendering engine R makes it possible to obtain a transformed image R(Xi, gi) in which the cosmetic product PC associated with the vector of characterizing parameters is applied to the person P6 shown in the image Xi.
The method comprises implementing 53 the rendering artificial neural network I. The rendering artificial neural network I is implemented by taking the predefined vector gi of characterizing parameters and the image Xi at input.
Implementing the rendering artificial neural network makes it possible to obtain a transformed image I(Xi, gi) in which the cosmetic product PC associated with the vector of characterizing parameters is applied to the person P6 shown in the image Xi.
The method then comprises computing 54 a cost Limitation in order to compare the transformed image I(Xi, gi) generated by the rendering artificial neural network I with the transformed image R(Xi, gi) generated by the rendering engine R. This comparison makes it possible to determine a difference between the transformed image I(Xi, gi) and the expected transformed image R(Xi, gi). In particular, the cost Llimitation may be computed using the following formula:
n being equal to the number of vectors gi used.
The training method then comprises adapting 55 the weights of the rendering artificial neural network I, in particular through gradient backpropagation, based on the computed cost Limitation, in order to reduce the difference between the transformed image I(Xi, gi) and the transformed image R(Xi, gi), and therefore to minimize the cost Limitation.
The method may be repeated for various predefined vectors gi and/or for multiple different images Xi in order to improve the performance of the rendering artificial neural network.
The system SYS comprises a memory MEM storing a computer program PRG. This computer program PRG comprises instructions that, when the program is executed by a computer, prompt said computer to implement the method for generating a photorealistic rendering of a cosmetic product. A reference image is also stored in the memory MEM. This reference image shows a person to whom a cosmetic product is applied.
The system SYS also comprises a photography device APH configured to acquire at least one source image of a person. This source image may then be stored in the memory MEM.
The system SYS comprises a processing unit UT configured to implement the computer program stored in the memory. The processing unit UT is thereby configured to implement the rendering generation method as described above. The processing unit UT is thus configured to generate a transformed image in which a photorealistic rendering of the cosmetic product is applied to the person from the source image based on the reference image stored in memory MEM.
The system furthermore comprises a screen ECR configured to display the transformed image.
The system may for example be a portable appliance, for example a smartphone, an electronic tablet, etc.
Number | Date | Country | Kind |
---|---|---|---|
2106030 | Jun 2021 | FR | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/EP2022/062478 | 5/9/2022 | WO |