1. Field of the Invention
The present invention relates to a 3D face model construction method, particularly a method which can reconstruct a 3D face model with the associated expressional deformation from a single 2D face image with facial expression.
2. Description of the Related Art
Facial recognition technology is one of the popular researches in the field of computer image and biometric recognition. The main challenge of 2D facial recognition is the varying facial expressions under different poses. To overcome such problem, many developed algorithms require enormous amount of training data under different head poses. However, in practice, it is fairly difficult to collect 2D face images under accurate head pose.
Recently, constructing a 3D face model from images is a very popular topic with many applications, such as facial animation and facial recognition, etc. Model-based statistical techniques have been widely used for robust human face modeling. Most of the previous 3D face reconstruction techniques require more than one face image to achieve satisfactory 3D human face modeling. Another approach for 3D face reconstruction from a single image is to simplify the problem by using a statistical head model as the priori. However, it is difficult to accurately reconstruct the 3D face model from a single face image with expression since the facial expression induces 3D face model deformation in a complex manner.
To solve aforementioned problems, one objective of the present invention is to propose a 3D human face construction method which can reconstruct a complete 3D face model from a single face image with expression deformation.
One objective of the present invention is to propose a 3D human face model construction method based on the probabilistic non-linear 2D expression manifold learned from a large set of expression data to decrease the complexity in constructing a face model.
In order to achieve abovementioned objective, one embodiment of the present invention discloses a 3D human face construction method comprising first, conducting a training step which includes registering and reconstructing data of a plurality of training faces to build a 3D neutral shape model, and calculating a 3D expression deformation for each expression of each said training face and projecting it onto a 2D expression manifold and calculating a probability distribution of expression deformations simultaneously. Next, conducting a face model reconstructing step comprising entering a 2D face image and obtaining a plurality of feature points from said 2D face image, conducting an initialization step for a 3D face model based on said feature points, conducting an optimization step for texture and illumination, conducting an optimization step for shape, and repeating optimization steps for texture and illumination and for shape until error converges;
The objectives, technical contents and characteristics of the present invention can be more fully understood by reading the following detailed description of the preferred embodiments, with reference made to the accompanying drawings, wherein:
a-
The present invention proposes a method which can reconstruct a 3D human face model from a single face image. This method is based on a trained 3D neutral shape model and a probabilistic 2D expression manifold model. The complexity of the 3D face model can be reduced by lowering the dimensions based on a manifold approach when processing the training data. In addition, an iterative algorithm is used to optimize the deformation parameters of the 3D face model.
The flowchart to construct the 3D model of one embodiment of the present invention is shown in
The next training step of the embodiment shown in
Δsifp=SEifp−SNifp Δsifp=SEifp−SNifp (1)
wherein SEifp={x1E,y1E,z1E, . . . xnE,ynE,znE}∈ denotes the ith 3D face geometry with expression and SNifp denotes the ith 3D neutral face geometry. M 3D expression deformations Δsifp for i=1 . . . M are projected onto a 2D expression manifold, as shown in
wherein sLLE is the projected 3D expression deformation onto 2D expression manifold by locally linear embedding(LLE), ωc is the probability of being in cluster C, and 0<ωc<1,
and μc and
denotes the mean and covariance matrix of the cth Gaussian distribution. The expectation maximization (EM) algorithm is employed to compute the maximum likelihood estimation of the model parameters.
In continuation of the aforementioned description, based on the trained 3D neutral shape model and the 2D expression manifold model, we proceed to reconstructing a human face model. First in face reconstruction steps, a 2D face image of unknown expression is entered, and multiple feature points are taken from the 2D face image (step S20). Then, we analyze the magnitude of the expression deformation to obtain the weighting of vertex in the 3D neutral shape model. In one embodiment, we quantify deformation of each vertex in the original 3D space to measure the magnitude of deformation. As shown in
wherein magmax, magmin and magj denote maximal, minimal, and the jth vertex's deformation magnitude, respectively.
Next, we proceed to an initialization step for the 3D human face model (step S22). We estimate a shape parameter vector α by minimizing the geometric distances of feature points, as shown in expression (4):
wherein definition of ωjN is described above, μj denotes the coordinate of the jth feature point of the 2D face image, P is the orthographic projection matrix, f is the scaling factor, R is the 3D rotation matrix, t is the translation vector and {circumflex over (x)}j(α) denotes the jth reconstructed 3D feature point, which is determined by the shape parameter vector α as in expression (5):
In one embodiment, the aforementioned minimization problem can be solved by using the Levenberg-Marquardt optimization to find the 3D face shape parameter vector and the pose of the 3D face as the initial solution for the 3D face model. In this step, the 3D neutral shape model is initialized and the effect of the deformation from facial expression can be alleviated by using the weighting ωjN. Since the magnitude, content and styles of expressions are all embedded into the low-dimensional expression manifold, the only parameters for facial expression are the coordinate of sLLE, and in one embodiment, the initial sLLE is set to (0,0.01), which is located at the common border of different expressions on the expression manifold.
In continuation, after the initialization step, all parameters are iteratively optimized in two steps. The first step includes an optimization for texture and illumination (step 24), which requires estimating a texture coefficient vector β and determine illumination bases B and the corresponding spherical harmonic (SH) coefficient vector . The illumination bases B are determined by a surface normal n and texture intensity T(β). Texture coefficient vector β and SH coefficient vector
can be determined by solving the following optimization problem:
In continuation of the abovementioned description, two areas—face feature area and skin area—of different reflection properties are defined for more accurate texture and illumination estimation. Since the feature area is less sensitive to illumination variations, the texture coefficient vector β is estimated based on minimizing the intensity errors for the vertices in the face feature area. On the other hand, the SH coefficient vector is determined by minimizing the image intensity errors in the skin area.
The second step includes an optimization step for shape (step S26). The facial deformation is estimated from the photometric approximation with the estimated texture parameters obtained from the previous step. In one embodiment, we employ a maximum a posteriori (MAP) estimator which finds the shape parameter vector α, an estimated expression parameter vector ŝLLE and a pose parameter vector ρ={f,R,t} by maximizing a posterior probability expressed as follows:
with
I
exp(α,β, f,R,t,ŝLLE)=I(fR(S(α)+φ(ŝLLE))+t) (8)
wherein ρI is the standard deviation of the image synthesis error and ψ(ŝLLE): →
is a non-linear mapping function that maps the estimated ŝLLE from the embedded space with dimension e=2 to the original 3D deformation space with dimension 3N. The nonlinear mapping function is of the following form:
wherein NB(ŝLLE) is the set of nearest neighbor training data points to said expression parameter vector ŝLLE on said 2D expression manifold, Δsk is the 3D deformation vector for the kth facial expression data in the corresponding set of expression deformation data of training faces, and the weight ωk is determined from the neighbors as described in LLE.
Since the prior probability of ŝLLE in the expression manifold is given by the Gaussian mixture model PGMM(ŝLLE) and the shape parameter vector α is estimated by PCA analysis, maximizing the log-likelihood of the posterior probability in Eq.(7) is equivalent to minimizing the following energy function:
wherein λi denotes the ith characteristic value estimated with PCA analysis for 3D neutral shape model. Then, iteratively repeating optimization for texture the illumination and for shape until error converges (step S28). Besides, since the probability distribution for an expression deformation and the associated expression parameter can be estimated for each input 2D face image, the expression can be removed to produce the corresponding 3D neutral expression model. Also, other expressions from the training data can be applied.
The experimental results of one embodiment of the present invention is shown in
Based on the above description, one characteristic of the present invention is being able to remove the expression of a reconstructed 3D face model by estimating the probability distribution of the expression deformation and the expression parameter of each input 2D face image. Besides, other expressions from the training data can be applied to the reconstructed 3D face model, which is of many applications. In conclusion, the present invention discloses a 3D human face reconstruction method which can reconstruct a complete 3D face model with expression deformation from a single face image. Besides, the complexity for building a 3D face model is reduced by building a probabilistic non-linear manifold for learning from a large amount of expression training model data.
The embodiments described above are to demonstrate the technical contents and characteristics of the preset invention to enable the persons skilled in the art to understand, make, and use the present invention. However, it is not intended to limit the scope of the present invention. Therefore, any equivalent modification or variation according to the spirit of the present invention is to be also included within the scope of the present invention.
Number | Date | Country | Kind |
---|---|---|---|
97146819 | Dec 2008 | TW | national |