Method of Localizing Landmark Points in Images

Abstract

A method of localizing landmark points and fitting appearance based models to image data. Image products are computed efficiently which improves the computational cost and improves performance of fitting algorithms for such models.

Description

BACKGROUND OF THE INVENTION

Here, relevant background material is presented and the relation to prior art explained. The technical details of the invention is presented in the following section Detailed Description and in the research paper [?].

Shape and appearance models can be applied to solve many different problems either by using the fitted model itself or using the model to locate landmark points in images. The most successful applications to this day are the analysis of medical images and images of faces, cf. e.g. [?] for examples. Early work like e.g. the active shape models [?] modeled only the variations in shape. This work was later extended so that the models also include the variations of the appearance (i.e. the image color) as well as the shape, the active appearance models [?] (AAM).

The building of such a model is done offline on a training set of annotated objects. In the online event of a new image containing an object of the modeled category, the model parameters have to be found by fitting the model to the image data. It is in this part that the contribution of the invention lies, by proposing an algorithm that drastically improves the computational cost of this fitting. There are several methods to chose from when performing this fitting. Many of them, most notably the robust simultaneous inverse compositional algorithm introduced in [?], involves the computation of a hessian matrix at each step of the optimization.

In the following section the invention, a way to speed up the computation of certain types of image inner products where the images are in a linear space, is introduced. This type of inner product is used e.g. in the computation of the hessian mentioned above. The computation of this hessian is the most expensive step of this iterative procedure and therefore the invention has considerable impact in reducing the computational load of systems and applications for image analysis and recognition. Under normal model assumptions the difference is a factor 9 to a factor 650 for the hessian computation and a factor 3 to a factor 7 of the actual model fitting, depending on image size.

The issue of computational efficiency has been addressed previously in the literature, see for instance [?]. The efficiency enhancement described in this reference is only achieved at a considerable loss in fitting performance [?]. The present invention gives a similar speedup, while maintaining fitting accuracy.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows an example of a shape and appearance representation for a face model, including landmark points.

FIG. 2 shows an example of a system or device for obtaining images, analyzing, and responding to results from the landmark localization.

ACTIVE APPEARANCE MODELS

Active appearance models (AAMs) [?, ?] are linear shape and appearance models that model a specific visual phenomenon. AAMs have successfully been applied to face modeling with applications such as face synthesis, face recognition [?, ?] and even facial action recognition [?] and medical image analysis with application such as diagnostics and aiding measurement.

In the AAM framework the shape is modeled as a base shape s₀ with a linear combination of shape modes S_i as

$\begin{array}{cc}S=S_0+\sum _{i=1}^mp_is_i,& \left(1\right)\end{array}$

where p_i are the shape coefficients and the shape s is represented as the 2D coordinates of the v vertices of a model mesh as S=(x₁, y₁, . . . , x_v, y_v), cf. FIG. 1. We will use p to denote the vector of p_i.

The appearance is modeled completely analogous as a base appearance image A₀ together with a linear combination of appearance modes A_i as

$\begin{array}{cc}A=A_0+\sum _{i=1}^n{\lambda }_iA_i,& \left(2\right)\end{array}$

where λ_i are the appearance coefficients and an appearance image is given by the set of pixels inside the same model mesh as above. We will use λ to denote the vector of λ_i. The shape and appearance modes are found using Principal Component Analysis (PCA) on aligned training data.

To be able to fit a model instance into an image additional parameters q are needed to describe scaling, rotation and translation. Setting

$r=\left(\begin{array}{c}q\\ p\end{array}\right),$

the warp W(r) is the piecewise affine warp from the base mesh S₀ to the current AAM shape under r. Thus I(W(r)) is an image on S₀ in which the pixel intensities are taken from the image I according to the warp W(r).

Simultaneous Inverse Compositional Image Alignment Algorithm

The simultaneous inverse compositional image alignment algorithm (SICIA) [?] is an algorithm for fitting the AAM to an input image I simultaneously with regards to appearance and shape. Inverse compositional signifies how the warp parameters r are updated.

The overall goal of the algorithm is to minimize the difference between the synthesized image of the model and the image I as

$\begin{array}{cc}{\left[\sum _{i=0}^n{\lambda }_iA_i-I\left(W\left(r\right)\right)\right]}^2,& \left(3\right)\end{array}$

where λ₀=1 (note the summation limits). In the inverse compositional formulation the minimization of equation (3) is carried out by iteratively minimizing

$\begin{array}{cc}{\left[\sum _{i=0}^n\left({\lambda }_i+{\mathrm{\Delta \lambda }}_i\right)A_i\left(W\left(\Delta r\right)\right)-I\left(W\left(r\right)\right)\right]}^2& \left(4\right)\end{array}$

simultaneously with respect to both λ and r. Note that the update of the warp is calculated on s₀ and not on the present AAM instance. The new parameters r_k+1 are then given as a composition of the warp update Δr_k and the present r_k so that

W(r_k+1)←W(r_k)∘W(Δr_k)⁻¹.  (5)

This means the gradient of the warp is constant [?]. The appearance parameters are updated by λ_k+1←λ_k+Δλ_k. Performing a first order Taylor expansion on expression (4) gives

$\begin{array}{cc}{\left[E+\left(\sum _{i=0}^n{\lambda }_i\nabla A_i\right)\frac{\partial W}{\partial r}\Delta r+\sum _{i=1}^nA_i{\mathrm{\Delta \lambda }}_i\right]}^2,& \left(6\right)\end{array}$

where the error image is

$\begin{array}{cc}E=\sum _{i=0}^n{\lambda }_iA_i-I\left(W\left(r\right)\right).& \left(7\right)\end{array}$

For notational convenience set

$t=\left(\begin{array}{c}r\\ \lambda \end{array}\right)\mathrm{and}\Delta t=\left(\begin{array}{c}\Delta r\\ \mathrm{\Delta \lambda }\end{array}\right).$

Also define the steepest descent images as

$\begin{array}{cc}{\mathrm{SD}}_{\Sigma }=\left(\left(\sum _{i=0}^n{\lambda }_i\nabla A_i\right)\frac{\partial W}{\partial r_1},\dots ,\left(\sum _{i=0}^n{\lambda }_i\nabla A_i\right)\frac{\partial W}{\partial r_{m+4}},A_1,\dots ,A_n\right).& \left(8\right)\end{array}$

The +4 comes from the fact that in a 2D case one needs 4 parameters in q. Using these reformulations (6) can be expressed as

[E−SD_ΣΔt]²,  (9)

which is minimized by

Δt=−H⁻¹SD_Σ^TE,  (10)

where the hessian is given by

H=SD_Σ^TSD_Σ.  (11)

DETAILED DESCRIPTION

In a preferred embodiment of the invention, a method for image model fitting and landmark localization is presented, the method comprising the steps of; —computation of the hessian matrix using the space defined by the image model to pre-compute the image inner products, —fitting the appearance model to image data, —storing the final model and landmark points for further use.

Yet another embodiment of the present invention, a computer program stored in a computer readable storage medium and executed in a computational unit for image model fitting and landmark localization comprising the steps of: —computation of the hessian matrix using the space defined by the image model to pre-compute the image inner products, —fitting the appearance model to image data, —storing the final model and landmark points for further use.

In another embodiment of the present invention, a system for image model fitting and landmark localization containing a computer program for image model fitting and landmark localization comprising the steps of: —computation of the hessian matrix using the space defined by the image model to pre-compute the image inner products, —fitting the appearance model to image data, —storing the final model and landmark points for further use.

In another embodiment of the present invention a system or device is used for obtaining images, analyzing, and responding to results from the landmark localization, as may be seen in FIG. 2. Such a system may include at least one image acquisition device 101 and a computational device 100.

The above mentioned and described embodiments are only given as examples and should not be limiting to the present invention. Other solutions, uses, objectives, and functions within the scope of the invention as claimed in the below described patent claims should be apparent for the person skilled in the art.

Below follows a detailed description of the invention.

Linear Space Inner Product

In this section we will detail a method of efficiently computing image inner products and show how this improves the computation of the hessian matrix in (11).

Formulating Inner Products using Linear Projections

Assume that the image I, represented as a vector, can be expressed as a linear combination of g appearance images A_i just as in equation (2). The inner product I_b^TI_c of two such images I_b and I_c is an operation taking as many multiplications to complete as there are elements (pixels) in the vector (image). If we rewrite the inner product using the appearance image representation it becomes

$\begin{array}{cc}\sum _{i=0}^g\sum _{j=0}^g{\lambda }_{b,i}{\lambda }_{c,j}a_{i,j}& \left(12\right)\end{array}$

where the scalar a_i,j=A_i^TA_j. The computations of all a_i,j can be done offline since they are fixed once the appearance images A_i are chosen. Assuming that we have obtained the coefficients λ_b,i and λ_c,i the inner product can be computed using 2g² multiplications instead of as many multiplications as there are pixels.

Linear Space Inner Product (LSIP) Applied to AAM

In one hessian calculation (n+m+4)² number of scalar products are performed while λ stay constant. This means that the hessian calculation is very suited to be performed using the LSIP.

Studying equations (8) and (11), one sees that the hessian will have four distinct areas computation-wise.

The Upper Left Quadrant.

Here each hessian element is given by

$\begin{array}{cc}{H}_{\mathrm{ij}}^{\mathrm{ul}}={\left(\left(\sum _{k=0}^n{\lambda }_k\nabla A_k\right)\frac{\partial W}{\partial r_i}\right)}^T\left(\left(\sum _{l=0}^n{\lambda }_l\nabla A_l\right)\frac{\partial W}{\partial r_j}\right),& \left(13\right)\end{array}$

with i,jε[1,m+4]. Analogously to Section 2(′)@ we rewrite

$\begin{array}{cc}{H}_{\mathrm{ij}}^{\mathrm{ul}}=\sum _{k=0}^n\sum _{l=0}^n{\lambda }_k{\lambda }_lh_{\mathrm{kl}}^{\mathrm{ul},i,j},& \left(14\right)\end{array}$

where

$h_{\mathrm{kl}}^{\mathrm{ul},i,j}={\left(\nabla A_k\frac{\partial W}{\partial r_i}\right)}^T\left(\nabla A_l\frac{\partial W}{\partial r_j}\right).$

Moving one multiplication outside and limiting the inner summation limit gives

$\begin{array}{cc}{H}_{\mathrm{ij}}^{\mathrm{ul}}=\sum _{k=0}^n\sum _{l=k}^n{\lambda }_{\mathrm{kl}}h_{\mathrm{kl}}^{\mathrm{ul},i,j},{\lambda }_{\mathrm{kl}}=\{\begin{array}{cc}{\lambda }_k{\lambda }_l& \mathrm{if}i=j\\ 2{\lambda }_k{\lambda }_l& \mathrm{if}i\ne j.\end{array}& \left(15\right)\end{array}$

The Lower Left and Upper Right Quadrant.

The upper right and lower left quadrants are symmetrical and therefore only the upper right quadrant will be described. The hessian elements are given by

$\begin{array}{cc}{H}_{\mathrm{ij}}^{\mathrm{ur}}={\left(\left(\sum _{k=0}^n{\lambda }_k\nabla A_k\right)\frac{\partial W}{\partial r_i}\right)}^TA_j,& \left(16\right)\end{array}$

with iε[1,m+4], jε[m+5, n+m+4]. This can be transformed into

$\begin{array}{cc}{H}_{\mathrm{ij}}^{\mathrm{ur}}=\sum _{k=0}^n{\lambda }_kh_k^{\mathrm{ur},i,j},h_{\mathrm{kl}}^{\mathrm{ur},i,j}={\left(\nabla A_k\frac{\partial W}{\partial r_i}\right)}^TA_j.& \left(17\right)\end{array}$

The Lower Right Quadrant.

This is simply the scalar products of the appearance images. This quadrant is therefore the identity matrix.

Theoretical Gain of Using the Linear Space Inner Product

Table 1 summarizes the time complexity of one iteration of SICIA [?]. The left column is the calculation performed and a reference to the corresponding equation (s). The first row is the computation of the error image including warping of input image and the image composite with a model appearance instance. The second step is the calculation of the steepest descent images and the third row is the scalar product of the steepest descent images and the error image. The fourth and main step is the calculation of the hessian and its inverse.

TABLE 1
Summary of the time complexity for one iteration of SICIA.
	Calculation	SICIA-Original	SICIA-LSIP
	E, (7)	O((n + m + 4)N	O((n + m + 4)N
	SDΣ, (8)	O((n + m + 4)N	O((n + m + 4)N
	SDΣE, (10)	O((n + m + 4)N	O((n + m + 4)N
	H−1, (10), (11)	O((n + m + 4)2N +	O((m + 4)2(n/2)2 +
		(n + m + 4)3)	(n + m + 4)3)
	Total	O((n + m + 4)2N +	O((m + 4)2(n/2)2 +
		(n + m)3)	(n + m)3 +
			4(n + m + 4)N)

The overwhelmingly largest time consumer for the original SICIA is the construction of the hessian. The computational cost is O((n+m+4)²N) where N is the size of the image. With the LSIP this task is converted to O((m+4)² (n/2)²).

We have described the underlying method used for the present invention together with a list of embodiments. Possible application areas for the above described invention range from object recognition, face recognition, facial expression analysis, object part analysis to image synthesis and computer graphics.

Claims

1. A method for efficient computation of the hessian matrix used in image-based model fitting that uses the space defined by the model to pre-compute the image inner products needed to construct this matrix.

2. The method according to claim 1, wherein said model is an active appearance model.

3. The method according to claim 1 wherein said space is a linear space defined by the modes of variation of the model.

4. A method for efficiently locating landmark points in images where the landmark points are obtained through a model fitting according to claim 1.

5. A computer program stored in a computer readable storage medium and executed in a computational unit for efficient computation of the hessian matrix used in image-based model fitting that uses the space defined by the model to pre-compute the image inner products needed to construct this matrix.

6. A system for fitting an image-based model containing a computational unit and a camera, e.g. a computer with camera or a mobile phone, where the image-based model is fitted according to claim 5.

7. A system for efficiently locating landmark points in images where the landmark points are obtained through a model fitting according to claim 6.