Feature point detecting device, feature point detecting method, and feature point detecting program

Abstract

A device and method for detecting feature points of an object from an image. A three-dimensional model is created in which a plurality of nodes corresponding to feature points in a learning image are defined. The model is projected onto an input image and a feature value is derived from a plurality of sampling points around a projection point of each node. An error estimated amount is computed based on the displacement of a feature point between a correct model and an error model. The three dimensional position of each feature point in the input image is estimated based on the error estimated amount and a three dimensional model.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B shows an example of feature points extracted from a face image.

FIG. 2 shows an example of a retina structure.

FIG. 3 shows a distribution example of sampling points when the retina structure is applied to a certain node of a face shape model.

FIGS. 4A and 4B shows examples of a displaced arrangement model.

FIG. 5 shows a flowchart illustrating a processing example of a training process.

FIG. 6 shows an example of function blocks of a feature point detecting device.

FIG. 7 shows a flowchart illustrating an operation example of the feature point detecting device.

Claims

1. A feature point detecting device for estimating a three dimensional position of a plurality of feature points in an image of a predetermined object from an input image, the device comprising: a model creating module for creating a three dimensional model in which three dimensional positions of a plurality of nodes each corresponding to the plurality of feature points are defined;a projecting module for projecting each node defined by the three dimensional model onto the image;a feature value acquiring module for acquiring a feature value from a plurality of sampling points around a projection point of each node projected by the projecting module as a node feature value of each node of the three dimensional model;a storing module for storing in advance information on a correlation of a difference between a correct model, in which the plurality of nodes are each three dimensionally arranged at correct positions of the corresponding feature points, and an error model, in which at least one of the plurality of nodes is three dimensionally arranged at a position displaced from the correct position of the corresponding feature point, and the node feature value acquired from the projection point of projecting the error model onto a learning image;an error estimated amount acquiring module for acquiring an error estimated amount indicating the displacement of the position of the corresponding feature point from the three dimensional model based on the information on the correlation stored in the storing module and the node feature value of each node acquired by the feature value acquiring module; andan estimating module for estimating the three dimensional position of each feature point in the input image based on the error estimated amount obtained by the error estimated amount acquiring module and the three dimensional model.

2. A feature point detecting device according to claim 1, further comprising an end determining module for determining an end of a repetitive process, wherein the model creating module moves the position of each node based on the error estimated amount and creates a new three dimensional model if the error estimated amount is obtained;the model creating module and the error estimated amount acquiring module repeatedly execute the processes; andthe estimating module estimates the three dimensional position of each feature point in the input image based on the error estimated amount at the point the end the determining module determines to end the process and the three dimensional model at the end.

3. A feature point detecting device according to claim 2, wherein the end determining module determines to end the repetitive process when the error estimated amount obtained by the error estimated amount acquiring module is less than or equal to a threshold value.

4. A feature point detecting device according to claim 2 or claim 3, wherein the model creating module: creates a new three dimensional model by performing movement and/or rotation while maintaining a current relative positional relationship of each node to change an orientation based on the error estimated amount in an initial stage of the repetitive process; andcreates a new three dimensional model by changing a current relative position of each node based on the error estimated amount after the error related to the orientation of the three dimensional model becomes less than or equal to the predetermined threshold value.

5. A feature point detecting device according to claim 1, wherein the object is a face of a person.

6. A feature point detecting device according to claim 5, further comprising: a face detecting module for detecting at least a position of the face of the person or an element of the face of the person from the input image; wherein,an arrangement module determines the three dimensional position of each node in the three dimensional model based on the detected position of the face of the person or the element of the face of the person.

7. A feature point detecting device according to claim 6, wherein. the face detecting module further detects a direction of the face of the person;the model creating module includes a three dimensional node model in which the three dimensional position of each node in the three dimensional model is defined and deforms the three dimensional node model based on the direction of the face detected by the face detecting module.

8. A feature point detecting device according to claim 6, wherein, the face detecting module further detects a size of the face of the person;the model creating module includes a three dimensional node model in which the three dimensional position of each node in the three dimensional model is defined, and deforms the three dimensional node model based on the size of the face detected by the face detecting module.

9. A feature point detecting device according to claim 6, wherein: the face detecting module further detects a direction of the face of the person;the model creating module includes a plurality of three dimensional node models in which the three dimensional position of each node in the three dimensional model is defined according to the direction of the face,the storing module stores the information on the correlation corresponding to each three dimensional node model obtained in advance using the learning image in which the face facing of the corresponding direction is imaged;the model creating module selects the three dimensional node model to use based on the direction of the face detected by the face detecting module; andthe error estimated amount acquiring module reads the information on the correlation corresponding to the three dimensional node model selected by the model creating module from the storing module and uses the information.

10. A feature point detecting device according to claim 6, wherein the face detecting module further detects a size of the face of the person;the model creating module includes a plurality of three dimensional node models in which the three dimensional position of each node in the three dimensional model is defined according to the size of the face;the storing module stores the information on the correlation corresponding to each three dimensional node model obtained in advance using the learning image in which the face of the corresponding size is imaged;the model creating module selects the three dimensional node model to use based on the size of the face detected by the face detecting module; andthe error estimated amount acquiring module reads the information on the correlation corresponding to the three dimensional node model selected by the model creating module from the storing module and uses the information.

11. A feature point detecting device according to claim 1, wherein the plurality of sampling points are discretely arranged around the projection point of the node.

12. A feature point detecting device for estimating a three dimensional position of a feature point in an input image using a three dimensional face shape model representing the three dimensional arrangement of a plurality of feature points in a face of a person, wherein the device comprises: a model creating module for creating the three dimensional face shape model;a projecting module for projecting each node of the three dimensional face shape model created by the model creating module onto an input image plane;a feature value acquiring module for discretely acquiring a plurality of feature values around the position of the projection point of each node projected by the projecting module and acquiring the plurality of feature values acquired based on one projection point as one node feature value;a storing module for storing a transformation vector matrix acquired in advance through a canonical correlation analysis, which uses as input a set of vectors, which indicate a difference in position of each node between a correct three dimensional face shape model and an error three dimensional face shape model and a set of the node feature values acquired from the projection points of projecting the error three dimensional face shape model, on a learning image, as an error estimation matrix;wherein each node in the three dimensional face shape model is arranged at a correct position and at least one of the nodes in the error three dimensional shape face model displaced from the correct position;an error estimated amount acquiring module for acquiring an error estimated amount, which indicates the displacement between a current position of each node of the three dimensional face shape model and the position of the feature point of the face based on the error estimation matrix stored in the storing module, and the set of node feature values acquired by the feature value acquiring module;a changing module for changing the position of each node of the current three dimensional face shape model based on the error estimated amount obtained by the error estimated amount acquiring module; andan estimating module for estimating the position of each node after the position is changed by the changing module as the three dimensional position of the feature point.

13. A feature point detecting method for estimating a three dimensional position of a plurality of feature points in an image of a predetermined object from an input image, wherein an information processing device executes the steps of: creating a three dimensional model in which three dimensional positions of a plurality of nodes each corresponding to the plurality of feature points are defined;projecting each node defined by the three dimensional model onto the image;acquiring a feature value from a plurality of sampling points around a projection point of each node projected by the projecting module as a node feature value of each node of the three dimensional model;acquiring an error estimated amount, which indicates the displacement of the position of the corresponding feature point from a current three dimensional model based on information on a correlation of a difference between a correct model in which the plurality of nodes are each three dimensionally arranged at correct positions of the corresponding feature points and an error model in which at least one of the plurality of nodes is three dimensionally arranged at a position displaced from the correct position of the corresponding feature point, and the node feature value acquired from the projection point of projecting the error model onto a learning image, acquired in advance based on the learning image, and the node feature value of each node acquired in the step of acquiring the feature value; andestimating the three dimensional position of each feature point in the input image based on the error estimated amount and the current three dimensional model.

14. A feature point detecting program for an information processing device to estimate a three dimensional position of a plurality of feature points in an image of a predetermined object from an input image, wherein the information processing device executes the steps of: creating a three dimensional model in which three dimensional positions of a plurality of nodes each corresponding to the plurality of feature points are defined;projecting each node defined by the three dimensional model onto the image;acquiring a feature value from a plurality of sampling points around a projection point of each node projected by the projecting module as a node feature value of each node of the three dimensional model;acquiring an error estimated amount, which indicates the displacement of the position of the corresponding feature point from the current three dimensional model based on information on a correlation of a difference between a correct model in which the plurality of nodes are each three dimensionally arranged at correct positions of the corresponding feature points and an error model in which at least one of the plurality of nodes is three dimensionally arranged at a position displaced from the correct position of the corresponding feature point, and the node feature value acquired from the projection point of projecting the error model onto a learning image, acquired in advance based on the learning image, and the node feature value of each node acquired in the step of acquiring the feature value; andestimating the three dimensional position of each feature point in the input image based on the error estimated amount and the current three dimensional model.

15. A feature point detecting method for estimating a three dimensional position of a feature point of a face in an input image using a three dimensional face shape model representing a three dimensional arrangement of a plurality of feature points in a face of a person, wherein an information processing device executes the steps of: creating the three dimensional face shape model;projecting each node of the created three dimensional face shape model onto an input image plane;discretely acquiring a plurality of feature values around the projection point of each projected node and acquiring the plurality of feature values acquired based on one projection point as one node feature value;acquiring an error estimated amount, which indicates the displacement between a current position of each node of the three dimensional face shape model and the position of the feature point of the face based on a transformation vector matrix acquired in advance through a canonical correlation analysis, which takes as input a set of vectors indicating the difference in position of each node between a correct three dimensional face shape model in which each node is arranged at a correct position and an error three dimensional face shape model in which at least one of the nodes is arranged displaced from the correct position and a set of node feature values acquired from the projection point of projecting the error three dimensional face shape model onto a learning image, acquired in advance based on the learning image, and a set of node feature values acquired in the step of acquiring the feature values;changing the position of each node of the current three dimensional face shape model based on the acquired error estimated amount; andestimating the position of each node after the position is changed as the three dimensional position of the feature point.

16. A feature point detecting program for an information processing device to estimate a three dimensional position of a feature point of a face in an input image using a three dimensional face shape model representing a three dimensional arrangement of a plurality of feature points in a face of a person, wherein an information processing device executes the steps of: creating the three dimensional face shape model;projecting each node of the created three dimensional face shape model onto an input image plane;discretely acquiring a plurality of feature values around the projection point of each projected node, and acquiring the plurality of feature values acquired based on one projection point as one node feature value;acquiring an error estimated amount indicating the displacement between a current position of each node of the three dimensional face shape model and the position of the feature point of the face based on a transformation vector matrix acquired in advance through a canonical correlation analysis, which takes as input a set of vectors indicating the difference in position of each node between a correct three dimensional face shape model in which each node is arranged at a correct position and an error three dimensional face shape model in which at least one of the nodes is arranged displaced from the correct position and a set of node feature values acquired from the projection point of projecting the error three dimensional face shape model onto a learning image, acquired in advance based on the learning image, and a set of node feature values acquired in the step of acquiring the feature values;changing the position of each node of the current three dimensional face shape model based on the acquired error estimated amount; andestimating the position of each node after the position is changed as the three dimensional position of the feature point.