Patent Grant
7050655
The present invention relates to head animation, and more particularly, to generating an animated three-dimensional video head based on two-dimensional video images.
Virtual spaces filled with avatars are an attractive way to allow for the experience of a shared environment. However, animation of a photo-realistic avatar generally requires intensive graphic processes, particularly for rendering facial features.
Accordingly, there exists a significant need for improved rendering of facial features. The present invention satisfies this need.
The present invention is embodied in a method for generating an animated three-dimensional video head. In the method, a sequence of two-dimensional video image frames of the face of an actor is captured. For each video image frame, the locations of the actor's facial features are sensed at predetermined node locations. Each node location is associated with a particular facial feature. Control points on a three-dimensional head mesh are driven based on the sensed node locations to generate a shaped three-dimensional head mesh. The video image frame used to generate the sensed node location is warped for projection onto the shaped head mesh. The warped video image frame is texture mapped onto the shaped head mesh to generate a three-dimensional frame head associated with the respective video image frame. The three-dimensional video head is animated by displaying a sequence of the three-dimensional frame heads associated with the sequence of video image frames.
In more detailed features of the invention, the step of sensing the locations of the facial features in the sequence of video image frames is performed using transformed facial image frames generated based on wavelet transformations, such as Gabor wavelet transformations.
Other features and advantages of the present invention should be apparent from the following description of the preferred embodiments taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the invention.
The present invention provides a technique for generating an animated three-dimensional video head based on sensed locations of facial features and texture mapping of corresponding two-dimensional video image frames onto a shaped head mesh generated using the sensed locations. The use of facial feature location sensing in combination with texture mapping of the same video image frame used in the facial feature location sensing, provides a animated three-dimensional video head having desirable photo-realistic properties.
With reference to
An example of an animated three-dimensional video head is shown in
The step of sensing the locations of the facial features in the sequence of video image frames may be performed using transformed facial image frames generated based on wavelet transformations, such as Gabor wavelet transformations.
Facial feature finding and tracking using Gabor wavelet tracking and jet graph matching is described in U.S. Pat. No. 6,272,231, titled WAVELET-BASED FACIAL MOTION CAPTURE FOR AVATAR ANIMATION. Imaging systems for acquiring images and image mapping are described in U.S. patent application Ser. No. 09/724,320, titled METHOD AND APPARATUS FOR RELIEF TEXTURE MAP FLIPPING (published in a continuation U.S. application Ser. No. 10/238,289, as U.S. Publication No. US 2003/0007666 A1). The entire disclosures of U.S. Pat. No. 6,272,231 and U.S. patent application Ser. No. 09/724,320 are incorporated herein by reference.
Although the foregoing discloses the preferred embodiments of the present invention, it is understood that those skilled in the art may make various changes to the preferred embodiments without departing from the scope of the invention. The invention is defined only the following claims.
This application is a continuation-in-part of U.S. application Ser. No. 09/871,370, filed May 31, 2001, now U.S. Pat. No. 6,580,811, which is a continuation of U.S. application Ser. No. 09/188,079, filed Nov. 6, 1998, now U.S. Pat. No. 6,272,231. This application is also a continuation-in-part of U.S. application Ser. No. 09/724,320, filed Nov. 27, 2000, which is a continuation-in-part of U.S. application Ser. No. 09/188,079, filed Nov. 6, 1998, now U.S. Pat. No. 6,272,231.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4725824 | Yoshioka | Feb 1988 | A |
| 4805224 | Koezuka et al. | Feb 1989 | A |
| 4827413 | Baldwin et al. | May 1989 | A |
| 5159647 | Burt | Oct 1992 | A |
| 5168529 | Peregrim et al. | Dec 1992 | A |
| 5187574 | Kosemura et al. | Feb 1993 | A |
| 5220441 | Gerstenberger | Jun 1993 | A |
| 5280530 | Trew et al. | Jan 1994 | A |
| 5333165 | Sun | Jul 1994 | A |
| 5383013 | Cox | Jan 1995 | A |
| 5430809 | Tomitaka | Jul 1995 | A |
| 5432712 | Chan | Jul 1995 | A |
| 5511153 | Azarbayejani et al. | Apr 1996 | A |
| 5533177 | Wirtz et al. | Jul 1996 | A |
| 5550928 | Lu et al. | Aug 1996 | A |
| 5581625 | Connell | Dec 1996 | A |
| 5588033 | Yeung | Dec 1996 | A |
| 5680487 | Markandey | Oct 1997 | A |
| 5699449 | Javidi | Dec 1997 | A |
| 5714997 | Anderson | Feb 1998 | A |
| 5715325 | Bang et al. | Feb 1998 | A |
| 5719954 | Onda | Feb 1998 | A |
| 5736982 | Suzuki et al. | Apr 1998 | A |
| 5764803 | Jacquin et al. | Jun 1998 | A |
| 5774591 | Black et al. | Jun 1998 | A |
| 5802220 | Black et al. | Sep 1998 | A |
| 5809171 | Neff et al. | Sep 1998 | A |
| 5828769 | Burns | Oct 1998 | A |
| 5917937 | Szeliski et al. | Jun 1999 | A |
| 5982853 | Liebermann | Nov 1999 | A |
| 5995119 | Cosatto et al. | Nov 1999 | A |
| 6011562 | Gagn{acute over ())}{acute over (})} | Jan 2000 | A |
| 6044168 | Tuceryan et al. | Mar 2000 | A |
| 6052123 | Lection et al. | Apr 2000 | A |
| 6072496 | Guenter et al. | Jun 2000 | A |
| 6430589 | Jennings, III | Aug 2002 | B1 |
| 6504546 | Cosatto et al. | Jan 2003 | B1 |
| 6556196 | Blanz et al. | Apr 2003 | B1 |
| 6580811 | Maurer et al. | Jun 2003 | B1 |
| 6593936 | Huang et al. | Jul 2003 | B1 |
| 6597736 | Fadel | Jul 2003 | B1 |
| 6611268 | Szeliski et al. | Aug 2003 | B1 |
| Number | Date | Country |
|---|---|---|
| 4406020 | Jun 1995 | DE |
| 0807902 | Nov 1997 | EP |
| Number | Date | Country | |
|---|---|---|---|
| 20030031381 A1 | Feb 2003 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 09188079 | Nov 1998 | US |
| Child | 09871370 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 09871370 | May 2001 | US |
| Child | 09929295 | US | |
| Parent | 09724320 | Nov 2000 | US |
| Child | 09188079 | US | |
| Parent | 09188079 | Nov 1998 | US |
| Child | 09724320 | US |
