Patent Application
20080170750
The present invention relates generally to motion capture, and more particularly to segment tracking using motion marker data.
Motion capture systems are used to capture the movement of an actor or object and map it onto a computer-generated actor/object as a way of animating it. These systems are often used in the production of motion pictures and video games for creating a digital representation of an actor or object for use as source data to create a computer graphics (“CG”) animation. In a typical system, an actor wears a suit having markers attached at various locations (e.g., small reflective markers are attached to the body and limbs). Appropriately placed digital cameras then record the actor's body movements in a capture volume from different angles while the markers are illuminated. The system later analyzes the images to determine the locations (e.g., spatial coordinates) of the markers on the actor's suit in each frame. By tracking the locations of the markers, the system creates a spatial representation of the markers over time and builds a digital representation of the actor in motion. The motion is then applied to a digital model in virtual space, which may be textured and rendered to produce a complete CG representation of the actor and/or the performance. This technique has been used by special effects companies to produce realistic animations in many popular movies.
Certain implementations as disclosed herein provide for methods, systems, and computer programs for providing segment tracking in motion capture.
In one aspect, a method as disclosed herein provides for segment tracking. The method includes: applying a marking material having a known pattern to a surface; acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of the known pattern covering the surface; deriving position and orientation information regarding the known pattern for each image frame of the sequence; and generating animation data incorporating the position and orientation information.
In another aspect, a system for segment tracking is disclosed. The system includes: an image acquisition module configured to generate a sequence of image frames, each image frame including a plurality of synchronized images of a known pattern disposed on a surface; and a segment tracking module configured to receive the sequence of image frames and generate animation data based on the known pattern disposed on the surface.
Other features and advantages of the present invention will become more readily apparent to those of ordinary skill in the art after reviewing the following detailed description and accompanying drawings.
The details of the present invention, both as to its structure and operation, may be gleaned in part by study of the accompanying drawings, in which:
Certain implementations as disclosed herein provide segment tracking in motion capture. Implementations include using one or more known patterns of markers applied on actors and/or objects. The marker(s) comprising a pattern are tracked as a group rather than individually. Thus, the pattern can provide information, such as identification, position/translation, and orientation/rotation, which significantly aids marker tracking.
After reading this description it will become apparent to one skilled in the art how to practice the invention in various alternative implementations and alternative applications. However, although various implementations of the present invention will be described herein, it is understood that these embodiments are presented by way of example only, and not limitation. As such, this detailed description of various alternative implementations should not be construed to limit the scope or breadth of the present invention as set forth in the appended claims.
According to implementations of the present invention, markers encoded with known patterns are applied to actors and/or objects, normally covering various surfaces of the actor and/or object. Identification, position/translation, and orientation/rotation of the actor and/or object are obtained by recording and digitizing images of the patterns. The patterns of markers can be selected to mitigate “missing-marker” effects in motion capture during marker occlusions. In another implementation, identifiable random patterns are used as “known patterns.”
The known (and random) patterns may be generated, for example, using materials including quantum nanodots, glow-in-the dark (i.e., fluorescent) material, tattoos, and virtually any visible, infra-red, or ultra-violet ink, paint, or material which can be applied in a sufficiently identifiable pattern. The patterns may also include patterns of inherent features (e.g., moles or wrinkles) of the actor and/or objects.
In one implementation, a pattern comprises a plurality of markers or features coupled to the actor's body. In another implementation, a pattern comprises a single marker (e.g., a marker strip) or feature. The pattern can be applied or affixed on or around the surfaces of an actor's limbs, hands, and feet. In one example, centroid information relating to the pattern can be derived from the circular disposition of a pattern strip wrapped around a limb (i.e., appendage). By applying known patterns to the actors and/or objects and then recording their movements with cameras, it is possible to obtain not only position but also marker identity and spatial orientation.
As shown, three motion capture cameras 120, 122, 124, are connected to the motion capture processor 110. Generally more than three motion capture cameras are required according to a variety of user- and animation-related needs and requirements. The motion capture cameras 120, 122, 124 are focused on the actor's body 140 and face 150, on which markers 160A-160F have been applied.
The placement of the markers 160A-160F is configured to capture motions of interest including, for example, the body 140, face 150, hands 170, arms 172, legs 174, 178, and feet 176 of the actor. In the implementation illustrated in
The motion capture cameras 120, 122, 124 are controlled by the motion capture processor 110 to capture synchronous sequences of two-dimensional (“2-D”) images of the markers. The synchronous images are integrated into image frames, each image frame representing one frame of a temporal sequence of image frames. That is, each individual image frame comprises an integrated plurality of simultaneously acquired 2-D images, each 2-D image generated by an individual motion capture camera 120, 122, or 124. The 2-D images thus captured may typically be stored, or viewed in real-time at the user workstation 130, or both.
The motion capture processor 110 performs the integration (i.e., performs a “reconstruction”) of the 2-D images to generate the frame sequence of three-dimensional (“3-D,” or “volumetric”) marker data. This sequence of volumetric frames is often referred to as a “beat,” which can also be thought of as a “take” in cinematography. Conventionally, the markers are discrete objects or visual points, and the reconstructed marker data comprise a plurality of discrete marker data points, where each marker data point represents a spatial (i.e., 3-D) position of a marker coupled to a target, such as an actor 140. Thus, each volumetric frame includes a plurality of marker data points representing a spatial model of the target. The motion capture processor 110 retrieves the volumetric frame sequence and performs a tracking function to accurately associate (or, “map”) the marker data points of each frame with the marker data points of preceding and subsequent frames in the sequence.
For example, each individual marker data point in a first volumetric frame corresponds to a single marker placed on an actor's body 140. A unique label is assigned to each such marker data point of the first volumetric frame. The marker data points are then associated with corresponding marker data points in a second volumetric frame, and the unique labels for the marker data points of the first volumetric frame are assigned to the corresponding marker data points of the second volumetric frame. When the labeling (i.e., tracking) process is completed for the entire volumetric frame sequence, the marker data points of the first volumetric frame are thus traceable through the sequence, resulting in an individual trajectory for each marker data point.
Discrete markers are conventionally used to capture the motion of rigid objects or segments of an object or body. For example, rigid markers attached at an elbow and a wrist define the positions of each end of a forearm. When the forearm is moved, the motions of the elbow and wrist markers are tracked and resolved as described above in a sequence of volumetric frames. The motion of the forearm is thus modeled as a rigid body (e.g., a rod) with only the ends defined by the elbow and wrist markers. However, while translational movements of the forearm are easily resolved by analyzing the changes in spatial positions of the elbow and wrist markers, a common twisting motion of the forearm is difficult to detect because a twist can be performed without substantially moving the wrist or elbow.
In one implementation, contrasting with the use of conventional discrete markers, markers disposed in a pattern are used, allowing the motion capture processor 110 to track the pattern as a group rather than individually tracking markers. Because the pattern provides identification information, movement of the markers of one pattern with respect to another pattern can be computed. In one implementation, a pattern tracked in this way is reconstructed in each volumetric frame as an individual object having spatial position information. The object is tracked through the sequence of volumetric frames, yielding a virtual animation representing the various spatial translations, rotations, and twists, for example, of the part of the actor to which the pattern is applied.
In one implementation, one or more known patterns are printed onto strips 160D. The strips 160D are then wrapped around each limb (i.e., appendage) of an actor such that each limb has at least two strips. For example, two strips 160D are depicted in
In one implementation, the motion capture processor 110 performs segment tracking according to techniques disclosed herein from which identification, positioning/translation, and orientation/rotation information is generated for a group of markers (or marked areas). While a conventional optical motion capture system typically records only position for a marker, segment tracking enables the motion capture processor 110 to identify which marker(s) are being captured and to locate the position and orientation of the captured markers of the segment. Once the markers are detected and identified, position and orientation/rotation information regarding the segment can be derived from the identified markers. Confidence in the determinations of position and orientation information for the segment increases as more markers are detected and identified.
Markers or marking material applied in known patterns (and identifiable random patterns) essentially encode identification and orientation information facilitating efficient segment tracking. In one implementation, a known pattern of smaller markers constitutes a single marker or marking area. For example, a marker labeled or patterned as representing a capital letter A, as shown in
Also, the marker placements on the 3-D model depicted in
A sequence of image frames is acquired next, at 620, according to methods and systems for motion capture described herein, and above in relation to
Animation data based on the movements of the markers are generated, at 640. For example, a virtual digital model is to be activated by the data derived from the movements of an actor captured in the image data. During the performance, the actor swings the leg according to a script. The actor's legs are equipped with markers at each major joint, i.e., the hip, knee, and ankle, for instance. The movements of the markers are determined and used to define the movements of the segments of the actor's leg. The segments of the actor's leg correspond to the segments of the leg of the digital model, and the movements of the segments of the actor's leg are therefore mapped to the leg of the virtual digital model to animate it.
As discussed in relation to
A flowchart depicting a method of utilizing strips of marking material 800 is provided in
A sequence of image frames is acquired next, at 820, according to methods and systems for motion capture described above in relation to
Centroid information is derived for the actor's limb equipped with one or more strips from the position and orientation information, at 840. As discussed above, the centroids of two or more strips can be used to approximate the skeletal structure (i.e., a bone) within the actor's limb. The movements of the centroids of the marker strips are then used to express the movements of the skeletal structure, and thus the movement of the limb.
Animation data based on the movements of the marker strips is generated, at 850. For example, during the performance, the actor swings his leg according to a script. The actor's legs are equipped with strips of marker material wrapped around as described above. That is, for instance, a strip may be wrapped around the upper thigh and another wrapped around near the knee. The movements of the centroids of the marker strips are determined and used to define the skeletal structure of the upper leg, and consequently a skeletal element within the upper leg of a virtual digital model corresponding to the actor. The movements of the centroids can therefore used to animate the leg of the virtual digital model.
In one implementation, the markers are printed or formed using quantum nanodots (also known as “Quantum Dots,” or “QDs”). The system would be similar in configuration to the traditional retro-reflective system, but with the addition of an exciter light source of a specific frequency (e.g., from existing filtered ring lights or from another source) and narrow band gap filters placed behind the lenses of the existing cameras, thus tuning the cameras to the wavelength of light emitted by the QDs.
QDs are configured to be excited by light of a particular wavelength, causing them to emit light (i.e., fluoresce) at a different wavelength. Because they can emit light that has been quantum shifted up the spectrum from the excitation wavelength, the excitation light can be filtered from the cameras. This causes any light that falls outside of the particular emission spectrum for the QDs to be substantially blocked at the camera.
QDs can be tuned to virtually any wavelength in the visible or invisible spectrum. This means that a group of cameras can be filtered to only see a specific group of markers and nothing else, significantly cutting down the workload required of a given camera, and allow the camera to work “wide open” in the narrow response range of the QDs.
In one implementation, the quantum nanodots are added to a medium such as ink, paint, plastic, temporary tattoo blanks, etc.
The image acquisition module 910 operates according to methods discussed above relating to the motion capture system 100 described in
The segment tracking module 920 operates according to methods and schemes described in relation to
The identification module 1100 includes capability to identify markers having known patterns, such as the markers 160A-160F shown in
In the example configuration illustrated in
The tracking module 1120 receives identification information and generates marker trajectory information. That is, the markers are tracked through the sequence of image frames, labeled, and a trajectory is determined. In one implementation, labeling and FACS approaches are employed according to U.S. patent application Ser. No. 11/467,503, filed Aug. 25, 2006, entitled “Labeling Used in Motion Capture”, and U.S. patent application Ser. No. 11/829,711, filed Jul. 27, 2007, entitled “FACS Cleaning in Motion Capture.” The labeling information (i.e., trajectory data) is passed to the animation module 1130.
The animation module 1130 receives orientation and labeling information and generates animation data. In one implementation, marker positions and orientations are mapped to positions on a (virtual) digital character model corresponding to the positions on the actor at which the markers were coupled. Similarly, a segment defined by markers on the body of the actor is mapped to a corresponding part on the digital character model. For example, the movement of the centroid approximating the skeletal structure of the actor's limb also models the movement of that part of the actor's body. The transformations describing the movements of the markers 160 and related centroids are then appropriately formatted and generated as animation data for animating a corresponding segment of the digital character.
The animation module 1130 receives the animation information and applies it to the digital character, resulting in its animation. The animation is typically examined for fidelity to the actor's movements, and for purposes of determining whether any, an how much, reprocessing may be required to obtain a desired result.
Memory 1020 stores data temporarily for use by the other components of the computer system 1000. In one implementation, memory 1020 is implemented as RAM. In one implementation, memory 1020 also includes long-term or permanent memory, such as flash memory and/or ROM.
Storage 1030 stores data temporarily or long term for use by other components of the computer system 1000, such as for storing data used by the segment tracking system 1090. In one implementation, storage 1030 is a hard disk drive.
The media device 1040 receives removable media and reads and/or writes data to the inserted media. In one implementation, for example, the media device 1040 is an optical disc drive.
The user interface 1050 includes components for accepting user input from the user of the computer system 1000 and presenting information to the user. In one implementation, the user interface 1050 includes a keyboard, a mouse, audio speakers, and a display. The controller 1010 uses input from the user to adjust the operation of the computer system 1000.
The I/O interface 1060 includes one or more I/O ports to connect to corresponding I/O devices, such as external storage or supplemental devices (e.g., a printer or a PDA). In one implementation, the ports of the I/O interface 660 include ports such as: USB ports, PCMCIA ports, serial ports, and/or parallel ports. In another implementation, the I/O interface 1060 includes a wireless interface for communication with external devices wirelessly.
The network interface 1070 includes a wired and/or wireless network connection, such as an RJ-45 or “Wi-Fi” interface (including, but not limited to 802.11) supporting an Ethernet connection.
The computer system 1000 includes additional hardware and software typical of computer systems (e.g., power, cooling, operating system), though these components are not specifically shown in
Various illustrative implementations of the present invention have been described. However, one of ordinary skill in the art will recognize that additional implementations are also possible and within the scope of the present invention. For example, known and identifiable random patterns may be printed, painted, and inked onto a surface of an actor or object. Further, any combination of printing, painting, inking, and tattoos, quantum nanodots, and inherent body features may be used to obtain a desired pattern.
It will be further appreciated that grouping functionalities within a module or block is for ease of description. Specific functionalities can be moved from one module or block to another without departing from the invention.
Accordingly, the present invention is not limited to only those embodiments described above.
This application claims the benefit of priority pursuant to 35 U.S.C. §119 of co-pending U.S. Provisional Patent Application No. 60/856,201, filed Nov. 1, 2006, entitled “Segment Tracking in Motion Picture,” the disclosure of which is hereby incorporated by reference. This application further incorporates by reference the disclosures of commonly assigned U.S. patent application Ser. No. 10/427,114, filed May 1, 2003, entitled “System and Method for Capturing Facial and Body Motion”; U.S. patent application Ser. No. 11/467,503, filed Aug. 25, 2006, entitled “Labeling Used in Motion Capture”; U.S. patent application Ser. No. 11/829,711, filed Jul. 27, 2007, entitled “FACS Cleaning in Motion Capture”; and U.S. patent application Ser. No. 11/776,358, entitled “Motion Capture Using Quantum Nano Dots,” filed Jul. 11, 2007, the disclosures of which are hereby incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 60856201 | Nov 2006 | US |
