Patent Application
20110046915
The invention pertains to the field of motion capture and, more particularly, to the use of a positioning aiding system concurrently with inertial motion capture systems.
In many fields, it is necessary or desirable to track the motion of an object, e.g., to analyze the motion or record an abstract of the motion. Although there are known methods of tracking motion via an external infrastructure of optical sensors, there are several benefits of using inertial motion tracking instead of optical tracking. Advantages include robust real-time tracking due to the absence of occlusion and marker swapping and the extremely large area tracking capabilities combined with the lack of a need for an installed infrastructure. However, unlike systems based on an installed infrastructure, an inertial based system will fundamentally build up position tracking errors over time and traversed distance.
Although the inventors hereof have used biomechanical joint constraints and physical external contact detection to resolve this problem to some extent, some degree of inertial position drift is present, and fundamentally unavoidable using solely inertial sensors, resulting in a horizontal position drift of the estimated movement of the characters over time as well as drift in traversed distance (typically 1% of traversed distance). For a single actor, this drift will not always be a problem. An animated environment could for example be adjusted to coincide with the actor's actions. Indeed, if the motion capture data is re-targeted to a character of a different size, this is the typical workflow anyway, even if the horizontal position tracking were perfect.
This option is not available when the interaction with the object is repeated after walking around or for real-time (pre) visualization purposes.
Moreover, in many applications, the simultaneous motion capture of a number of interacting actors is required. Correcting for multi-actor interaction in combination with movement of the actors is very difficult because the actors will not experience the same drift. This has heightened consequences when the actors interact with each other. To a certain extent, the relative drift can be corrected during post-processing (i.e., by editing foot contacts) to have the actors interact properly later on. However, this corrective action involves additional work and does not permit real-time visualization.
One way to mitigate drift is to use an external force such as the measured gravitational acceleration to provide a reference direction. In particular, the magnetic field sensors determine the earth's magnetic field as a reference for the forward direction in the horizontal plane (north), also known as “heading.” The sensors measure the motion of the segment on which they are attached, independently of other system with respect to an earth-fixed reference system. The sensors consist of gyroscopes, which measure angular velocities, accelerometers, which measure accelerations including gravity, and magnetometers measuring the earth magnetic field. When it is known to which body segment a sensor is attached, and when the orientation of the sensor with respect to the segments and joints is known, the orientation of the segments can be expressed in the global frame. By using the calculated orientations of individual body segments and the knowledge about the segment lengths, orientation between segments can be estimated and a position of the segments can be derived under strict assumptions of a linked kinematic chain (constrained articulated model). This method is well-known in the art and assumes a fully constrained articulated rigid body in which the joints only have rotational degrees of freedom.
The need to utilize the earth magnetic field as a reference is cumbersome, however, since the earth magnetic field can be heavily distorted inside buildings, or in the vicinity of cars, bikes, furniture and other objects containing magnetic materials or generating their own magnetic fields, such as motors, loudspeakers, TVs, etc.
It is an object of the invention to support robust real-time motion capture, using an inertial motion capture system, aided with a positioning system, of multiple closely interacting actors and to position the actor exactly in space with respect to a pre-defined reference frame. It is a further object of the invention to track or locate floors, walls or other objects in general in the motion capture volume to further improve the robustness and accuracy of the system without increasing demands for very high accuracy positioning systems. It is a further object of the invention to use such positioning systems to aid the inertial motion capture system that the known advantages of using inertial motion capture technology is not compromised to a great extent. It is disclosed that such positioning systems include pressure sensors, UWB positioning systems and GPS or other GNSS systems. It is a further object of the invention to avoid the use of the earth magnetic field as a reference direction as much as possible, due to the known problems of distortion thereof. Suitable use of kinematic coupling algorithms using inertial sensors is disclosed, including use on subsegments of a body, such as for example only the leg
For fully ambulatory motion capture systems that do not require horizontal plane position tracking, but have requirements for vertical position tracking, the system can be extended using pressure sensors, optionally using one or more reference pressure sensors at known altitudes. Such systems can not only be used in the atmosphere but are also suitable for accurate tracking of depth under water. In a preferred embodiment of the invention each body segment is fitted with an inertial sensor and at that same location is also fitted a pressure sensor.
For use outside of an inertial motion capture system it is preferred to extend the system using position aiding based on global navigation satellite systems (GNSS) such as GPS. Although with systems such as GPS, position estimates can be obtained that are accurate, it may be preferable to rely additionally on GPS velocity aiding of the inertial motion capture system since GPS systems are capable of accurate velocity estimates.
Especially for larger set-ups indoors or other locations where GPS or other GNSS systems are not available, or applications outdoors that require higher positional accuracy than can be obtained using GPS, and applications that require horizontal position tracking, the use of UWB positioning systems provides distinctive benefits that are unforeseen in the art compared to the other mentioned positioning technologies. For example, it does not necessarily require line of-sight and is therefore much more robust to occlusion than optical systems. Moreover, large motion capture areas can be constructed for only a fraction of the costs and installed hardware compared to optical systems, and due to the low installed hardware intensity per motion capture area, the system is easy to set-up and re-locate. The system is easy scalable to very large volumes and does not suffer from restrictions in lighting conditions or other environmental conditions (e.g., air pressure, moisture, temperature). Moreover, the inventors have found that a much higher degree of robustness is unexpectedly achieved with described system compared to other RF-based positioning options.
Instead of physically aligning a magnetometer (electromagnetic compass) to the reader setup, the direction of the magnetic field with respect to the setup can be determined using a device containing an inertial sensor/UWB tag combination as described in U.S. patent application Ser. No. 12/534,607 filed Aug. 3, 2009. This device could be used to determine the direction of the magnetic field over the motion capture volume prior to performing a motion capture. However, to account for local deviations in the earth magnetic field as well as to relieve the user from performing an additional calibration, the combined inertial sensor/UWB device could also be placed on the body as to dynamically track the local magnetic field with respect to the UWB system. The inertiaUUWB device should be mounted sufficiently close to the segment(s) for which the magnetic field update is to be applied to ensure that the magnetic field at the device is representative for the magnetic field at the segment.
In some cases the inertial/UWB device can not be placed sufficiently close to the segment for which the heading is to be determined. This is the case e.g. when the UWB tag is to be placed on the head while moving on a floor containing steel reinforcements. In this case, the local magnetic field around the legs is disturbed and not representative for the (earth) magnetic field near the head.
If this is the case, the heading between each of the legs and upper body can be made observable by considering the connection between these three. The linkage between the legs is obviously the pelvis. By feeding the velocity of both legs to the pelvis sensor and feeding the velocity after the biomechanical fusion engine update back to the legs, the orientation of lower body is consistent without magnetometers. This interrelationship can be seen in
Moreover, although the UWB signal does not require line of sight (LOS) for positioning, the signal is delayed when travelling through body parts, causing the positioning to shift a little away from the reader that was blocked. Absorption of the LOS signal might also cause the signal to noise ratio to drop, causing more noise in the TOA and/or causing a signal-lock to a reflection. However, since in the use of an inertial motion capture system the position of all body parts, and their size and orientation is known, and the location of the UWB Tags on the body and UWB Readers are known, it is possible to “ray-trace” the path between the Tag and the Reader and check if a body part, and if so which and its orientation, is in the path of the “ray”, i.e. the UWB RF pulse. Combined with the UWB system RSSI (Received Signal Strength Indicator), a very robust measure can be obtained for the likelihood of a multi-path (reflection) UWB measurement, or if the UWB signal from the Tag is likely to have been absorbed or delayed due to the transmission through the human body. In such a case the time delay caused by the path length through the body, which has a refraction index close to that of water, can be accurately estimated. This estimate can be accurate because the size, position and orientation of the body segment is known (tracked). The advantage of this approach is that the UWB measurement can still be used accurately and does not have to be discarded because it has been transmitted through the human body.
Whenever the UWB aiding system is temporarily not consistent with the position solution obtained from inertial sensors and biomechanical relations, state augmentation can be used to temporarily bridge the inconsistency as to ensure a smooth animation and overcome incidental errors that could be caused by e.g. wrong footstep detection.
For reasons of e.g. optimal line of sight, it may not always be desirable to mount the tag in the same position as the inertial sensor units. In case a tag is not mounted near the inertial sensor units, or not even on the same segment, the lever arm between the inertial sensor and the tag has to be taken into account. Computation of this arm may involve the crossing of different segments with known orientation, including modeling of uncertainties therein, e.g., the pelvis orientation and position could be determined using a) the algorithm described in U.S. application Ser. No. 12/534,526, filed Aug. 3, 2009, b) inertial sensor information from the inertial sensor unit mounted on the pelvis c) UWB readings taken from the tag on the head and d) taking into account the dynamically changing vector between the pelvis and the head, computed using different inertial sensor units.
Before discussing the overall system, this section gives some basic technical background for basic UWB positioning systems in order to give the reader an understanding of the capabilities and limitations of such systems. In a UWB positioning set-up, a small and mobile radio transmitter, or tag, periodically (e.g., 10 times per second) emits a burst RF-signal. This signal travels with the speed of light (˜300,000 km/s) in the ambient medium (largely air) to receivers, or readers installed at fixed locations around the motion caption area.
The UWB RF-signal comprises a series of very short (nano second) EM-pulses that contains the unique ID of the tag. Because of the wide-band nature of the signal, the reader can determine the exact time at which the signal is received. The clock of the reader is sufficiently precise so as to determine the time-of-arrival (TOA) with a resolution of about 39 picoseconds (10−12 seconds).
Although the signal travels very fast, it will take time for the signal to travel from the tag to a reader. For example, in the minimum time step that the reader can measure of 39 picoseconds it will travel approximately 1 cm. Thus, the system positioning resolution is about 1 cm.
So, if the reader could know the exact time at which the tag transmitted the signal, simply taking the difference between this time-of-transmission (TOT) and the TOA would give the time passed since the signal was transmitted, i.e., the time-of-flight (TOF). Theoretically, this could then be used to calculate the range, as it is simply the speed of light times the TOF. Unfortunately, the reader does not know the TOT, because it has no knowledge of the internal clock of the tag. Therefore, one reader alone will not give any range information. However, if a configuration is created with a number of synchronized readers, the TOA at each reader will differ from the other reader with a measure of the difference in the distance to the tag. Unfortunately, the reader does not know the TOT, because it has no knowledge of the internal clock of the tag. Therefore, one reader alone will not give any range information. However, if a configuration is created with a number of synchronized readers, the TOA at each reader will differ from the other reader with a measure of the difference in the distance to the tag.
Referring to an exemplary environment, the body of interest 100, e.g., and actor, is outfitted with one or several transmitters (tags). The transmissions of the tags are picked up by a set of readers (not shown in
An entire system implementation according to an embodiment of the invention is illustrated schematically in
A tag (also named transmitter) emits a short pulse (nanosecond duration) at some initially unknown time TOE (time of emission). This pulse is received by different receivers (readers) at different times (because of the speed of light and the different distances of the tag to each of the receivers, see arrows with dashed lines). The reader clocks are synchronized to high accuracy using a master clock device. The time of arrival (TOA) of this pulse at the different receivers is recorded and send to a PC.
For synchronization the readers are connected to a Synchronization and Distribution (SD) master, i.e. a master clock device. Via this connection the SD master also powers (Power-over-Ethernet) the readers in an embodiment of the invention. The SD master is connected to a local Ethernet and serves as a transparent link for the readers to transmit UDP packets containing the TOA to the motion capture system. Given the different TOA's, and optionally inertial sensor signals and height input, the position of the tag is computed. This position is in turn used to correct any positional drift in the movement that is tracked (using software we named MVN studio).
To determine a 3D position, at least 4 readers are required. So, the minimal set-up is created with four readers, two of which are preferably placed on the floor and two on the ceiling. The resulting accuracy of the UWB position information that is used for drift correction is given in
As can be seen from
The robustness of the minimal set-up is limited since at least 4 readers are required to calculate a stable position. This means that if any reader is blocked, e.g, due to absorption of the transmitted RF-pulse, a full solution can not be calculated. Moreover, the theoretical limitation, the DoP, of the achievable accuracy is larger when the number of readers is increased.
A more robust and accurate UWB constellation 700 is shown in
The configurations displayed in the previous sections are influenced by the range limit of the readers. However, it is possible to extend the area to beyond the range of the individual readers as illustrated in
Another environment within which the present system is advantageous is that of a stage such as a movie stage. In such environments, it is generally a requirement that an unobstructed view to one side is created so that no readers are in the view of the scene camera. It is important to note that in the examples above, the actual motion capture area may be much larger than the area in which accurate drift correction can be performed. The total motion capture area is only limited by the range of the wireless receivers. Thus, the actor is not restricted to the drift-free area but can wander outside the area if no interaction with other objects is required outside the area. Once the actor re-enters the drift-free area the position of the MVN character is gradually converged back to the actual position.
The illustrated constellations to this point have been 3D constellations, meaning that there are readers present above and below the area. However, it might not always be possible to create such a constellation. For example, in some cases it is only possible to create a minimal constellation in which the readers are fixed to the ceiling as illustrated in
As described in the related applications, the integration of the UWB positioning data with the inertial system uses very advanced algorithms to combine the UWB TOA data on the very lowest level with the inertial data. This method is known as “tight coupling.” This means that the system does not first calculate a position based on UWB only and subsequently combine that position with the inertial data. Instead, each individual UWB measurement (TOA) is used directly in the algorithm, yielding superior robustness, accuracy and coverage.
It has been discussed and illustrated above how the readers could be placed (the constellation) and how this influences the achievable accuracy of position tracking. However, once a constellation is selected, the readers must still be physically mounted in the area and the positions of the readers must be accurate recorded or determined while doing so. Also, for synchronization, the system needs to determine the clock-offset for each reader to a level of picosecond accuracy, which depends on cabling lengths and associated delays of the synchronization signal (speed of light).
In an embodiment of the invention, the described system is used in conjunction with one or more traditional systems such as optical tracking and/or computer vision based tracking. Optical tracking can deliver the required sub-millimeter accuracy needed for some applications, but it can not do so robustly. Computer vision based image tracking is important because it can deliver “through the lens” tracking. Even though sometimes quite inaccurate, this is important in practice because the perceived accuracy (in the image plane) is automatically “optimized” resulting in a readily visually acceptable image.
There are two issues related to defining planes in the described system, including measuring the height of the plane and determining the location and orientation of the plane. With respect to measuring the height of the plane, pressure sensors (optionally differential) may be used to alleviate/reduce the need for a full 3D setup of readers. This implementation requires the tags to be equipped with a barometer and requires integration of the associated data in the transmitted packet of the tag.
With respect to determining the location and orientation of the plane, this is performed automatically in an embodiment of the invention as follows. First, the surfaces are placed in the set-up. Tags are then placed at the corners of the surfaces and are detected using the default height of the location algorithm, and the tags corresponding to the same surface are linked together automatically or by hand. Due to the default height, the location of the tags have an offset. The height of the tags is then defined, e.g., by defining the height of the corresponding surface in case of a horizontal surface, and this information is used to create the objects in the virtual representation which can then additionally be used for external contact detection. If the user wants to use an arbitrary shaped plane, tags can be attached at precisely defined positions on the plane.
It is also possible to integrate props into the system by attaching an IMU to a freely moveable object, the IMU being equipped with a tag as well. Again, it should be noted that the vertical position of the prop is not known, so that either a 3D set-up must be created or additional algorithms are used to determine whether an actor picks up the prop in which case the movement of the prop becomes part of the motion model of the actor. Alternatively, a pressure sensor can aid in vertical location resolution.
There are a number of ways to place readers, but consider a configuration in which readers are placed on high tripods. With respect to this set-up there are the following advantages compared to motion caption systems: (1) The number of readers is low; for an area of approximately 15×15 meters only 4 readers are required (3 will also do, but will be less robust); (2) More readers can be placed to increase accuracy, but are redundant; (3) Larger area can be covered; (4) although the number of readers is low for a basic set-up, the number of readers is not limited; (5) If required, an arbitrarily large area can be covered, divided in, for example, cells of 15×15 meters; (6) Cost—a reader can be offered at a much lower price compared to a high speed camera.
To be able to set a height reference, the configuration information defines the ID of the tag for each shoulder of the tagged actor 1205, 1207. Then, using the body model, the heights are determined for the shoulder tags and the heights are sent to the TDOA location algorithm which uses the heights to calculate the locations of the shoulder tags. The determined locations are then sent back for use in the virtual body model where they can be used in the position aiding algorithm.
Referring to
The speed of light in a body is approximately half the speed of light in vacuum due to the refraction index of the body, which is mainly water. Other materials such as glass also cause a time delay in the signal due to the refraction index. This causes the positioning to shift slightly away from the reader that was blocked by body parts, since the position is derived from the Time of Arrival (TOA) compared between different readers. Moreover, absorption of the LOS signal might cause the signal to noise ratio to drop. This can have two effects: more noise in the TOA and a signal-lock to a reflection as shown in view 1301.
However, since the position of all body parts, and their size and orientation is known, and the location of the UWB Tags on the body and UWB Readers are known, it is possible to “ray-trace” the path between the Tag and the Reader and check if a body part, and if so which and its orientation, is in the path of the “ray”, i.e., the UWB RF pulse. Combined with the UWB system RSSI (Received Signal Strength Indicator), a very robust measure can be obtained for the likelihood of a multi-path (reflection) UWB measurement, or if the UWB signal from the Tag is likely to have been absorbed or delayed due to the transmission through the human body. In such a case the time delay caused by the path length through the body, which has a refraction index close to that of water, can be accurately estimated. This estimate can be accurate because the size, position and orientation of the body segment is known (tracked). The advantage of this approach is that the UWB measurement can still be used accurately and does not have to be discarded simply because it has been transmitted through the human body.
Referring now to
It is possible to define contact points within the studio and to define planes by defining the z-coordinate as a function of the horizontal position. Normally, this will only work in a limited number of scenarios with no magnetic disturbances, slow movement and during short periods. In all other cases, the exact position is not known without a proper location system.
With the exact location available using the inertial motion capture system utilizing UWB positioning set-up, it makes sense to use this feature in the fusion software. There are two issues related to defining planes, for the purpose of external contact detection, in inertial motion capture systems, namely measuring the height of the plane and determining the location and orientation of the plane. Measuring the height of the plane is something that could be left to the user as it is a simple action. However, as was stated during the discussion of the requirements, preferably not of course; it does introduce a possibility of user-error and a time load on the user to have to measure manually. Another option is to use pressure sensors (optionally differential) to alleviate/reduce the need for a full 3D setup of readers. This would of course require the tags to be equipped with a barometer and integrate the measurement in the transmitted packet of the tag.
Determining the location and orientation of the plane is something that should not be left to the user without requiring the user to survey the position of the plane and determine the exact orientation and setting the parameters in MVN Studio. So, preferably, this is done automatically. In the following section it is explained how this can be done.
By way of example, consider the set-up 1600 as illustrated in
In an embodiment of the invention, the objects that create the plane (e.g. a table) can be moved around, and the changed location is determined automatically. The delay depends on the desired averaging to acquire the required accuracy. It will be appreciated that attached tags can be used to determine the position and orientation of an arbitrary shaped plane as well. If the user wants to use an arbitrary shaped plane, tags can be attached at precisely defined positions on the plane. Such a plane may be defined in any suitable way by the application, e.g., via polynomial definition.
It will be appreciated that since it is, with this innovation, now possible to locate objects by using the location system, it is also possible to integrate wireless IMUs into MVN and have actors interact with objects to which this IMU is attached. Other advantages, features and consequences of the invention will be appreciated by those of skill in the art from the description herein.
All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.
The use of the terms “a” and “an” and “the” and similar referents in the context of describing the invention (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms “comprising,” “having,” “including,” and “containing” are to be construed as open-ended terms (i.e., meaning “including, but not limited to,”) unless otherwise noted. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., “such as”) provided herein, is intended merely to better illuminate the invention and does not pose a limitation on the scope of the invention unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention.
Certain examples of the invention are described herein, including the best mode known to the inventors for carrying out the invention. Variations of those examples will be apparent to those of ordinary skill in the art upon reading the foregoing description. The inventors expect skilled artisans to employ such variations as appropriate, and the inventors intend for the invention to be practiced otherwise than as specifically described herein. Accordingly, this invention includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the invention unless otherwise indicated herein or otherwise clearly contradicted by context.
This application claims the benefit of U.S. Provisional Application No. 61/273,517, filed Aug. 5, 2010. This application also is a continuation in part of U.S. application Ser. No. 12/534,607 filed Aug. 3, 2009, U.S. application Ser. No. 12/534,526, filed Aug. 3, 2009, and U.S. application Ser. No. 11/748,963, May 15, 2007, all of which are herein incorporated by reference in their entireties, for all that they teach, disclose, and suggest, without exclusion of any portion thereof.
| Number | Date | Country | |
|---|---|---|---|
| 61273517 | Aug 2009 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 12534607 | Aug 2009 | US |
| Child | 12850370 | US | |
| Parent | 12534526 | Aug 2009 | US |
| Child | 12534607 | US | |
| Parent | 11748963 | May 2007 | US |
| Child | 12534526 | US |
