Patent Application
20110292036
The present invention relates generally to methods and systems for three-dimensional (3D) mapping, and specifically to processing of 3D map data.
A number of different methods and systems are known in the art for creating depth maps. In the present patent application and in the claims, the term “depth map” refers to a representation of a scene as a two-dimensional matrix of pixels, in which each pixel corresponds to a respect lye location. In the scene and has a respective pixel value indicative of the distance from a certain reference location to the respective scene location. (In other words, the depth map has the form of an image in which the pixel values indicate topographic information, rather than brightness and/or color of the objects in the scene.) Depth maps may be created, for example, by detection and processing of an image of an object onto which a laser speckle pattern is projected, as described in POT International Publication WO 2007/043036 A1, whose disclosure is incorporated herein by reference.
Depth maps may be processed in order to segment and identify objects in the scene. Identification of humanoid forms (meaning 3D shapes whose structure resembles that of a human being) in a depth map, and changes in these forms from scene to scene, may be used as a means for controlling computer applications. For example, PCT International Publication WO 2007/132451, whose disclosure is incorporated herein by reference, describes a computer-implemented method in which a depth map is segmented so as to find a contour of a humanoid body. The contour is processed in order to identify a torso and one or more limbs of the body. An input is generated to control an application program running on a computer by analyzing a disposition of at least one of the identified limbs in the depth map.
Computer interfaces based on three-dimensional sensing of parts of the user's body have also been proposed. For example, PCT International Publication WO 2003/071410, whose disclosure is incorporated herein by reference, describes a gesture recognition system using depth-perceptive sensors. A three-dimensional sensor provides position information, which is used to identify gestures created by a body part of interest. The gestures are recognized based on the shape of the body part and its position and orientation over an interval. The gesture is classified for determining an input into a related electronic device.
Embodiments of the present invention provide an enhanced interface between sensors and software that are used in creating a depth map and application programs that make use of the depth map information.
There is therefore provided, in accordance with an embodiment of the present invention, a method for processing data, including receiving a depth map of a scene containing a body of a humanoid subject, the depth map including a matrix of pixels, each pixel corresponding to a respective location in the scene and having a respective pixel depth value indicative of a distance from a reference plane to the respective location. The depth map is processed in a digital processor to extract a skeleton of at least a part of the body, the skeleton including multiple joints having respective coordinates. An application program interface (API) is provided, indicating at least the coordinates of the joints.
In a disclosed embodiment, the skeleton includes two shoulder joints having different, respective depth values. The different depth values of shoulder joints define a coronal plane of the body that is rotated by at least 10° relative to the reference plane.
In one embodiment, the API includes a first interface providing the coordinates of the joints and a second interface providing respective depth values of the pixels in the depth map.
Additionally or alternatively, receiving the depth map includes receiving a sequence of depth maps as the body moves, and processing the depth map includes tracking movement of one or more of the joints over the sequence, wherein the API includes a first interface providing the coordinates of the joints and a second interface providing an indication of gestures formed by the movement of the one or more of the joints.
In some embodiments, the scene contains a background, and processing the depth map includes identifying one or more parameters of at least one element of the background, wherein the API includes a first interface providing the coordinates of the joints and a second interface providing the one or more parameters of the at least one element of the background. In one embodiment, the at least one element of the background includes a planar element, and the one or more parameters indicate a location and orientation of a plane corresponding to the planar element.
Additionally or alternatively, when the scene contains respective bodies of two or more humanoid subjects, processing the depth map may include distinguishing the bodies from one another and assigning a respective label to identify each of the bodies, wherein the API identifies the coordinates of the joints of each of the bodies with the respective label. In one embodiment, distinguishing the bodies includes identifying an occlusion of a part of one of the bodies in the depth map by another of the bodies, wherein the API identifies the occlusion.
Further additionally or alternatively, processing the depth map includes computing a confidence value associated with an identification of an element in the scene, wherein the API indicates the identification and the associated confidence value.
There is also provided, in accordance with an embodiment of the present invention, apparatus for processing data, including an imaging assembly, which is configured to generate a depth map of a scene containing a body of a humanoid subject. A processor is configured to process the depth map to extract a skeleton of at least a part of the body, the skeleton including multiple joints having respective coordinates, and to provide an application program interface (API) indicating at least the coordinates of the joints.
There is additionally provided, in accordance with an embodiment of the present invention, a computer software product, including a computer-readable medium in which program instructions are stored, which instructions, when read by a processor, cause the processor to receive a depth map of a scene containing a body of a humanoid subject, to process the depth map to extract a skeleton of at least a part of the body, the skeleton including multiple joints having respective coordinates, and to provide an application program interface (API) indicating at least the coordinates of the joints.
There is further provided, in accordance with an embodiment of the present invention, a method for processing data, including receiving a depth map of a scene including a matrix of pixels, each pixel corresponding to a respective location in the scene and having a respective pixel depth value indicative of a distance from a reference plane to the respective location. The depth map is segmented in a digital processor to identify one or more objects in the scene. A label map is generated, including respective labels identifying the pixels belonging to the one or more objects. An indication of the label map is provided via an application program interface (API).
In a disclosed embodiment, receiving the depth map includes receiving a sequence of depth maps as the objects move, and generating the label map includes updating the label map over the sequence responsively to movement of the objects.
Additionally or alternatively, when at least one of the objects includes multiple segments, generating the label map includes assigning a single label to all of the segments.
Further additionally or alternatively, segmenting the depth map includes recognizing an occlusion of a part of one of the identified objects in the depth map by another object, and generating the label map includes identifying the occlusion in the label map.
There is moreover provided, in accordance with an embodiment of the present invention, apparatus for processing data, including an imaging assembly, which is configured to generate a depth map of a scene including a matrix of pixels. A processor is configured to segment the depth map to identify one or more objects in the scene, to generate a label map including respective labels identifying the pixels belonging to the one or more objects, and to provide an indication of the label map via an application program interface (API).
There is furthermore provided, in accordance with an embodiment of the present invention, a computer software product, including a computer-readable medium in which program instructions are stored, which instructions, when read by a processor, cause the processor to receive a depth map of a scene including a matrix of pixels, to segment the depth map to identify one or more objects in the scene, to generate a label map including respective labels identifying the pixels belonging to the one or more objects, and to provide an indication of the label map via an application program interface (API).
The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which:
Depth maps provide a wealth of information, particularly when they are presented in a continuous stream over time. Handling this large volume of information is a challenge for software application developers, whose interest and skills are not generally directed to processing of the depth information, but rather to using high-level information regarding people in the scene and their movements in controlling interactive applications. For example, various computer games have been developed that use motion input from an implement held by a user or a marker attached to the user in order to interact with objects on the display screen. Games and other applications based on depth maps, however, have developed only slowly due to the difficulties inherent in capturing, processing, and extracting high-level information from such maps.
Embodiments of the present invention that are described hereinbelow address this problem by providing middleware—supporting software for extracting information from a depth map—with an application program interface (API) for application software developers. The middleware processes depth maps of a scene that are output by an imaging assembly in order to extract higher-level information about the scene, and particularly about humanoid forms in the scene. Methods for processing depth maps that may be used in this context are described, for example, in U.S. patent application Ser. Nos. 12/854,187 and 12/854,188, both filed Aug. 11, 2010, whose disclosures are incorporated herein by reference. The API enables applications to access this information in a structured and straightforward way.
In some embodiments, the middleware reconstructs a skeleton of at least a part of the body of a humanoid subject in the scene. The API indicates coordinates of the joints of the skeleton, which may include both location and orientation coordinates. These coordinates—particularly the coordinates of the two shoulders may indicate that the body is turned at least partly away from the imaging assembly, i.e., that the coronal plane of the body (defined as a vertical plane that divides the body into anterior and posterior sections) is rotated relative to the reference plane of the imaging assembly. The middleware and API can be configured to measure and give an indication of this rotation angle, as well as rotations of the skeleton about horizontal axes. The measured angles may be anywhere in the range between 0° and 360°, typically with angular resolution of 10° or better, including rotations of 90°, at which the body is turned sideways relative to the reference plane, with the coronal plane parallel to the optical axis of the imaging assembly.
Other information provided by the API regarding the skeleton may include, for example, confidence values associated with joint coordinates, as well as identifiers associated with the parts of different skeletons, particularly when there are multiple bodies in the scene. The identifiers are useful to application developers in separating the actions of two simultaneous users, such as game participants, even when one of the bodies partly occludes the other in the depth map. The confidence values can be useful in making application-level decisions under conditions of conflicting input information due to noise or other uncertainty factors.
The API may provide different levels of information—not only the coordinates of the joints, but also other objects at lower and higher levels of abstraction. For example, at a lower level, the API may provide the actual depth values of the pixels in the depth map. Alternatively or additionally, at a higher level, the middleware may track movement of one or more of the joints over a sequence of frames, and the API may then provide an indication of gestures formed by the movement. As a further option, the middleware may identify elements of the background in the scene, and the API may provide parameters of these elements, such as the locations and orientations of planes corresponding to the floor and/or walls of a room in which the scene is located.
The middleware processes data generated by assembly 22 in order to reconstruct a 3D map, including at least a part of the user's body. The term “3D map” refers to a set of 3D coordinates representing the surface of a given object or objects, such as the user's body. In one embodiment, assembly 22 projects a pattern of spots onto the scene and captures an image of the projected pattern. Assembly 22 or computer 24 then computes the 3D coordinates of points in the scene (including points on the surface of the user's body) by triangulation, based on transverse shifts of the spots in the pattern. This approach is advantageous in that it does not require the user to hold or wear any sort of beacon, sensor, or other marker. It gives the depth coordinates of points in the scene relative to a predetermined reference plane, at a certain distance from assembly 22. Methods and devices for this sort of triangulation-based 3D mapping using a projected pattern are described, for example, in PCT International Publications WO 2007/043036, WO 2007/105205 and WO 2008/120217, whose disclosures are incorporated herein by reference. Alternatively, system 20 may use other methods of 3D mapping, based on single or multiple cameras or other types of sensors, as are known in the art.
In the present embodiment, system 20 captures and processes a sequence of three-dimensional (3D) maps containing user 28, while the user moves his hands and possibly other parts of his body. Middleware running on assembly 22 and/or computer 24 processes the 3D map data to extract a skeleton of the body, including 3D locations and orientations of the user's hands and joints. It may also analyze the trajectory of the hands over multiple frames in order to identify gestures delineated by the hands. The skeleton and gesture information are provided via an API to an application program running on computer 24. This program may, for example, move and modify objects 30 presented on display 26 in response to the skeleton and/or gesture information. For example, the application program may be an interactive game, in which the user interacts with objects 30 in a virtual space by moving his or her body appropriately.
Computer 24 typically comprises a general-purpose computer processor, which is programmed in software to carry out the functions described hereinbelow. The software may be downloaded to the processor in electronic form, over a network, for example, or it may alternatively be provided on tangible media, such as optical, magnetic, or electronic memory media. Alternatively or additionally, some or all of the functions of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although computer 24 is shown in
As another alternative, at least some of these processing functions may be carried out by a suitable processor that is integrated with display screen 26 (in a television set, for example) or with any other suitable sort of computerized device, such as a game console or media player. The sensing functions of assembly 22 may likewise be integrated into the computer or other computerized apparatus that is to be controlled by the sensor output.
A processor 40 receives the images from subassembly 34 and compares the pattern in each image to a reference pattern stored in a memory 42. The reference pattern is typically captured in advance by projecting the pattern onto a reference plane at a known distance from assembly 22. Generally, this plane is perpendicular to the optical axis of subassembly 34. Processor 40 computes local shifts of parts of the pattern over the area of the depth map and translates these shifts into depth coordinates. Details of this process are described, for example, in PCT International Publication WO 2010/004542, whose disclosure is incorporated herein by reference. Alternatively, as noted earlier, assembly 22 may be configured to generate depth maps by other means that are known in the art, such as stereoscopic imaging or time-of-flight measurements.
Processor 40 outputs the depth maps via a communication link 44, such as a Universal Serial Bus (USB) connection, to a suitable interface 46 of computer 24. The computer comprises a central processing unit (CPU) 48 with a memory 50 and a user interface 52, which drives display 26 and may include other components, as well. As noted above, imaging assembly 22 may alternatively output only raw images from subassembly 34, and the depth map computation described above may be performed in software by CPU 48. Middleware for extracting higher-level information from the depth maps may run on processor 40, CPU 48, or both. CPU 48 runs one or more application programs, which drive user interface 52 based on information provided by the middleware via an API, as described further hereinbelow.
Computer 24 runs a package of middleware 60 for processing depth maps provided by imaging assembly 22 and outputting control commands to the imaging assembly as needed. Middleware 60 comprises the following layers:
An API 70 provides a set of objects that can be called by application 72 to access information generated by different layers of middleware 60. The API may include some or all of the following items:
As noted above, the label map provided by layer 64 may be applied not only to humanoid bodies, but also to any sort of object in the scene. It facilitates identifying and maintaining distinction between objects in a frame and over multiple frames, notwithstanding changes in apparent shape as objects move and occlusion of one object by another.
As noted above, each joint may be labeled with a number of items of information, which are available via API 70. For example, as shown in
Shoulder joints 82 and hip joints 86 define the coronal plane of skeleton 80. In the example shown in
Middleware 60 identifies the planar structures in the scene corresponding to walls 106 and floor 108 and is able to provide information about these structures to application 72 via API 70. The information may be either in parametric form, in terms of plane equations, or as a mask of background pixels. In most applications, the background elements are not of interest, and they are stripped out of the frame using the information provided through API 70 or simply ignored.
When a scene includes more than one moving body, as in scene 100, applications using the scene information generally need accurate identification of which parts belong to each body. In
As was explained above in reference to
It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.
This application claims the benefit of U.S. Provisional Patent Application 61/349,894, filed May 31, 2010, which is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61349894 | May 2010 | US |
