Patent Application
20130260885
The invention refers to an electronic entertainment system and to a method of providing entertainment.
In recent times video games that use motion tracking input devices have become popular. These input devices enable users to control and interact with the game console without the need to touch a game controller, through a natural user interface using gestures. The respective software is able to interpret specific gestures, making completely hands-free control of electronic devices possible by using an infrared projector and a video camera. By using a special microchip it is possible to track the movement of objects and individuals in three dimensions. Motion tracking input devices are commercialized under the trademarks Kinect™ (for Microsoft's Xbox 360) and PlayStation Eye (for Sony's Playstation 3).
Dance Central is a music video game for the Xbox 360 that uses the Kinect™ motion peripheral. The gameplay involves the player performing given dance moves, which are tracked by Kinect™ and represented on the screen by an avatar.
Conventional motion tracking input devices such as Kinect™ are configured to be connected to a gaming console or a personal computer. It is reported that Kinect™ is capable of simultaneously tracking only up to two active players for motion analysis with a feature extraction of 20 joints per player. The number of people the device can see is further limited by how many will fit in the field-of-view of the device's camera. The number of people the device can track simultaneously is further limited by the maximum data transfer rate and the maximum calculating capacity of the input device or the computer hardware to which it is connected. In particular, it has to be noted that motion tracking usually requires a motion analysis comprising a feature extraction of a plurality of characteristic body parts per player in order to obtain user body motion tracking data of said player(s), which is a computationally intensive task.
There is a need for providing motion tracking functionality for entertainment purposes making it possible to simultaneously track a larger number of players. In particular, the system should be capable of simultaneously tracking at least 8, preferably at least 16, more preferably at least 32 players. While conventional motion tracking input devices are mostly used in a private environment only, the simultaneous tracking of a higher number of players would allow to use the motion tracking systems in public spaces, such as in entertainment parks, amusement halls, fitness centers etc.
The instant invention aims at providing an entertainment system and a method of providing entertainment that allows the simultaneous motion tracking of at least 4 players. The entertainment system shall be easy to set up and inexpensive. In particular, the entertainment system shall not require any large-capacity computer or mainframe computer, but shall work on standard personal computers.
To solve these objects the invention, according to a first aspect thereof, provides an entertainment system comprising
a server computer,
a plurality of client computers each connected to the server computer via a data communication network,
a plurality of 3D scanners connected to each client computer, and
an electronic video display device,
each 3D scanner being configured to continuously capture 3D data of at least one player and to transmit said 3D data to the client computer,
each client computer comprising data processing means for subjecting said 3D data to a motion analysis comprising a feature extraction of a plurality of characteristic body parts per player thereby obtaining user body motion tracking data of said at least one player,
each client computer further comprising sending means for continuously sending said user body motion tracking data of each player to the server computer,
said server computer continuously receiving said user body motion tracking data from said plurality of client computers and comprising processing means for comparing said user body motion tracking data from each player with reference body motion tracking data, thereby obtaining matching quality data for each player,
said server computer being connected to said electronic video display device for displaying said matching quality data on said electronic video display device.
The invention provides for a system that is scalable to a large number of 3D scanners and an accordingly large number of players. Capturing and analyzing 3D data provided by a 3D scanner is computationally intensive. Therefore, a personal computer is usually capable of handling only a limited number of 3D scanners, such as a maximum of four scanners. A further limiting factor is the (PCI)-Bus used for peripheral devices, such as the 3D scanners. In addition, computational power is needed for synchronizing the motion tracking data of each player with reference data in order to analyze how close the player's movements match with the reference data. According to the invention, a client-server system is proposed in which the computationally intensive tasks are distributed according to the following principle. The capturing and analysis of 3D data provided by the 3D scanners is performed in the client computers in order to obtain motion tracking data for each player. The synchronization of all player's motion tracking data with reference data and the comparing step in order to obtain matching quality data is performed in the server computer. The number of client computers needed depends on the number of 3D scanners required, which in turn depends on the number of players. Usually the maximum possible number of 3D scanners per client computer is four.
The term 3D scanner generally refers to a hardware component that is capable of capturing 3D data of a moving body. Preferably, the 3D scanner comprises an infrared projector and a camera. The infrared projector preferably collects distance information about surfaces within its field of view. The picture produced by the infrared projector describes the distance to a surface at each point in the picture. This allows the three dimensional position of each point in the picture to be identified. Preferably the 3D scanner has a frame rate of >15 frames per second, more preferably a frame rate of 30 frames per second.
The 3D data obtained by the 3D scanner preferably is a digital image as captured by the camera, such as a raster image consisting of a plurality of pixels, wherein each pixel contains color information as obtained by a grayscale or a color camera and wherein distance information as obtained by the infrared projector is assigned to each pixel.
The 3D scanner can be connected to the client computer by USB or other high-speed bus type connection.
The term motion analysis refers to the processing of the 3D data by the use of algorithms in order to detect specific body parts of the player. The motion analysis comprises feature extraction of characteristic body parts of the player. Feature extraction is a mathematical procedure, which results in a transformation of the 3D data into a reduced representation set of features, namely the specific body parts, such as torso, head, left hand, right hand, right hip, left hip, etc). Preferably, up to 15 body parts are detected and their coordinates identified.
The system can be used for any kind of entertainment or exercise that involves a specific course of motions of human body parts, such as dancing, physical exercise, fitness training, aerobic etc. The motion tracking data of the players are compared with the motion tracking data of a reference person, such as a teacher, in order to determine any deviations. The goal of the game for each player is to achieve a synchronization of his movements with those of the reference person. The reference person can either be displayed on the video display device or a person that is dancing or exercising in real time. By comparing the user body motion tracking data from each player with reference body motion tracking data the server computer obtains matching quality data, which are representative of the player's degree of conformity with the reference person's movements.
According to a preferred embodiment said 3D scanner is configured to capture a sequence of frames containing 3D data of at least one player.
According to a preferred embodiment said data processing means of said client computer is configured to separately subject each frame of 3D data to said motion analysis thereby obtaining a data set of user body motion tracking data for each frame.
According to a preferred embodiment said client computer comprises a clock that creates a timestamp each time a frame of 3D data is captured and is configured to add said time stamp to the corresponding data set.
According to a preferred embodiment said sending means are configured to separately send each data set of user body motion tracking data to the server computer as a data packet.
According to a preferred embodiment said sending means are configured to send each data packet by using the User Datagram Protocol (UDP). Using UDP results in a very quick transfer of the data packets, although it does not guarantee the completeness of the transmitted data and the correct sequence of the data packets.
According to a preferred embodiment said reference body motion tracking data comprise a plurality of data sets, each comprising reference body motion tracking data and a time stamp.
According to a preferred embodiment said processing means of said server computer is configured to synchronize said data sets of user body motion tracking data with said data sets of reference body motion tracking data by using the time stamps contained in each data set.
According to a preferred embodiment said processing means of said server computer is configured to transform, normalize, filter and/or scale said user body motion tracking data.
According to a preferred embodiment said system further comprises means for generating said reference body motion tracking data in real-time, including a further 3D scanner being configured to capture 3D data of a reference player, processing means for subjecting said 3D data to a motion analysis comprising a feature extraction of a plurality of characteristic body parts of said reference player thereby obtaining said reference body motion tracking data.
According to a preferred embodiment the server computer further comprises a scoring engine for generating a player score based on said matching quality data.
According to a preferred embodiment the 3D scanners are arranged in an array of at least three rows and at least three columns.
According to a preferred embodiment the 3D scanners each comprise a digital video camera and a depth sensor, the depth sensor adding depth information to each pixel of the digital image obtained by the video camera.
According to a preferred embodiment the 3D scanners are positioned above and in front of each player, the optical axis of the device being oriented at an angle of 30-60° with a horizontal plane.
According to a preferred embodiment the sensing range of the depth sensor is adjusted such that neighboring depth sensors do not interfere.
According to a preferred embodiment neighboring 3D scanners have an angular field of view such that their fields of view do not overlap each other.
To solve the objects underlying the instant invention a second aspect of the invention refers to a method for providing entertainment, the method comprising providing
a server computer,
a plurality of client computers each connected to the server computer via a data communication network,
a plurality of 3D scanners connected to each client computer, and
an electronic video display device,
each 3D scanner capturing 3D data of at least one player and transmitting said 3D data to the client computer,
subjecting said 3D data to a motion analysis, said motion analysis being performed in each client computer and comprising a feature extraction of a plurality of characteristic body parts per player thereby obtaining user body motion tracking data of said at least one player,
continuously sending said user body motion tracking data of each player from each client computer to the server computer,
continuously receiving said user body motion tracking data in the server computer from said plurality of client computers,
comparing said user body motion tracking data from each player with reference body motion tracking data, thereby obtaining matching quality data for each player,
displaying said matching quality data on an electronic video display device connected to said server computer.
Preferred embodiments of the method are described in the sub claims and will become apparent from the description of the accompanying drawings.
The invention will now be further explained with reference to the drawings:
Aspects of the present invention are disclosed in the following description and related figures directed to specific embodiments of the invention. Those skilled in the art will recognize that alternate embodiments may be devised without departing from the spirit or the scope of the claims. Additionally, well-known elements of exemplary embodiments of the invention will not be described in detail or will be omitted so as not to obscure the relevant details of the invention.
It should be understood that the described embodiments are not necessarily to be construed as preferred or advantageous over other embodiments. Moreover, the terms “embodiments of the invention”, “embodiments” or “invention” do not require that all embodiments of the invention include the discussed feature, advantage or mode of operation.
Further, many of the embodiments described herein are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It should be recognized by those skilled in the art that the various sequence of actions described herein can be performed by specific circuits (e.g., application specific integrated circuits (ASICs)) and/or by program instructions executed by at least one processor. Additionally, the sequence of actions described herein can be embodied entirely within any form of computer-readable storage medium such that execution of the sequence of actions enables the processor to perform the functionality described herein. Thus, the various aspects of the present invention may be embodied in a number of different forms, all of which have been contemplated to be within the scope of the claimed subject matter. In addition, for each of the embodiments described herein, the corresponding form of any such embodiments may be described herein as, for example, “a computer configured to” perform the described action.
The entertainment system further comprises an electronic video display device 9, which is arranged to be visible to each player while standing in his field 2.
The entertainment system can be located in an entertainment park, in a fitness center or in a private location.
The entertainment system shown in
The hardware components of the entertainment system are illustrated in
In
The data sets are transformed into data packages and transmitted to the server computer 10. The transmission is performed by using UDP, which is faster than TCP. Each data set is transmitted separately so that up to 30 data packets have to be sent per second in case the 3D scanner has a frame rate of 30 frames per second.
The server computer 10 not only receives data sets 231, 232, 233 . . . 23i representing user body motion tracking data, but also data sets 241, 242, 243 . . . 24i representing reference body motion tracking data. The reference body motion tracking data can be loaded onto the server computer 10 from a data carrier 25 and stored in the server computer for later use (playback mode). In playback use the server computer 10 processes the reference body motion tracking data according to a defined frame rate and feeds the stored data sets 241, 242, 243 . . . 24i to an animation engine that generates a moving body image, which can be displayed on the electronic display device 9 as an avatar.
Alternatively, the reference body motion tracking data can be generated and transmitted in real time (live mode) by using a further 3D scanner 26 and a further client computer 27, whereby the procedure is the same as explained with reference to 3D scanner 5 and client computer 11. The 3D scanner 26 captures the movements of a reference user, such as an instructor, and the 3D scanners 5 capture the movements of the plurality of players, the players' task being to synchronize their movements with the instructor's real time movements as good as possible. In order to facilitate synchronization, the instructor's movements can be synchronized with music. The instructor can be located in the same room as the players, but can also be located remotely, such as in a different city or country. In the latter case, client computer 27 is connected to server computer 10 via an internet connection.
In the live mode, the server computer 10 continuously receives reference body motion tracking data from client computer 27 and temporarily stores the data in a processing unit 28, where said user body motion tracking data from each player is compared with reference body motion tracking data, thereby obtaining matching quality data for each player. The processing unit 28 synchronizes said data sets 231, 232, 233 . . . 23i representing user body motion tracking data with the data sets 241, 242, 243 . . . 24i representing reference body motion tracking data by using the time stamps contained in each data set. The synchronizing step comprises sorting at least part of the data sets 231, 232, 233 . . . 23i such that they match with the chronologically arranged time stamps of the data sets 241, 242, 243 . . . 24i. The comparing step comprises comparing the coordinates assigned to the body parts in data sets with matching time stamps and identifying how close the coordinates match. The comparing step may optionally comprise comparing the coordinates assigned to the body parts in data sets that differ with regard to their time stamps according to a defined time tolerance value. The time tolerance value may be preselected, thereby offering the possibility of adjusting the level of difficulty.
Before synchronizing and comparing the data sets the user body motion data may optionally be subjected to transformation, normalization, filtering and/or scaling These steps serve to compensate for circumstances that result in that the user body motion tracking data cannot be perfectly compared with the reference body motion tracking data. Such circumstances can lie in that the 3D scanners optical axis is arranged at an angle with the horizontal plane resulting in a distortion of the image captured by the 3D scanner. Other circumstances that influence the comparability of the user body motion tracking data are differences among the players and between the players and the instructor in body height and in the body's proportions.
The server computer 10 optionally further comprises a scoring engine 29 for generating a player score based on the matching quality of the individual players. The player score may be displayed on the electronic display device 9, which is connected to the server computer 10. Further, an animation engine 30 may be provided that generates an moving body image based on the user body motion tracking data for displaying the players as moving avatars on the electronic display device 9.
The foregoing description and accompanying figures illustrate the principles, preferred embodiments and modes of operation of the invention. However, the invention should not be construed as being limited to the particular embodiments discussed above. Additional variations of the embodiments discussed above will be appreciated by those skilled in the art.
Therefore, the above-described embodiments should be regarded as illustrative rather than restrictive. Accordingly, it should be appreciated that variations to those embodiments can be made by those skilled in the art without departing from the scope of the invention as defined by the following claims.
