Multiple cameras are used to capture activity in a scene. Subsequent processing of the captured images enables end users to view the scene and move throughout the scene in over a full 360-degree range of motion. For example, multiple cameras may be used to capture a sports game and end users can move throughout the re-created field of play freely. For example, the end user may also view the game from a virtual camera.
The same numbers are used throughout the disclosure and the figures to reference like components and features. Numbers in the 100 series refer to features originally found in
Sporting events and other competitions are often broadcast for the entertainment of end users. These games may be rendered in a variety of formats. For example, a game can be rendered as a two-dimensional video or a three-dimensional video. The games may be captured using one or more high-resolution cameras positioned around an entire field of play. The plurality of cameras may capture an entire three-dimensional volumetric space, including the field of play. In embodiments, the camera system may include multiple super high-resolution cameras for volumetric capture. The end users can view the action of the game and move through the captured volume freely by being presented with a sequence of images representing the three-dimensional volumetric space. Additionally, an end user can view the game from a virtual camera that follows the action within the field by following the ball or a specific player in the three-dimensional volumetric space.
In embodiments, a camera system used for volumetric capture may include one or more physical cameras with 5120×3072 resolution, configured throughout a stadium to capture the field of play. For example, the number of cameras in the camera system may be thirty-eight. A subset of cameras may be selected, such as eighteen cameras from among the thirty-eight cameras, to cover the entire field of play and ensure that each pixel in the field of play is captured by at least three cameras. The camera system may capture a real-time video stream from a plurality of cameras. The plurality of cameras may capture the field of play at 30 frames per second (fps). The subset of cameras selected may be different in different scenarios. For example, depending on the structure surrounding the field of play, each location may be captured by at least three cameras using a varying number of cameras.
For ease of description the present techniques are described by referencing an American football game. In the example of an American football game, thirty-eight cameras may be located throughout a stadium to capture volumetric content. Segmentation and three-dimensional (3D) reconstruction may be applied to the captured content to build a 3D scene (e.g., voxel) for every frame. An optical tracking algorithm may be applied to track the ball and each player automatically on the playfield. As used herein, optical tracking data may be trajectories or other movement paths generated according to the visible trajectories or movement paths as captured by one or more cameras of the system. The optical tracking data may be generated to control the movement of a virtual camera. In embodiments, the virtual camera can follow a specific object such as a ball or player of interest to present an immersive viewing experience.
The present techniques enable player and ball trajectory post-processing. For example, the present techniques include a player and/or ball post-processing method that optimizes received player and/or ball trajectories for high quality immersive volumetric video generation. In embodiments, affinity player tracking is implemented to estimate the target player's position by leveraging the information from surrounding players. In examples, affinity player tracking may be implemented when portions or the target player's trajectory is missing or otherwise incomplete. The present techniques can recover the missing portions of the target player's trajectory in a huddle situation. Additionally, the present techniques use a dynamic buffer based on a quadratic Bezier filtering method by using all positions as Bezier control points. The ball/player trajectory is efficiently smoothed without a large position discrepancy as compared to conventional approaches. The present techniques also include game status detection to automatically classify the game as being in one or more states of play. The states of play may be used to stabilize the movement of ball tracking during game breaks. The present techniques provide high quality virtual camera movement without broken or missing trajectory data. The post-processing provided by the present techniques enables an immersive media experience by following the ball or player. The present techniques can significantly improve accuracy of a virtual camera.
As used herein, a game may refer to a form of play according to a set of rules. The game may be played for recreation, entertainment, or achievement. A competitive game may be referred to as a sport, sporting event, or competition. Accordingly, a sport may also be a form of competitive physical activity. The game may have an audience of spectators that observe the game. The spectators may be referred to as end-users when the spectators observe the game via an electronic device, as opposed to viewing the game live and in person. The game may be competitive in nature and organized such that opposing individuals or teams compete to win. A win refers to a first individual or first team being recognized as triumphing over other individuals or teams. A win may also result in an individual or team meeting or securing an achievement. Often, the game is played on a field, court, within an arena, or some other area designated for gameplay. The area designated for gameplay typically includes markings, goal posts, nets, and the like to facilitate gameplay.
A game may be organized as any number of individuals configured in an opposing fashion and competing to win. A team sport is a game where a plurality of individuals is organized into opposing teams. The individuals may be generally referred to as players. The opposing teams may compete to win. Often, the competition includes each player making a strategic movement to successfully overcome one or more players to meet a game objective. An example of a team sport is football.
Generally, football describes a family of games where a ball is kicked at various times to ultimately score a goal. Football may include, for example, association football, gridiron football, rugby football. American football may be a variation of gridiron football. In embodiments, the American football described herein may be as played according to the rules and regulations of the National Football League (NFL). While American football is described, the present techniques may apply to any event where an individual makes strategic movements within a defined space. In embodiments, a strategic movement may be referred to as a trajectory. An end user can be immersed in a rendering of the event based on this trajectory according to the techniques described herein. In particular, the present techniques enable trajectory refinement, which is based on the identification of affinity players with respect to a target player. Again, for ease of description, the present techniques are described using an American football game as an example. However, any game, sport, sporting event, or competition may be used according to the present techniques.
High precision player and ball tacking are crucial to generate high quality and smooth volumetric video. However, due to the complexity of American football, optical tracking is inaccurate due to large huddles, occlusion, lighting, etc. These issues with optical tracking cause a player to not be visible in one or more camera views. As a result, with optical tracking the object trajectory, such as a player or ball, will be lost or wrong. The missing/wrong player and ball path will lead to non-contiguous or abnormal virtual camera movement which significantly impacts the user experience if the virtual camera doesn't follow the player or ball in real practice.
For example,
Referring again to
At block 106, trajectory refinement refines the trajectory of the player and the ball. The trajectory may be refined using a quadratic Bezier calculation. At block 108, ball tracking stabilization stabilizes the ball tracking during a game break. At block 110, a virtual camera calculator calculates the movement of a virtual camera.
The diagram of
The operation of the virtual camera may depend on the positions of players or ball, which are typically the focus of the game. In embodiments, the player location and trajectory control or dictate the virtual camera path movement. In embodiments, the player trajectory may provide a path along which a virtual camera may progress through the field of play. The ability to progress through the field of play enables an end user to “see” the same view as a player saw during real-time gameplay.
The diagram of
Accurate player tracking and trajectory generation can be a very challenging task in team sports due to the heavy occlusion of players, large variation of player body shapes, and the generally similar appearance of players of the same team. Thus, the present techniques enable optical trajectory refinement based on an identification of affinity players in a field of play, at any frame from any camera using only the data captured by that particular camera. In particular, the present techniques connect a target player to corresponding affinity players, and in the event that the target player is occluded/not visible in a camera view, the visible trajectory of the affinity players are used to estimate the trajectory of the target player.
As used herein, an affinity player is a player that is likely to guard a target player when the target player is on offense. When the target player is on defense, an affinity player is a player that is likely to be guarded by the target player. Typically, it can be found that heavy huddle/occlusion is the major reason why the target player's position is correct. However, the jersey or other identifying information cannot be detected due to occlusion. As the target player is occluded by at least 1-2 surrounded players, they can be treated as a bundle that generally moves in the same direction. These surrounding players largely represent the behavior of the missing, invisible target player. These surrounding players are affinity players. In examples, the target player and affinity players are from the opposite team. Typically, in case of huddle/occlusion, the target player and affinity players have close body contact and run in the same direction.
When the target player is no longer visible in optical tracking or otherwise unidentifiable, a mean location of the identified affinity players is used to estimate the location of the player. These estimated locations are used to derive or refine the missing or jittery portions trajectory of the target player. As illustrated in
As described above, player/ball path jittering as calculated using optical tracking techniques may degrade the calculated trajectory, and ultimately yield poor virtual camera movement. Accordingly, player/ball path jittering is another important factor—which leads to a poor virtual camera movement experience. It causes volumetric video shaking due to non-smooth camera motion. Traditionally, a mean or quadric filter is applied to trajectories to smooth the trajectory. In these traditional solutions, the smoothness of the resulting trajectory is dependent on a length of a buffer. In particular, if the buffer is too long, the processed position would be far from the original one. If the buffer is too short, the refined position would not be smooth as expected.
In addition to smoothing the ball and player trajectories, the ball tracking may be stabilized to ensure accurate virtual camera tracking with respect to the ball. For example, at several times in an American football gamer, breaks occur and more than one ball is present within the field of play. When more than one ball is present within a field of play, unexpected virtual camera movement may occur in an attempt to follow the ball. The present techniques provide game status detection to determine various states of an event, such as an American football game. The various states may be used to stabilize ball tracking during game breaks or other situations where multiple balls are present.
A current state, also referred to as a game state or status, can be estimated according to the velocity information from the ball and player trajectories. In embodiments, inter-frame information 706 includes velocity, position, and other data such as orientation for the ball 702 and the players 704. The temporal convolutional network (TCN) 710 is used to determine the game status with a look-ahead buffer. In embodiments, the TCN outputs a state every 10 frames. The TCN 710 includes a plurality of temporal convolution layers 712, a fully connected layer 714, and a SoftMax function 716. In the example of
In embodiments, ball position is stabilized as described with respect to
In this manner, the present techniques enable a smooth and seamless way to track a player and ball position virtually. In an exemplary virtual camera simulator to use to simulate “follow ball” and “follow player” feature, after applying the post-processing methods described herein, the error rate of ball tracking is reduced from 6% to 2%, and player tracking error rate is reduced from 4% to 1.6%. It greatly improves the capability of virtual camera to keep track of the ball and player of interest.
Accordingly, the present techniques enable an estimation of missing tracking by affinity player. The present techniques also enable a refinement of player and object trajectory by using a quadratic Bezier of a dynamic sliding window across points of the trajectory. The number of points used within the sliding window may change as necessary. Further, the present techniques enable stabilization of a virtual camera by game status detection. With these new features, we can create high quality virtual camera movement for compelling immersive media.
This process flow diagram is not intended to indicate that the blocks of the example process of method 1000 is to be executed in any particular order, or that all of the blocks are to be included in every case. Further, any number of additional blocks not shown may be included within the method 1000, depending on the details of the specific implementation.
Referring now to
The computing device 1100 may also include a graphics processing unit (GPU) 1108. As shown, the CPU 1102 may be coupled through the bus 1106 to the GPU 1108. The GPU 1108 may be configured to perform any number of graphics operations within the computing device 1100. For example, the GPU 1108 may be configured to render or manipulate graphics images, graphics frames, videos, or the like, to be displayed to a viewer of the computing device 1100.
The CPU 1102 may also be connected through the bus 1106 to an input/output (I/O) device interface 1110 configured to connect the computing device 1100 to one or more I/O devices 1112. The I/O devices 1112 may include, for example, a keyboard and a pointing device, wherein the pointing device may include a touchpad or a touchscreen, among others. The I/O devices 1112 may be built-in components of the computing device 1100, or may be devices that are externally connected to the computing device 1100. In some examples, the memory 1104 may be communicatively coupled to I/O devices 1112 through direct memory access (DMA).
The CPU 1102 may also be linked through the bus 1106 to a display interface 1116 configured to connect the computing device 1100 to a display device 1118. The display devices 1116 may include a display screen that is a built-in component of the computing device 1100. The display devices 1118 may also include a computer monitor, television, or projector, among others, that is internal to or externally connected to the computing device 1100. The display device 1118 may also include a head mounted display.
The computing device 1100 also includes a storage device 1120. The storage device 1120 is a physical memory such as a hard drive, an optical drive, a thumbdrive, an array of drives, a solid-state drive, or any combinations thereof. The storage device 1120 may also include remote storage drives.
The computing device 1100 may also include a network interface controller (NIC) 1122. The NIC 1122 may be configured to connect the computing device 1100 through the bus 1106 to a network 1124. The network 1124 may be a wide area network (WAN), local area network (LAN), or the Internet, among others. In some examples, the device may communicate with other devices through a wireless technology. For example, the device may communicate with other devices via a wireless local area network connection. In some examples, the device may connect and communicate with other devices via Bluetooth® or similar technology.
The computing device 1100 further includes camera 1126. Although the camera is illustrated as included in the computing device 1100, the camera array may be remotely located from the computing device 1100 and the computing device 1100 may be communicatively coupled with the camera array 1126. In the example of an American football game, the camera array 1126 may be disposed around the field of play.
Additionally, the computing device 1100 includes a trajectory post-processor 1128. The trajectory post-processor 1128 may be configured to enable a 360° view of a sporting event from any angle. In particular, ball and player trajectories generated may be processed such that an end user can virtually experience any location within the field of play. In particular, the end user may establish a viewpoint in the game, regardless of particular camera locations used to capture images of the sporting event. The trajectory post-processor 1128 includes an optical tracker receiver 1130 to determine an optical track of a target player and a ball. An affinity tracker 1132 is to estimate the missing portions of the target player trajectory or ball trajectory based on the trajectory of an affinity player. A trajectory refiner 1134 is to refine the trajectory of the player and ball via a quadratic Bezier function. A ball trajectory stabilizer 1136 is to stabilize the trajectory of the ball. A virtual camera calculator 1138 calculates the movement of a virtual camera based on the target player and ball trajectories.
The block diagram of
The various software components discussed herein may be stored on one or more computer readable media 1200, as indicated in
The block diagram of
Example 1 is a method for player and ball tracking post processing. The method includes optically tracking a trajectory of a player and ball and determining at least one affinity player for the currently tracked player. The method also includes completing the optical trajectory for the currently tracked player based on an estimated trajectory from the affinity player, stabilizing ball tracking during game break, and calculating virtual camera movement.
Example 2 includes the method of example 1, including or excluding optional features. In this example, the affinity player is from the opposing team.
Example 3 includes the method of any one of examples 1 to 2, including or excluding optional features. In this example, the method includes selecting a plurality of affinity players, wherein a mean location of the affinity players is used to derive the location of the current player.
Example 4 includes the method of any one of examples 1 to 3, including or excluding optional features. In this example, the affinity player location is used to determine the missing trajectory of the current player until the current player is recovered in normal optical tracking.
Example 5 includes the method of any one of examples 1 to 4, including or excluding optional features. In this example, refining the trajectory comprises using a quadratic Bezier curve to smooth the trajectory.
Example 6 includes the method of any one of examples 1 to 5, including or excluding optional features. In this example, stabilizing the ball tracking comprises determining a game break and stabilizing the ball trajectory during game break.
Example 7 includes the method of any one of examples 1 to 6, including or excluding optional features. In this example, stabilizing the ball tacking the virtual camera is fixed to reset for a new play.
Example 8 includes the method of any one of examples 1 to 7, including or excluding optional features. In this example, the virtual camera focus area is calculated by determining a 3D region of interest (the temporal association is found by determining a bounding box including the first player in multiple frames of a same camera view of the captured field of view.)
Example 9 includes the method of any one of examples 1 to 8, including or excluding optional features. In this example, the method includes constructing a virtual camera within a three-dimensional volumetric representation of the captured field of view, and progressing through the three-dimensional volumetric representation according to the generated trajectory.
Example 10 is a system for trajectory generation based on player tracking. The system includes an optical trajectory receiver to receive an optical trajectory of a player and a ball, an affinity tracker to determine at least one affinity player for the currently tracked player, and a trajectory refiner to complete the optical trajectory for the currently tracked player based on an estimated trajectory from the affinity player. The system further includes a trajectory stabilizer to stabilize ball tracking during game break and a virtual camera calculator to calculate virtual camera movement.
Example 11 includes the system of example 10, including or excluding optional features. In this example, the affinity player is from the opposing team.
Example 12 includes the system of any one of examples 10 to 11, including or excluding optional features. In this example, the system includes selecting a plurality of affinity players, wherein a mean location of the affinity players is used to derive the location of the current player.
Example 13 includes the system of any one of examples 10 to 12, including or excluding optional features. In this example, the affinity player location is used to determine the missing trajectory of the current player until the current player is recovered in normal optical tracking.
Example 14 includes the system of any one of examples 10 to 13, including or excluding optional features. In this example, refining the trajectory comprises using a quadratic Bezier curve to smooth the trajectory.
Example 15 includes the system of any one of examples 10 to 14, including or excluding optional features. In this example, stabilizing the ball tracking comprises determining a game break and stabilizing the ball trajectory during game break.
Example 16 includes the system of any one of examples 10 to 15, including or excluding optional features. In this example, stabilizing the ball tacking the virtual camera is fixed to reset for a new play.
Example 17 includes the system of any one of examples 10 to 16, including or excluding optional features. In this example, the virtual camera focus area is calculated by determining a 3D region of interest (the temporal association is found by determining a bounding box including the first player in multiple frames of a same camera view of the captured field of view.)
Example 18 includes the system of any one of examples 10 to 17, including or excluding optional features. In this example, the system includes constructing a virtual camera within a three-dimensional volumetric representation of the captured field of view, and progressing through the three-dimensional volumetric representation according to the generated trajectory.
Example 19 is at least one non-transitory computer-readable medium. The computer-readable medium includes instructions that direct the processor to optically track a trajectory of a player and ball and determine at least one affinity player for the currently tracked player. The computer-readable medium also includes instructions that direct the processor to complete the optical trajectory for the currently tracked player based on an estimated trajectory from the affinity player, stabilize ball tracking during game break, and calculate virtual camera movement.
Example 20 includes the computer-readable medium of example 19, including or excluding optional features. In this example, the affinity player is from the opposing team.
Example 21 includes the computer-readable medium of any one of examples 19 to 20, including or excluding optional features. In this example, the computer-readable medium includes selecting a plurality of affinity players, wherein a mean location of the affinity players is used to derive the location of the current player.
Example 22 includes the computer-readable medium of any one of examples 19 to 21, including or excluding optional features. In this example, the affinity player location is used to determine the missing trajectory of the current player until the current player is recovered in normal optical tracking.
Example 23 includes the computer-readable medium of any one of examples 19 to 22, including or excluding optional features. In this example, refining the trajectory comprises using a quadratic Bezier curve to smooth the trajectory.
Example 24 includes the computer-readable medium of any one of examples 19 to 23, including or excluding optional features. In this example, stabilizing the ball tracking comprises determining a game break and stabilizing the ball trajectory during game break.
Example 25 includes the computer-readable medium of any one of examples 19 to 24, including or excluding optional features. In this example, stabilizing the ball tacking the virtual camera is fixed to reset for a new play.
Not all components, features, structures, characteristics, etc. described and illustrated herein need be included in a particular aspect or aspects. If the specification states a component, feature, structure, or characteristic “may”, “might”, “can” or “could” be included, for example, that particular component, feature, structure, or characteristic is not required to be included. If the specification or claim refers to “a” or “an” element, that does not mean there is only one of the element. If the specification or claims refer to “an additional” element, that does not preclude there being more than one of the additional element.
It is to be noted that, although some aspects have been described in reference to particular implementations, other implementations are possible according to some aspects. Additionally, the arrangement and/or order of circuit elements or other features illustrated in the drawings and/or described herein need not be arranged in the particular way illustrated and described. Many other arrangements are possible according to some aspects.
In each system shown in a figure, the elements in some cases may each have a same reference number or a different reference number to suggest that the elements represented could be different and/or similar. However, an element may be flexible enough to have different implementations and work with some or all of the systems shown or described herein. The various elements shown in the figures may be the same or different. Which one is referred to as a first element and which is called a second element is arbitrary.
It is to be understood that specifics in the aforementioned examples may be used anywhere in one or more aspects. For instance, all optional features of the computing device described above may also be implemented with respect to either of the methods or the computer-readable medium described herein. Furthermore, although flow diagrams and/or state diagrams may have been used herein to describe aspects, the techniques are not limited to those diagrams or to corresponding descriptions herein. For example, flow need not move through each illustrated box or state or in exactly the same order as illustrated and described herein.
The present techniques are not restricted to the particular details listed herein. Indeed, those skilled in the art having the benefit of this disclosure will appreciate that many other variations from the foregoing description and drawings may be made within the scope of the present techniques. Accordingly, it is the following claims including any amendments thereto that define the scope of the present techniques.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/CN2020/091490 | 5/21/2020 | WO |