MULTI-INPUT FREE VIEWPOINT VIDEO PROCESSING PIPELINE

BACKGROUND

A given video generally includes one or more scenes, where each scene in the video can be either relatively static (e.g., the objects in the scene do not substantially change or move over time) or dynamic (e.g., the objects in the scene substantially change and/or move over time). In a traditional video the viewpoint of each scene is chosen by the director when the video is recorded/captured and this viewpoint cannot be controlled or changed by an end user while they are viewing the video. In other words, in a traditional video the viewpoint of each scene is fixed and cannot be modified when the video is being rendered and displayed. In a free viewpoint video an end user can interactively control and change their viewpoint of each scene at will while they are viewing the video. In other words, in a free viewpoint video each end user can interactively request different synthetic (i.e., virtual) viewpoints of each scene on-the-fly when the video is being rendered and displayed.

SUMMARY

This Summary is provided to introduce a selection of concepts, in a simplified form, that are further described hereafter in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Free viewpoint video processing pipeline technique embodiments described herein are generally applicable to generating a free viewpoint video of a scene and presenting it to a user. In one exemplary embodiment an arrangement of sensors is used to capture the scene, where the arrangement includes a plurality of video capture devices and generates a plurality of streams of sensor data each of which represents the scene from a different geometric perspective. The streams of sensor data are then input and calibrated. A scene proxy is then generated from the calibrated streams of sensor data, where the scene proxy geometrically describes the scene as a function of time and includes one or more types of geometric proxy data which is matched to a first set of current pipeline conditions in order to maximize the photo-realism of the free viewpoint video that results from the scene proxy at each point in time. This scene proxy generation includes the following actions. The current pipeline conditions in the first set are periodically analyzed. The results of this periodic analysis are then used to select one or more different 3D (three-dimensional) reconstruction methods which are matched to these current pipeline conditions. The selected 3D reconstruction methods are then used to generate one or more different 3D reconstructions of the scene from the calibrated streams of sensor data. The 3D reconstructions of the scene and the results of the periodic analysis are then used to generate the scene proxy.

In another exemplary embodiment the scene proxy is input. A current synthetic viewpoint of the scene is then generated from the scene proxy, where this current synthetic viewpoint generation maximizes the photo-realism of the current synthetic viewpoint based upon a second set of current pipeline conditions. The current synthetic viewpoint of the scene is then displayed. The current synthetic viewpoint generation includes the following actions. The current pipeline conditions in the second set are periodically analyzed. The results of this periodic analysis are then used to select one or more different image-based rendering methods which are matched to these current pipeline conditions. The selected image-based rendering methods and the results of the period analysis are then used to generate the current synthetic viewpoint of the scene.

DESCRIPTION OF THE DRAWINGS

The specific features, aspects, and advantages of the free viewpoint video processing pipeline technique embodiments described herein will become better understood with regard to the following description, appended claims, and accompanying drawings where:

FIG. 1 is a diagram illustrating an exemplary embodiment, in simplified form, of the various stages in the free viewpoint video processing pipeline.

FIG. 2 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for generating a free viewpoint video of a scene.

FIG. 3 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for calibrating the streams of sensor data which are generated by an arrangement of sensors that is being used to capture the scene.

FIG. 4 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for generating a scene proxy from the calibrated streams of sensor data.

FIG. 5 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for presenting a free viewpoint video of a scene to an end user.

FIG. 6 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for generating a current synthetic viewpoint of the scene from the scene proxy.

FIG. 7 is a diagram illustrating an exemplary embodiment, in simplified form, of a continuum of the various exemplary image-based rendering methods which can be employed by the free viewpoint video processing pipeline technique embodiments described herein.

FIG. 8 is a diagram illustrating the various degrees of viewpoint navigation freedom that can be supported by the pipeline technique embodiments describe herein.

FIG. 9 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for using a point cloud 3D (three-dimensional) reconstruction method to generate 3D reconstructions of the scene from the calibrated streams of sensor data when these calibrated streams include a plurality of different streams of depth map images of the scene.

FIG. 10 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for using the point cloud 3D reconstruction method to generate 3D reconstructions of the scene from the calibrated streams of sensor data when these calibrated streams include a plurality of different streams of infrared images of the scene.

FIG. 11 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for using the point cloud 3D reconstruction method to generate 3D reconstructions of the scene from the calibrated streams of sensor data when these calibrated streams include a plurality of different streams of color images of the scene.

FIG. 12 is a diagram illustrating a simplified example of a general-purpose computer system on which various embodiments and elements of the free viewpoint video processing pipeline technique, as described herein, may be implemented.

DETAILED DESCRIPTION

In the following description of free viewpoint video (FVV) processing pipeline technique embodiments (hereafter simply referred to as pipeline technique embodiments) reference is made to the accompanying drawings which form a part hereof, and in which are shown, by way of illustration, specific embodiments in which the pipeline technique can be practiced. It is understood that other embodiments can be utilized and structural changes can be made without departing from the scope of the pipeline technique embodiments.

It is also noted that for the sake of clarity specific terminology will be resorted to in describing the pipeline technique embodiments described herein and it is not intended for these embodiments to be limited to the specific terms so chosen. Furthermore, it is to be understood that each specific term includes all its technical equivalents that operate in a broadly similar manner to achieve a similar purpose. Reference herein to “one embodiment”, or “another embodiment”, or an “exemplary embodiment”, or an “alternate embodiment”, or “one implementation”, or “another implementation”, or an “exemplary implementation”, or an “alternate implementation” means that a particular feature, a particular structure, or particular characteristics described in connection with the embodiment or implementation can be included in at least one embodiment of the pipeline technique. The appearances of the phrases “in one embodiment”, “in another embodiment”, “in an exemplary embodiment”, “in an alternate embodiment”, “in one implementation”, “in another implementation”, “in an exemplary implementation”, and “in an alternate implementation” in various places in the specification are not necessarily all referring to the same embodiment or implementation, nor are separate or alternative embodiments/implementations mutually exclusive of other embodiments/implementations. Yet furthermore, the order of process flow representing one or more embodiments or implementations of the pipeline technique does not inherently indicate any particular order not imply any limitations of the pipeline technique.

The term “sensor” is used herein to refer to any one of a variety of scene-sensing devices which can be used to generate a stream of sensor data that represents a given scene. Generally speaking and as will be described in more detail hereafter, the pipeline technique embodiments described herein employ a plurality of sensors which can be configured in various arrangements to capture a scene, thus allowing a plurality of streams of sensor data to be generated each of which represents the scene from a different geometric perspective. Each of the sensors can be any type of video capture device (VCD) (e.g., any type of video camera), or any type of audio capture device, or any combination thereof. Each of the sensors can also be either static (i.e., the sensor has a fixed spatial location and a fixed rotational orientation which do not change over time), or moving (i.e., the spatial location and/or rotational orientation of the sensor change over time). The pipeline technique embodiments can employ a combination of different types of sensors to capture a given scene.

The term “baseline” is used herein to refer to a ratio of the actual physical distance between a given pair of VCDs to the average of the actual physical distance from each VCD in the pair to the viewpoint of the scene. When this ratio is larger than a prescribed value the pair of VCDs is referred to herein as a wide baseline stereo pair of VCDs. When this ratio is smaller than the prescribed value the pair of VCDs is referred to herein as a narrow baseline stereo pair of VCDs.

1.0 Multi-Input Free Viewpoint Video (FVV) Processing Pipeline

The pipeline technique embodiments described herein generally involve an FVV processing pipeline for generating an FVV of a given scene and presenting the FVV to one or more end users. The pipeline technique embodiments are advantageous for various reasons including, but not limited to, the following. Generally speaking and as will be appreciated from the more detailed description that follows, the pipeline technique embodiments create a feeling of immersion for any end user who is viewing a rendering of the captured scene, thus enhancing their viewing experience. The pipeline technique embodiments also enable optimal viewpoint navigation for up to six degrees of viewpoint navigation freedom.

Furthermore, the pipeline technique embodiments described herein do not rely upon having to constrain the FVV processing pipeline in order to produce a desired visual result. In other words, the pipeline technique embodiments eliminate the need to place constraints on the FVV processing pipeline in order to generate various synthetic viewpoints of the scene which are photo-realistic and thus are free of discernible artifacts. More particularly and by way of example but not limitation, the pipeline technique embodiments eliminate having to constrain the arrangement of the sensors that are used to capture the scene. Accordingly, the pipeline technique embodiments are operational with any arrangement of sensors. The pipeline technique embodiments also eliminate having to constrain the complexity or composition of the scene that is being captured (e.g., neither the environment(s) in the scene, nor the types of objects in the scene, nor the number of people of in the scene, among other things has to be constrained). Accordingly, the pipeline technique embodiments are operational with any type of scene, including both relatively static and dynamic scenes. The pipeline technique embodiments also eliminate having to constrain the number or types of sensors that are used to capture the scene. Accordingly, the pipeline technique embodiments are operational with any number of sensors and all types of sensors. The pipeline technique embodiments also eliminate having to constrain the number of degrees of viewpoint navigation freedom that are provided during the rendering and end user viewing of the captured scene. Accordingly, the pipeline technique embodiments can produce visual results having as many as six degrees of viewpoint navigation freedom. The pipeline technique embodiments can also produce visual results having just one degree of viewpoint navigation freedom.

Yet, furthermore, the pipeline technique embodiments described herein do not rely upon having to use a specific 3D (three-dimensional) reconstruction method in the FVV processing pipeline to generate a 3D reconstruction of the captured scene. Accordingly, the pipeline technique embodiments support the use of any one or more 3D reconstruction methods in the pipeline and therefore provide the freedom to use whatever 3D reconstruction method(s) produces the desired visual result (e.g., the highest degree of photo-realism for the particular scene being captured and the desired number of degrees of viewpoint navigation freedom) based on the particular characteristics of the streams of sensor data that are generated by the sensors (e.g., based on factors such as the particular number and types of sensors that are used to capture the scene, and the particular arrangement of these sensors that is used), along with other current pipeline conditions.

Yet furthermore, the pipeline technique embodiments described herein do not rely upon having to use a specific image-based rendering method in the FVV processing pipeline during the rendering and end user viewing of the captured scene. Accordingly, the pipeline technique embodiments support the use of any one or more image-based rendering methods in the pipeline and therefore provide the freedom to use whatever image-based rendering method(s) produces the desired visual result based on the particular characteristics of the streams of sensor data that are generated by the sensors, along with other current pipeline conditions. By way of example but not limitation, in an exemplary situation where just two VCDs are used to capture a scene, an image-based rendering method that renders a lower fidelity 3D geometric proxy of the captured scene (herein simply referred to as a scene proxy) may produce an optimally photo-realistic visual result when the end user's viewpoint is close to the axis of one of the VCDs (such as with billboards). In another exemplary situation where a large number of VCDs configured in a circular arrangement are used to capture a scene, a conventional image warping/morphing image-based rendering method may produce an optimally photo-realistic visual result. In yet another exemplary situation where a large number of VCDs configured in either a 2D (two-dimensional) or 3D array arrangement are used to capture a scene, a conventional view interpolation image-based rendering method may produce an optimally photo-realistic visual result. In yet another exemplary situation where an even larger number of VCDs is used, a conventional lumigraph or light-field image-based rendering method may produce an optimally photo-realistic visual result.

It will thus be appreciated that the pipeline technique embodiments described herein result in a flexible, robust and commercially viable next generation FVV processing pipeline that meets the needs of today's various creative video producers and editors. By way of example but not limitation and as will be appreciated from the more detailed description that follows, the pipeline technique embodiments are applicable to various types of video-based media applications such as consumer entertainment (e.g., movies, television shows, and the like) and video-conferencing/telepresence, among others. The pipeline technique embodiments support a broad range of features that provide for the capture (i.e., recording), processing, storage, distribution, rendering, and end user viewing of any type of FVV that can be generated. Various implementations of the pipeline technique embodiments are possible, where each different implementation supports a different type of FVV. Exemplary types of supported FVV are described in more detail hereafter.

Additionally, the pipeline technique embodiments described herein allow any one or more parameters in the FVV processing pipeline to be freely modified without introducing artifacts into the FVV that is presented to the one or more end users. This allows the photo-realism of the FVV that is presented to each end user to be maximized (i.e., the artifacts are minimized) regardless of the characteristics of the various sensors that are used to capture the scene, and the characteristics of the various streams of sensor data that are generated by the sensors. Exemplary pipeline parameters which can be modified include, but are not limited to, the following. The number and types of sensors that are used to capture the scene can be modified. The arrangement of the sensors can also be modified. Which if any of the sensors is static and which is moving can also be modified. The complexity and composition of the scene can also be modified. Whether the scene is relatively static or dynamic can also be modified. The 3D reconstruction methods and image-based rendering methods that are used can also be modified. The number of degrees of viewpoint navigation freedom that are provided during the rendering and end user viewing of the captured scene can also be modified.

1.1 Pipeline Stages Overview

FIG. 1 illustrates an exemplary embodiment, in simplified form, of the various stages in the FVV processing pipeline. As exemplified in FIG. 1, the FVV processing pipeline 100 starts with a capture stage 102 during which, and generally speaking, the following actions take place. An arrangement of sensors is used to capture a given scene, where the arrangement includes a plurality of VCDs and generates a plurality of streams of sensor data each of which represents the scene from a different geometric perspective. These streams of sensor data are input from the sensors and calibrated in a manner which will be described in more detail hereafter. The calibrated streams of sensor data are then output to a processing stage 104. As described heretofore, the pipeline technique embodiments described herein support the use of various types, various numbers and various combinations of sensors which can be configured in various arrangements, including both 2D and 3D arrangements, where each of the sensors can be either static or moving.

Referring again to FIG. 1, the processing stage 104 of the FVV processing pipeline 100 inputs the calibrated streams of sensor data. A scene proxy which geometrically describes the captured scene as a function of time is then generated from the calibrated streams of sensor data. The scene proxy is then output to a storage and distribution stage 106. As will be described in more detail hereafter, the scene proxy includes one or more types of geometric proxy data which is matched to a first set of current conditions in the pipeline 100, where these conditions are generally associated with the specific implementation of the pipeline technique embodiments that is being used. As will also be described in more detail hereafter, the scene proxy is generated using one or more different 3D reconstruction methods which extract 3D geometric information from the calibrated streams of sensor data. The particular 3D reconstruction methods that are used and the particular manner in which the scene proxy is generated are determined based on a periodic analysis of the first set of current conditions. As will be appreciated from the more detailed description that follows, unlike the 3D reconstruction methods that are used in the arts of video gaming and special effects, the pipeline technique embodiment described herein use automated computer-vision-type 3D reconstruction methods which can operate without human input.

Referring again to FIG. 1, the storage and distribution stage 106 of the FVV processing pipeline 100 inputs the scene proxy and based on the specific implementation of the pipeline technique embodiments described herein and the related type of FVV that is being processed in the pipeline (i.e., the type of FVV that is being generated and presented to the one or more end users), can either store the scene proxy, or output the scene proxy and distribute it to one or more end users who either are, or will be, viewing the FVV, or both. In an exemplary embodiment of the pipeline technique described herein where the capture and processing stages 102 and 104 are implemented on one computing device (or a collection of computing devices) and each of the end users either is, or will be, viewing the FVV on another computing device, this distribution takes place by transmitting the scene proxy over whatever one or more data communication networks the end users' computing devices are connected to. It will be appreciated that this transmission is implemented in a manner that meets the needs of the specific implementation of the pipeline technique embodiments and the related type of FVV that is being processed in the pipeline 100.

Referring again to FIG. 1, a rendering stage 108 of the FVV processing pipeline 100 inputs the scene proxy which is output from the storage and distribution stage 106. A current synthetic viewpoint of the captured scene is then generated from the scene proxy, where this current synthetic viewpoint generation maximizes the photo-realism of the current synthetic viewpoint based upon a second set of current conditions in the pipeline 100. As will be described in more detail hereafter, this second set of current conditions may include viewpoint navigation information which is output by a user viewing experience stage 110, and may also include temporal navigation information which can also be output by the user viewing experience stage. The current synthetic viewpoint is then output to the user viewing experience stage 110. In addition to including this viewpoint and temporal navigation information, the second set of current conditions is also generally associated with the specific implementation of the pipeline technique embodiments that is being used. Generally speaking, the current synthetic viewpoint of the captured scene is generated using one or more different image-based rendering methods. The particular image-based rendering methods that are used and the particular manner in which the current synthetic viewpoint of the captured scene is generated are determined based on a periodic analysis of the second set of current conditions.

Referring again to FIG. 1, the user viewing experience stage 110 of the FVV processing pipeline 100 generally provides the one or more end users with the ability to view the current synthetic viewpoint of the captured scene on a display device and spatiotemporally navigate/control this viewpoint on-the-fly at will. In other words, the user viewing experience stage 110 provides each end user with the ability to continuously and interactively navigate/control their viewpoint of the scene that is being displayed on their display device. The user viewing experience stage 110 may also provide each end user with the ability to interactively temporally navigate/control the FVV at will. During the user viewing experience stage 110 the current synthetic viewpoint of the captured scene is input from the rendering stage 108 and displayed on each end user's display device. Each end user can interactively navigate their viewpoint of the scene and based on this viewpoint navigation the rendering stage 108 will modify the current synthetic viewpoint of the scene accordingly. In situations where the capture and processing stages 102 and 104 operate asynchronously from the rendering and user viewing experience stages 108 and 110 (such as in the asynchronous FVV implementation which is described in more detail hereafter), each end user can also interactively temporally control the FVV and based on this temporal control the rendering stage 108 will either temporally pause/stop, or rewind, or fast forward the FVV accordingly. As described heretofore, the pipeline technique embodiments described herein provide the freedom to use whatever image-based rendering method(s) produces the desired visual result (e.g., produces an optimally photo-realistic visual result) based on the current pipeline conditions.

1.2 Supported FVV Types

As noted heretofore, various implementations of the pipeline technique embodiments described herein are possible, where each different implementation supports a different type of FVV and a different user viewing experience. As will now be described in more detail, each of these different implementations differs in terms of the user viewing experience it provides, its latency characteristics (i.e., how rapidly the streams of sensor data have to be processed through the FVV processing pipeline), its storage characteristics, its transmission and related bandwidth characteristics, and the types of computing device hardware it necessitates.

Referring again to FIG. 1, one implementation of the pipeline technique embodiments described herein supports asynchronous (i.e., non-live) FVV, which corresponds to a situation where the streams of sensor data that are generated by the sensors are pre-captured 102, then post-processed 104, and the resulting scene proxy is then stored and can be transmitted in a one-to-many manner (i.e., broadcast) to one or more end users 106. As such, there is effectively an unlimited amount of time available for the processing stage 104. This allows an FVV producer to optionally manually “touch-up” the streams of sensor data that are input during the capture stage 102, and also optionally manually remove any 3D reconstruction artifacts that are introduced in the processing stage 104. This particular implementation is referred to hereafter as the asynchronous FVV implementation. Exemplary types of video-based media that work well in the asynchronous FVV implementation include movies, documentaries, sitcoms and other types of television shows, music videos, digital memories, and the like. Another exemplary type of video-based media that works well in the asynchronous FVV implementation is the use of special effects technology where synthetic objects are realistically modeled, lit, shaded and added to a pre-captured scene. It will be appreciated that the streams of sensor data generated by the sensors are captured 102 and processed 104, and the resulting scene proxy is stored and distributed 106 in a manner that supports the particular number of degrees of viewpoint navigation freedom that are being provided in the user viewing experience stage 110, and also supports the particular geometric proxy that is processed by the image-based rendering method in the rendering stage 108.

Referring again to FIG. 1, another implementation of the pipeline technique embodiments described herein supports unidirectional (i.e., one-way) live FVV, which corresponds to a situation where the streams of sensor data that are being generated by the sensors are concurrently captured 102 and processed 104, and the resulting scene proxy is stored and transmitted in a one-to-many manner on-the-fly (i.e., live) to one or more end users 106. As such, each end user can view 110 the scene live (i.e., each use can view the scene at substantially the same time it is being captured 102). This particular implementation is referred to hereafter as the unidirectional live FVV implementation. Exemplary types of video-based media that work well in the unidirectional live FVV implementation include sporting events, news programs, live concerts, and the like. As with the asynchronous FVV implementation, the streams of sensor data generated by the sensors are captured 102 and processed 104, and the resulting scene proxy is stored and distributed 106 in a manner that supports the particular number of degrees of viewpoint navigation freedom that are being provided in the user viewing experience stage 110, and also supports the particular geometric proxy that is processed by the image-based rendering method in the rendering stage 108.

Referring again to FIG. 1, yet another implementation of the pipeline technique embodiments described herein supports bidirectional (i.e., two-way) live FVV such as that which is associated with various video-conferencing/telepresence applications. This particular implementation is referred to hereafter as the bidirectional live FVV implementation. This bidirectional live FVV implementation is generally the same as the unidirectional live FVV implementation with the following exception. In the bidirectional live FVV implementation a computing device at each physical location that is participating in a given video-conferencing/telepresence session is able to concurrently capture 102 streams of sensor data that are being generated by sensors which are capturing a local scene and process 104 these locally captured streams of sensor data, store and transmit the resulting local scene proxy in a one-to-many manner on the fly to the other physical locations that are participating in the session 106, receive a remote scene proxy from each of the remote physical locations that are participating in the session 106, and render 108 each received proxy. As with the asynchronous FVV implementation, the streams of sensor data generated by the sensors are captured 102 and processed 104, and the resulting local scene proxy is stored and distributed 106 in a manner that supports the particular number of degrees of viewpoint navigation freedom that are being provided in the user viewing experience stage 110, and also supports the particular geometric proxy that is processed by the image-based rendering method in the rendering stage 108.

Referring again to FIG. 1, it will be appreciated that in the unidirectional and bidirectional live FVV implementations of the pipeline technique embodiments described herein in order for an end user to be able to view the scene live, the capture, processing, storage and distribution, rendering, and user viewing experience stages 102/104/106/108/110 have to be completed within a very short period of time.

1.3 FVV Generation

This section provides a more detailed description of the capture and processing stages of the FVV processing pipeline. The pipeline technique embodiments described herein generally employ a plurality of sensors which are configured in a prescribed arrangement to capture a given scene. The pipeline technique embodiments are operable with any type of sensor, any number (two or greater) of sensors, any arrangement of sensors (where this arrangement can include a plurality of different geometries and different geometric relationships between the sensors), and any combination of different types of sensors. The pipeline technique embodiments are also operable with both static and moving sensors. A given sensor can be any type of VCD (examples of which are described in more detail hereafter), or any type of audio capture device (such as a microphone, or the like), or any combination thereof. Each VCD generates a stream of video data which includes a stream of images (also known as and referred to herein as frames) of the scene from the specific geometric perspective of the VCD. Similarly, each audio capture device generates a stream of audio data representing the audio emanating from the scene from the specific geometric perspective of the audio capture device.

Exemplary types of VCDs that can be employed include, but are not limited to, the following. A given VCD can be a conventional visible light video camera which generates a stream of video data that includes a stream of color images of the scene. A given VCD can also be a conventional light-field camera (also known as a “plenoptic camera”) which generates a stream of video data that includes a stream of color light-field images of the scene. A given VCD can also be a conventional infrared structured-light projector combined with a conventional infrared video camera that is matched to the projector, where this projector/camera combination generates a stream of video data that includes a stream of infrared images of the scene. This projector/camera combination is also known as a “structured-light 3D scanner”. A given VCD can also be a conventional monochromatic video camera which generates a stream of video data that includes a stream of monochrome images of the scene. A given VCD can also be a conventional time-of-flight camera which generates a stream of video data that includes both a stream of depth map images of the scene and a stream of color images of the scene. For simplicity sake, the term “color camera” is sometimes used herein to refer to any type of VCD that generates color images of the scene.

It will be appreciated that variability in factors such as the composition and complexity of a given scene, and each end user's viewpoint navigation during the user viewing experience stage of the FVV processing pipeline, among other factors, can impact the determination of how many sensors to use to capture the scene, the particular type(s) of sensors to use, and the particular arrangement of the sensors to use. The pipeline technique embodiments described herein generally employ a minimum of one VCD which generates color image data for the scene, along with one or more other VCDs that can be used in combination to generate 3D geometry data for the scene. In situations where an outdoor scene is being captured or the sensors are located far from the scene, it is advantageous to capture the scene using both a wide baseline stereo pair of color cameras and a narrow baseline stereo pair of color cameras. In situations where an indoor scene is being captured, it is advantageous to capture the scene using a narrow baseline stereo pair of VCDs both of which generate video data that includes a stream of infrared images of the scene in order to eliminate the dependency on scene lighting variables.

Generally speaking, it is advantageous to increase the number of sensors being used as the complexity of the scene increases. In other words, as the scene becomes more complex (e.g., as additional people are added to the scene), the use of additional VCDs serves to reduce the number of occluded areas within the scene. It may also be advantageous to capture the entire scene using a given arrangement of static VCDs, and at the same time also capture a specific higher complexity region of the scene using one or more additional moving VCDs. In a situation where a large number of VCDs is used to capture a complex scene, different combinations of the VCDs can be used during the processing stage of the FVV processing pipeline (e.g., a situation where a specific VCD is part of both a narrow baseline stereo pair and a different wide baseline stereo pair involving a third VCD).

FIG. 2 illustrates an exemplary embodiment, in simplified form, of a process for generating an FVV of a scene. As exemplified in FIG. 2, the process starts in block 200 with using an arrangement of sensors to capture the scene, where the arrangement includes a plurality of VCDs and generates a plurality of streams of sensor data each of which represents the scene from a different geometric perspective. The streams of sensor data are then input (block 202) and calibrated (block 204). It will be appreciated that a given stream of sensor data will include video data whenever the sensor that generated the stream is a VCD. A given stream of sensor data will include audio data whenever the sensor that generated the stream is an audio capture device. A given stream of sensor data will include both video and audio data whenever the sensor that generated the stream is a combined video and audio capture device. It will also be appreciated that various methods can be used to calibrate the streams of sensor data. One such method will now be described in more detail.

FIG. 3 illustrates an exemplary embodiment, in simplified form, of a process for calibrating the streams of sensor data which are generated by the arrangement of sensors. As exemplified in FIG. 3, the process starts in block 300 with determining the number of VCDs in the arrangement of sensors that is being used to capture the scene. Intrinsic characteristics of each of the VCDs are then determined (block 302). Exemplary intrinsic characteristics which can be determined for a given one of the VCDs include one or more of the VCD type, or the VCD's frame rate, or the VCD's shutter speed, or the VCD's mosaic pattern, or the VCD's white balance, or the bit depth and pixel resolution of the images that are generated by the VCD, or the focal length of the VCD's lens, or the principal point of the VCD's lens, or the VCD's skew coefficient, or the distortions of the VCD's lens, or the VCD's field of view, among others. It will be appreciated that knowing such intrinsic characteristics for each of the VCDs allows the FVV processing pipeline to understand the governing physics and optics of each of the VCDs. Extrinsic characteristics of each of the VCDs at each point in time during the capture of the scene are also determined (block 304). Exemplary extrinsic characteristics which can be determined for a given VCD include one or more of the VCD's current rotational orientation (i.e., the direction that the VCD is currently pointing), or the VCD's current spatial location (i.e., the VCD's current location within the arrangement), or whether the VCD is static or moving, or the current geometric relationship between the VCD and each of the other VCDs in the arrangement (i.e., the VCD's current position relative to each of the other VCDs in the arrangement), or the current position of the VCD relative to the scene, or whether or not the VCD is genlocked (i.e., temporally synchronized) with the other VCDs in the arrangement, among others. The determination of the intrinsic and extrinsic characteristics of each of the VCDs can be made using various conventional methods, examples of which will be described in more detail hereafter. The knowledge of the number of VCDs in the arrangement, and the intrinsic and extrinsic characteristics of each of the VCDs, is then used to temporally and spatially calibrate the streams of sensor data (block 306).

As is appreciated in the art of video recording, the intrinsic and extrinsic characteristics of each of the VCDs in the arrangement are commonly determined by performing one or more calibration procedures which calibrate the VCDs, where these procedures are specific to the particular types of VCDs that are being used to capture the scene, and the particular number and arrangement of the VCDs. In the unidirectional and bidirectional live FVV implementations of the pipeline technique embodiments described herein the calibration procedures are performed and the streams of sensor data which are generated thereby are input before the scene capture. In the asynchronous FVV implementation of the pipeline technique embodiments the calibration procedures can be performed and the streams of sensor data which are generated thereby can be input either before or after the scene capture. Exemplary calibration procedures will now be described.

In a situation where the VCDs that are being used to capture the scene are genlocked and include a combination of color cameras, VCDs which generate a stream of infrared images of the scene, and one or more time-of-flight cameras, and this combination of cameras is arranged in a static array, the cameras in the array can be calibrated and the intrinsic and extrinsic characteristics of each of the cameras can be determined in the following manner. A stream of calibration data can be input from each of the cameras in the array while a common physical feature (such as a ball, or the like) is internally illuminated with an incandescent light (which is visible to all of the cameras) and moved throughout the scene. These streams of calibration data can then be analyzed using conventional methods to determine both an intrinsic and extrinsic calibration matrix for each of the cameras.

In another situation where the VCDs that are being used to capture the scene include a plurality of color cameras which are arranged in a static array, the cameras in the array can be calibrated and the intrinsic and extrinsic characteristics of each of the cameras can be determined in the following manner. A stream of calibration data can be input from each camera in the array while it is moved around the scene but in close proximity to its static location (thus allowing each camera in the array to view overlapping parts of the static background of the scene). After the scene is captured by the static array of color cameras and the streams of sensor data generated thereby are input, the streams of sensor data can be analyzed using conventional methods to identify features in the scene, and these features can then be used to calibrate the cameras in the array and determine the intrinsic and extrinsic characteristics of each of the cameras by employing a conventional structure-from-motion method.

In yet another situation where one or more of the VCDs that are being used to capture the scene are moving VCDs (such as when the spatial location of a given VCD changes over time, or when controls on a given VCD are used to optically zoom in on the scene while it is being captured (which is commonly done during the recording of sporting events, among other things)), each of these moving VCDs can be calibrated and its intrinsic and extrinsic characteristics can be determined at each point in time during the scene capture by using a conventional background model to register and calibrate relevant individual images that were generated by the VCD. In yet another situation where the VCDs that are being used to capture the scene include a combination of static and moving VCDs, the VCDs can be calibrated and the intrinsic and extrinsic characteristics of each of the VCDs can be determined by employing conventional multistep calibration procedures.

In yet another situation where there is no temporal synchronization between the VCDs that are being used to capture the scene and the arrangement of the sensors can randomly change over time (such as when a plurality of mobile devices are held up by different users and the VCDs on these devices are used to capture the scene), the pipeline technique embodiments described herein will both spatially and temporally calibrate the streams of sensor data generated by the VCDs at all points in time during the scene capture before the streams are processed in the processing stage. In an exemplary embodiment of the pipeline technique this spatial and temporal calibration can be performed as follows. After the scene is captured and the streams of sensor data representing the scene are input, the streams of sensor data can be analyzed using conventional methods to separate the static and moving elements of the scene. The static elements of the scene can then be used to generate a background model. Additionally, the moving elements of the scene can be used to generate a global timeline that encompasses all of the VCDs, and each image in each stream of sensor data is assigned a relative time. The intrinsic characteristics of each of the VCDs can be determined by using conventional methods to analyze each of the streams of sensor data.

In an embodiment of the pipeline technique described herein where the capture stage of the FVV processing pipeline is directly connected to the VCDs that are being used to capture the scene, the intrinsic characteristics of each of the VCDs can also be determined by reading appropriate hardware parameters directly from each of the VCDs. In another embodiment of the pipeline technique where the capture stage is not directly connected to the VCDs but rather the streams of sensor data are pre-recorded and then imported into the capture stage, the number of VCDs and various intrinsic properties of each of the VCDs can be determined by analyzing the streams of sensor data using conventional methods.

Referring again to FIG. 2, after the streams of sensor data have been calibrated (block 204), a scene proxy is generated from the calibrated streams of sensor data (block 206). As will be described in more detail hereafter, the scene proxy geometrically describes the scene as a function of time and includes one or more types of geometric proxy data which is matched to a set of current pipeline conditions in order to maximize the photo-realism of the FVV that results from the scene proxy at each point in time. These conditions can be in any one or more of the aforementioned stages of the FVV processing pipeline. After the scene proxy has been generated (block 206), it can be stored (block 208). In a situation where a given end user either is, or will be, viewing the FVV on another computing device which is connected to a data communication network, the scene proxy can also be distributed to the end user by transmitting it over the network to the other computing device (block 210).

FIG. 4 illustrates an exemplary embodiment, in simplified form, of a process for generating the scene proxy from the calibrated streams of senor data. As exemplified in FIG. 4, the process starts in block 400 with periodically analyzing the set of current pipeline conditions. The set of current pipeline conditions can include one or more conditions in the capture stage of the FVV processing pipeline such as the number of VCDs that either is being, or was, used to capture the scene, or one or more of the intrinsic characteristics of each of the VCDs (e.g., the VCD type, among others), or one or more of the extrinsic characteristics of each of the VCDs (e.g., the current position of the VCD relative to the scene, and whether the VCD is static or moving, among others), or the like. The set of current pipeline conditions can also include one or more conditions in the processing stage of the pipeline such as whether the scene proxy is being generated on-the-fly during the rendering and end user viewing of the captured scene, or the scene proxy is being generated asynchronously from the rendering and end user viewing of the scene (i.e., the particular type of FVV that is being generated and the related speed at which the scene proxy has to be generated), or the like.

The set of current pipeline conditions can also include one or more conditions in the storage and distribution stage of the FVV processing pipeline such as the amount of storage space that is currently available to store the scene proxy, or the network transmission bandwidth that is currently available, or the like. The set of current pipeline conditions can also include one or more conditions in the user viewing experience stage of the pipeline such as the type of display device upon which the FVV either is, or will be, viewed, or the particular characteristics of the display device (e.g., one or more of its aspect ratio, or its pixel resolution, or its form factor, among others), or the level of data fidelity that is desired in the free viewpoint video, or the like.

Referring again to FIG. 4, after the analysis of the set of current pipeline conditions has been completed (block 400), the results of this analysis are then used to select one or more different 3D reconstruction methods which are matched to the current pipeline conditions (block 402). The selected 3D reconstruction methods are then used to generate one or more different 3D reconstructions of the scene from the calibrated streams of sensor data (block 404). The 3D reconstructions of the scene and the results of the period analysis are then used to generate the scene proxy (block 406). The actions of blocks 400, 402, 404 and 406 are repeated for the duration of the scene (block 408, No). As such, it will be appreciated that the 3D reconstruction methods which are selected, the types of 3D reconstructions of the scene which are generated, and thus the types of geometric proxy data in the scene proxy can change over time based upon changes in the current pipeline conditions. It will also be appreciated that the current pipeline conditions can be analyzed using different periodicities. By way of example but not limitation, in one embodiment of the pipeline technique described herein the current pipeline conditions can be analyzed on a frame-by-frame basis (i.e., for each image in the streams of sensor data). In another embodiment of the pipeline technique the current pipeline conditions can be analyzed using a periodicity of a prescribed number of sequential frames, where this number is greater than one. In yet another embodiment of the pipeline technique the current pipeline conditions can be analyzed using a periodicity of a prescribed period of time.

It will thus be appreciated that the pipeline technique embodiments described herein can use a wide variety of 3D reconstruction methods in various combinations, where the particular types of 3D reconstruction methods that are being used depend upon various current conditions in the FVV processing pipeline. Accordingly and as will be described in more detail hereafter, the scene proxy will include one or more types of geometric proxy data examples of which include, but are not limited to, the following. The scene proxy can include a stream of depth map images of the scene. The scene proxy can also include a stream of calibrated point cloud reconstructions of the scene. As is appreciated in the art of 3D reconstruction, these point cloud reconstructions are a low order geometric representation of the scene. The scene proxy can also include one or more high order geometric models, where these models can include one or more of planes, or billboards, or existing (i.e., previously created) generic object models (e.g., human body models, or human face models, or clothing models, or furniture models, or the like) which can be either modified, or animated, or both, among others. Such high order geometric models can be advantageously used to fill in occlusions that may exist in the captured scene. The scene proxy can also include other high fidelity proxies such as a stream of mesh models of the scene and a corresponding stream of texture maps which define texture data for each of the mesh models, among others. It will further be appreciated that since the particular 3D reconstruction methods that are used and the related manner in which the scene proxy is generated are based upon a period analysis (i.e., monitoring) of the various current conditions in the FVV processing pipeline, the 3D reconstruction methods that are used and the resulting types of data in the scene proxy can change over time based on changes in the pipeline conditions.

Generally speaking, for the unidirectional and bidirectional live FVV implementations of the pipeline technique embodiments described herein, due to the fact that the capture, processing, storage and distribution, rendering, and user viewing experience stages of the FVV processing pipeline have to be completed within a very short period of time, the types of 3D reconstruction methods that can be used in these implementations are limited to high speed 3D reconstruction methods. By way of example but not limitation, in the unidirectional and bidirectional live FVV implementations of the pipeline technique embodiments the scene proxy that is generated will include a stream of calibrated point cloud reconstructions of the scene, and may also include one or more high order geometric models which can be either modified, or animated, or both. It will be appreciated that 3D reconstruction methods which can be implemented in hardware are also favored in the unidirectional and bidirectional live FVV implementations of the pipeline technique embodiments. The use of VCDs which generate infrared images of the scene is also favored in the unidirectional and bidirectional live FVV implementations of the pipeline technique embodiments.

For the asynchronous FVV implementation of the pipeline technique embodiments described herein, due to the fact that the capture and processing stages of the FVV processing pipeline operate asynchronously from the rendering and user viewing experience stages (and as such, there is effectively an unlimited amount of time available for the processing stage), more rigorous (and thus slower) 3D reconstruction methods can be used in this implementation. By way of example but not limitation, in the asynchronous FVV implementation of the pipeline technique embodiments the scene proxy that is generated can include both a stream of calibrated point cloud reconstructions of the scene, as well as one or more higher fidelity geometric proxies of the scene (such as when the calibrated point cloud reconstructions of the scene are used to generate a stream of mesh models of the scene, among other possibilities). The asynchronous FVV implementation of the pipeline technique embodiments also allows a plurality of 3D reconstruction steps to be used in sequence when generating the scene proxy. By way of example but not limitation, consider a situation where a stream of calibrated point cloud reconstructions of the scene has been generated, but there are some noisy or error prone stereo matches present in these reconstructions that extend beyond a human silhouette boundary in the scene. It will be appreciated that these noisy or error prone stereo matches can lead to the wrong texture data appearing in the mesh models of the scene, thus resulting in artifacts in the rendered scene. These artifacts can be eliminated by running a segmentation process to separate the foreground from the background, and then points outside of the human silhouette can be rejected as outliers.

FIG. 9 illustrates an exemplary embodiment, in simplified form, of a process for using a point cloud 3D reconstruction method to generate one or more different 3D reconstructions of the scene from the calibrated streams of sensor data when these calibrated streams include a plurality of different streams of depth map images of the scene (hereafter simply referred to as different depth map image streams). As exemplified in FIG. 9, whenever the calibrated streams of sensor data include a plurality of different depth map image streams (block 900, Yes), these different depth map image streams are merged into a stream of calibrated point cloud reconstructions of the scene (block 902). It is noted that these point cloud reconstructions are unordered and as such, overlaps may exist therein. Depending on the current pipeline conditions, the stream of calibrated point cloud reconstructions of the scene can then optionally be used to generate one or more different high fidelity geometric proxies of the scene (block 904). By way of example but not limitation, the stream of calibrated point cloud reconstructions of the scene can be used to generate a stream of mesh models of the scene, where this mesh model generation can be performed using conventional methods such as Poisson, among others.

FIG. 10 illustrates an exemplary embodiment, in simplified form, of a process for using the point cloud 3D reconstruction method to generate one or more different 3D reconstructions of the scene from the calibrated streams of sensor data when these calibrated streams include a plurality of different streams of infrared images of the scene (hereafter simply referred to as different infrared image streams). As exemplified in FIG. 10, whenever the calibrated streams of sensor data include a plurality of different infrared image streams (block 1000, Yes), the following actions occur. Any narrow baseline stereo pairs of VCDs that exist in the arrangement of sensors and generate pairs of infrared image streams are identified (block 1002). A first set of different depth map image streams is then created from the pairs of infrared image streams generated by the identified narrow baseline stereo pairs of VCDs (block 1004). Any wide baseline stereo pairs of VCDs that exist in the arrangement of sensors and generate pairs of infrared image streams are then identified (block 1006). A second set of different depth map image streams is then created from the pairs of infrared image streams generated by the identified wide baseline stereo pairs of VCDs (block 1008). The different depth map image streams in the first set and the second set are then merged into a stream of calibrated point cloud reconstructions of the scene (block 1010). It is noted that these point cloud reconstructions are unordered and as such, overlaps may exist therein. Depending on the current pipeline conditions, the stream of calibrated point cloud reconstructions of the scene can then optionally be used to generate one or more different high fidelity geometric proxies of the scene (block 1012). By way of example but not limitation and as just described, the stream of calibrated point cloud reconstructions of the scene can be used to generate a stream of mesh models of the scene.

FIG. 11 illustrates an exemplary embodiment, in simplified form, of a process for using the point cloud 3D reconstruction method to generate one or more different 3D reconstructions of the scene from the calibrated streams of sensor data when these calibrated streams include a plurality of different streams of color images of the scene (hereafter simply referred to as different color image streams). As exemplified in FIG. 11, whenever the calibrated streams of sensor data include a plurality of different color image streams (block 1100, Yes), the following actions occur. Any narrow baseline stereo pairs of VCDs that exist in the arrangement of sensors and generate pairs of color image streams are identified (block 1102). A first set of different depth map image streams is then created from the pairs of color image streams generated by the identified narrow baseline stereo pairs of VCDs (block 1104). Any wide baseline stereo pairs of VCDs that exist in the arrangement of sensors and generate pairs of color image streams are then identified (block 1106). A second set of different depth map image streams is then created from the pairs of color image streams generated by the identified wide baseline stereo pairs of VCDs (block 1108). The different depth map image streams in the first set and the second set are then merged into a stream of calibrated point cloud reconstructions of the scene (block 1110). It is noted that these point cloud reconstructions are unordered and as such, overlaps may exist therein. Depending on the current pipeline conditions, the stream of calibrated point cloud reconstructions of the scene can then optionally be used to generate one or more different high fidelity geometric proxies of the scene (block 1112). By way of example but not limitation and as just described, the stream of calibrated point cloud reconstructions of the scene can be used to generate a stream of mesh models of the scene.

It will be appreciated that depending on the particular arrangement of sensors that is used to capture the scene, a given VCD can be in a plurality of narrow baseline stereo pairs of VCDs, and can also be in a plurality of wide baseline stereo pairs of VCDs. This serves to maximize the number of different depth map image streams that are created, which in turn serves to maximize the precision of the scene proxy.

1.3.1 FVV Generation Examples

Referring again to FIG. 1, this section provides an overview description, in simplified form, of several additional implementations of the capture and processing stages 102 and 104 of the FVV processing pipeline 100. It will be appreciated that the implementations described in this section are merely exemplary. Many other implementations of the capture and processing stages 102 and 104 are also possible which use other types of sensor arrangements and generate other types of scene proxies.

In one implementation of the capture and processing stages of the FVV processing pipeline a circular arrangement of eight genlocked VCDs is used to capture a scene which includes one or more human beings, where each of the VCDs includes a combination of one infrared structured-light projector, two infrared video cameras, and one color camera. Accordingly, the VCDs each generate a different stream of video data which includes both a stereo pair of infrared image streams and a color image stream. As described heretofore, the pair of infrared image streams and the color image stream generated by each VCD are first used to generate different depth map image streams. The different depth map image streams are then merged into a stream of calibrated point cloud reconstructions of the scene. These point cloud reconstructions are then used to generate a stream of mesh models of the scene. A conventional view-dependent texture mapping method which accurately represents specular textures such as skin is then used to extract texture data from the color image stream generated by each VCD and map this texture data to the stream of mesh models of the scene.

In another implementation of the capture and processing stages of the FVV processing pipeline four genlocked visible light video cameras are used to capture a scene which includes one or more human beings, where the cameras are evenly placed around the scene. Accordingly, the cameras each generate a different stream of video data which includes a color image stream. An existing 3D geometric model of a human body can be used in the scene proxy as follows. Conventional methods can be used to kinematically articulate the model over time in order to fit (i.e., match) the model to the streams of video data generated by the cameras. The kinematically articulated model can then be colored as follows. A conventional view-dependent texture mapping method can be used to extract texture data from the color image stream generated by each camera and map this texture data to the kinematically articulated model.

In yet another implementation of the capture and processing stages of the FVV processing pipeline three unsynchronized visible light video cameras are used to capture a soccer game, where each of the cameras is moving and is located far from the game (e.g., rather than the spatial location of each of the cameras being fixed to a specified arrangement, each of the cameras is hand held by a different user who is capturing the game while they freely move about). Accordingly, the cameras each generate a different stream of video data which includes a stream of color images of the game. Articulated billboards can be used to represent the moving players in the scene proxy of the game as follows. For each stream of video data, conventional methods can be used to generate a segmentation mask for each body part of each player in the stream. Conventional methods can then be used to generate an articulated billboard model of each of the moving players in the game from the appropriate segmentation masks. The articulated billboard model can then be colored as just described.

1.4 FVV Presentation to End User

This section provides a more detailed description of the rendering and user viewing experience stages of the FVV processing pipeline.

FIG. 5 illustrates an exemplary embodiment, in simplified form, of a process for presenting an FVV of a scene to an end user. As exemplified in FIG. 5, the process starts in block 500 with inputting a scene proxy which geometrically describes the scene as a function of time. A current synthetic viewpoint of the scene is then generated from the scene proxy, where this current synthetic viewpoint generation maximizes the photo-realism of the current synthetic viewpoint based upon a set of current pipeline conditions (block 502). These conditions can be in any one or more of the aforementioned stages of the FVV processing pipeline. The current synthetic viewpoint of the scene is then displayed on a display device (block 504) so that it can be viewed and navigated by the end user. As described herein, a prescribed number of degrees of viewpoint navigation freedom are provided to the end user, where this number is greater than or equal to one and less than or equal to six.

FIG. 6 illustrates an exemplary embodiment, in simplified form, of a process for generating the current synthetic viewpoint of the scene from the scene proxy. As exemplified in FIG. 6, the process starts in block 600 with periodically analyzing the set of current pipeline conditions. The set of current pipeline conditions can include one or more conditions in the capture stage of the FVV processing pipeline such as the number of VCDs that either is being, or was, used to capture the scene, or one or more of the intrinsic characteristics of each of the VCDs (e.g., the VCD type, among others), or one or more of the extrinsic characteristics of each of the VCDs (e.g., the positioning of the VCD relative to the scene, and whether the VCD is static or moving, among others), or the complexity and composition of the scene, or whether the scene is relatively static or dynamic, or the like. The set of current pipeline conditions can also include one or more conditions in the processing stage of the pipeline such as the particular 3D reconstruction methods that are being (or were) used to generate the scene proxy, or the types of geometric proxy data that are in the scene proxy, or the level of data fidelity that is desired in the free viewpoint video, or the like.

The set of current pipeline conditions can also include one or more conditions in the rendering and user viewing experience stages of the FVV processing pipeline such as the graphics processing capabilities/features that are available in the hardware of the computing device which is being used by a given end user to generate the current synthetic viewpoint of the scene, or the type of display device upon which the current synthetic viewpoint of the scene is being displayed, or the particular characteristics of the display device (described heretofore), or the number of degrees of viewpoint navigation freedom that are being provided to the end user, or the view frustum of the current synthetic viewpoint, or whether or not this computing device includes a natural user interface (and if so, the particular natural user interface modalities that are anticipated to be used by the end user), or the like. The set of current pipeline conditions can also include information which is generated by the end user in the user viewing experience stage that specifies desired changes to (i.e., controls) the current synthetic viewpoint of the scene. Such information can include one or more of viewpoint navigation information which is being output by this stage based upon the FVV navigation that is being performed by the end user, or temporal navigation information which may also be output by this stage based upon this FVV navigation. The set of current pipeline conditions can also include the type of FVV that is being presented to the end user.

Referring again to FIG. 6, after the analysis of the set of current pipeline conditions has been completed (block 600), the results of this analysis are then used to select one or more different image-based rendering methods which are matched to the current pipeline conditions (block 602). The selected image-based rendering methods and the results of the period analysis are then used to generate the current synthetic viewpoint of the scene (block 604). The actions of blocks 600, 602 and 604 are repeated for the duration of the scene (block 606, No). As such, it will be appreciated that the one or more image-based rendering methods which are used and the current synthetic viewpoint of the scene that is generated can change over time based upon changes in the current pipeline conditions. As described heretofore, the current pipeline conditions can be analyzed using different periodicities.

It will thus be appreciated that the pipeline technique embodiments described herein can use a wide variety of image-based rendering methods in various combinations, where the particular types of image-based rendering methods that are being used depend upon various current conditions in the FVV processing pipeline. Unlike the rendering methods that are employed in conventional 3D computer graphics applications where the 3D geometry of the scene that is being rendered is known (i.e., the geometric primitives for the scene are known), the image-based rendering methods that are employed by the pipeline technique embodiments described herein can render novel views (i.e., synthetic viewpoints) of the scene directly from a collection of images in the scene proxy without having to know the scene geometry. An overview exemplary image-based rendering methods which can be employed by the pipeline technique embodiments is provided hereafter.

The pipeline technique embodiments described herein support using any type of display device to view the FVV including, but not limited to, the very small form factor display devices used on conventional smart phones and other types of mobile devices, the small form factor display devices used on conventional tablet computers and netbook computers, the display devices used on conventional laptop computers and personal computers, conventional televisions and 3D televisions, conventional autostereoscopic 3D display devices, conventional head-mounted transparent display devices, and conventional wearable heads-up display devices such as those that are used in virtual reality applications. In a situation where the end user is using an autostereoscopic 3D display device to view the FVV, then the rendering stage of the FVV processing pipeline will simultaneously generate both left and right current synthetic viewpoints of the scene at an appropriate aspect ratio and resolution in order to create a stereoscopic effect for the end user. In another situation where the end user is using a conventional television to view the FVV, then the rendering stage will generate just a single current synthetic viewpoint.

The pipeline technique embodiments described herein also support using any type of user interface modality to control the current viewpoint while viewing the FVV including, but not limited to, conventional keyboards, conventional pointing devices (such as a mouse, or a graphics tablet, or the like), and conventional natural user interface modalities (such as voice, or a touch-sensitive display screen, or the head tracking functionality that is integrated into wearable heads-up display devices, or a motion and location sensing device (such as the Microsoft Kinect™ (a trademark of Microsoft Corporation), among others), or the like). It will be appreciated that if the end user either is, or will be, using one or more natural user interface modalities while they are viewing the FVV, this can influence the spatiotemporal navigation capabilities that are provided to the end user. In other words, the FVV processing pipeline can process the streams of sensor data differently in order to enable different end user viewing experiences based on the particular type(s) of user interface modality that is anticipated to be used by the end user. By way of example but not limitation, in a situation where a given end user is using the wearable heads-up display device to view and navigate the FVV, then all six degrees of viewpoint navigation freedom could be provided to the end user. In the bidirectional live FVV implementation of the pipeline technique embodiments, if the end user at each physical location that is participating in a given video-conferencing/telepresence session is using the wearable heads-up display device to view and navigate the FVV, then parallax functionality can be implemented in order to provide each end user with an optimally realistic viewing experience when they control/change their viewpoint of the FVV using head movements; the pipeline can also provide for corrected conversational geometry between two end users, thus providing the appearance that both end users are looking directly at each other. In another situation where a given end user is using the motion and location sensing device navigate the FVV, then the rendering stage can optimize the current synthetic viewpoint that is being displayed based on the end user's current spatial location in front of their display device. In this way, the end user's current spatial location can be mapped to the 3D geometry within the FVV.

FIG. 8 illustrates the various degrees of viewpoint navigation freedom that can be supported by the pipeline technique embodiments describe herein. As described heretofore, the pipeline technique embodiments generally support spatiotemporal (i.e., space-time) navigation of the FVV. More particularly, the asynchronous FVV, unidirectional live FVV, and bidirectional live FVV implementations described herein can each support spatial viewpoint navigation of the FVV having as many as six degrees of freedom, which can be appropriate when the end user is viewing and navigating an FVV that includes high fidelity geometric information. As exemplified in FIG. 8, these six degrees of freedom include viewpoint navigation along the x axis, viewpoint navigation rotationally about the x axis (θx), viewpoint navigation along the y axis, viewpoint navigation rotationally about the y axis (θy), viewpoint navigation along the z axis, and viewpoint navigation rotationally about the z axis (θz). The asynchronous FVV, unidirectional live FVV, and bidirectional live FVV implementations can also each support spatial viewpoint navigation of the FVV having just one degree of viewpoint navigation freedom, which can be appropriate when the viewpoint navigation of the FVV is constrained to a straight line that connects the sensors. The asynchronous FVV implementation can also support temporal navigation of the FVV.

In some embodiments of the pipeline technique described herein, such as the asynchronous FVV implementation described herein, a producer or editor of the FVV may want to specify the particular types of viewpoint navigation that are possible at different times during the FVV. By way of example but not limitation, in one scene a movie director may want to confine the end user's viewpoint navigation to a limited area of the scene or a specific axis, but in another scene the director may want to allow the end user to freely navigate their viewpoint throughout the entire area of the scene.

1.4.1 Image-Based Rendering Methods

As described heretofore, the current synthetic viewpoint of the scene is generated using one or more image-based rendering methods which are selected based upon a periodic analysis of the aforementioned set of current pipeline conditions. Accordingly, the particular image-based rendering methods that are used can change over time based upon changes in the current pipeline conditions. It will thus be appreciated that in one situation where the scene has a low degree of complexity and the arrangement of sensors which either is being, or was, used to capture the scene are located close to the scene, just a single image-based rendering method may be used to generate the current synthetic viewpoint of the scene. In another situation where the scene has a high degree of complexity and the arrangement of sensors which either is being, or was, used to capture the scene are located far from the scene, a plurality of image-based rendering methods may be used to generate the current synthetic viewpoint of the scene depending on the location of the current viewpoint relative to the scene and the particular types of geometric proxy data that are in the scene proxy.

FIG. 7 illustrates an exemplary embodiment, in simplified form, of a continuum of the various exemplary image-based rendering methods which can be employed by the pipeline technique embodiments described herein. As exemplified in FIG. 7, for didactic purposes these various image-based rendering methods can be classified into three categories according to the amount and type of scene geometry information that is included in the scene proxy and thus is available to be used in the rendering stage, namely rendering with without scene geometry 706 (i.e., the scene geometry is unknown), rendering with implicit scene geometry 704 (i.e., correspondence), and rendering with explicit scene geometry 702 (which can be either approximate or accurate). These categories 702, 704 and 706 are to be viewed as a continuum 700 rather than strict and discrete categories since it will be appreciated that certain of the image-based rendering methods defy strict categorization.

As also exemplified in FIG. 7, a trade-off exists between the amount and type of scene geometry information that is available to be used in the rendering stage, and the number of images that are needed to be available in the scene proxy in order to generate synthetic viewpoints of the scene which are photo-realistic. Generally speaking, the higher the VCD density in the arrangement of sensors that is being used to capture the scene (i.e., the larger the number of VCDs that is used in the arrangement), the larger the number of images that is available in the scene proxy, and thus the less scene geometry information that is needed to be available in the scene proxy in order to generate synthetic viewpoints of the scene which are photo-realistic. However, it is noted that having less scene geometry information in the scene proxy will generally decrease the end user's options for navigating the current synthetic viewpoint of the scene (i.e., the synthetic viewpoints will generally be limited to positions between VCDs or near VCDs. Correspondingly, the lower the VCD density (i.e., the smaller the number of VCDs that is used in the arrangement), the smaller the number of images that is available in the scene proxy, and thus the more scene geometry information that is needed to be available in the scene proxy in order to generate synthetic viewpoints of the scene which are photo-realistic. However, it is noted that having more scene geometry information in the scene proxy will generally increase the end user's options for navigating the current synthetic viewpoint of the scene (i.e., the synthetic viewpoints can be navigated to positions which are far away from the real VCD viewpoints).

On the left side 706 of the continuum 700 exemplified in FIG. 7 the scene proxy includes a large number of images but does not include any scene geometry or correspondence information. In this situation the current synthetic viewpoint of the scene can be generated by using a conventional light-field method, or a conventional lumigraph method, or a conventional concentric mosaics method, among others, to process the scene proxy. As is appreciated in the art of image-based rendering, each of these methods relies on the characterization of the conventional plenoptic function, and constructs a continuous representation of the plenoptic function from the images in the scene proxy. The light-field method is generally applicable when the images of the scene are uniformly captured. The light-field method generates new images of the scene by appropriately filtering and interpolating the images in the scene proxy. The lumigraph method is similar to the light-field method except that the lumigraph method is generally applicable when the images of the scene are not uniformly captured. The lumigraph method enhances the rendering performance by applying approximated geometry to compensate for this non-uniform capture. Unlike the light-field and lumigraph methods which are applicable when the arrangement of VCDs is a 2D grid, the concentric mosaics method is applicable when the arrangement of VCDs is circular. Conventional image mosaicing methods can also be used to construct a complete plenoptic function at a fixed viewpoint from an incomplete set of images of the scene.

In the middle 704 of the continuum 700 exemplified in FIG. 7 the scene proxy does not include explicit scene geometry information, but rather it includes implicit scene geometry information in the form of feature (e.g., point) correspondences between images, where these correspondences can be computed using conventional computer vision methods. In this situation the current synthetic viewpoint of the scene can be generated by using various conventional transfer methods (such as a conventional view interpolation method, or a conventional view morphing method, among others) to process the scene proxy. As is appreciated in the art of image-based rendering, such transfer methods are characterized by the use of a relatively small number of images with the application of geometric constraints (which are either recovered or known a priori) to project image pixels appropriately at a given synthetic viewpoint. These geometric constraints can be in the form of known depth values at each pixel, or epipolar constraints between stereo pairs of images, or trifocal/tri-linear tensors that link correspondences between triplets of images. The view interpolation method generates synthetic viewpoints of the scene by interpolating optical flow between corresponding points. The view morphing method generates synthetic viewpoints that reside on a line which links the optical centers of two different VCDs based on point correspondences.

On the right side 702 of the continuum 700 exemplified in FIG. 7 the scene proxy includes explicit and accurate scene geometry information and a small number of images, where this geometry information can be in form of either depth along known lines-of-sight, or 3D coordinates, among other things. In this situation the current synthetic viewpoint of the scene can be generated by using conventional 3D warping methods, or a conventional layered depth images method, or a conventional layered depth images tree method, or a conventional view-dependent texture mapping method, or a conventional view-dependent geometry method, among others, to process the scene proxy. As is appreciated in the art of image-based rendering, the 3D warping methods, or the layered depth images method, or the layered depth images tree method can be used when the scene proxy includes both depth map images and color (or monochrome) images of the scene. When the scene proxy includes depth information for all the points in an image, the 3D warping methods can be used to render the image from any nearby point of view by projecting the pixels of the image to their proper 3D locations and then re-projecting them onto a new picture. The rendering speed of such 3D warping methods can be increased by using conventional relief texture methods which factor the warping process into a relatively simple pre-warping operation and a conventional texture mapping operation (which may be performed by conventional graphics processing hardware). It is noted that the 3D warping methods can be applied to both traditional perspective images as well as multi-perspective images. The view-dependent geometry method was first used in the context of 3D cartoons and trades off geometry and images, and may be used to represent the current synthetic viewpoint of the scene more compactly. A conventional texture-mapped models method can also be used to generate the current synthetic viewpoint of the scene.

2.0 Additional Embodiments

While the pipeline technique has been described by specific reference to embodiments thereof, it is understood that variations and modifications thereof can be made without departing from the true spirit and scope of the pipeline technique. By way of example but not limitation, rather than the capture and processing stages of the FVV processing pipeline being implemented on one computing device (or a collection of computing devices), and the rendering and user viewing experience stages of the pipeline being implemented on another computing device(s) which is being used by an end user(s) to view the FVV, an alternate embodiment of the pipeline technique described herein is possible where the capture, processing, rendering and user viewing experience stages of the pipeline are implemented on a single computing device (i.e., the FVV can be rendered and viewed on the same computing device that is used to input/calibrate the streams of sensor data and generate the scene proxy). Furthermore, in addition to the sensors being any type of VCD, or any type of audio capture device, or any combination thereof as described heretofore, the sensors can also be a wearable body-suit that provides a stream of depth data.

It is also noted that any or all of the aforementioned embodiments can be used in any combination desired to form additional hybrid embodiments. Although the pipeline technique embodiments have been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described heretofore. Rather, the specific features and acts described heretofore are disclosed as example forms of implementing the claims.

3.0 Computing Environment

The pipeline technique embodiments described herein are operational within numerous types of general purpose or special purpose computing system environments or configurations. FIG. 12 illustrates a simplified example of a general-purpose computer system on which various embodiments and elements of the pipeline technique, as described herein, may be implemented. It is noted that any boxes that are represented by broken or dashed lines in FIG. 12 represent alternate embodiments of the simplified computing device, and that any or all of these alternate embodiments, as described below, may be used in combination with other alternate embodiments that are described throughout this document.

For example, FIG. 12 shows a general system diagram showing a simplified computing device 1200. Such computing devices can be typically be found in devices having at least some minimum computational capability, including, but not limited to, personal computers (PCs), server computers, handheld computing devices, laptop or mobile computers, communications devices such as cell phones and personal digital assistants (PDAs), multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and audio or video media players.

To allow a device to implement the pipeline technique embodiments described herein, the device should have a sufficient computational capability and system memory to enable basic computational operations. In particular, as illustrated by FIG. 12, the computational capability is generally illustrated by one or more processing unit(s) 1210, and may also include one or more graphics processing units (GPUs) 1215, either or both in communication with system memory 1220. Note that the processing unit(s) 1210 may be specialized microprocessors (such as a digital signal processor (DSP), a very long instruction word (VLIW) processor, a field-programmable gate array (FPGA), or other micro-controller) or can be conventional central processing units (CPUs) having one or more processing cores including, but not limited to, specialized GPU-based cores in a multi-core CPU.

In addition, the simplified computing device 1200 of FIG. 12 may also include other components, such as, for example, a communications interface 1230. The simplified computing device 1200 of FIG. 12 may also include one or more conventional computer input devices 1240 (e.g., pointing devices, keyboards, audio (e.g., voice) input/capture devices, video input/capture devices, haptic input devices, devices for receiving wired or wireless data transmissions, and the like). The simplified computing device 1200 of FIG. 12 may also include other optional components, such as, for example, one or more conventional computer output devices 1250 (e.g., display device(s) 1255, audio output devices, video output devices, devices for transmitting wired or wireless data transmissions, and the like). Exemplary types of input devices (herein also referred to as user interface modalities) and display devices that are operable with the pipeline technique embodiments described herein have been described heretofore. Note that typical communications interfaces 1230, additional types of input and output devices 1240 and 1250, and storage devices 1260 for general-purpose computers are well known to those skilled in the art, and will not be described in detail herein.

The simplified computing device 1200 of FIG. 12 may also include a variety of computer readable media. Computer readable media can be any available media that can be accessed by the computer 1200 via storage devices 1260, and includes both volatile and nonvolatile media that is either removable 1270 and/or non-removable 1280, for storage of information such as computer-readable or computer-executable instructions, data structures, program modules, or other data. By way of example but not limitation, computer readable media may include computer storage media and communication media. Computer storage media includes, but is not limited to, computer or machine readable media or storage devices such as digital versatile disks (DVDs), compact discs (CDs), floppy disks, tape drives, hard drives, optical drives, solid state memory devices, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other device which can be used to store the desired information and which can be accessed by one or more computing devices.

Storage of information such as computer-readable or computer-executable instructions, data structures, program modules, and the like, can also be accomplished by using any of a variety of the aforementioned communication media to encode one or more modulated data signals or carrier waves, or other transport mechanisms or communications protocols, and includes any wired or wireless information delivery mechanism. Note that the terms “modulated data signal” or “carrier wave” generally refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. For example, communication media includes wired media such as a wired network or direct-wired connection carrying one or more modulated data signals, and wireless media such as acoustic, radio frequency (RF), infrared, laser, and other wireless media for transmitting and/or receiving one or more modulated data signals or carrier waves. Combinations of the any of the above should also be included within the scope of communication media.

Furthermore, software, programs, and/or computer program products embodying the some or all of the various embodiments of the pipeline technique described herein, or portions thereof, may be stored, received, transmitted, or read from any desired combination of computer or machine readable media or storage devices and communication media in the form of computer executable instructions or other data structures.

Finally, the pipeline technique embodiments described herein may be further described in the general context of computer-executable instructions, such as program modules, being executed by a computing device. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The pipeline technique embodiments may also be practiced in distributed computing environments where tasks are performed by one or more remote processing devices, or within a cloud of one or more devices, that are linked through one or more communications networks. In a distributed computing environment, program modules may be located in both local and remote computer storage media including media storage devices. Additionally, the aforementioned instructions may be implemented, in part or in whole, as hardware logic circuits, which may or may not include a processor.

MULTI-INPUT FREE VIEWPOINT VIDEO PROCESSING PIPELINE

Information

Publication Number

Date Filed

Date Published

Inventors

Original Assignees

CPC

US Classifications

International Classifications

Abstract

Description

Claims

CROSS REFERENCE TO RELATED APPLICATION

Provisional Applications (1)