BACKGROUND
A given video generally includes one or more scenes, where each scene in the video can be either relatively static (e.g., the objects in the scene do not substantially change or move over time) or dynamic (e.g., the objects in the scene substantially change and/or move over time). As is appreciated in the art of computer graphics, polygonal modeling is commonly used to represent three-dimensional objects in a scene by approximating the surface of each object using polygons. A polygonal model of a given scene includes a collection of vertices. Two neighboring vertices that are connected by a straight line form an edge in the polygonal model. Three neighboring and non-co-linear vertices that are interconnected by three edges form a triangle in the polygonal model. Four neighboring and non-co-linear vertices that are interconnected by four edges form a quadrilateral in the polygonal model. Triangles and quadrilaterals are the most common types of polygons used in polygonal modeling, although other types of polygons may also be used depending on the capabilities of the renderer that is being used to render the polygonal model. A group of polygons that are interconnected by shared vertices are referred to as a mesh and as such, a polygonal model of a scene is also known as a mesh model. Each of the polygons that makes up a mesh is referred to as a face in the polygonal/mesh model. Accordingly, a polygonal/mesh model of a scene includes a collection of vertices, edges and polygonal (i.e., polygon-based) faces that represents/approximates the shape of each object in the scene.
SUMMARY
This Summary is provided to introduce a selection of concepts, in a simplified form, that are further described hereafter in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
Video generation technique embodiments described herein are generally applicable to generating a video of a scene and presenting it to a user. In an exemplary embodiment of this generation, one or more streams of sensor data that represent the scene are received. Scene proxies are then generated from the streams of sensor data. This scene proxies generation includes the following actions. A stream of mesh models of the scene is generated from the streams of sensor data. Then, for each of the mesh models, the following actions take place. The mesh model is sliced using a series of planes that are parallel to each other, where each of the planes in the series defines one or more contours each of which defines a specific region on the plane where the mesh model intersects the plane. A texture map is then generated for the mesh model which defines texture data corresponding to each of the contours that is defined by the series of planes.
In an exemplary embodiment of the just-mentioned presentation, the scene proxies are received. The scene proxies include a stream of mathematical equations describing contours that are defined by a series of planes that are parallel to each other, and a stream of texture maps defining texture data corresponding to each of the contours that is defined by the series of planes. Images of the scene are then rendered from the scene proxies and displayed. This image rendering includes the following actions. The series of planes is constructed using data specifying the spatial orientation and geometry of the series of planes. The contours that are defined by the series of planes are then constructed using the stream of mathematical equations. A series of point locations is then constructed along each of the contours that is defined by the series of planes, where this construction is performed in a prescribed order across each of the planes in the series of planes, and this construction is also performed starting from a prescribed zero position on each of these contours. The point locations that are defined by the series of planes are then tessellated, where this tessellation generates a stream of polygonal models, and each polygonal model includes a collection of polygonal faces that are formed by neighboring point locations on corresponding contours on neighboring planes in the series of planes. The stream of texture maps is then sampled to identify the texture data that corresponds to each of the polygonal faces in the stream of polygonal models. This identified texture data is then used to add texture to each of the polygonal faces in the stream of polygonal models.
DESCRIPTION OF THE DRAWINGS
The specific features, aspects, and advantages of the video generation technique embodiments described herein will become better understood with regard to the following description, appended claims, and accompanying drawings where:
FIG. 1 is a diagram illustrating an exemplary embodiment, in simplified form, of a video processing pipeline for implementing the video generation technique embodiments described herein.
FIG. 2 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for generating a video of a scene.
FIG. 3 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for generating scene proxies that geometrically describe the scene as a function of time.
FIGS. 4A-4C are diagrams illustrating an exemplary embodiment, in simplified form, of a series of three planes that are parallel to each other and are used to slice a mesh model of a human.
FIG. 5 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for generating a texture map for a given mesh model in a stream of mesh models of the scene.
FIG. 6 is a diagram illustrating an exemplary embodiment, in simplified form, of a contour and three-dimensional that are identified along the contour.
FIG. 7 is a diagram illustrating an exemplary embodiment, in simplified form, of a contour point map.
FIG. 8 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for storing the scene proxies.
FIG. 9 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for distributing the scene proxies to an end user who either is, or will be, viewing the video.
FIG. 10 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for presenting the video of the scene to an end user.
FIG. 11 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for rendering images of the scene from the scene proxies.
FIG. 12 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for assigning texels in a given scanline of the texture map to contours that are defined by a given plane in a series of planes that are used to slice the mesh model.
FIG. 13 is a diagram illustrating a simplified example of a general-purpose computer system on which various embodiments and elements of the video generation technique, as described herein, may be implemented.
FIG. 14 is a diagram illustrating a simplified example of the assignment of the texels in two neighboring scanlines of an exemplary texture map to exemplary contours that are defined by two neighboring planes that correspond to the two neighboring scanlines.
FIG. 15 is a flow diagram illustrating an exemplary embodiment, in simplified form, of a process for sampling a stream of texture maps to identify texture data that corresponds to polygonal faces in a stream of polygonal models that is generated from a stream of mathematical equations describing contours that are defined by the series of planes.
DETAILED DESCRIPTION
In the following description of video generation technique embodiments reference is made to the accompanying drawings which form a part hereof, and in which are shown, by way of illustration, specific embodiments in which the video generation technique can be practiced. It is understood that other embodiments can be utilized and structural changes can be made without departing from the scope of the video generation technique embodiments.
It is also noted that for the sake of clarity specific terminology will be resorted to in describing the video generation technique embodiments described herein and it is not intended for these embodiments to be limited to the specific terms so chosen. Furthermore, it is to be understood that each specific term includes all its technical equivalents that operate in a broadly similar manner to achieve a similar purpose. Reference herein to “one embodiment”, or “another embodiment”, or an “exemplary embodiment”, or an “alternate embodiment”, or “one implementation”, or “another implementation”, or an “exemplary implementation”, or an “alternate implementation” means that a particular feature, a particular structure, or particular characteristics described in connection with the embodiment or implementation can be included in at least one embodiment of the video generation technique. The appearances of the phrases “in one embodiment”, “in another embodiment”, “in an exemplary embodiment”, “in an alternate embodiment”, “in one implementation”, “in another implementation”, “in an exemplary implementation”, and “in an alternate implementation” in various places in the specification are not necessarily all referring to the same embodiment or implementation, nor are separate or alternative embodiments/implementations mutually exclusive of other embodiments/implementations. Yet furthermore, the order of process flow representing one or more embodiments or implementations of the video generation technique does not inherently indicate any particular order not imply any limitations of the video generation technique.
As is known in the arts of human anatomy and medical research, the Visible Human Project was conceived in the late 1980s and run by the U.S. National Library of Medicine. The goal of the Project was to create a detailed human anatomy data set using cross-sectional photographs of the human body in order to facilitate anatomy visualization applications. A convicted murderer named Joseph Paul Jernigan was executed in 1993 and his cadaver was used to provide male data for the Project. More particularly, Jernigan's cadaver was encased and frozen in a gelatin and water mixture in order to stabilize the cadaver for cutting thereof. Jernigan's cadaver was then segmented (i.e., “cut”) along its axial plane (also known as its transverse plane) at one millimeter intervals from the top of the cadaver's scalp to the soles of its feet, resulting in 1,871 “slices”. Each of these slices was photographed and digitized at a high resolution. The term “convict hull” is accordingly used herein to refer to a given plane that is used to “slice” a given mesh model of a given scene.
The term “sensor” is used herein to refer to any one of a variety of scene-sensing devices which can be used to generate a stream of sensor data that represents a given scene. Generally speaking and as will be described in more detail hereafter, the video generation technique embodiments described herein employ one or more sensors which can be configured in various arrangements to capture a scene, thus allowing one or more streams of sensor data to be generated each of which represents the scene from a different geometric perspective. Each of the sensors can be any type of video capture device (e.g., any type of video camera), or any type of audio capture device (such as a microphone, or the like), or any combination thereof. Each of the sensors can also be either static (i.e., the sensor has a fixed spatial location and a fixed rotational orientation which do not change over time), or moving (i.e., the spatial location and/or rotational orientation of the sensor change over time). The video generation technique embodiments described herein can employ a combination of different types of sensors to capture a given scene.
1.0 Video Generation Using Convict Hulls
The video generation technique embodiments described herein generally involve using convict hulls to generate a video of a given scene and then present the video to one or more end users. The video generation technique embodiments support the generation, storage, distribution, and end user presentation of any type of video. By way of example but not limitation, one embodiment of the video generation technique supports various types of traditional, single viewpoint video in which the viewpoint of the scene is chosen by the director when the video is recorded/captured and this viewpoint cannot be controlled or changed by an end user while they are viewing the video. In other words, in a single viewpoint video the viewpoint of the scene is fixed and cannot be modified when the video is being rendered and displayed to an end user. Another embodiment of the video generation technique supports various types of free viewpoint video in which the viewpoint of the scene can be interactively controlled and changed by an end user at will while they are viewing the video. In other words, in a free viewpoint video an end user can interactively generate synthetic (i.e., virtual) viewpoints of the scene on-the-fly when the video is being rendered and displayed. Exemplary types of single viewpoint and free viewpoint video that are supported by the video generation technique embodiments are described in more detail hereafter.
The video generation technique embodiments described herein are advantageous for various reasons including, but not limited to, the following. Generally speaking and as will be appreciated from the more detailed description that follows, the video generation technique embodiments serve to minimize the size of (i.e., minimize the amount of data in) the video that is generated, stored and distributed. Based on this video size/data minimization, it will also be appreciated that the video generation technique embodiments minimize the cost and maximize the performance associated with storing and transmitting the video in a client-server framework where the video is generated and stored on a server computing device, and then transmitted from the server over a data communication network to one or more client computing devices upon which the video is rendered and then viewed and navigated by the one or more end users. Furthermore, the video generation technique embodiments maximize the photo-realism of the video that is generated when it is rendered and then viewed and navigated by the end users. As such, the video generation technique embodiments provide the end users with photo-realistic video that is free of discernible artifacts, thus creating a feeling of immersion for the end users and enhancing their viewing experience.
Additionally, the video generation technique embodiments described herein eliminate having to constrain the complexity or composition of the scene that is being captured (e.g., neither the environment(s) in the scene, nor the types of objects in the scene, nor the number of people of in the scene, among other things has to be constrained). Accordingly, the video generation technique embodiments are operational with any type of scene, including both relatively static and dynamic scenes. The video generation technique embodiments also provide a flexible, robust and commercially viable method for generating a video, and then presenting it to one or more end users, that meets the needs of today's various creative video producers and editors. By way of example but not limitation and as will be appreciated from the more detailed description that follows, the video generation technique embodiments are applicable to various types of video-based media applications such as consumer entertainment (e.g., movies, television shows, and the like) and video-conferencing/telepresence, among others.
1.1 Video Processing Pipeline
FIG. 1 illustrates an exemplary embodiment, in simplified form, of a video processing pipeline for implementing the video generation technique embodiments described herein. As noted heretofore, the video generation technique embodiments support the generation, storage, distribution, and end user presentation of any type of video including, but not limited to, various types of single viewpoint video and various types of free viewpoint video. As exemplified in FIG. 1, the video processing pipeline 100 start with a generation stage 102 during which, and generally speaking, scene proxies of a given scene are generated. The generation stage 102 includes a capture sub-stage 104 and a processing sub-stage 106 whose operation will now be described in more detail.
Referring again to FIG. 1, the capture sub-stage 104 of the video processing pipeline 100 generally captures the scene and generates one or more streams of sensor data that represent the scene. More particularly, in an embodiment of the video generation technique described herein where a single viewpoint video is being generated, stored, distributed and presented to one or more end users (hereafter simply referred to as the single viewpoint embodiment of the video generation technique), during the capture sub-stage 104 a single sensor is used to capture the scene, where the single sensor includes a video capture device and generates a single stream of sensor data which represents the scene from a single geometric perspective. The stream of sensor data is received from the sensor and then output to the processing sub-stage 106. In another embodiment of the video generation technique described herein where a free viewpoint video is being generated, stored, distributed and presented to one or more end users (hereafter simply referred to as the free viewpoint embodiment of the video generation technique), during the capture sub-stage 104 an arrangement of sensors is used to capture the scene, where the arrangement includes a plurality of video capture devices and generates a plurality of streams of sensor data each of which represents the scene from a different geometric perspective. These streams of sensor data are received from the sensors and calibrated, and then output to the processing sub-stage 106.
Referring again to FIG. 1, the processing sub-stage 106 of the video processing pipeline 100 receives the stream(s) of sensor data from the capture sub-stage 104, and then generates scene proxies that geometrically describe the captured scene as a function of time from the stream(s) of sensor data. The scene proxies are then output to a storage and distribution stage 108.
Referring again to FIG. 1, the storage and distribution stage 108 of the video processing pipeline 100 receives the scene proxies from the processing sub-stage 106, stores the scene proxies, outputs the scene proxies and distributes them to one or more end users who either are, or will be, viewing the video, or both. In an exemplary embodiment of the video generation technique described herein where the generation stage 102 is implemented on one computing device (or a collection of computing devices) and an end user presentation stage 110 of the pipeline 100 is implemented on one or more end user computing devices, this distribution takes place by transmitting the scene proxies over whatever one or more data communication networks the end user computing devices are connected to. It will be appreciated that this transmission is implemented in a manner that meets the needs of the specific implementation of the video generation technique embodiments and the related type of video that is being processed in the pipeline 100.
Referring again to FIG. 1 and generally speaking, the end user presentation stage 110 of the video processing pipeline 100 receives the scene proxies that are output from the storage and distribution stage 108, and then presents each of the end users with a rendering of the scene proxies. The end user presentation stage 110 includes a rendering sub-stage 112 and a user viewing experience sub-stage 114 whose operation will now be described in more detail.
Referring again to FIG. 1, in the single viewpoint embodiment of the video generation technique described herein the rendering sub-stage 112 of the video processing pipeline 100 receives the scene proxies that are output from the storage and distribution stage 108, and then renders images of the captured scene from the scene proxies, where these images have a fixed viewpoint that cannot be modified by an end user. The fixed viewpoint images of the captured scene are then output to the user viewing experience sub-stage 114 of the pipeline 100. The user viewing experience sub-stage 114 receives the fixed viewpoint images of the captured scene from the rendering sub-stage 112, and then displays these images on a display device for viewing by a given end user. In situations where the generation stage 102 operates asynchronously from the end user presentation stage 110 (such as in the asynchronous single viewpoint video implementation that is described in more detail hereafter), the user viewing experience sub-stage 114 can provide the end user with the ability to interactively temporally navigate/control the single viewpoint video at will, and based on this temporal navigation/control the rendering sub-stage 112 will either temporally pause/stop, or rewind, or fast forward the single viewpoint video accordingly.
Referring again to FIG. 1, in the free viewpoint embodiment of the video generation technique described herein the rendering sub-stage 112 receives the scene proxies that are output from the storage and distribution stage 108, and then renders images of the captured scene from the scene proxies, where these images have a synthetic viewpoint that can be modified by an end user. The synthetic viewpoint images of the captured scene are then output to the user viewing experience sub-stage 114. The user viewing experience sub-stage 114 receives the synthetic viewpoint images of the captured scene from the rendering sub-stage 112, and then displays these images on a display device for viewing by a given end user. Generally speaking, the user viewing experience sub-stage 114 can provide the end user with the ability to spatio/temporally navigate/control the synthetic viewpoint images of the captured scene on-the-fly at will. In other words, the user viewing experience sub-stage 114 can provide the end user with the ability to continuously and interactively navigate/control their viewpoint of the images of the scene that are being displayed on the display device, and based on this viewpoint navigation the rendering sub-stage 112 will modify the images of the scene accordingly. In situations where the generation stage 102 operates asynchronously from the end user presentation stage 110 (such as in the asynchronous free viewpoint video implementation that is described in more detail hereafter), the user viewing experience sub-stage 114 can also provide the end user with the ability to interactively temporally navigate/control the free viewpoint video at will, and based on this temporal navigation/control the rendering sub-stage 112 will either temporally pause/stop, or rewind, or fast forward the free viewpoint video accordingly.
1.2 Video Generation
This section provides a more detailed description of the generation stage of the video processing pipeline. As described heretofore, the video generation technique embodiments described herein generally employ one or more sensors which can be configured in various arrangements to capture a scene. These one or more sensors generate one or more streams of sensor data each of which represents the scene from a different geometric perspective.
FIG. 2 illustrates an exemplary embodiment, in simplified form, of a process for generating a video of a scene. As exemplified in FIG. 2, the process starts in block 200 with receiving the one or more streams of sensor data that represent the scene. Scene proxies are then generated from these streams of sensor data (block 202), where the scene proxies geometrically describe the scene as a function of time. The scene proxies can then be stored (block 204). In a situation where a given end user either is, or will be, viewing the video on another computing device which is connected to a data communication network, the scene proxies can also be distributed to the end user (block 206).
FIG. 3 illustrates an exemplary embodiment, in simplified form, of a process for generating the scene proxies from the one or more streams of sensor data that represent the scene. As exemplified in FIG. 3, the process starts in block 300 with generating a stream of mesh models of the scene from the streams of sensor data, where each of the mesh models includes a collection of vertices and a collection of polygonal faces that are formed by the vertices. The following actions then take place for each of the mesh models (block 302). The mesh model is sliced using a series of planes that are parallel to each other, where each of the planes in the series of planes defines one or more contours each of which defines a specific region on the plane where the mesh model intersects the plane (block 304). A texture map for the mesh model is then generated, where the texture map defines texture data corresponding to each of the contours that is defined by the series of planes (block 306). It will be appreciated that this texture map can be generated using various methods, one example of which is described in more detail hereafter.
FIGS. 4A-4C illustrate an exemplary embodiment, in simplified form, of a series of three planes that are parallel to each other and are used to slice a mesh model of a human. As exemplified in FIGS. 4A-4C, the series of planes 400/402/404 has a horizontal spatial orientation. As exemplified in FIG. 4A, the top-most plane 400 in the series of planes slices the mesh model 406 substantially along the shoulder region of the human and defines a single contour 408 along this region. As exemplified in FIG. 4B, the middle plane 402 in the series of planes slices the mesh model 406 substantially along the right bicep region, chest region and left bicep region of the human and defines two different contours 410 and 412. More particularly, the middle plane 402 defines a contour 410 along the right bicep region of the human. The middle plane 402 defines another contour 412 along the chest and left bicep regions of the human. As exemplified in FIG. 4C, the bottom-most plane 404 in the series of planes slices the mesh model 406 substantially along the right elbow region, upper stomach region, right hand region, left forearm region, and left hand region of the human and defines three different contours 414/416/418. More particularly, the bottom-most plane 404 defines a contour 414 along the right elbow region of the human. The bottom-most plane 404 defines another contour 416 along the upper stomach, left hand and left forearm regions of the human. The bottom-most plane 404 defines yet another contour 418 along the right hand region of the human.
FIG. 5 illustrates an exemplary embodiment, in simplified form, of a process for generating a texture map for a given mesh model which defines texture data corresponding to each of the contours that is defined by the series of planes. This particular embodiment assumes that the texture map includes a series of scanlines each of which corresponds to a different one of the planes in the series of planes, where each of the scanlines includes a series of texels, and the total number of texels in each of the scanlines is the same. It will be appreciated that any number of texels per scanline can be used. In an exemplary embodiment of the video generation technique described herein the total number of texels in each of the scanlines is 1024. As exemplified in FIG. 5, the process starts in block 500 with the following actions taking place for each of the planes in the series of planes. Each of the contours that is defined by the plane is analyzed in a prescribed order across the plane to identify a series of point locations along the contour (block 502). This analysis of each of the contours that is defined by the plane is performed starting from a prescribed zero position on the contour, where this zero position is the same for each of the contours that is defined by the plane. In an exemplary embodiment of the video generation technique described herein successive point locations along the contour are separated by a prescribed distance which is measured along the contour, and the just-described order, distance and zero position are the same for each of the planes in the series of planes.
Referring again to FIG. 5, once each of the contours that is defined by the plane has been analyzed (block 502), for each of these contours, the series of point locations that is identified along the contour is used to determine a mathematical equation describing the contour (block 504), where this determination is made using conventional methods. The point locations that are identified along each of the contours that is defined by the plane can then optionally be entered into a contour point map (block 506), where these point locations are entered into the contour point map in the order in which they are identified. In one embodiment of the video generation technique described herein each of the point locations in the contour point map is represented by its two-dimensional coordinates on the particular plane on which it lies. In another embodiment of the video generation technique each of the point locations in the contour point map is represented by its angle from the point location that immediately precedes it on its contour. It will be appreciated that the contour point map may be used for additional types of processing. Once a mathematical equation describing each of the contours that is defined by the plane has been determined (block 504), the texels in the scanline of the texture map that corresponds to the plane are assigned to these contours, where this texel assignment is performed in the aforementioned prescribed order across the plane (block 508). It will be appreciated that the assignment (block 508) can be performed in various ways, one example of which is described in more detail hereafter. Information specifying the texel assignment (e.g., which of the texels in the scanline is assigned to which of the contours defined by the plane) is then entered into the texture map (block 510).
Referring again to FIG. 5, once the actions of block 500 have been completed, the one or more streams of sensor data that represent the scene are used to compute texture data for each of the texels that is in the texture map (block 512), and this computed texture data is entered into the texture map (block 514). In an exemplary embodiment of the video generation technique described herein, this texture data computation is performed in the following manner. For each of the texels that is in the texture map, a projective texture mapping method is used to sample each of the streams of sensor data that represent the scene and combine texture information from each of these samples to generate texture data for the texel. It is noted that in a situation where just a single stream of sensor data is captured (i.e., in the aforementioned single viewpoint implementation of the video generation technique), just a single stream of sensor data will be sampled. Generally speaking and as is appreciated in the art of parametric texture mapping, the video generation technique embodiments described herein can compute various textures in order to maximize the photo-realism of the objects that are represented by the mesh models. In other words, the texture data that is computed for each of the texels that is in the texture map can include one or more of color data, or specular highlight data, or transparency data, or reflection data, or shadowing data, among other things.
Referring again to FIG. 5, once the computed texture data for each of the texels has been entered into the texture map (block 514), each of the scanlines in the texture map can optionally be replicated a prescribed number of times, where this number is greater than or equal to one (block 516). By way of example but not limitation, in the case where this prescribed number of times is one, each of the scanlines in the texture map will be replicated once so that two neighboring scanlines in the texture map will correspond to each of the planes in the series of planes. It is noted that a trade-off exists in the selection of the number of times each of the scanlines in the texture map is replicated. More particularly, replicating each of the scanlines a larger number of times creates more texture resolution in the images of the scene that are rendered from the scene proxies and displayed to the end users, but can also increase the amount of storage and network communication resources that are used in the aforementioned storage and distribution stage of the video processing pipeline exemplified in FIG. 1, and can also increase the amount of processing that is associated with the aforementioned rendering sub-stage of the video processing pipeline. On the other hand, replicating each of the scanlines a smaller number of times creates less texture resolution in the images of the scene that are rendered, but can also decrease the amount of storage and network communication resources that are used in the storage and distribution stage, and can also decrease the amount of processing that is associated with the rendering sub-stage.
It is noted that the contours that are defined by a given plane can be analyzed in any order across the plane. By way of example but not limitation, in an exemplary embodiment of the video generation technique described herein the contours that are defined by each of the planes are analyzed in a left-to-right order across the plane. It is also noted that the just-described zero position on the contour can be any position thereon as long as it is the same for each of the contours being analyzed. In an exemplary embodiment of the video generation technique this zero position is the left-most point on the contour. It is also noted that a trade-off exists in the selection of the distance which separates successive point locations along the contours that are defined by the series of planes. More particularly, using a smaller distance between successive point locations along the contours creates a finer approximation of each of the mesh models, but also increases the amount of processing that is associated with the aforementioned processing sub-stage of the video processing pipeline exemplified in FIG. 1. On the other hand, using a larger distance between successive point locations along the contours creates a coarser approximation of each of the mesh models, but also decreases the amount of processing that is associated with the processing sub-stage.
FIG. 6 illustrates an exemplary embodiment, in simplified form, of a contour and point locations that are identified along the contour. As exemplified in FIG. 6, a series of point locations 602-620 are identified along the contour 600. Successive point locations along the contour 600 (e.g., point location 614 and point location 616) are separated by the distance D which is measured along the contour.
FIG. 7 illustrates an exemplary embodiment, in simplified form, of a contour point map. The contour point map 700 exemplified in FIG. 7 corresponds to the series of three planes 400/402/404 that are used to slice the mesh model of the human 406 in FIGS. 4A-4C. The contour point map 700 includes three lines of data 702/704/706 each of which corresponds to a different one of the three planes 400/402/404. More particularly, a first line of data 702 in the contour point map 700 corresponds to the top-most plane 400. The first line of data 702 specifies that the top-most plane 400 defines one contour (namely, contour 408) and lists each of the A total point locations that are identified along this contour (namely, P1,1, P1,2, . . . , P1,A). The second line of data 704 specifies that the middle plane 402 defines two contours (namely, contour 410 and contour 412). The second line of data 704 also lists each of the B total point locations that are identified along contour 410 (namely, P1,1, P1,2, . . . , P1,B), and then lists each of the C total point locations that are identified along contour 412 (namely, P2,1, P2,2, . . . , P2,C). The third line of data 706 specifies that the bottom-most plane 404 defines three contours (namely, contour 414, contour 416, and contour 418). The third line of data 706 also lists each of the D total point locations that are identified along contour 414 (namely, P1,1, P1,2, . . . , P1,D), and then lists each of the E total point locations that are identified along contour 416 (namely, P2,1, P2,2, . . . , P2,E), and then lists each of the F total point locations that are identified along contour 418 (namely, P3,1, P3,2, . . . , P3,F).
FIG. 12 illustrates an exemplary embodiment, in simplified form, of a process for assigning the texels in the scanline of the texture map that corresponds to the plane to the contours that are defined by the plane. As will now be described in more detail, the process embodiment exemplified in FIG. 12 generally determines what percentage of the texels in the scanline are to be assigned to each of the contours that is defined by the plane. The process starts in block 1200 with the following actions taking place for each of the contours that are defined by the plane. The length of the contour is calculated (block 1202). The normalized length of the contour is then calculated by dividing the length of the contour by the sum of the lengths of all of the contours that are defined by the plane (block 1204). The number of texels in the scanline that are to be assigned to the contour is then calculated by multiplying the normalized length of the contour and the total number of texels that is in the scanline (1206), and this calculated number of texels is assigned to the contour (block 1208).
FIG. 14 illustrates a simplified example of the assignment of the texels in two neighboring scanlines of an exemplary texture map to the exemplary contours that are defined by two neighboring planes that correspond to the two neighboring scanlines. As exemplified in FIG. 14, a first scanline 1400 in the texture map (not shown) has a series of six texels T—1,1-T—1,6, and a second scanline 1402 which immediate succeeds the first scanline in the texture map has another series of six texels T—2,1-T—2,6. There are two contours 1404 and 1406 that are defined by a first plane (not shown) in a series of planes (not shown), and there are another two contours 1408 and 1410 that are defined by a second plane (not shown) which immediately succeeds the first plane in the series of planes. Based on the normalization of the lengths of the two contours 1404 and 1406 that are defined by the first plane, the first four texels T—1,1-T—1,4 in the first scanline 1400 will be assigned to contour 1404, and the last two texels T—1,5 and T—1,6 in the first scanline will be assigned to contour 1406. Similarly, based on the normalization of the lengths of the two contours 1408 and 1410 that are defined by the second plane, the first two texels T—2,1 and T—2,2 in the second scanline 1402 will be assigned to contour 1408, and the last four texels T—2,3-T—2,6 in the second scanline will be assigned to contour 1410.
The series of planes that is used to slice each of the mesh models has a prescribed spatial orientation and a prescribed geometry. It will be appreciated that the geometry of the series of planes can be specified using various types of data. Examples of such types of data include, but are not limited to, data specifying the number of planes in the series of planes, data specifying a prescribed spacing that is used between successive planes in the series of planes, and data specifying the shape and dimensions of each of the planes in the series of planes. Both the spatial orientation and the geometry of the series of planes are arbitrary and as such, various spatial orientations and geometries can be used for the series of planes. In one embodiment of the video generation technique the series of planes has a horizontal spatial orientation. In another embodiment of the video generation technique the series of planes has a vertical spatial orientation. It will be appreciated that any number of planes can be used, any spacing between successive planes can be used, any plane shape/dimensions can be used, and any distance which separates successive point locations along the contours can be used. In an exemplary embodiment of the video generation technique described herein the spacing that is used between successive planes in the series of planes is selected such that the series of planes intersects a maximum number of vertices in each of the mesh models. This is advantageous in a situation where each of the mesh models of the scene includes a mesh texture map that defines texture data which has already been computed for the vertices of the mesh model.
In one embodiment of the video generation technique described herein the spatial orientation and geometry of the series of planes, the order across each of the planes in the series of planes by which each of the contours that is defined by the plane is analyzed (hereafter simply referred to as the contour analysis order), the number of texels in each of the scanlines, the zero position on each of the contours (hereafter simply referred to as the contour zero position), and the number of times each of the scanlines in the texture map is replicated are pre-determined and thus are known to each of the end user computing devices in advance of the scene proxies being distributed thereto. In another embodiment of the video generation technique one or more of the spatial orientation of the series of planes, or the geometry of the series of planes, or the contour analysis order, or the number of texels in each of the scanlines, or the contour zero position, or the number of times each of the scanlines in the texture map is replicated, may not be pre-determined and thus may not be known to each of the end user computing devices in advance of the scene proxies being distributed thereto.
FIG. 8 illustrates an exemplary embodiment, in simplified form, of a process for storing the scene proxies. As exemplified in FIG. 8, the following actions take place for each of the mesh models (block 800). The mathematical equation describing each of the contours that is defined by the series of planes is stored (block 802). Data specifying which contours on neighboring planes in the series of planes correspond to each other (e.g., which neighboring contours are associated with either the same object or the same surface of a given object in the scene) is also stored (block 804). The texture map for the mesh model is also stored (block 806). Whenever the spatial orientation of the series of planes is not pre-determined, data specifying this spatial orientation will also be stored (block 808). Whenever the geometry of the series of planes is not pre-determined, data specifying this geometry will also be stored (block 810). Whenever the contour analysis order is not pre-determined, data specifying the contour analysis order will also be stored (block 812). Whenever the number of texels in each of the scanlines is not pre-determined, data specifying the number of texels in each of the scanlines will also be stored (block 814). Whenever the contour zero position is not pre-determined, data specifying the contour zero position will also be stored (block 816). Whenever the number of times each of the scanlines in the texture map is replicated is not pre-determined, data specifying the number of times each of the scanlines in the texture map is replicated will also be stored (block 818).
In one embodiment of the video generation technique described herein the mathematical equation describing a given contour specifies a polygon approximation of the contour. In another embodiment of the video generation technique the mathematical equation describing a given contour specifies a non-uniform rational basis spline (NURBS) curve approximation of the contour.
FIG. 9 illustrates an exemplary embodiment, in simplified form, of a process for distributing the scene proxies to an end user who either is, or will be, viewing the video on another computing device which is connected to a data communication network. As exemplified in FIG. 9, the following actions take place for each of the mesh models (block 900). The mathematical equation describing each of the contours that is defined by the series of planes is transmitted over the network to the other computing device (block 902). Data specifying which contours on neighboring planes in the series of planes correspond to each other is also transmitted over the network to the other computing device (block 904). The texture map for the mesh model is also transmitted over the network to the other computing device (block 906). Whenever the spatial orientation of the series of planes is not pre-determined, data specifying this spatial orientation will also be transmitted over the network to the other computing device (block 908). Whenever the geometry of the series of planes is not pre-determined, data specifying this geometry will also be transmitted over the network to the other computing device (block 910). Whenever the contour analysis order is not pre-determined, data specifying the contour analysis order will also be transmitted over the network to the other computing device (block 912). Whenever the number of texels in each of the scanlines is not pre-determined, data specifying the number of texels in each of the scanlines will also be transmitted over the network to the other computing device (block 914). Whenever the contour zero position is not pre-determined, data specifying the contour zero position will also be transmitted over the network to the other computing device (block 916). Whenever the number of times each of the scanlines in the texture map is replicated is not pre-determined, data specifying the number of times each of the scanlines in the texture map is replicated will also be transmitted over the network to the other computing device (block 918).
1.3 Video Presentation To End User
This section provides a more detailed description of the end user presentation stage of the video processing pipeline.
FIG. 10 illustrates an exemplary embodiment, in simplified form, of a process for presenting a video of a scene to an end user. As exemplified in FIG. 10, the process starts in block 1000 with receiving scene proxies that geometrically describe the scene as a function of time. The scene proxies include a stream of mathematical equations which describe contours that are defined by a series of planes that are parallel to each other. The scene proxies also include a stream of data specifying which contours on neighboring planes in the series of planes correspond to each other. The scene proxies also include a stream of texture maps which define texture data corresponding to each of the contours that is defined by the series of planes. After the scene proxies have been received (block 1000), images of the scene are rendered from the scene proxies (block 1002). The images of the scene are then displayed on a display device (block 1004) so that they can be viewed and navigated by the end user. As described heretofore, the video that is being presented to the end user can be any type of video including, but not limited to, asynchronous single viewpoint video, or asynchronous free viewpoint video, or unidirectional live single viewpoint video, or unidirectional live free viewpoint video, or bidirectional live single viewpoint video, or bidirectional live free viewpoint video.
FIG. 11 illustrates an exemplary embodiment, in simplified form, of a process for rendering images of the scene from the scene proxies. As exemplified in FIG. 11, the process starts in block 1100 with constructing the series of planes using data specifying the spatial orientation of the series of planes and the geometry of the series of planes. The contours that are defined by the series of planes are then constructed using the stream of mathematical equations (block 1102). A series of point locations is then constructed along each of the contours that is defined by the series of planes (block 1104), where this construction is performed in the aforementioned prescribed order across each of the planes in the series of planes, and this construction is also performed starting from the aforementioned prescribed zero position on each of the contours that is defined by each of the planes in the series of planes. In an exemplary embodiment of the video generation technique described herein a common number of point locations is constructed along each of the contours, and these point locations can be interspersed along the contour such that they are equidistant from each other. It will be appreciated that the actions of blocks 1100, 1102 and 1104 generate a stream of 3D point models that generally correspond to the stream of mesh models of the scene that was originally sliced using the series of planes. It will also be appreciated that this common number of point locations can have any value; a trade-off associated with the selection of this value is described in more detail hereafter.
The video generation technique embodiments described herein assume that each of the mathematical equations specifies how the particular contour it describes is positioned on the particular plane that defines this contour (e.g., by specifying one or more control points for the contour, among other ways). An alternate embodiment of the video generation technique is also possible where each of the mathematical equations does not specify how the particular contour it describes is positioned on the particular plane that defines this contour, in which case this positioning information will be separately specified, stored and distributed to the end user.
Referring again to FIG. 11, after the series of point locations has been constructed along each of the contours that is defined by the series of planes (block 1104), the point locations that are defined by the series of planes are tessellated using conventional methods, where this tessellation generates a stream of polygonal models, and each polygonal model includes a collection of polygonal faces that are formed by neighboring point locations on corresponding contours on neighboring planes in the series of planes (block 1106). Different embodiments of the video generation technique described herein are possible where the polygonal faces are either triangles, or quadrilaterals, or any other prescribed type of polygon. It will be appreciated that the action of block 1106 serves to recreate the stream of mesh models of the scene. It is noted that a trade-off exists in the selection of the common number of point locations that is constructed along each of the contours. More particularly, using a larger common number of point locations creates denser polygonal models and thus a higher resolution in the images of the scene that are rendered from the scene proxies and displayed to the end users, but also increases the amount of processing that is associated with the aforementioned rendering sub-stage of the video processing pipeline exemplified in FIG. 1. On the other hand, using a smaller common number of point locations creates less dense polygonal models and thus a lower resolution in the images of the scene that are rendered from the scene proxies, but also decreases the amount of processing that is associated with the rendering sub-stage.
Referring again to FIG. 11, after the point locations that are defined by the series of planes are tessellated (block 1106), the stream of texture maps is sampled to identify the texture data that corresponds to each of the polygonal faces in the stream of polygonal models (block 1108). Conventional methods are then employed to use this identified texture data to add texture to each of the polygonal faces in the stream of polygonal models (block 1110).
FIG. 15 illustrates an exemplary embodiment, in simplified form, of a process for sampling the stream of texture maps to identify the texture data that corresponds to each of the polygonal faces in the stream of polygonal models. As will be appreciated from the more detailed description that follows, this particular embodiment serves to minimize any sampling errors that may occur during the sampling of the stream of texture maps. As exemplified in FIG. 15, the process starts in block 1500 with an action of, for each of the scanlines in each of the texture maps, adapting the number of texels in the scanline that are assigned to each one of the contours that is defined by the plane corresponding to the scanline to be the average of the number of texels in the scanline that are assigned to this one of the contours and the number of texels in the next scanline in the series of scanlines that are assigned to another contour that corresponds to this one of the contours, where this adaption results in a modified version of each of the texture maps. The modified version of each of the texture maps is then sampled to identify the texture data that corresponds to each of the polygonal faces in the stream of polygonal models (block 1502).
In an exemplary embodiment of the video generation technique described herein, the just-described adaption operates in the following manner. In the case where the number of texels in a given scanline that are assigned to a particular contour that is defined by the plane corresponding to the scanline is greater than the average of the number of texels in the scanline that are assigned to the particular contour and the number of texels in the next scanline in the series of scanlines that are assigned to another contour that corresponds to the particular contour, the adaption of the number of texels in the scanline that are assigned to the particular contour involves using conventional methods to contract a series of texels in the scanline. In the case where the number of texels in a given scanline that are assigned to a particular contour that is defined by the plane corresponding to the scanline is less than the average of the number of texels in the scanline that are assigned to the particular contour and the number of texels in the next scanline in the series of scanlines that are assigned to another contour that corresponds to the particular contour, the adaption of the number of texels in the scanline that are assigned to the particular contour involves using conventional methods to expand a series of texels in the scanline.
In addition to performing the just-described adaption of the number of scanline texels that are assigned to each one of the contours that is defined by the plane corresponding to each of the scanlines in each of the texture maps, any sampling errors that may occur during the sampling of the stream of texture maps can further be minimized by optionally performing an action of, for each of the scanlines in each of the texture maps, inserting one or more “gutter texels” between neighboring texel series in the scanline, and optionally also inserting one or more “gutter scanlines” between neighboring scanlines in each of the texture maps. It will be appreciated that implementing such gutter texels and gutter scanlines serves to prevent bleed-over when the stream of texture maps is sampled using certain sampling methods such as the conventional bilinear interpolation method.
1.4 Supported Video Types
This section provides a more detailed description of exemplary types of single viewpoint video and exemplary types of free viewpoint video that are supported by the video generation technique embodiments described herein.
Referring again to FIG. 1, one implementation of the single viewpoint embodiment of the video generation technique described heretofore supports asynchronous (i.e., non-live) single viewpoint video, and a similar implementation of the free viewpoint embodiment of the video generation technique described heretofore supports asynchronous free viewpoint video. Both of these implementations correspond to a situation where the streams of sensor data that are generated by the sensors are pre-captured 104, then post-processed 106, and the resulting scene proxies are then stored and can be transmitted in a one-to-many manner (i.e., broadcast) to one or more end users 108. As such, there is effectively an unlimited amount of time available for the processing sub-stage 106. This allows a video producer to optionally manually “touch-up” the streams of sensor data that are received during the capture sub-stage 104, and also optionally manually remove any three-dimensional reconstruction artifacts that are introduced in the processing sub-stage 106. These particular implementations are referred to hereafter as the asynchronous single viewpoint video implementation and the asynchronous free viewpoint video implementation respectively. Exemplary types of video-based media that work well in the asynchronous single viewpoint video and asynchronous free viewpoint video implementations include movies, documentaries, sitcoms and other types of television shows, music videos, digital memories, and the like. Another exemplary type of video-based media that works well in the asynchronous single viewpoint video and asynchronous free viewpoint video implementations is the use of special effects technology where synthetic objects are realistically modeled, lit, shaded and added to a pre-captured scene.
Referring again to FIG. 1, another implementation of the single viewpoint embodiment of the video generation technique supports unidirectional (i.e., one-way) live single viewpoint video, and a similar implementation of the free viewpoint embodiment of the video generation technique supports unidirectional live free viewpoint video. Both of these implementations correspond to a situation where the streams of sensor data that are being generated by the sensors are concurrently captured 104 and processed 106, and the resulting scene proxies are stored and transmitted in a one-to-many manner on-the-fly (i.e., live) to one or more end users 108. As such, each end user can view 114 the scene live (i.e., each use can view the scene at substantially the same time it is being captured 104). These particular implementations are referred to hereafter as the unidirectional live single viewpoint video implementation and the unidirectional live free viewpoint video implementation respectively. Exemplary types of video-based media that work well in the unidirectional live single viewpoint video and unidirectional live free viewpoint video implementations include sporting events, news programs, live concerts, and the like.
Referring again to FIG. 1, yet another implementation of the single viewpoint embodiment of the video generation technique supports bidirectional (i.e., two-way) live single viewpoint video (such as that which is associated with various video-conferencing/telepresence applications), and a similar implementation of the free viewpoint embodiment of the video generation technique supports bidirectional live free viewpoint video. These particular implementations are referred to hereafter as the bidirectional live single viewpoint video implementation and the bidirectional live free viewpoint video implementation respectively. The bidirectional live single/free viewpoint video implementation is generally the same as the unidirectional live single/free viewpoint video implementation with the following exception. In the bidirectional live single/free viewpoint video implementation a computing device at each physical location that is participating in a given video-conferencing/telepresence session is able to concurrently capture 104 streams of sensor data that are being generated by sensors which are capturing a local scene and process 106 these locally captured streams of sensor data, store and transmit the resulting local scene proxies in a one-to-many manner on the fly to the other physical locations that are participating in the session 108, receive remote scene proxies from each of the remote physical locations that are participating in the session 108, and render 112 each of the received proxies.
Referring again to FIG. 1, it will be appreciated that in the unidirectional and bidirectional live single and free viewpoint video implementations in order for an end user to be able to view the scene live, the generation, storage and distribution, and end user presentation stages 102/108/110 have to be completed within a very short period of time. The video generation technique embodiments described herein make this possible based on the aforementioned video size/data minimization that is achieved by the video generation technique embodiments.
2.0 Additional Embodiments
While the video generation technique has been described by specific reference to embodiments thereof, it is understood that variations and modifications thereof can be made without departing from the true spirit and scope of the video generation technique. By way of example but not limitation, rather than supporting the generation, storage, distribution, and end user presentation of video, alternate embodiments of the video generation technique described herein are possible which support any other digital image application where a scene is represented by a mesh model and a corresponding mesh texture map which defines texture data for the mesh model.
It is also noted that any or all of the aforementioned embodiments can be used in any combination desired to form additional hybrid embodiments. Although the video generation technique embodiments have been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described heretofore. Rather, the specific features and acts described heretofore are disclosed as example forms of implementing the claims.
3.0 Computing Environment
The video generation technique embodiments described herein are operational within numerous types of general purpose or special purpose computing system environments or configurations. FIG. 13 illustrates a simplified example of a general-purpose computer system on which various embodiments and elements of the video generation technique, as described herein, may be implemented. It is noted that any boxes that are represented by broken or dashed lines in FIG. 13 represent alternate embodiments of the simplified computing device, and that any or all of these alternate embodiments, as described below, may be used in combination with other alternate embodiments that are described throughout this document.
For example, FIG. 13 shows a general system diagram showing a simplified computing device 1300. Such computing devices can be typically be found in devices having at least some minimum computational capability, including, but not limited to, personal computers (PCs), server computers, handheld computing devices, laptop or mobile computers, communications devices such as cell phones and personal digital assistants (PDAs), multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and audio or video media players.
To allow a device to implement the video generation technique embodiments described herein, the device should have a sufficient computational capability and system memory to enable basic computational operations. In particular, as illustrated by FIG. 13, the computational capability is generally illustrated by one or more processing unit(s) 1310, and may also include one or more graphics processing units (GPUs) 1315, either or both in communication with system memory 1320. Note that that the processing unit(s) 1310 may be specialized microprocessors (such as a digital signal processor (DSP), a very long instruction word (VLIW) processor, a field-programmable gate array (FPGA), or other micro-controller) or can be conventional central processing units (CPUs) having one or more processing cores including, but not limited to, specialized GPU-based cores in a multi-core CPU.
In addition, the simplified computing device 1300 of FIG. 13 may also include other components, such as, for example, a communications interface 1330. The simplified computing device 1300 of FIG. 13 may also include one or more conventional computer input devices 1340 (e.g., pointing devices, keyboards, audio (e.g., voice) input/capture devices, video input/capture devices, haptic input devices, devices for receiving wired or wireless data transmissions, and the like). The simplified computing device 1300 of FIG. 13 may also include other optional components, such as, for example, one or more conventional computer output devices 1350 (e.g., display device(s) 1355, audio output devices, video output devices, devices for transmitting wired or wireless data transmissions, and the like). Exemplary types of input devices (herein also referred to as user interface modalities) and display devices that are operable with the video generation technique embodiments described herein have been described heretofore. Note that typical communications interfaces 1330, additional types of input and output devices 1340 and 1350, and storage devices 1360 for general-purpose computers are well known to those skilled in the art, and will not be described in detail herein.
The simplified computing device 1300 of FIG. 13 may also include a variety of computer readable media. Computer readable media can be any available media that can be accessed by the computer 1300 via storage devices 1360, and includes both volatile and nonvolatile media that is either removable 1370 and/or non-removable 1380, for storage of information such as computer-readable or computer-executable instructions, data structures, program modules, or other data. By way of example but not limitation, computer readable media may include computer storage media and communication media. Computer storage media includes, but is not limited to, computer or machine readable media or storage devices such as digital versatile disks (DVDs), compact discs (CDs), floppy disks, tape drives, hard drives, optical drives, solid state memory devices, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other device which can be used to store the desired information and which can be accessed by one or more computing devices.
Storage of information such as computer-readable or computer-executable instructions, data structures, program modules, and the like, can also be accomplished by using any of a variety of the aforementioned communication media to encode one or more modulated data signals or carrier waves, or other transport mechanisms or communications protocols, and includes any wired or wireless information delivery mechanism. Note that the terms “modulated data signal” or “carrier wave” generally refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. For example, communication media includes wired media such as a wired network or direct-wired connection carrying one or more modulated data signals, and wireless media such as acoustic, radio frequency (RF), infrared, laser, and other wireless media for transmitting and/or receiving one or more modulated data signals or carrier waves. Combinations of the any of the above should also be included within the scope of communication media.
Furthermore, software, programs, and/or computer program products embodying the some or all of the various embodiments of the video generation technique described herein, or portions thereof, may be stored, received, transmitted, or read from any desired combination of computer or machine readable media or storage devices and communication media in the form of computer executable instructions or other data structures.
Finally, the video generation technique embodiments described herein may be further described in the general context of computer-executable instructions, such as program modules, being executed by a computing device. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The video generation technique embodiments may also be practiced in distributed computing environments where tasks are performed by one or more remote processing devices, or within a cloud of one or more devices, that are linked through one or more communications networks. In a distributed computing environment, program modules may be located in both local and remote computer storage media including media storage devices. Additionally, the aforementioned instructions may be implemented, in part or in whole, as hardware logic circuits, which may or may not include a processor.
Patent Application
20130321413
