The present invention generally pertains to a system and method for generating geometry and texture in a 3D video where the 3D video was generated from input images taken from a limited viewpoint.
There is considerable interest in generating 3D video from 2D images. For example, a 3D video could be generated of a person's grandmother waltzing with a famous dancer such as Fred Astaire. Many photographs and movies of Fred Astaire exist so that it would not be difficult to generate a volumetric image of Mr. Astaire in, for example, a white tie and tails, or to generate a 3D video of him dancing a waltz, for any angle or angles desired. However, only a limited number of images of the grandmother exist, almost all of them photographs taken at different times with different clothes on, and with the woman facing (or nearly facing) the camera so that no images were available of her back or the back of her head.
Therefore, in order to generate the desired volumetric video of the woman dancing with Fred Astaire, geometry and texture would have to be generated for portions of her body and clothing where no input images exist.
Typically, in the prior art, generating geometry and texture for the portions of a volumetric image that were not visible in the original image or images created artifacts in the geometry, the texture or both. Artifacts can comprise such things as discontinuities in geometry, texture or both, unexpected changes in geometry, texture or both, or blurring or jaggedness in the image.
It is therefore a long felt need to provide a system and method for generating geometry and texture for volumetric video where 2D images showing a large range of input angles are not available.
It is an object of the present invention to disclose a system for generating geometry and texture in a 3D video where the 3D video was generated from input images taken from a limited viewpoint.
It is another object of the present invention to disclose a method for generating a volumetric image of a subject from at least one 2 dimensional image, said at least one 2 dimensional image having a limited number of viewpoints, said volumetric image insertable into an environment, comprising steps of:
It is another object of the present invention to disclose the method as described above, additionally comprising a step of providing said texture as a pattern.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of fixing said pattern to a layer on a virtual camera.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of fixing said pattern to said initial volumetric image.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of fixing said pattern to said skeleton.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of fixing said pattern to said center of mass.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of fixing said pattern to a fixed point in space.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said pattern from a group consisting of a pattern of said environment, a proprietary pattern, a user-selected pattern, or a user-generated pattern.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of providing said pattern either changing over time or fixed over time.
It is another object of the present invention to disclose the method as described above, additionally comprising generating said reducing of said quality by a means comprising a member selected from a group consisting of:
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said higher quality comprising a member of a group consisting of a higher resolution, more detail, fewer artifacts, and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said at least a part of said initial volumetric image to be at least a part of said volumetric image back.
It is another object of the present invention to disclose a set of instructions that, when executed, are configured to generate a volumetric image of a subject from at least one 2 dimensional image, said at least one 2 dimensional image having a limited number of viewpoints, said volumetric image insertable into an environment, said instructions comprising steps configured to:
It is another object of the present invention to disclose the set of instructions as described above, wherein said texture is provided as a pattern.
It is another object of the present invention to disclose the set of instructions as described above, wherein said pattern is fixed to a layer on a virtual camera.
It is another object of the present invention to disclose the set of instructions as described above, wherein said pattern is fixed to said initial volumetric image.
It is another object of the present invention to disclose the set of instructions as described above, wherein said pattern is fixed to said skeleton.
It is another object of the present invention to disclose the set of instructions as described above, wherein said pattern is fixed to said center of mass.
It is another object of the present invention to disclose the set of instructions as described above, wherein said pattern is fixed to a fixed point in space.
It is another object of the present invention to disclose the set of instructions as described above, wherein said pattern is selected from a group consisting of a pattern of said environment, a proprietary pattern, a user-selected pattern, or a user-generated pattern.
It is another object of the present invention to disclose the set of instructions as described above, wherein said pattern is provided either changing over time or fixed over time.
It is another object of the present invention to disclose the set of instructions as described above, wherein said reducing of said quality is generated by a means comprising a member selected from a group consisting of:
It is another object of the present invention to disclose the set of instructions as described above, wherein said higher quality comprises a member selected from a group consisting of a higher resolution, more detail, fewer artifacts, and any combination thereof.
It is another object of the present invention to disclose the set of instructions as described above, wherein said at least a part of said initial volumetric image is selected to be at least a part of said volumetric image back.
In order to better understand the invention and its implementation in practice, a plurality of embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, wherein
The following description is provided, alongside all chapters of the present invention, so as to enable any person skilled in the art to make use of said invention and sets forth the best modes contemplated by the inventor of carrying out this invention. Various modifications, however, will remain apparent to those skilled in the art, since the generic principles of the present invention have been defined specifically to provide a means and method for generating geometry and texture in a 3D video where the 3D video was generated from input images taken from a limited viewpoint.
The term ‘volumetric image front’ hereinafter refers to the portion(s) of a volumetric image derived from those parts of the subject visible in at least one input 2D image.
The term ‘volumetric image back’ hereinafter refers to the portion(s) of a volumetric image derived from those parts of the subject not visible in any of the input 2D image(s).
The present invention discloses a system and method for generating volumetric video from input images taken from limited viewpoint locations, for example, from a single point of view.
Generating a volumetric image of those portions of a subject that are visible in the input 2D image(s) is well known in the art, as is avoiding having a visible mark where a visible portion from one image abuts a visible portion of another image. Inserting the volumetric image into a background, changing the point of view from which the volumetric image is viewed and changing the positions of features in the volumetric image, such as, but not limited to, moving or bending limbs are also well-known in the art. Therefore, in the prior art, volumetric video of acceptable quality can be generated if the portions of the subject visible in the original 2D image comprise substantially all of the subject.
However, difficulties can arise in generating a volumetric image for portions of the subject that were not visible in the input 2D image(s); artifacts such as mismatches of texture (color, pattern) or geometry between adjacent areas are all too common. In addition, there can be mismatch between higher-resolution portions of the volumetric image and lower-resolution portions of the volumetric image; the higher-resolution portions typically generated from portions of the subject visible in the initial 2D image(s) and the lower-resolution portions typically generated by algorithms as fill-in for the originally-invisible portions of the subject.
There are several methods which can be used to mitigate or hide the such artifacts. Non-limiting examples comprise: reducing the resolution of the volumetric image in the higher-resolution portions thereof, thereby blurring the artifact; simplifying the geometry or the texture by removing detail; or superimposing a predetermined pattern on the lower-resolution portions of the volumetric image. The predetermined pattern can be, for non-limiting example, a simplified version of an existing pattern in the texture; or it can be a pattern or fixed type, for non-limiting example, a proprietary identifier unique to a game or supplier.
It can also be desired to enable customization of the effect that is superimposed on top of the volumetric video.
Any or all of the above can be carried out by changing at least one portion of the texture, the geometry or both of the volumetric image to fit a type of “mold”. Non-limiting examples of a change of this kind comprise:
Combinations of the above can also be used. For non-limiting example, changing to a volumetric image having a geometry with limited degrees of freedom and incorporating a pattern on the volumetric image back.
If the texture of the volumetric image back is changed, the user can be given a choice—a texture providing a look and feel characteristic of the environment into which the subject is to be inserted (for non-limiting example, a Minecraft texture, a Roblox Studio texture, etc.), a proprietary texture providing a look and feel characteristic of a supplier (for non-limiting example, the texture shown in
These effects can be instituted by:
The texture can be moved with the body of the subject, with the camera (by providing a texture layer on the camera and terminating the texture layer at the edges of the subject), or with the environment (by fixing the texture layer to the environment and terminating the texture layer at the edges of the subject).
In
In
In
An exemplary method (100) of generating geometry, texture and resolution from at least one 2D image is shown in
In order to provide a result of acceptable quality, modifications to the volumetric image can be selected (115) to enable reduction of or hiding of the discrepancy between the volumetric image front and the volumetric image back. This can be done by reducing the resolution of the volumetric image front to match that of the volumetric image back, by simplifying the geometry, by attaching a predetermined texture to at least a part of the volumetric image back, or any combination thereof.
Typical types of modification comprise:
The volumetric image geometry can be simplified by combining features, for non-limiting example, by combining the fingers and palm of a hand into a single block, by reducing the number of joints in the volumetric image, or by treating the volumetric image, for the purpose of adding texture, as a center of mass. Simplifying the geometry can also reduce or eliminate the discrepancy in resolution. The type of simplification can match the subject to the environment into which the subject is inserted. For non-limiting example, a subject to be inserted into a Minecraft environment would be reduced to a head, a torso, two arms, two legs and, sometimes, a neck, each of these being a cuboid. These blocks can move relative to each other. The blocks have appropriate texture; for example, the head block comprises eyes, ears, nose, mouth and hair.
Another type of simplification, which also reduces or eliminates the discrepancy in resolution, reduces the subject to a skeleton with an extent; in yet another type of simplification, the subject is reduced to a center of mass with an extent.
Simplification can also comprise reducing the complexity of features, joining features to other feature, or eliminating features. Features of this type can comprise clothing, wrinkles in clothing, belts, buckles, or fasteners (buttons, ties, snaps, etc.). For non-limiting example, a shirt, waistcoat and jacket could be combined into a single, wrinkle-free unit forming a colored layer integral with the body.
A pattern can be superimposed on the volumetric image back, to hide the discrepancy in resolution.
The pattern can be selected to match the types of pattern in the environment, it can be a proprietary pattern (such as the enlarging and shrinking circles of
The superimposition can be relative to the camera (e.g., a pattern layer at the virtual location of the camera, the pattern layer “trimmed” frame-by-frame to match the 2D shape and size of the subject as seen by the camera. The superimposition can be relative to a skeleton of the volumetric image, relative to the volumetric image, relative to a center of mass of the volumetric image, or relative to a fixed point in the space of the environment.
Another type of simplification reduces the resolution of the volumetric image front to match that of the volumetric image back, thus eliminating the discrepancy in resolution. This is the easiest simplification, but it can be problematic, in that it can result in a subject who appears blurred relative to a sharper environment.
Once the type(s) of modification have been selected (115), they are applied (120) to the volumetric image, generating (125), frame-by-frame, a result showing the subject in the environment, the subject having an acceptable, although not necessarily realistic, look-and-feel.
This application claims the benefit of priority of U.S. Provisional Patent Application No. 63/394,686, filed Aug. 3, 2022, the contents of which are all incorporated herein by reference in their entirety.
Number | Date | Country | |
---|---|---|---|
63394686 | Aug 2022 | US |