Patent Application
20140087877
This disclosure relates generally to video game graphics and, more particularly, to the technology for constituting video game graphics by dynamically superimposing a foreground image having interactive game objects and a pre-recorded video content.
The approaches described in this section could be pursued, but are not necessarily approaches that have previously been conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
There have been steady improvements in visual presentations via graphical user interfaces (GUIs), and in particular, improvements associated with graphical characteristics of video games. Most modem video games provide virtual environments, through which the players can interact with one another or various game objects. The video game developers wish to create realistic gaming environments to enhance the overall game experience. To this end, game environments may include three-dimensional images having non-interactive objects, such as a background, as well as interactive objects such as user avatars or other game artifacts. A Graphics Processing Unit (GPU) can render images of game objects in real time during the play. Several graphics techniques, such as scaling, transformation, texture mapping, lighting modeling, physics-based modeling, collision detection, animation, anti-aliasing, and others can be used to create the visual appearance of the gameplay in real time. However, to improve game graphics and, correspondingly, user experience, better computational resources and complex graphics techniques are needed.
However, computational resources are limited and often not sufficient to create realistic virtual game environments. For example, a GPU may need to create a single frame for a video game every 33 milliseconds, which is a frame rate of 30 frames per second (FPS). High computational demands result in trade-offs associated with rendering images, which may prevent the game graphics from achieving high levels of quality.
To improve the experience of playing video games, some developers may include pre-recorded video fragments that can be shown during certain actions or scenes, for example when a player completes a particular game level. These video fragments can be of a higher quality and can have more realistic graphics than those shown during a regular gameplay. Moreover, the playback of such pre-recorded video fragments may not utilize large computational resources compared to the resources required for real-time rendering of such fragments. However, during the playback of these pre-recorded video fragments, the players may have limited or no control over the game.
Some other video games, such as music-based games, may play real live video over which graphic elements are overlaid. However, these games have a fixed game camera, which means the players may not have any control over the game camera, and thus the displayed video cannot be transformed. Similarly, augmented-reality games may utilize live video captured directly by a video camera connected to a game console, but the game camera cannot be controlled by the players.
Therefore, utilization of pre-recorded video content has been traditionally considered an obstacle to interactivity in video games. Furthermore, to provide high-quality three-dimensional graphics, large computational resources may be needed. As a result, today's game graphics technologies makes it extremely challenging to provide realistic video game graphics of high quality for the interactive games in which players exercise control over the virtual game camera.
This summary is provided to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
The present disclosure involves compositing realistic video game graphics for the video games in which the players have control over virtual game cameras, and for games in which the game developer can predict the movement of the virtual game camera. The technology involves rendering images of interactive game objects based on the current gameplay and virtual game camera parameters such as a pan angle, tilt angle, roll angle, and zoom data. The rendered images constitute the foreground of game scene that is displayed, which is then superimposed with pre-recorded video content. The video content constitutes the background of the game scene and refers to a real live video or high definition animation transformed from a prerecorded panoramic video based on the same virtual game camera parameters and the gameplay. The generation and superimposition of the foreground images and background videos are repeatedly performed to dynamically reflect the user actions within the virtual game environment. The superimposition process may also include any kind of synchronization process for the foreground images and the background images so that they are overlaid without any visual artifacts.
Thus, the present technology allows creating realistic game graphics with very high visual fidelity and detailed background graphical presentation, while providing freedom to the players to control the virtual game camera and the timing of user input. The import and transformation of pre-recorded video content do not require large computational resources compared to the resources required for real-time rendering of such scenes using traditional methods, and hence the present technology can be effectively employed in a number of game consoles, computers, mobile devices, and so forth.
According to one or more embodiments of the present disclosure, there is provided a method for compositing video game graphics, which includes the above steps. In further example embodiments, the method steps are stored on a machine-readable medium comprising instructions, which when implemented by one or more processors, perform the steps. In yet further example embodiments, hardware systems or devices can be adapted to perform the recited steps. Other features, examples, and embodiments are described below.
Embodiments are illustrated by way of example, and not by limitation in the figures of the accompanying drawings, in which like references indicate similar elements and in which:
The following detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show illustrations in accordance with example embodiments. These example embodiments, which are also referred to herein as “examples,” are described in enough detail to enable those skilled in the art to practice the present subject matter. The embodiments can be combined, other embodiments can be utilized, or structural, logical, and electrical changes can be made without departing from the scope of what is claimed. The following detailed description is therefore not to be taken in a limiting sense, and the scope is defined by the appended claims and their equivalents. In this document, the terms “a” and “an” are used, as is common in patent documents, to include one or more than one. In this document, the term “or” is used to refer to a nonexclusive “or,” such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated.
The techniques of the embodiments disclosed herein may be implemented using a variety of technologies. For example, the methods described herein may be implemented in software executing on a computer system or in hardware utilizing either a combination of microprocessors or other specially designed application-specific integrated circuits (ASICs), programmable logic devices, or various combinations thereof. In particular, the methods described herein may be implemented by a series of computer-executable instructions residing on a storage medium such as a disk drive, or computer-readable medium. It should be noted that methods disclosed herein can be implemented by a computer (e.g., a desktop computer, tablet computer, laptop computer), game console, handheld gaming device, cellular phone, smart phone, smart television system, and so forth.
In general, the embodiments of the present disclosure teach methods for creation of realistic and very high quality virtual environment graphics. A “virtual world” is an example of such virtual environment that is widely used in video and computer games. In virtual worlds, users can take the form of avatars, which are able to interact with other avatars or various game objects within the same virtual game environment. The ability for the users to explore the virtual world using input mechanisms, such as a game controller, is a basic requirement for interactive video games. In particular, the users can control the virtual game camera by manipulating the game controller to look around the virtual world and interactively perform various actions. When a user operates the virtual game camera, the technology described herein can be used to dynamically generate graphics of all that is captured by the virtual game camera and display on a user device.
Every game object used in the virtual world can be classified as interactive or non-interactive. Interactive game objects are those that could be affected by the actions of the player. Non-interactive game objects are those that the player cannot modify such as background elements including a sky, clouds, waterfalls, landscapes, nature scenes, and so forth. The technology, according to the embodiments of the present disclosure, uses pre-recorded video content to form some or all non-interactive game objects, while interactive game objects are rendered and then overlaid over the video content depending on the current position and parameters of the virtual game camera. The use of pre-recorded video content applied as a background can significantly reduce the need for computational resources and also increase the quality of game graphics. The pre-recorded video can be a real life video or a very high definition animation, and it may provide greater experience of enjoying playing the video game.
The principles described above are illustrated by
The foreground layer 110 can comprise images associated with interactive game objects, including an avatar that the player controls, other game characters, active game elements, and so forth. The images of interactive game objects can be dynamically rendered depending on the position and orientation of the virtual game camera 130. The rendering can be performed by a GPU or other rendering device so that the rendered images are either two- or three-dimensional. The images can be created in such a way they are placed on a transparent layer. In other words, there can be a number of opaque parts (pixels) of the layer 110 related to a particular interactive game object and also a number of transparent parts (pixels).
The background layer 120 can comprise pre-recorded video content or animation associated with non-interactive game elements including a sky, clouds, waterfalls, landscapes, nature scenes, city environments, and so forth. The video content can be two- or three-dimensional, and, moreover, it may also optionally include still images. As will be described below in greater details, the video content presented in the background layer 120 can be transformed from any kind of prerecorded panoramic video (e.g., spherical, cubical, or cylindrical prerecorded panoramic video) or its part by generating a corresponding equirectangular or rectilinear projections. The transformation and selection of a particular part of the prerecorded panoramic video are based on the current position, orientation, or other parameters of the virtual game camera 130. It should be also mentioned that the video content presented in the background layer 120 can be looped so that a certain video can be displayed repeatedly. The video content can be transformed or otherwise generated by a dedicated processor, such as a GPU or video decoder, or by a computing means, such as a central processing unit (CPU). The video content can reside in a machine readable medium or memory.
The term “virtual game camera,” as used herein, refers to a virtual system for capturing two-dimensional images of a three-dimensional virtual world. The virtual game camera 130 can be controlled by a user (i.e., a player) so that the images captured refer to the current position of the virtual game camera 130, its characteristics, position, and orientation. In the interactive games, such as first-person games, the game image is rendered from the viewpoint of the player character, which coincides with the view of the virtual game camera 130. In other words, the user sees the virtual world just like the avatar he controls. Accordingly, actions performed by the user will effect position of the virtual game camera 130, its orientation, and various parameters including a pan angle, a tilt angle, a roll angle, and zoom data.
The prerecorded panoramic video content used as a background layer 120 can be captured using a surround video capturing camera system such as the Dodeca® 2360 from Immersive Media Company (IMC) or LadyBug® 3 from Point Grey Research, Inc. However, the prerecorded panoramic video can also be created from footage captured using multiple cameras, each capturing a different angle of the panorama. The background video can also be rendered by GPU or other rendering device in different viewing angles to cover the complete field of view, and then these different views can be combined together to form a single frame of the prerecorded panoramic video. The process of creating the background video using various computational resources need not be done in real time, and therefore could incorporate complex lighting and physics effects, animation, and other visual effects having a great importance for the players.
In various embodiments, the number of frames in the video depends on the amount of time the background non-interactive game objects need to be shown on a display screen and the frame rate of the video game. If the background video consists of a pattern that repeats, the video could be looped to reduce the number of frames that need to be stored. A looping background video could be used, for example, in racing games that take place in a racing circuit, or for games that feature waves on water surfaces, to name a few.
The section of the panorama that needs to be displayed based on the values of the control parameters can be transformed using a rectilinear projection to remove various visual distortions.
All or some units 610-660 can be integrated within a single apparatus, or, alternatively, can be remotely located and optionally accessed via a third party. The system 600 may further include additional units, such as CPU or High Definition Video Processor (HDVP), but the disclosure of such modules is omitted so as to not burden the entire description of the present teachings. In various additional embodiments, the functions of the disclosed herein units 610-660 can be performed by other devices (e.g., CPU, HDVP, etc.).
The communication unit 610 can be configured to provide communication between the GPU 620, the video decoder 630, the storage 640, the game controller 650, and the display 660. In particular, the communication unit 610 can receive user control commands, which can then be used to determine the current position and orientation of the virtual game camera 130. Furthermore, the communication unit 610 can also transmit data, such as superimposed foreground images and background videos, to the display 660 for displaying to the user. The communication unit 610 can also provide transmit data from and to the storage 640.
The GPU 620 can be configured, generally speaking, to process graphics. More specifically, the GPU 620 is responsible for rendering images of the foreground layer 110 based on game data, the current gameplay, the current position and orientation of the virtual game camera 130, user commands, and so forth. The GPU can also superimpose the images of the foreground layer 110 and the video content of the background layer 120 to provide the resulting image to be transformed to the display 660 for presenting to the user.
In order to perfectly match the background layer 120 with the foreground layer 110, one or more synchronization methods can be also implemented by the GPU 620 using either time stamps or techniques such as vertical synchronization that can enforce a constant frame rate.
The video decoder 630 can be configured to process video content. More specifically, the video decoder 630 can be responsible for decoding or compressing video content and also for transformation of prerecorded panoramic video content from equirectangular projections to corresponding rectilinear projections based on game data, predetermined settings, the current gameplay, the current position and orientation of the virtual game camera 130, user commands, and so forth. This transformation may also be performed using the GPU as mentioned above with reference to
When the video decoder 630 decompresses the pre-recorded video data, any distortions will be avoided. For example, if the width of the final composited frame that is displayed is w, the width of the equirectangular projection shall be at least 3 w. The height of the equirectangular projections shall be at least 3w/2. However, for a limited horizontal or vertical field of view, the width and the height of the displayable frame can be smaller than 3 w and 3w/2, respectively. Any compression standard could be used that allows high speed decoding of the video frames at this resolution.
The storage 640 can be configured to store game data needed for running a video game, data necessary for generating the foreground layer 110 images, and pre-recorded videos for the background layer 120. The pre-recorded video can be adaptively selected based on the current gameplay (e.g., there can be the same scenes for day and night). The storage 640 may also store various processor-executable instructions.
The game controller 650 can be configured to provide input to a video game (typically to control an avatar, a game object or character in the video game). The game controller 650 can include keyboards, mice, game pads, joysticks, and so forth.
As shown in
At operation 720, the GPU 620 generates a foreground image, which is associated with one or more interactive game objects. The foreground image may be generated based on the parameters of the virtual game camera 130. The foreground images include both opaque and transparent parts (pixels). In various embodiments, the GPU 620 can generate multiple foreground images.
At operation 730, the video decoder 630 generates a background video by selecting and transforming at least a part of virtual prerecorded panoramic video from an equirectangular projection to a corresponding rectilinear projection. This transformation may also be performed using the GPU as mentioned above with reference to
At operation 740, the GPU 620 superimposes the background video and the foreground image(s). The superimposition process may include synchronization of the background video and the foreground image(s) to exclude visual artifacts.
At operation 750, the GPU 620 displays superimposed the background video and the foreground image through the display 660.
The example computer system 800 includes a processor or multiple processors 805 (e.g., a CPU, a GPU, or both), and a main memory 810 and a static memory 815, which communicate with each other via a bus 820. The computer system 800 can further include a video display unit 825 (e.g., a LCD or a cathode ray tube (CRT)). The computer system 800 also includes at least one input device 830, such as an alphanumeric input device (e.g., a keyboard), a cursor control device (e.g., a mouse), a microphone, a digital camera, a video camera, and so forth. The computer system 800 also includes a disk drive unit 835, a signal generation device 840 (e.g., a speaker), and a network interface device 845.
The disk drive unit 835 includes a computer-readable medium 850, which stores one or more sets of instructions and data structures (e.g., instructions 855) embodying or utilized by any one or more of the methodologies or functions described herein. The instructions 855 can also reside, completely or at least partially, within the main memory 810 and/or within the processors 805 during execution thereof by the computer system 800. The main memory 810 and the processors 805 also constitute machine-readable media.
The instructions 855 can further be transmitted or received over the communications network 860 via the network interface device 845 utilizing any one of a number of well-known transfer protocols (e.g., Hyper Text Transfer Protocol (HTTP), CAN, Serial, and Modbus).
While the computer-readable medium 850 is shown in an example embodiment to be a single medium, the term “computer-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “computer-readable medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the machine and that causes the machine to perform any one or more of the methodologies of the present application, or that is capable of storing, encoding, or carrying data structures utilized by or associated with such a set of instructions. The term “computer-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media. Such media can also include, without limitation, hard disks, floppy disks, flash memory cards, digital video disks, random access memory (RAM), read only memory (ROM), and the like.
The example embodiments described herein can be implemented in an operating environment comprising computer-executable instructions (e.g., software) installed on a computer, in hardware, or in a combination of software and hardware. The computer-executable instructions can be written in a computer programming language or can be embodied in firmware logic. If written in a programming language conforming to a recognized standard, such instructions can be executed on a variety of hardware platforms and for interfaces to a variety of operating systems. Although not limited thereto, computer software programs for implementing the present method can be written in any number of suitable programming languages such as, for example, Hypertext Markup Language (HTML), Dynamic HTML, XML, Extensible Stylesheet Language (XSL), Document Style Semantics and Specification Language (DSSSL), Cascading Style Sheets (CSS), Synchronized Multimedia Integration Language (SMIL), Wireless Markup Language (WML), Java™ Jini™ C, C++, C#, .NET, Adobe Flash, Perl, UNIX Shell, Visual Basic or Visual Basic Script, Virtual Reality Markup Language (VRML), ColdFusion™ or other compilers, assemblers, interpreters, or other computer languages or platforms.
Thus, methods and systems for compositing video game graphics are disclosed. Although embodiments have been described with reference to specific example embodiments, it will be evident that various modifications and changes can be made to these example embodiments without departing from the broader spirit and scope of the present application. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.
