Patent Application
20160217760
Recently, computer content has increasingly moved towards cloud platforms where dedicated servers render and stream content over a wide area network to client computing devices. In the genre of cloud-based streaming games, such streaming platforms allow service providers to centralize the rendering process for graphical scenes in games at datacenter servers. Server-side rendering offers several advantages. First, client computing devices can utilize the high-end graphics provided by powerful server GPUs, allowing the users to enjoy high-end graphics on less powerful client computing devices. Second, developers can specifically target their software for the datacenter servers, alleviating platform compatibility problems and increasing efficiency in platform performance tuning. Third, centralizing to datacenter servers increases the ease of bug fixing, software updates, hardware updates, and content additions. Lastly, users can select a vast library of content already loaded on the datacenter servers and instantly stream that content to their client computing devices.
One drawback of cloud-based streaming platforms is that wide-area networks can be subject to periods of high latency due to network congestion, packet loss, etc., and/or continually high latency caused by a large distance between the user and the server, which can degrade the user's ability to experience the streaming content in real-time without frequent pauses and delays. Current streaming platforms attempt to solve this problem through the use of content buffers. To implement a content buffer, servers stream content to the client computing device, where the content is placed into a content buffer that stores the content for display to the user. The client computing device waits until sufficient content has been buffered prior to playback to the user. After sufficient content is stored in the buffer, the content is displayed to the user. As the content is displayed, the buffer is refilled with streaming content from the server. Without a buffer, a period of high latency would result in a pause in playback; however, with a buffer, a pause in playback is not needed because a pause merely occurs in downloading the data stream to the buffer. In this manner, streaming content can be played to a user without perceived delays, since delays in downloading due to latency are absorbed by the buffer. However, while buffering can reduce user perceptions of delays in streaming content, buffering is subject to a rather large drawback when applied to applications such as gaming, in which user input at the client computing device is sent to the server and influences the rendered content. In these rendering applications, buffering can actually increase the user's perception of latency, due to the communication time lag between receipt of the content at the computing device and display of the content to the user. A small communication time lag may present an annoyance to the user as a small lag develops between user input and the response of the displayed content. However, a large communication time lag may make real-time game play over such networks frustrating and impractical.
To address these issues a client device and method are provided for reconstructing a viewport from a rendered rendering surface of a predicted user viewpoint in order to reduce user perceived latency of the network. The client device may comprise a user input device that receives a stream of user input, a display device, a processor, and a storage device holding instructions for a client application program, executable by the processor to: receive, from a server device over a network, a rendered rendering surface of a predicted scene having a predicted user viewpoint, wherein the rendered rendering surface has a wider field of view than a viewport of the client application program and includes a plurality of non-displayed rendered pixel data items in regions not displayed in the viewport of the client application program centered at the predicted user viewpoint, receive, from the user input device, a subsequent user navigation input in the stream of user input, determine an actual user viewpoint based on the subsequent user navigation input, determine a user viewpoint misprediction based on the predicted user viewpoint and the actual user viewpoint, reconstruct a viewport for the actual user viewpoint from the rendered rendering surface, wherein the viewport for the actual user viewpoint includes a portion of the plurality of non-displayed rendered pixel data items, and display the viewport on the display device.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
Computing system 10 typically includes a client device 12 configured to communicate with a server device 14 over a network 16, which may be a wide area network such as the Internet, or other suitable computer network. The client device 12 includes a processor 18, volatile memory 20, a non-volatile storage device 22, and at least one user input device 24 configured to receive user input from a user of the client device 12. The non-volatile storage device 22 holds instructions for a client application program 26 executable by the processor 18 to perform a variety of client-side functions, as described below. Similarly, the server device 14 includes a processor 28, volatile memory 30, and an associated non-volatile storage device 32. The non-volatile storage device 32 holds instructions for a server application program 34 executable by the processor 28 to perform a variety of server-side functions, as described below.
Within computing system 10 generally, the client device 12 gathers client-side user input and forwards that user input over the network to the server device 14, which in turn renders content based on the user input, and sends the rendered content back to the client device 12 over the network 16. As briefly described above, data traveling over network 16 may experience transmission delays due to network congestion or packet loss due to equipment failure, thereby triggering TCP slow start and requiring time-consuming retransmission under the TCP/IP protocols or continual high latency caused by a large distance between the user and the server. As a result, client device and server device may communicate over network 16 during periods of high latency. To reduce the perceived communication time lag between user input at the client device and subsequent display at the client device of responsive server-rendered content, a predictive rendering scheme is implemented according to which the server device renders extra content based on a future predicted input state, and the client device performs run-time decision making logic based on current user inputs, to determine the particular portion of the rendered content to display, and interpolates pixel data items to fill any visual holes left from any insufficiencies in the received rendered content, as described in detail below.
Turning first to the client device 12, it will be appreciated that the client device 12 receives a stream of user input 36 from user input device 24 at a user input buffer 38. The user input buffer 38 stores and organizes the stream of user input 36 and sends a current user navigation input 40 from the stream of user input 36 to the server application program 34 on the server device 14. It will be appreciated that the client device 12 may be a mobile computing device such as a smartphone, tablet, or head-mounted augmented reality computing device, a personal computing device, or a game console, as a few examples. The user input device 24 may be, for example, a touchscreen, keyboard, mouse, and/or sensors (e.g., gyroscope, accelerometers, depth camera, and/or RGB camera) mounted to a wearable computing device such as a head mounted augmented reality device, for example. It will be appreciated that navigational inputs such as the current user navigation input 40 are inputs that are interpreted by the client application 26 and server application 34 to navigate the user through a virtual space. For example, the client application program 26 and server application program 34 may be configured as a game that includes a virtual world that occupies a two- or three-dimensional game space. Navigation inputs may be interpreted by the program logic implemented in the client application program 26 and server application program 34 to cause a player character to travel through the game space in a direction indicated by the navigation inputs. It will be further appreciated that the user input may also include one or more actual impulse inputs 41, which are non-navigational inputs such as activating an object, activating an ability, etc. In the context of a first person perspective battle game, for example, an impulse input may cause a player character to wield a weapon, attack with the weapon, turn invisible, light a torch, etc. These inputs do not themselves cause navigation of the player character within the virtual world of the game.
The server application program 34 in turn receives the current user navigation input 40 at a prediction module 42. The prediction module 42 calculates a predicted future navigation input 44. It will be appreciated that predicted future navigation input 44 may be calculated using a neural network time-series prediction model, a linear and polynomial regression model, or a Markov model, as a few examples. In one exemplary embodiment, a discrete time Markov chain is applied to calculate the predicted future navigation input 44, as described below. The non-navigational actual impulse inputs 41 are received at the server device 14, as well. While these inputs do affect the application state 47 of the server application program, in one embodiment they are not used to predict navigational inputs, and therefore in the Figures are not shown as being passed to the prediction module 42, but are shown being received generally by the server application program 34.
In one example Markov model that may be implemented by prediction module 42, time is quantized, with each discrete interval representing a clock tick of the server application program 34. The random variable navigation vector Nt represents the change in 3-D translation and rotation at time t: Nt={δx,t, δy,t, δz,t, θx,t, θy,t, θz,t}
Each component above is quantized. Nt represents an actual empirical navigation vector received from the client as current user navigation input 40. The prediction module 42 calculates a state estimation problem to find the maximum likelihood estimator {circumflex over (N)}t+λ where λ is the round trip time (RTT) over the network 16 between the client device 12 and the server device 14.
Using the Markov model, the probability distribution of the navigation vector at the next time step is dependent only upon the navigation vector from the current time step: p(Nt+1|Nt). The prediction module 42 predicts the most likely navigation vector {circumflex over (N)}t+1 at the next time step as:
where Nt=nt indicated that the current time step has been assigned a fixed value by sampling the actual user input nt. In many cases, the RTT is longer than a single time step (e.g., 32 ms). To handle this case, the prediction module 42 predicts the most likely value after one RTT as:
where λ represents the RTT latency expressed in units of clock ticks.
The above Markov model when implemented by the prediction module 42 has been found to have the desirable effect that, Nt+1 is memoryless (i.e., independent of the past given Nt), and additional user input history (in the form of longer Markov chains) does not show a measurable benefit in terms of prediction accuracy. The prediction module 42 treats each component of the vector N independently, and constructs six separate models. The beneficial effect of this approach is that less training is required when estimating {circumflex over (N)}, and it has been found that treating the vector components independently does not hurt prediction accuracy.
After calculating the predicted user navigation input 44, the prediction module 42 determines a plurality of alternative impulse input sequences 46 for different permutations of impulse inputs that can occur based on a current application state of application state 47 within a communication lag time (e.g. one RTT) between the server device and client device. As described above, impulse inputs are non-navigational user inputs such as wielding a weapon, attacking with the weapon, activating an object such as a torch, activating an ability such as invisibility, etc. In one exemplary embodiment, the prediction module 42 considers all possible sequences of impulse inputs that can occur in one RTT. For example, according to the program logic of the server application program 34 and the client application program 26, it may be possible for a user to activate the impulse input of firing a weapon at every clock tick of the client application program 26. Accordingly, if the RTT is two clock ticks, then the prediction module 42 considers impulse input sequences corresponding to firing a weapon at both clock ticks, as well as firing the weapon at the first clock tick and not the second clock tick, not firing the weapon at the first clock tick and firing the weapon at the second clock tick, or not firing the weapon at either clock tick. Accordingly, these four alternative possibilities would be included within the plurality of alternative impulse input sequences 46. In cases where the network latency is high, such as if the RTT is 8 clock ticks long, the state space for possible impulse input sequences may become undesirably large.
To reduce the number of possible alternative impulse input sequences that can occur during one RTT, the prediction module 42 typically only allows impulse input activations to occur at a subsampling periodicity U that is greater than one clock tick of the server application program 34 and the client application program 26. This subsampling reduces the state space of all possible impulse input sequences considered by the prediction module 42 to 2λ/σ.
The prediction module 42 sends the predicted future navigation input 44 and the plurality of alternative impulse input sequences 46 to the rendering module 48. The rendering module 48 retrieves a plurality of rendering surfaces 50 from a rendering surface library 52. Based on the predicted future navigation input 44, the plurality of alternative impulse input sequences 46, and the current application state of application state 47 of the server application program 34, the rendering module 48 renders a plurality of future scenes having a predicted user viewpoint to the plurality of rendering surfaces 50 and a corresponding plurality of depth maps 51. Each depth map 51 contains depth values representing distances in the virtual world (game space) from a point corresponding to the predicted user viewpoint to each pixel (or group of pixels) in each rendered rendering surface. The rendering module 48 sends the rendered rendering surfaces 54 to a joint encoder 56. It is expected that the predicted future navigation input 44 will not always completely match the actual navigation input entered from the user. To account for user viewpoint mispredictions, rendering module 48 renders the future scene with a larger field of view than a viewport of the client application program 26 at the client device 12. By rendering with a larger field of view, even if the actual user viewpoint has a different orientation than the predicted user viewpoint, such as the user is looking further to the right than was predicted, that actual user viewpoint has a high probability of still being contained within the rendered larger field of view. When rendering with the larger field of view, the plurality of rendering surfaces 50 will not necessarily be a rectangle as in a viewport for a game application. In some embodiments, the plurality of rendering surfaces may be a cubemap, a clipped cubemap, an edge, a tetrahedron, or other type of rendering surface.
To perform the rendering, an exemplary embodiment of rendering module 48 creates a plurality of slave instances, each of which is a modified instance of the original server application program 34. The slave instances are configured to execute the predicted future navigation input 44 and one alternative impulse input sequence from the plurality of alternative impulse input sequences 46 according to the program logic of the server application program 34. For each slave instance a predicted user viewpoint is determined based on the predicted future navigation input 44. After execution of the program logic of the server application program, each slave instance renders its own future scene from the predicted user viewpoint based on the current application state of application state 47 and the alternative impulse input sequence it received to a rendering surface and creates a corresponding depth map of the future scene. After rendering, each slave sends its own rendered rendering surface and depth map to a master process in rendering module 48. The master process may vary the number of slave instances dependent upon the length of the RTT, and is responsible for forwarding the renderings completed by each slave instance for downstream processing by the joint encoder 56.
It will be appreciated that not all of the speculative sequences for impulse inputs and navigation inputs will be correct, and indeed it is possible that none of the speculative sequences is correct. Consequently, the slave instances may have executed speculative sequences for impulse inputs and navigation inputs that were mispredictions, and will have incorrect application states. To account for this, the system may roll back the incorrect slave instances to a correct application state using one or more of the following techniques: page-level checkpointing, object-level checkpointing, and restore functions.
For page-level checkpointing, the server application program keeps copies of pages of memory corresponding to past application states of the server application program in order to roll back to a past application state if the current application state resulted from incorrect speculative input sequences of navigational inputs and impulse inputs. To accomplish this, the server application program implements page-level copy-on-write. At the start of a speculation sequence for navigation inputs or impulse inputs, the server application program marks all pages read-only. When a page fault occurs, the server application program makes a copy of the original page and sets the protection level of the faulted page to read-write. When new input arrives, the server application program invalidates and discards speculative input sequences that do not match the new input. In order to roll back to a checkpoint, the server application program copies back the original content using the copies of the original page that it created. The server application program discards any pages created as a result of new object allocations since the last checkpoint. Additionally, page deallocation resulting from deleted objects during a speculative sequence are deferred as those deleted objects may need to be restored if the speculative sequence is incorrect.
For object-level checkpointing, the lifetime of each object is tracked by the server application program. To roll back a speculation sequence, objects that did not exist at the time of the checkpoint being rolled back to are deleted, and objects that were deleted during the incorrect speculation sequence since the checkpoint being rolled back to are restored.
After receiving the rendered rendering surfaces 54 from the rendering module 48, the joint encoder 56 jointly encodes the rendered rendering surfaces 54 and sends encoded rendered rendering surfaces 58 over the network 16 to the decoder 60 at the client device 12. In an exemplary embodiment, the joint encoder 56 divides the rendered rendering surfaces 54 into macroblocks (e.g. 64×64 bit). A search process of the joint encoder identifies macroblocks that are equivalent (in a lossy domain) intra-frame, inter-frame, and across the different streams of rendered rendering surfaces corresponding to each slave instance. The purpose of the encoding by the joint encoder 56 is to reduce the amount of data that represents the encoded rendered rendering surfaces, to increase the speed of data transmission and reduce bandwidth consumption, which contributes to the user's decreased perception of latency at the client device 12.
At the decoder 60 of the client device 12, the encoded rendered rendering surfaces 58 are decoded, and the decoded rendered rendering surfaces 61 are sent to the view interpolation module 62. The view interpolation module 62 additionally receives the plurality of alternative impulse input sequences 46 from the prediction module 42 and the plurality of depth maps 51 from the rendering module 48.
The user input buffer 38 receives a subsequent user navigation input 64 and a subsequent impulse input sequence 66 in the stream of user input 38 from user input device 24, and sends the subsequent user navigation input 64 and the subsequent impulse input sequence 66 to the view interpolation module 62. Thus, it will be appreciated that the plurality of alternative impulse input sequences 46, the encoded rendered rendering surfaces 58, and depth maps 51 received from the server device 14 were calculated and predicted based on user input at a quantized time step (e.g., T=0) that was current for the server device but is now past input for the client device due to the time-lag in client server communications. It will be further appreciated that the subsequent user navigation input and the subsequent impulse input sequence 66 received from the user input buffer 38 represents user input from a later quantized time step (e.g., T=1) that is the most current user input received at the client device.
The view interpolation module 62 is configured to select an alternative impulse input sequence from the plurality of alternative impulse input sequences 46 based on (i.e., that corresponds to) the subsequent impulse input sequence 66. In an exemplary embodiment, the view interpolation module 62 is configured to compare each alternative impulse input sequence of the plurality of alternative impulse input sequences 46 to the subsequent impulse input sequence 66 and determine which alternative impulse input sequence matches the subsequent impulse input sequence 66. Thus, for example, if the current impulse input sequence 66 is input to fire a weapon twice in a row, then the alternative impulse input sequence 66 corresponding to firing a weapon twice in a row is selected.
The view interpolation module 62 is further configured to select the rendered rendering surface that corresponds to the selected alternative impulse input sequence. The view interpolation module 62 reconstructs a viewport for an actual user viewpoint by reprojecting from the selected rendered rendering surface to a viewport of the client application program 26 based on the plurality of depth maps 51 and the subsequent user navigation input. The view interpolation module 62 sends the reconstructed viewport 68 be displayed on the display device 70.
Thus, the client application program 26 is configured to communicate with server application program 34 to send user input to the server device 14 and in response download predictive rendered content from the server application program 34 and perform image based rendering techniques on the predictive rendered content to reconstruct and display a rendered viewport that is responsive to the user's current input.
Now turning to
One type of misprediction that may be made by the server device is a user orientation misprediction.
When rendering to a wider field of view in 3D space, rendering surfaces such as a cubemap, clipped cubemap, edge, tetrahedron, etc., can be used.
Generally, rendering to all 6 planes of the cubemap is unnecessary as the actual user viewpoint is unlikely to diverge from the predicted user viewpoint so significantly as to necessitate rendering in the opposite direction of a current user perspective, for example. An embodiment of the clipped cubemap is shown in
In another embodiment of the rendering surface, an edge shape is used.
It will be appreciated that typically there is a high likelihood that the actual user viewpoint will be close to the predicted user viewpoint. Accordingly, the rendering quality for the actual user viewpoint can be increased while simultaneously reducing server, client, and network bandwidth costs through likelihood based foveation, a technique that reallocates pixels to areas which are more likely to be viewed. With likelihood based foveation, the rendered rendering surface is rendered with a high resolution at areas with a high likelihood of actually being viewed and rendered with a lower resolution at areas with a low likelihood of actually being viewed.
In embodiments using the clipped cubemap and likelihood based foveation, the most likely location to be actually be viewed is the front face of the cube, which is the predicted viewed area. Accordingly, the front face of the clipped cubemap is rendered with a higher resolution than the above, below, left, and right planes of the clipped cubemap.
Now turning to
In an exemplary embodiment, each alternative impulse input sequence from
In some embodiments, the communication lag time between the server device and the lag device may be high. In the embodiment depicted by
Only allowing impulse input activations to occur at a subsampling periodicity larger that is greater than one clock tick of the server application program and the client application program may cause the subsequent impulse input activations received at the client device from the user input device that do not fall on a subsampled tick to be lost. The quantization logic depicted in
Method 1400 begins at 1402 and includes receiving, at a server application program executed on the server device, a current user navigation input in a stream of user navigation inputs from a client device over a network. Method 1400 advances to 1404 and includes calculating a predicted future navigation input based on the current user navigation input and a current application state of the server application program, as described above.
Method 1400 advances to 1406 including determining whether impulse inputs can be activated in the server application program. In an embodiment of the server application program that only considers navigational inputs and does not have impulse inputs, method 1400 proceeds to 1408 and includes rendering a future scene based on the predicted future navigation input to a rendering surface. The server device may render the future scene with a wider field of view than a viewport of the client application program executed on the client device. The rendering surface may be a cubemap, a clipped cubemap, an edge, or a tetrahedron as a few non-limiting examples, as discussed above. Method 1400 proceeds to 1410 and includes sending the rendered rendering surface to the client device over the network.
In another embodiment of the server application program that considers both navigational inputs and impulse inputs, method 1400 proceeds from 1404 to 1406 to 1412 and includes determining, based on the current application state of the server application program, a plurality of alternative impulse input sequences for different permutations of impulse inputs that can occur within a communication lag time between the server device and the client device. To reduce the state space of possible impulse input sequences that may occur within the communication lag time, the server device may determine impulse inputs to only occur at a subsampling periodicity that is greater than one clock tick of the server application program.
Method 1400 advances to 1414 and includes rendering a plurality of future scenes based on each of the alternative impulse input sequences and the predicted future navigation input to a plurality of rendering surfaces. The server device may render the plurality of future scenes with a wider field of view than a viewport of the client application program executed on the client device. The rendering surfaces may be a cubemap, a clipped cubemap, an edge, or a tetrahedron as a few non-limiting examples. Method 1400 proceeds to 1416 and includes sending the plurality of alternative impulse input sequences and the plurality of rendered rendering surfaces to the client device over the network.
Now turning to
Next, the client device reconstructs a viewport for the actual user viewpoint, from the rendered rendering surface by reprojecting from the rendered rendering surface to the reconstructed viewport. In an exemplary embodiment of the client device, in order to reproject from the rendered rendering surface to the viewport for the actual user viewpoint, the client redraws the scene from the viewpoint of the actual user viewpoint using the RGBD(epth) data of the rendered rendering surface and depth map. To redraw the scene, the client device casts a ray from the actual user viewpoint to the scene defined by the rendered rendering surface and corresponding depth map. For example, ray 1600 is cast from the actual user viewpoint and intersects with a portion of the square in the predicted scene. After determining the intersection of ray 1600 and the square, the pixel information for that portion of the square that intersected with ray 1600 can be copied to a location in the reconstructed viewport corresponding to the origination point from which ray 1600 was cast. Similarly, ray 1602 also intersects with an object, the hexagon, and the pixel data items for the portion of the hexagon intersecting with ray 1602 can be copied to the reconstructed viewport in a location corresponding to the origination point of ray 1602. It will be appreciated that Ray Casting, Ray Marching, Ray Tracing, and other techniques may be used for casting the ray in the above described embodiment.
The user viewpoint misprediction shown in
In the embodiment where the future scene is rendered to a wider field of view as shown in
However, even in embodiments that include rendering to a wider field of view than a viewport of the client application program, there would be no visual information for ray 1604 shown in
To fill any visual holes left after reprojecting from the rendered rendering surface to the reconstructed viewport of the actual user viewpoint, the view interpolation module of the client application program on the client device interpolates one or more pixel data items to fill the one or more visual holes in the reconstructed viewport using image based rendering methods. It will be appreciated that multiple image based rendering methods may be used, for example, spatial view interpolation, temporal view interpolation, depth peeling, etc. In one embodiment, a spatial view interpolation method is used, wherein a screen-space grid mesh is projected onto a depth buffer for the depth map corresponding to the rendered rendering surface of the predicted scene having the predicted user viewpoint.
In an exemplary embodiment, the view interpolation module casts a ray from the actual user viewpoint to the screen-space grid mesh. As shown in
In some embodiments, the edges of the screen-space grid mesh projected onto the depth buffer of the depth map are dilated by one or more pixels in width by applying a minimum filter over small window sizes (e.g. 5×5), such that the foreground object appears larger. Dilating the edges can help preserve the shape of the foreground object and reduce the impact of resultant visual artifact.
In another embodiment, the view interpolation module interpolates one or more pixel data items using a temporal view interpolation method, by reprojecting from a portion of a previously rendered viewport. An example of this embodiment is shown in
In one embodiment of the above configuration, the number of depth layers in the plurality of depth layers rendered to the rendering surface does not exceed an adjustable threshold value. Some objects or scenes may have complex depth profiles, and have a potentially large number of depth layers (e.g. foliage or a particle system). The large number of depth layers could lead to potentially unbounded memory requirements. By limiting the number of potential depth layers to a threshold value, the problem of potentially unbounded memory requirements is prevented.
Method 2000 begins at 2002 and includes determining whether the client application program executed on the client device includes impulse inputs. If the client application program only allows for navigational inputs to be entered and not impulse inputs, then method 2000 proceeds to 2004 and includes receiving, from a server device over a network, a rendered rendering surface of a predicted scene having a predicted user viewpoint, wherein the rendered rendering surface has a wider field of view than a viewport of the client application program and includes a plurality of non-displayed rendered pixel data items in regions not displayed in the viewport of the client application program centered at the predicted user viewpoint.
Method 2000 advances to 2006 and includes receiving a subsequent user navigation input in a stream of user input. Upon receiving the subsequent user navigation input, method 200 proceeds to 2014 and includes determining an actual user viewpoint based on the subsequent user navigation input. Next, method 2000 advances to 2016 and includes determining a user viewpoint misprediction based on the predicted user viewpoint and the actual user viewpoint.
Upon determining that there was a user viewpoint misprediction, method 2000 proceeds to 2018 and includes reconstructing a viewport for the actual user viewpoint from the rendered rendering surface, wherein the viewport for the actual user viewpoint includes a portion of the plurality of non-displayed rendered pixel data items. It will be appreciated that the portion of the plurality of non-displayed rendered pixel data items that are included in the viewport for the actual user viewpoint will be displayed to the user, while the remaining portion of the plurality of non-displayed rendered pixel data items that were not included in the viewport for the actual user viewpoint will remain un-displayed.
During reconstruction step 2018, method 2000 proceeds through sub-steps 2020 and 2022. Step 2020 includes reprojecting from a portion of the rendered rendering surface to the viewport for the actual user viewpoint. It will be appreciated that after reprojection, visual holes may remain in the viewport for the actual user viewpoint due to the user viewpoint misprediction as described above. To fill these visual holes, method 2000 advances to step 2022 and includes interpolating one or more pixel data items to fill one or more visual holes in the reconstructed viewport. It will be appreciated that the one or more pixel data items may be interpolated through multiple different methods, for example through spatial interpolation, temporal interpolation, or depth peeling.
One example manner of accomplishing step 2022 uses spatial interpolation, as illustrated at substep 2022A. At substep 2022A, the method may include projecting a screen-space grid mesh onto a depth buffer for a depth map corresponding to the rendered rendering surface of the predicted scene. The screen-space grid mesh may include an edge that extends between a foreground object and a background object in the predicted scene. Next, substep 2022A may include dilating the edge in the screen-space grid mesh by one or more pixels in width. Then, substep 2022A may include casting a ray from the actual user viewpoint to the screen-space grid mesh. Next, substep 2022A determining that the ray intersects with the edge in the screen-space grid mesh. Finally, substep 2022A may include calculating a pixel color for the interpolated one or more pixel data items based on a color of the background object in the predicted scene. It will be appreciated that when casting a ray, Ray Casting, Ray Marching, Ray Tracing, and other ray techniques may be used.
Another substep of step 2022 that uses temporal interpolation includes reprojecting from a portion of a previously rendered viewport is illustrated at substep 2022B. A substep of step 2022 that uses depth peeling is illustrated at 2022C, and includes reprojecting from the rendered rendering surface. According to substep 2022C, the rendered rendering surface may be a composite of a plurality of depth layers, and a number of depth layers in the plurality of depth layers typically does not exceed an adjustable threshold value.
After reconstructing the viewport for the actual user viewpoint, method 2000 proceeds to step 2024 including displaying the viewport for the actual user viewpoint on a display device.
Turning back to step 2002, if the client application program includes impulse inputs, then method 2000 advances to step 2008 and includes receiving, from a server device over a network, a plurality of alternative impulse input sequences and a plurality of rendered rendering surfaces corresponding to the plurality of alternative impulse input sequences. It will be appreciated that the plurality of rendered rendering surfaces may have each been rendered with the predicted user viewpoint and with a wider field of view than a viewport of the client application program and include a plurality of non-displayed rendered pixel data items in regions not displayed in the viewport of the client application program centered at the predicted user viewpoint.
Method 2000 advances to step 2010 and includes receiving a subsequent user navigation input and a subsequent impulse input sequence in the stream of user input. Next, method 2000 proceeds to 2012 and includes selecting an alternative impulse input sequence from the plurality of alternative impulse input sequences based on the subsequent impulse input sequence and selecting a rendered rendering surface from the plurality of rendered rendering surfaces corresponding to the selected alternative impulse input sequence. It will be appreciated that this selections may be made using a variety of criterions. In one embodiment, the alternative impulse input sequence that matches the subsequent impulse input sequence is selected. In other embodiments, the alternative impulse input sequence that is closest to the subsequent impulse input sequence is selected. It will also be appreciated that the alternative impulse input sequences may have been determined accordingly with a subsampled periodicity that is greater than one clock tick of the client application program. In such an embodiment, method 2012 may include time-shifting each impulse input of the subsequent impulse input sequence to a nearest subsampled tick of the plurality of alternative impulse input sequences before selecting an alternative impulse input sequence.
Upon selection of an alternative impulse input sequence and a corresponding rendered rendering surface, method 2000 proceeds through steps 2014, 2016, 2018, 2020, 2022, and 2024 in the same manner as described above.
As shown in
In another embodiment, to reduce bandwidth consumption, the server device only executes method 1400 for the predicted user viewpoint of a user's first eye. The server device may alternate the first eye between the user's left eye and the user's right eye at each rendering pass, such that the HMD client device receives predictive content that alternates between a predicted user viewpoint of the user's left eye and a predicted user viewpoint of the user's right eye. In this embodiment, the HMD client device executes the method 2500 to reconstruct both a viewport for the actual user viewpoint of the user's first eye and a second viewport for the actual user viewpoint of the user's second eye based on current predictive content that was just received from the server device and previous predictive content that was previously received from the server device. It will be appreciated that in this embodiment, the current predictive content was rendered from the predicted user viewpoint of a first eye, and the previous predictive content was rendered from the predicted user viewpoint of a second eye. It will be further appreciated that the predictive content may be a rendered rendering surface, or a plurality of rendered rendering surfaces and a plurality of alternative impulse input sequences produced by the server device according to method 1400.
Now turning to
In the example illustrated in
However, rays 2504 and 2506 point to locations that are not contained within the rendered content of the rendered rendering surface having a user viewpoint for the first eye. In one embodiment, the view interpolation module of the HMD client device interpolates pixel data items to fill these visual holes left by the rays that do not intersect with rendered content in the rendered rendering surface. It will be appreciated that pixel data items for the visual holes for ray 2504 can be interpolated using the spatial view interpolation method shown in
In the embodiment where the server device alternates between rendering the scene from the user viewpoint for the left eye and user viewpoint for the right eye, the HMD client device synthesizes the viewport for the second eye based on not only a current rendered rendering surface representing a current view of a scene for a first eye of the user, but also a previously rendered rendering surface representing a past view of the scene for a second eye of the user. It will be appreciated that because the server device is alternating between eyes after each rendering pass in this embodiment, the past view of the scene represented by the previously rendered rendering surface will only be on the order of 30-100 ms in the past. It will also be appreciated that the server device may alternate between the left eye and then right eye according to other periodicities, such as every two or three rendering passes.
In the example depicted in
The visuals holes that arise after reprojection from the current rendered first eye rendering surface due to depth disparity and interpupillary distance as described in
Turning back to
Reprojection from the previously rendered second eye rendering surface will cause visual holes in the current synthesized second eye viewport when a rate of change of user viewpoint is high. In some embodiments, the rate of change of user viewpoint is calculated based on the stream of user input received from the user (e.g. total head rotation distance over one RTT). It will be appreciated that if the rate of change of user viewpoint is low or zero (e.g. user has not recently entered a navigational input), then the previously rendered second eye rendering surface will have the same user viewpoint for the second eye as the current synthesized second eye viewport. Accordingly, if the rate of change of user viewpoint is low or zero, reprojection from the previously rendered second eye rendering surface to the current synthesized second eye viewport will result in minimal visual holes.
After reprojection from the previously rendered second eye rendering surface to the current synthesized second eye viewport, there may still be remaining visual holes in the current synthesized second eye viewport. In some embodiments, these remaining visual holes are filled using screen-space color picking from the previously rendered second eye rendering surface. In these embodiments, the view interpolation module of the HMD client device may determine the locations in the current synthesized second eye viewport that are visual holes, and copy pixel data from corresponding locations in the previously rendered second eye rendering surface.
Method 2500 begins at 2502 and includes obtaining, at a view interpolation module of the client application program, a current rendered rendering surface representing a current view of a scene for a first eye of a user and a previously rendered rendering surface representing a past view of the scene for a second eye of the user. It will be appreciated that the current rendered rendering surface and the previously rendered rendering surface may be rendered with a wider field of view than a viewport of the client device. Additionally, the rendering surfaces may be many kinds of shapes such as cubemaps, clipped cubemaps, edges, or tetrahedrons as a few examples. It will be further appreciated that the first eye being rendered at each rendering pass may alternate between the left eye of the user and the right eye of the user. In one embodiment, this rendering process may be completed on a server device that sends the rendered rendering surfaces to a client device that enacts method 2500. In another embodiment, the server device may render predicted scene according to method 1400, and send the rendered rendering surfaces of predicted scenes for a predicted viewpoint of either a left eye or a right eye of the user to the client device over a network.
Next, the view interpolation module synthesizes a current second eye viewport representing a current view of the scene for the second eye of the user based on the current rendered rendering surface and the previously rendered rendering surface. This synthesis is accomplished in the remaining steps of method 2500. Method 2500 proceeds to 2504 and includes reprojecting from the current rendered rendering surface to the current second eye viewport.
Method 2500 advances to 2506 and includes detecting one or more visual holes in the current second eye viewport. In one embodiment, detecting one or more visual holes in the current second eye viewport is based on a detected depth disparity in a depth map for the current rendered rendering surface, the depth map being obtained at the view interpolation module of the client application program. Next, method 2500 interpolates one or more pixel data items for the one or more visual holes based on the previously rendered rendering surface.
Method 2500 interpolates one or more pixel data items by proceeding to 2508 and includes reprojecting from the previously rendered rendering surface to the current second eye viewport. Method 2500 advances to 2510 and includes detecting one or more remaining visual holes in the current second eye viewport after reprojection from the previously rendered rendering surface. In one embodiment, detecting one or more remaining visual holes is based on a calculated rate of change in user viewpoint that is calculated based on a stream of user input received from the user. Next, method 2500 fills the one or more remaining visual holes based on the previously rendered rendering surface.
Method 2500 fills the one or more remaining visual holes by proceeding to 2512 and includes determining a pixel location in the current second eye viewport for each of the one or more remaining visual holes. Method 2500 advances to 2514 and includes copying pixel data from pixel locations in the previously rendered rendering surface corresponding to the pixel location in the current second eye viewport of each of the one or more remaining visual holes. In one embodiment, the pixel data is directly copied from the previously rendered rendering surface into the corresponding location in the current second eye viewport.
In some embodiments, method 2500 is combined with method 2000 for embodiments of the HMD client device that receive predictive rendered content from the server device. In these embodiments, the server device renders content according to method 1400, alternating between a left eye and right eye of the user. The HMD client device receives a current rendered rendering surface of a predicted scene having a predicted user viewpoint for a first eye of a user and obtains, at a view interpolation module of the client application program, a previously rendered rendering surface of a previously predicted scene having a previously predicted user viewpoint for a second eye of the user. In some embodiments, the previously rendered rendering surface is obtained from memory HMD client device.
Next, the HMD client device receives, from the user input device, a subsequent user navigation input in the stream of user input. Then, determines an actual user viewpoint for the first eye and an actual user viewpoint for the second eye based on the subsequent user navigation input. The HMD client device then proceeds to determine a user viewpoint misprediction based on the predicted user viewpoint for the first eye and the actual user viewpoint for the first eye. It will be appreciated that the first eye in this embodiment is the eye that the server device rendered to the rendering surface, and was received by the HMD client device as a current rendered rendering surface. If there is a user viewpoint misprediction, the HMD client device cannot simply project the rendered rendering surface to the viewport for the first eye, but needs to use view interpolation to reconstruct a first viewport for the actual user viewpoint for the first eye of the user from the current rendered rendering surface. To reconstruct the first viewport for the actual user viewpoint, the HMD client device may use method 2000 or a combination of steps contained in method 2000.
Next, the HMD client device proceeds to synthesize a second viewport for the actual user viewpoint for the second eye of the user based on the current rendered rendering surface and the previously rendered rendering surface. To synthesize the second viewport for the actual user viewpoint, the HMD client device may use method 2500 or a combination of step contained in method 2500. It will be appreciated that the actual user viewpoint for the second eye of the user will be translated to the left or right of the actual user viewpoint for the first eye by the interpupillary distance. It will be further appreciated that the previously predicted viewpoint for the second eye of the user of the previously rendered rendering surface was predicted for a past user viewpoint for the second eye, and not for the current user viewpoint for the second eye. However, method 2500 can still be applied using the previously rendered rendering surface and the current rendered rendering surface.
After reconstructing the first viewport for the first eye and synthesizing the second viewport for the second eye, the HMD client device proceeds to display the first viewport and the second viewport on the stereoscopic display device. It will be appreciated that the above embodiment that combines methods 2000 and 2500 is not limited to HMD client device. The above embodiment can also be applied to a desktop computer with a stereoscopic screen as another non-limiting example.
In some embodiments, the methods and processes described herein may be tied to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer-application program or service, an application-programming interface (API), a library, and/or other computer-program product.
Computing system 10 includes a logic machine 18 on client device 12 and a logic machine 28 on server device 14. Computing system 10 also includes and a storage machine 22 on client device 12 and a storage machine 32 on server device 14. Computing system 10 may optionally include a display subsystem 70, input subsystem 24, and/or other components not shown in
Logic machines 18 and 28 include one or more physical devices configured to execute instructions. For example, the logic machines may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.
The logic machines may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machines may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the logic machines optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the logic machines may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration.
Storage machines 22 and 32 include one or more physical devices configured to hold instructions executable by the logic machines to implement the methods and processes described herein. When such methods and processes are implemented, the state of storage machines 22 and 32 may be transformed—e.g., to hold different data.
Storage machines 22 and 32 may include removable and/or built-in devices. Storage machines 22 and 32 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), among others. Storage machines 22 and 32 may include volatile, nonvolatile, dynamic, static, read/write, read-only, random-access, sequential-access, location-addressable, file-addressable, and/or content-addressable devices.
It will be appreciated that storage machines 22 and 32 include one or more physical devices. However, aspects of the instructions described herein alternatively may be propagated by a communication medium (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for a finite duration.
Aspects of logic machines 18 and 28 and storage machines 22 and 32 may be integrated together into one or more hardware-logic components. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.
The terms “module,” “program,” and “engine” may be used to describe an aspect of computing system 10 implemented to perform a particular function. In some cases, a module, program, or engine may be instantiated via logic machines 18 and 28 executing instructions held by storage machines 22 and 32. It will be understood that different modules, programs, and/or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and/or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.
It will be appreciated that a “service”, as used herein, is an application program executable across multiple user sessions. A service may be available to one or more system components, programs, and/or other services. In some implementations, a service may run on one or more server-computing devices.
When included, display subsystem 70 may be used to present a visual representation of data held by storage machines 22 and 32. This visual representation may take the form of a graphical user interface (GUI). As the herein described methods and processes change the data held by the storage machine, and thus transform the state of the storage machine, the state of display subsystem 70 may likewise be transformed to visually represent changes in the underlying data. Display subsystem 70 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic machines 18 and 28 and/or storage machines 22 and 32 in a shared enclosure, or such display devices may be peripheral display devices.
When included, input subsystem 24 may comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may comprise or interface with selected natural user input (NUI) componentry. Such componentry may be integrated or peripheral, and the transduction and/or processing of input actions may be handled on- or off-board. Example NUI componentry may include a microphone for speech and/or voice recognition; an infrared, color, stereoscopic, and/or depth camera for machine vision and/or gesture recognition; a head tracker, eye tracker, accelerometer, and/or gyroscope for motion detection and/or intent recognition; as well as electric-field sensing componentry for assessing brain activity.
It will be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated and/or described may be performed in the sequence illustrated and/or described, in other sequences, in parallel, or omitted. Likewise, the order of the above-described processes may be changed.
The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.
The subject matter of the present disclosure is further described in the following paragraphs. One aspect provides a client device for reconstructing a viewport from a rendered rendering surface of a predicted user viewpoint, comprising a user input device that receives a stream of user input, a display device, a processor, and a storage device holding instructions for a client application program, executable by the processor to: receive, from a server device over a network, a rendered rendering surface of a predicted scene having a predicted user viewpoint, wherein the rendered rendering surface has a wider field of view than a viewport of the client application program and includes a plurality of non-displayed rendered pixel data items in regions not displayed in the viewport of the client application program centered at the predicted user viewpoint, receive, from the user input device, a subsequent user navigation input in the stream of user input, determine an actual user viewpoint based on the subsequent user navigation input, determine a user viewpoint misprediction based on the predicted user viewpoint and the actual user viewpoint, reconstruct a viewport for the actual user viewpoint from the rendered rendering surface, wherein the viewport for the actual user viewpoint includes a portion of the plurality of non-displayed rendered pixel data items, and display the viewport on the display device. The client device may additionally or alternatively include wherein reconstructing the viewport for the actual user viewpoint includes: reprojecting from a portion of the rendered rendering surface to the viewport, and interpolating one or more pixel data items to fill one or more visual holes in the reconstructed viewport. The client device may additionally or alternatively include wherein interpolating one or more pixel data items includes: projecting a screen-space grid mesh onto a depth buffer for a depth map corresponding to the rendered rendering surface of the predicted scene, wherein the screen-space grid mesh includes an edge that extends between a foreground object and a background object in the predicted scene, casting a ray from the actual user viewpoint to the screen-space grid mesh, determining that the ray intersects with the edge in the screen-space grid mesh, and calculating a pixel color for the interpolated one or more pixel data items based on a color of the background object in the predicted scene. The client device may additionally or alternatively include wherein the edge in the screen-space grid mesh projected onto the depth buffer is dilated by one or more pixels in width. The client device may additionally or alternatively include wherein interpolating one or more pixel data items includes reprojecting from a portion of a previously rendered viewport. The client device may additionally or alternatively include wherein the rendered rendering surface is a composite of a plurality of depth layers. The client device may additionally or alternatively include wherein a number of depth layers in the plurality of depth layers does not exceed an adjustable threshold value. The client device may additionally or alternatively include holding further instructions to: receive, from the server device, a plurality of alternative impulse input sequences and a plurality of rendered rendering surfaces corresponding to the plurality of alternative impulse input sequences, receive, from the user input device, a subsequent impulse input sequence from the stream of user input, select an alternative impulse input sequence from the plurality of alternative impulse input sequences based on the subsequent impulse input sequence, and wherein the viewport for the actual user viewpoint is reconstructed from the rendered rendering surface corresponding to the selected speculative impulse input sequence. The client device may additionally or alternatively include wherein selecting an alternative impulse input sequence includes determining which alternative impulse input sequence of the plurality of alternative impulse input sequences matches the subsequent impulse input sequence. The client device may additionally or alternatively include wherein selecting an alternative impulse input sequence further includes time-shifting each impulse input of the subsequent impulse input sequence to a nearest subsampled tick in the plurality of alternative impulse input sequences.
Another aspect provides a method for reconstructing a viewport from a rendered rendering surface of a predicted user viewpoint comprising at a client device executing a client application program: receiving, from a server device over a network, a rendered rendering surface of a predicted scene having a predicted user viewpoint, wherein the rendered rendering surface has a wider field of view than a viewport of the client application program and includes a plurality of non-displayed rendered pixel data items in regions not displayed in the viewport of the client application program centered at the predicted user viewpoint, receiving a subsequent user navigation input in a stream of user input, determining an actual user viewpoint based on the subsequent user navigation input, determining a user viewpoint misprediction based on the predicted user viewpoint and the actual user viewpoint, and reconstructing a viewport for the actual user viewpoint from the rendered rendering surface, wherein the viewport for the actual user viewpoint includes a portion of the plurality of non-displayed rendered pixel data items. The method may additionally or alternatively include wherein reconstructing the viewport for the actual user viewpoint includes: reprojecting from a portion of the rendered rendering surface to the viewport, and interpolating one or more pixel data items to fill one or more visual holes in the reconstructed viewport. The method may additionally or alternatively include wherein interpolating one or more pixel data items includes: projecting a screen-space grid mesh onto a depth buffer for a depth map corresponding to the rendered rendering surface of the predicted scene, wherein the screen-space grid mesh includes an edge that extends between a foreground object and a background object in the predicted scene, casting a ray from the actual user viewpoint to the screen-space grid mesh, determining that the ray intersects with the edge in the screen-space grid mesh, and calculating a pixel color for the interpolated one or more pixel data items based on a color of the background object in the predicted scene. The method may additionally or alternatively include wherein the edge in the screen-space grid mesh projected onto the depth buffer is dilated by one or more pixels in width. The method may additionally or alternatively include wherein interpolating one or more pixel data items includes reprojecting from a portion of a previously rendered viewport. The method may additionally or alternatively include wherein the rendered rendering surface is a composite of a plurality of depth layers, wherein a number of depth layers in the plurality of depth layers does not exceed an adjustable threshold value. The method may additionally or alternatively include receiving, from the server device, a plurality of alternative impulse input sequences and a plurality of rendered rendering surfaces corresponding to the plurality of alternative impulse input sequences, receiving a subsequent impulse input sequence in the stream of user input, selecting an alternative impulse input sequence from the plurality of alternative impulse input sequences based on the subsequent impulse input sequence, and wherein the viewport for the actual user viewpoint is reconstructed from the rendered rendering surface corresponding to the selected alternative impulse input sequence. The method may additionally or alternatively include wherein selecting an alternative impulse input sequence further includes time-shifting each impulse input of the subsequent impulse input sequence to a nearest subsampled tick of the plurality of alternative impulse input sequences.
Another aspect provides a client device for reconstructing a viewport from a rendered rendering surface of a predicted user viewpoint, comprising, a user input device that receives a stream of user input, a display device, a processor, and a storage device holding instructions for a client application program executable by the processor to: receive, from a server device over a network, a plurality of alternative impulse input sequences and a corresponding plurality of rendered rendering surfaces of a predicted scene having a predicted user viewpoint, wherein the plurality of rendered rendering surfaces have a wider field of view than a viewport of the client application program and each include a plurality of non-displayed rendered pixel data items in regions not displayed in the viewport of the client application program centered at the predicted user viewpoint, receive, from the user input device, a subsequent user navigation input and a subsequent impulse input sequence in the stream of user input, select an alternative impulse input sequence from the plurality of alternative impulse input sequences based on the subsequent impulse input sequence, determine an actual user viewpoint based on the subsequent user navigation input, determine a user viewpoint misprediction based on the predicted user viewpoint and the actual user viewpoint, select a rendered rendering surface from the plurality of rendered rendering surfaces that corresponds to the selected alternative impulse input sequence, reconstruct a viewport for the actual user viewpoint from the selected rendered rendering surface, wherein the viewport for the actual user viewpoint includes a portion of the plurality of non-displayed rendered pixel data items for the selected rendered rendering surface, and display the viewport on the display device. The client device may additionally or alternatively include wherein reconstructing the viewport for the actual user viewpoint includes: reprojecting from a portion of the selected rendered rendering surface to the viewport, and interpolating one or more pixel data items to fill one or more visual holes in the reconstructed viewport.
