Patent Grant
12182925
The present disclosure relates generally to eXtended Reality (XR) applications, and more particularly, to an improved split-rendering process for reducing or eliminating undesirable visual effects, such as judder, in XR applications.
The quality of a video image is very important in a variety of applications. Often, however, various undesirable “visual artifacts” can decrease the quality of a video image. One example of an undesirable “visual effect” is judder.
Judder is a perceptual effect. Generally defined, judder is a combination of smearing and strobing that can significantly reduce the visual quality on a Virtual Reality (VR) headset (e.g., a head mounted display—HMD). It usually occurs when a game that a user is playing cannot maintain a desired frame rate. In order to correct for this, the game renders the same image twice instead of rendering a new image. However, as the user moves his/her head and eyes, the light from the subsequently rendered images fall on a different part of the user's retina. This results in the user seeing a double image.
There are some techniques that have been developed to diminish the effect of judder. One such technique is referred to as “asynchronous time-warp,” which generates intermediate frames in situations when the game is unable maintain frame rate. This technique warps the rendered image before sending it to a display (e.g., the HMD worn by the user) thereby correcting for head motion that occurs after a scene is rendered. In essence, this technique helps to reduce perceived latency.
A more basic version of the asynchronous time-warp technique is referred to as “orientation-only time-warp.” With this technique, only the rotational movement of the user's head is corrected for based on HMD rotational data.
Another technique for reducing judder is referred to as the “positional time-warp” technique. In general, the positional time-warp technique is an extension of the broader time-warp techniques, and thus, also reduces perceived latency. However, the positional time-warp technique can also compensate for the user's translational head movement by displacing parts of the image depending on depth information.
Yet another technique, referred to as “asynchronous space-warp” technique, generates extrapolated frames from previous frames generated by a Virtual Reality (VR) application based on sensor data received from the user's HMD. This technique is generally a frame-rate smoothing technique that can reduce the amount of CPU/GPU time that is required to produce similar output from the same content. The difference between the asynchronous space-warp technique and the so-called time-warp techniques is that the asynchronous space-warp technique generates a new frame where the time-warp techniques shift an existing frame based on extrapolation using movement information related to the movement. Such movement includes, but is not limited to, the movement of a game character (or any object in a game being rendered to a user), a camera that is capturing images to be rendered to the user, a touch-sensitive controller, and the user's own movement. Currently, time-warp and space-warp techniques can be used in parallel.
A method for split rendering is described in U.S. Pat. No. 10,446,119 entitled “Method for Supporting Multiple Layers in Split Rendering.” In this method, a rendered image is split into a number of different graphics layers known as “Z-layers” to reduce video stream bandwidth. Layers that change less frequently are sent to a decoder with lower framerate, while layers that change more frequently are sent to the decoder with a higher framerate.
Recently developed XR applications (i.e., both Virtual Reality (VR) and Augmented Reality (AR) applications) place critical rendering processes in a remote cloud processing environment, such as an Edge Cloud (EC), for example. Such schemes, which are commonly referred to as “remote rendering,” place the processing power needed by these applications close to the client location (e.g., an HMD), thereby working to improve response times and preserve bandwidth. Some example XR applications that utilize the EC are those related to gaming and training.
Embodiments of the present disclosure provide an improved split-rendering process for reducing undesirable visual artifacts, such as judder, for example, in a video image having one or more frames.
In a first aspect, a method for reducing undesirable visual artifacts in a video image having one or more composite frames is implemented by a server node in an Edge Data Network (EDN) and comprises creating a plurality of graphics layers comprising one or more objects by rendering the one or more objects to a number of different Z-order planes, grouping the plurality of graphics layers into graphics layer groups based on the Z-order of the graphics layers and motion information for the one or more objects, encoding the graphics layer groups separately into a composite video frame such that each graphics layer group is separately decodable and the composite video frame includes the motion information, and sending the composite video frame to a decoder.
In a second aspect, the method for reducing undesirable visual artifacts in a video image having one or more composite frames is implemented by a client device. In this aspect, each composite video frame includes a plurality of graphics layers arranged in graphics layer groups, motion information for one or more objects in the plurality of graphics layers, and Z-order information associated with the plurality of graphics layers. The method comprises the client device extracting the motion information from a composite video frame, and separately decoding each graphics layer group in the composite video frame into the plurality of graphics layers. Then, for each of the one or more objects in the plurality of graphics layers, the method comprises the client device compensating for motion of the object based on the motion information for the object and compensating the object for movement of a user's head based on the Z-order information, and rendering the plurality of graphics layers on a display to the user.
In a third aspect, the present disclosure provides a server node in an Edge Data Network (EDN) for reducing undesirable visual artifacts in a video image having one or more frames. In this aspect, the server node is configured to create a plurality of graphics layers comprising one or more objects by rendering the one or more objects to a number of different Z-order planes, group the plurality of graphics layers into graphics layer groups based on the Z-order of the graphics layers and motion information for the one or more objects, encode the graphics layer groups separately into a composite video frame such that each graphics layer group is separately decodable and the composite video frame includes the motion information, and send the composite video frame to a decoder.
In a fourth aspect, the present disclosure provides a server node in an Edge Data Network (EDN) for reducing undesirable visual artifacts in a video image having one or more frames. In this aspect, the server node comprises communications interface circuitry and processing circuitry. The communications interface circuitry is configured to communicate with a client device operatively connected to the EDN. The processing circuitry configured to create a plurality of graphics layers comprising one or more objects by rendering the one or more objects to a number of different Z-order planes, group the plurality of graphics layers into graphics layer groups based on the Z-order of the graphics layers and motion information for the one or more objects, encode the graphics layer groups separately into a composite video frame such that each graphics layer group is separately decodable and the composite video frame includes the motion information, and send the composite video frame to a decoder.
In a fifth aspect, the present disclosure provides a non-transitory computer-readable storage medium having a computer program comprising executable instructions stored thereon that, when executed by a processing circuitry of a server node in an Edge Data Network (EDN), causes server node to create a plurality of graphics layers comprising one or more objects by rendering the one or more objects to a number of different Z-order planes, group the plurality of graphics layers into graphics layer groups based on the Z-order of the graphics layers and motion information for the one or more objects, encode the graphics layer groups separately into a composite video frame such that each graphics layer group is separately decodable and the composite video frame includes the motion information, and send the composite video frame to a decoder.
In a sixth aspect, the present disclosure provides a client device for reducing undesirable visual artifacts in a video image having one or more frames. In this aspect, the client is configured to create a plurality of graphics layers comprising one or more objects by rendering the one or more objects to a number of different Z-order planes, group the plurality of graphics layers into graphics layer groups based on the Z-order of the graphics layers and motion information for the one or more objects, encode the graphics layer groups separately into a composite video frame such that each graphics layer group is separately decodable and the composite video frame includes the motion information, and send the composite video frame to a decoder.
In a seventh aspect, the present disclosure provides a client device for reducing undesirable visual artifacts in a video image having one or more frames. In this aspect, the client device comprises communications interface circuitry and processing circuitry. The communications interface circuitry is configured to communicate with a client device operatively connected to the EDN. The processing circuitry is configured to create a plurality of graphics layers comprising one or more objects by rendering the one or more objects to a number of different Z-order planes, group the plurality of graphics layers into graphics layer groups based on the Z-order of the graphics layers and motion information for the one or more objects, encode the graphics layer groups separately into a composite video frame such that each graphics layer group is separately decodable and the composite video frame includes the motion information, and send the composite video frame to a decoder.
In an eighth aspect, the present disclosure provides a non-transitory computer-readable storage medium having a computer program comprising executable instructions stored thereon that, when executed by a processing circuit of a client device, causes client device to extract the motion information from a composite video frame, separately decode each graphics layer group in the composite video frame into the plurality of graphics layers, for each of the one or more objects in the plurality of graphics layers compensate for motion of the object based on the motion information for the object and compensate the object for movement of a user's head based on the Z-order information, and render the plurality of graphics layers on a display to the user.
Time-warp technology is very useful in helping to prevent or minimize some undesirable side effects (e.g., motion sickness) experienced by users using HMDs. This is true even when the XR applications render the graphics locally. However, when a client device like an HMD is unable to maintain a given framerate, time-warp technologies has some well-known problems.
For example, asynchronous time-warp techniques perform 2D transformation. Therefore, asynchronous time-warp techniques can only correct for the changes caused by the rotational movement of a user's head. They do not compensate for translational head movement, thereby causing an undesirable visual artifact known as positional judder. This effect is generally very noticeable by users in spaces with near field objects.
Positional time-warp techniques, however, are able to compensate for the translational movement of the user's head. Although this technique does reduce positional judder, it also uses a single image as a source, which can cause disocclusion artifacts in the image. Disocclusion is a term commonly understood to those in the graphics art, and is used to describe a situation where a first object in the image, previously occluded by a second object in the image, becomes visible. More particularly, as an object in a video image moves, the processing logic for whichever technique is being employed (e.g., the positional time-warp technique or the asynchronous space warp technique) needs something to fill the space the object leaves behind (i.e., the space that was previously occluded by the object before it moved). However, these techniques are not capable of knowing what is going to fill the previously occluded space. Therefore, they stretch or “warp” the part of the image that was not occluded in order to fill the previously occluded space. Regardless of the particular technique used, however, moving objects still causes judder artifacts because the warped image is created without knowledge of the motion of the object.
One notable aspect of remote rendering by XR applications is that of “interaction latency.” Interaction latency is defined as the time between when a user presses a button or moves the HMD or controller, and when the resultant video image is updated on the display of the HMD. Typically, interaction latency is higher for XR applications with local rendering (generally, on the order of 80-100 ms higher). This higher interaction latency, however, indicates the degradation of the user's experience. Hence, the motivation to use the Edge Cloud (EC) for remote rendering.
The “motion to photon” latency requirement (i.e., the time between when the user moves his/her head and the time when the resultant video image is updated on the display of the HMD), is much shorter (on the order of about 20 ms). This means that for remote rendering, time-warp techniques must be applied to compensate for head movement, even in normal scenarios (that is, even in situations where there are no delayed or lost frames). Therefore, minimizing the problems related to time-warp techniques is crucial for XR applications with remote rendering.
The improved split-rendering process of the present disclosure addresses these issues. In particular, embodiments of the present disclosure configure a server node and a client node to reduce undesirable visual artifacts in a video image having one or more frames. One such undesirable visual artifact is “judder.” More particularly, a graphics application such as a game engine, for example, executes on the server node, which is disposed in an Edge Data Network (EDN). The game engine generates graphics layers from 3D objects, and augments the layers with the Z-layer information and motion information, such as the velocity and direction of motion of the 3D objects. The server node then encodes the graphics layers into a video stream such that every video frame of the stream is a composite video frame of the graphics layers. The encoded video stream is then sent to a decoder, such as the decoder in a client device.
Upon receipt, the client device (e.g., an HMD worn by a user) extracts the motion information from a composite video frame and decodes the composite video frame. The client device compensates the positions of the graphics layers using the Z-layer information and the motion information, and applies a selected positional time-warp algorithm to compensate for the translational and rotational movement of the user's head. The resultant layers are then combined into a video image and displayed to the user on the HMD.
The improved split-rendering technique described in the following embodiments provides advantages and benefits not achieved by conventional split-rendering methods. For example, the improved split-rendering technique of the present embodiments improves the quality of rendering and diminishes or eliminates problems associated with “time-warp” judder. This is especially useful for applications using remote rendering.
In another benefit, the improved positional time-warp of the present embodiments use the graphic layer information to reduce or eliminate the undesirable effects of disocclusion. Additionally, the present embodiments use the motion information added to graphics layers to diminish judder caused by the moving objects. This eliminates the need to perform space-warp processing on a video image, thereby reducing the processing at the client device in cases where remote rendering is performed.
It should be noted that the following aspects of the present disclosure are discussed in terms of reducing or eliminating an undesirable visual effect known as “judder.” However, this is merely for illustrative purposes and ease of discussion. Those of ordinary skill in the art will readily appreciate that the present disclosure is not limited solely to the reduction or elimination of judder, but rather, may be applied to reduce or limit undesirable visual effects in addition to, or in lieu of, judder.
Turning now to the drawings,
The access network 12 may be any type of communications network (e.g., WiFi, ETHERNET, Wireless LAN (WLAN), 3G, LTE, etc.), and functions to connect subscriber devices, such as HMD 18, to one or more service provider nodes, such as server node 16. The cloud network 14 provides such subscriber devices with “on-demand” availability of computer resources (e.g., memory, data storage, processing power, etc.) without requiring the user to directly, actively manage those resources. According to embodiments of the present disclosure, such resources include, but are not limited to, one or more XR applications being executed on server node 16. The XR applications may comprise, for example, gaming applications and/or simulation applications used for training.
In general, one or more sensors (not shown) on HMD 18 measure the translational and/or rotational movement of the user's head as the user views images rendered by server node 16 on HMD 18. Signals representing the detected and measured movement are then sent to server node 16. Upon receipt, server node 16 utilizes those signals to compensate the video images for the user's movement, and sends the compensated video to HMD 18. To help reduce and/or eliminate latency associated with the communications between HMD 18 and server node 116, the present embodiments place server node 16 in an Edge Data Network (EDN).
Generally, the server node 16 functions to generate ordered graphics layer groups based on the Z-order of the graphics layers and motion information. In more detail, the XR application executing on server node 16 creates a plurality of graphics layers, each of which comprises one or more 3D objects, by rendering those 3D objects to a number of different, Z-order planes (box 22). For example, in one embodiment, server node 16 renders the 3D objects to the Z-order planes based on their Z-order distance from a rendering camera. Server node 16 then uses well-known techniques to determine the motion information for the objects (i.e., the velocity and the direction of movement of the 3D objects) (box 24), and assigns the determined motion information for each 3D object to the graphics layer corresponding to that 3D object (box 26). Once assigned, server node 16 determines whether or not to group the plurality of graphics layers into graphics layer groups.
In this embodiment, for example, server node 16 determines whether to group the different graphics layers based on the Z-order of the graphics layer and the motion information associated with that graphics layer. To accomplish this, one embodiment of the present disclosure configures server node 16 to compare the motion information for the different graphics layers (box 28). Graphics layers on the same Z-order plane are grouped together if they have the same motion information (box 30). Graphics layers on the same Z-order plane having different motion information, however, are grouped together, but rather, are put into separate groups (box 32).
Once the graphics layers are grouped, server node 16 assembles a composite video frame from the graphics layer groups. Particularly, in this embodiment, server node 16 encodes the graphics layer groups separately into the composite video frame ensuring that each graphics layer group is separately decodable independently of the other graphics layer groups (box 34), and then adds the motion information for the graphics layer groups as meta-information to the composite video frame (box 36). In one embodiment, for example, server node 16 adds the motion information to the beginning of the composite video frame so that a client device can extract the information upon receipt of the composite video frame. Once the composite video frame has the motion information, server node 16 sends the composite video frame to a client device having a decoder, such as HMD 18, for example (box 38).
In some embodiments, the motion information comprises a velocity vector and a direction of motion vector for each of the one or more objects.
Additionally, in some embodiments, the meta-information comprises an identity for each graphics layer in the composite video frame, the motion information for the objects in each graphics layer, and a position of each graphics layer in the composite video frame.
In some embodiments, the one or more objects are 3D objects in a video image.
In some embodiments, the number of Z-order planes is predetermined.
As seen in
Next, for each of the objects in the plurality of graphics layers, HMD 18 compensates for the motion of the object based on the motion information for the object, and compensates the object for translational movement of the user's head based on the Z-order information (box 50). For example, one embodiment of the present disclosure applies a positional time-warp technique to the objects. Conventional positional time-warp techniques can cause disocclusion artifacts in cases where there is no information on the image content that becomes visible after object displacement (i.e., once a previously occluded object becomes visible). However, the positional time-warp technique of the present embodiments utilizes the Z-layer information to shift the graphics layers in the Z-plane based on the user's head movement. In this manner, there is always image information on the graphics layers that become visible (i.e., the objects that were previously occluded), thereby greatly reducing or eliminating disocclusion. Additionally, judder is also effectively reduced or eliminated because the present embodiments correct the position of the objects in the graphics layers before applying the positional time-warp.
Once compensated, the HMD 18 renders the plurality of graphics layers as a 2D image on a display of the HMD 18 for the user.
In some embodiments, compensating the object for movement of a user's head based on the Z-order information compensates for the translational and/or the rotational movement of the user's head.
An apparatus can perform any of the methods herein described by implementing any functional means, modules, units, or circuitry. In one embodiment, for example, the apparatuses comprise respective circuits or circuitry configured to perform the steps shown in the method figures. The circuits or circuitry in this regard may comprise circuits dedicated to performing certain functional processing and/or one or more microprocessors in conjunction with memory. For instance, the circuitry may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include Digital Signal Processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory may include program instructions for executing one or more telecommunications and/or data communications protocols as well as instructions for carrying out one or more of the techniques described herein, in several embodiments. In embodiments that employ memory, the memory stores program code that, when executed by the one or more processors, carries out the techniques described herein.
The communications interface circuitry 96 comprises network interface circuitry for communicating with other nodes in and/or communicatively connected to communication network 10. Such nodes include, but are not limited to, one or more network nodes and/or functions disposed in cloud network 14, access network 12, and one or more client devices, such as HMD 18.
Processing circuitry 90 controls the overall operation of server node 16 and is configured to perform the steps of method 20 shown in
Memory circuitry 92 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuitry 90 for operation. Memory circuitry 92 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory circuitry 92 stores computer program 98 comprising executable instructions that configure the processing circuitry 90 to implement method 20 shown in
The user interface 102 comprises one or more buttons, actuators, and software-driven controls that facilitates a user's ability to interact with and control the operation of HMD 18. The display 106 comprises one or more display devices configured to display rendered video images to a user. In this regard, the display devices may comprise, for example, Cathode Ray Tubes (CRTs), Liquid Crystal Displays (LCDs), Liquid Crystal on Silicon (LCos) displays, and Light-Emitting Diodes (LEDs). Other types of display devices not explicitly described herein may also be possible.
The communications interface circuitry 108 comprises network interface circuitry for communicating with other nodes in and/or communicatively connected to communication network 10. Such nodes include, but are not limited to, one or more network nodes and/or functions disposed in cloud network 14, such as server node 16, and access network 12. Although not specifically shown in the figures, some embodiments of the present disclosure configure HMD 18 to communicate with a local computing device that, in turn, is connected to server node 14 via access network 12.
Processing circuitry 100 controls the overall operation of HMD 18 and is configured to perform the steps of method 40 shown in
Memory circuitry 104 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuitry 100 for operation. Memory circuitry 104 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory circuitry 104 stores computer program 110 comprising executable instructions that configure the processing circuitry 100 to implement method 40 shown in
The rendering unit/module 120 is configured to create a plurality of graphics layers comprising one or more objects by rendering the one or more objects to a number of different Z-order planes. The graphics layer grouping unit/module 122 is configured to group the plurality of graphics layers into graphics layer groups based on the Z-order of the graphics layers and motion information for the one or more objects. The encoding unit/module 124 is configured to encode the graphics layer groups separately into a composite video frame such that each graphics layer group is separately decodable, and the composite video frame includes the motion information. The communications unit/module 126 is configured to send the encoded composite video frame to HMD 18, and further, to receive sensor signals and indications regarding the user's head movement from HMD 18.
The motion information extraction unit/module 130 is configured to extract the motion information from a composite video frame received from a server node 16. The composite video frame comprises one or more graphics layer groups, as described above. The decoding unit/module 132 is configured to separately decode each graphics layer group in the composite video frame into a plurality of graphics layers. The motion compensation unit/module 134 is configured to compensate for motion of the objects in the graphics layers based on the motion information for the objects. The translational movement unit/module 136 is configured to compensate the object for movement of a user's head based on the Z-order information. The rendering unit/module 138 is configured to render the plurality of graphics layers on a display to the user, and the communications unit/module 140 is configured to receive the composite video frames from server node 14, and to send sensor signals and indications regarding the user's head movement to the server node 14.
Those skilled in the art will also appreciate that embodiments herein further include corresponding computer programs. A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.
Embodiments further include a carrier containing such a computer program. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium.
In this regard, embodiments herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform as described above.
Embodiments further include a computer program product comprising program code portions for performing the steps of any of the embodiments herein when the computer program product is executed by a computing device. This computer program product may be stored on a computer readable recording medium.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2020/071947 | 8/5/2020 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2022/028684 | 2/10/2022 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 10446119 | Lakshmikantha | Oct 2019 | B1 |
| 20100127968 | Kramer | May 2010 | A1 |
| 20140359508 | Otero Diaz | Dec 2014 | A1 |
| 20160018896 | Kramer | Jan 2016 | A1 |
| 20160070755 | Kramer | Mar 2016 | A1 |
| 20170346792 | Nataros | Nov 2017 | A1 |
| 20170346864 | Nataros | Nov 2017 | A1 |
| 20180048888 | Zhao | Feb 2018 | A1 |
| 20230260191 | Mihály | Aug 2023 | A1 |
| Number | Date | Country |
|---|---|---|
| 2019212643 | Nov 2019 | WO |
| 2020068406 | Apr 2020 | WO |
| Entry |
|---|
| Reinert B, Kopf J, Ritschel T, Cuervo E, Chu D, Seidel HP. Proxy-guided image-based rendering for mobile devices. InComputer Graphics Forum Oct. 2016 (vol. 35, No. 7, pp. 353-362). |
| Number | Date | Country | |
|---|---|---|---|
| 20230260191 A1 | Aug 2023 | US |
