Patent Application
20210360330
This disclosure relates to storage and transport of encoded media data.
Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio telephones, video teleconferencing devices, and the like. Digital video devices implement video compression techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263 or ITU-T H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 (also referred to as High Efficiency Video Coding (HEVC)), and extensions of such standards, to transmit and receive digital video information more efficiently.
After video data and other media data have been encoded, the media data may be packetized for transmission or storage. The media data may be assembled into a video file conforming to any of a variety of standards, such as the International Organization for Standardization (ISO) base media file format and extensions thereof.
In general, this disclosure describes techniques related to evaluating configuration of various components involved in performing split rendering for interactive media, such as online cloud video gaming. These techniques may be implemented by a computing device or a system of computing devices. These techniques include an evaluation framework for evaluating efficacy of these devices and configuration of these devices. For example, different portions of the system may be evaluated in different ways to determine different configuration settings. In particular, these techniques include evaluations at the video level, slice level, and packet level. Evaluations at the packet level can be used to determine whether a corresponding slice has been properly received. Evaluations at the slice level can be used to determine qualities of the slices and corresponding frames including the slices. Evaluations at the video level can be used to determine whether overall sets of configuration parameters for all components of the system are adequate.
In one example, a method of processing media data includes receiving tracking and sensor information from an extended reality (XR) client device; generating scene data using the tracking and sensor information, the scene data comprising one or more video frames; encoding the video frames to form encoded video frames; performing a radio access network (RAN) simulation of delivering the encoded video frames via a radio access network; decoding the encoded video frames delivered according to the RAN simulation to form decoded video frames; calculating values representing individual frame quality for each of the video frames from the generated one or more video frames and the decoded video frames; and determining an overall quality value from the values representing the individual frame quality for each of the video frames.
In another example, a device for processing media data includes a memory configured to store video data; and one or more processors implemented in circuitry and configured to: receive tracking and sensor information from an extended reality (XR) client device; generate scene data using the tracking and sensor information, the scene data comprising one or more video frames of the video data; encode the video frames to form encoded video frames; perform a radio access network (RAN) simulation of delivering the encoded video frames via a radio access network; decode the encoded video frames delivered according to the RAN simulation to form decoded video frames; calculate values representing individual frame quality for each of the video frames from the generated one or more video frames and the decoded video frames; and determine an overall quality value from the values representing the individual frame quality for each of the video frames.
In another example, a computer-readable storage medium has stored thereon instructions that, when executed, cause a processor to: receive tracking and sensor information from an extended reality (XR) client device; generate scene data using the tracking and sensor information, the scene data comprising one or more video frames; encode the video frames to form encoded video frames; perform a radio access network (RAN) simulation of delivering the encoded video frames via a radio access network; decode the encoded video frames delivered according to the RAN simulation to form decoded video frames; calculate values representing individual frame quality for each of the video frames from the generated one or more video frames and the decoded video frames; and determine an overall quality value from the values representing the individual frame quality for each of the video frames.
The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.
In some examples, XR scene generation unit 112 may correspond to an interactive media entertainment application, such as a video game, which may be executed by one or more processors implemented in circuitry of XR server device 110. XR viewport pre-rendering rasterization unit 114 may format scene data generated by XR scene generation unit 112 as pre-rendered two-dimensional (2D) media data (e.g., video data) for a viewport of a user of XR client device 140. 2D media encoding unit 116 may encode formatted scene data from XR viewport pre-rendering rasterization unit 114, e.g., using a video encoding standard, such as ITU-T H.264/Advanced Video Coding (AVC), ITU-T H.265/High Efficiency Video Coding (HEVC), ITU-T H.266 Versatile Video Coding (VVC), or the like. XR media content delivery unit 118 represents a content delivery sender, in this example. In this example, XR media content delivery unit 148 represents a content delivery receiver, and 2D media decoder 144 may perform error handling.
In general, XR client device 140 may determine a user's viewport, e.g., a direction in which a user is looking and a physical location of the user, which may correspond to an orientation of XR client device 140 and a geographic position of XR client device 140. Tracking/XR sensors 146 may determine such location and orientation data, e.g., using cameras, accelerometers, magnetometers, gyroscopes, or the like. Tracking/XR sensors 146 provide location and orientation data to XR viewport rendering unit 142 and 5GS delivery unit 150. XR client device 140 provides tracking and sensor information 132 to XR server device 110 via network 130. XR server device 110, in turn, receives tracking and sensor information 132 and provides this information to XR scene generation unit 112 and XR viewport pre-rendering rasterization unit 114. In this manner, XR scene generation unit 112 can generate scene data for the user's viewport and location, and then pre-render 2D media data for the user's viewport using XR viewport pre-rendering rasterization unit 114. XR server device 110 may therefore deliver encoded, pre-rendered 2D media data 134 to XR client device 140 via network 130, e.g., using a 5G radio configuration.
XR scene generation unit 112 may receive data representing a type of multimedia application (e.g., a type of video game), a state of the application, multiple user actions, or the like. XR viewport pre-rendering rasterization unit 114 may format a rasterized video signal. 2D media encoding unit 116 may be configured with a particular er/decoder (codec), bitrate for media encoding, a rate control algorithm and corresponding parameters, data for forming slices of pictures of the video data, low latency encoding parameters, error resilience parameters, intra-prediction parameters, or the like. XR media content delivery unit 118 may be configured with real-time transport protocol (RTP) parameters, rate control parameters, error resilience information, and the like. XR media content delivery unit 148 may be configured with feedback parameters, error concealment algorithms and parameters, post correction algorithms and parameters, and the like.
Raster-based split rendering refers to the case where XR server device 110 runs an XR engine (e.g., XR scene generation unit 112) to generate an XR scene based on information coming from an XR device, e.g., XR client device 140 and tracking and sensor information 132. XR server device 110 may rasterize an XR viewport and perform XR pre-rendering using XR viewport pre-rendering rasterization unit 114.
In the example of
In some examples, latency from rendering video data by XR server device 110 and XR client device 140 receiving such pre-rendered video data may be in the range of 50 milliseconds (ms). Latency for XR client device 140 to provide location and position (e.g., pose) information may be lower, e.g., 20 ms, but XR server device 110 may perform asynchronous time warp to compensate for the latest pose in XR client device 140.
The following call flow is an example highlighting steps of performing these techniques:
According to TR 26.928, clause 4.2.2, the relevant processing and delay components are summarized as follows:
The roundtrip interaction delay is therefore the sum of the Age of Content and the User Interaction Delay. If part of the rendering is done on an XR server and the service produces a frame buffer as a rendering result of the state of the content, then for raster-based split rendering in cloud gaming applications, the following processes contribute to such a delay:
As XR client device 140 applies ATW, the motion-to-photon latency requirements (of at most 20 ms) are met by internal processing of XR client device 140. What determines the network requirements for split rendering is time of pose-to-render-to-photon and the roundtrip interaction delay. According to TR 26.928, clause 4.5, the permitted downlink latency is typically 50-60 ms.
Rasterized 3D scenes available in frame buffers (see clause 4.4 of TR 26.928) are provided by XR scene generation unit 112 and need to be encoded, distributed, and decoded. According to TR 26.928, clause 4.2.1, relevant formats for frame buffers are 2 k by 2 k per eye, potentially even higher. Frame rates are expected to be at least 60 fps, potentially higher up to 90 fps. The formats of frame buffers are regular texture video signals that are then directly rendered. As the processing is graphics centric, formats beyond commonly used 4:2:0 signals and YUV signals may be considered.
In practical considerations, the NVIDIA Encoding functions may be used. The parameters of such an encoder are documented at developer.nvidia.com/nvidia-video-codec-sdk.
The techniques of this disclosure may be used to address certain challenges and achieve certain tasks. For instance, these techniques may be used to evaluate basic system design options and their performance, generate traffic models for evaluation of radio access network (RAN) options, provide guidelines for good parameter settings on encoding, content delivery, and RAN configurations, identify capacities for such types of applications, and define potential optimizations. These techniques may simulate these various elements with a reasonable setup.
In a first example, using split rendering, a full simulation for system 100 may be performed. In this example, 5G New Radio (NR) settings and simulations may be performed for communications via network 130, such as, for example, tracking and sensor information 132 and pre-rendered 2D media data 134. Tracking/XR sensors 146 may track and sense example user movements. Quality of video data presented at display device 152 may then be measured during these various simulations.
To perform this first example, simulations may be performed using a variety of models performing separate tasks. A source video model may include actions performed by XR scene generation unit 112, XR viewport pre-rendering rasterization unit 114, XR viewport rendering unit 142, 5GS delivery unit 120, and display device 152. A content delivery model may include actions performed by 2D media encoding unit 116, XR media content delivery unit 118, 5GS delivery unit 120, 5GS delivery unit 150, and 2D media decoder 144. An uplink model may include actions performed by tracking/XR sensors 146 and 5GS delivery unit 150. Tracking and sensor information 132 may be generated as part of a traffic model uplink, and RAN simulations may be performed to generate example pre-rendered 2D media data 134.
The various components of XR server device 110, XR client device 140, and display device 152 may be implemented using one or more processors implemented in circuitry, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. The functions attributed to these various components may be implemented in hardware, software, or firmware. When implemented in software or firmware, it should be understood that instructions for the software or firmware may be stored on a computer-readable medium and executed by requisite hardware.
Based on the system design in S4-200771, some challenges for a potential simulation and for generation of traffic models exist as several aspects may be addressed, e.g., using certain techniques of this disclosure. Such challenges include:
Simulation and modeling per system 100 may be broken into separate individual components:
Game engine 162 receives data 178, including pose model and traces data, game data, and game/engine configuration data, for simulating an extended reality (XR) game using received pose model and traces data. Game engine 162 generates video frames 180 from this received data and provides video frames 180 to model encoding device 164. Model encoding device 164 encodes video frames 180 to generate encoded video frames and v-trace data and provides the encoded video frames and v-trace data 182 to content encoding and delivery model 166.
Content encoding and delivery model uses encoded video frames and v-trace data 182 to generate slice trace (s-trace) data 184 and packets representing encapsulated, encoded slices of the encoded video frames using example configuration parameters 188. Content encoding and delivery model 166 provides s-trace data 184 and the packets to 5GS simulation unit 168. 5GS simulation unit 168 performs a RAN simulation using received s-trace data 184 and the packets and configuration parameters 190 to generate s′-trace data 186. S-trace data 184 represent slices of video data before radio transmission, while s′-trace data 186 represent the slices of video data after radio transmission (which may, for example, be corrupted due to radio transmission). 5GS simulation unit 168 provides s′-trace data 186 and delivered packets encapsulating encoded slices of video data to content delivery and decoding model 170. Content delivery and decoding model 170 performs an example decoding process on the data of the packets.
Using encoded and decoded data from content encoding and delivery model 166 and content delivery and decoding model 170, values representing quality per frame 172 can be generated, to test performance of the RAN simulation performed by 5GS simulation unit 168 and configuration parameters 190. Using a combination/aggregation of individual values of quality per frame 172, an overall quality value 174 can be generated.
Goals may be set for the values for quality per frame 172 and overall quality 174 for adjusting the various configuration parameters. In one example, a goal may be to have no change of video quality as measured by quality per frame 172 and overall quality 174. One or a small number of application metrics may be derived from RAN simulations performed by 5GS simulation unit 168. For each scenario, few v-traces (e.g., one to three) may be performed but over the duration of several hours. Quantization parameters may be unchanged in configuration parameters 188. Several various values for configuration parameters 188 may be adjusted, which may cause bitrate changes as side effects and changes to traffic characteristics. However, no quality changes should be made. There may be a relatively small number of configurations tested, e.g., three to ten.
Game engine 162, model encoding device 164, content encoding and delivery model 166, 5GS simulation unit 168, and content delivery and decoding model 170 may be implemented using one or more processors implemented in circuitry, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. The functions attributed to these various components may be implemented in hardware, software, or firmware. When implemented in software or firmware, it should be understood that instructions for the software or firmware may be stored on a computer-readable medium and executed by requisite hardware.
Based on the concepts discussed above, several parameters are relevant for the overall system design:
Video quality may be impacted by various factors. One factor is coding artifacts based on encoding. Such artifacts may be determined, for example, by the peak signal to noise ratio (PSNR). Another factor is artifacts due to lost packets and the resulting error propagation.
According to the techniques of this disclosure, quality may be modeled for each simulation as a combination of:
In general, references to macroblocks above and throughout this disclosure refer to macroblocks of ITU-T H.264/AVC. However, it should be understood that the concept of a macroblock may be replaced with a coding unit (CU) or coding tree unit (CTU) of ITU-T H.265/HEVC or ITU-T H.266/VVC, without loss of generality.
In the example of
The following elements may be used by system 160 and other systems described herein:
Referring again to
To perform the various test scenarios discussed above, representative source data may be generated to perform v-traces. S-traces may also be performed using content encoding, delivery, and quality, including, for example, on various system designs, delivery parameters, mappings to radio, modeling, and quality definitions. RAN delivery simulations may be based on the S-traces referred to above.
Game engine 202 generates video frames 210 and provides video frames 210 to each of predictive model encoding unit 204 and intra model encoding unit 206. Video frames 210 may be, for example, extended reality (XR) split data at 60 fps, with one frame per eye to achieve the XR effect. In this example, v-trace system model 200 includes two tracks: one for intra-prediction and one for inter-prediction. In video coding, intra-prediction is performed by predicting blocks of video data with neighboring, previously decoded blocks of the same frame, and inter-prediction is performed by predicting blocks of video data using reference blocks of previously decoded frames.
Predictive model encoding unit 204 may generate VP-trace data 212, while intra model encoding unit 206 may generate VI-trace data 214. Trace combination unit 208 receives VP-trace data 212 and VI-trace data 214 and generates V-trace data 216. V-trace data 216 may be formatted according to a passlogfile format of FFMPEG after a coding pass. Such formats are described in, e.g., ffmpeg.org/wiki/Encode/H.264 and slhck.info/video/2017/03/01/rate-control.html. Trace combination unit 208 may also use configuration parameters “-pass[: stream_specifier] n (output,per-stream)” and “-passlogfile.” Pseudocode for an example algorithm for generating v-trace data 216 in this format is shown below:
As an alternative to ITU-T H.264/AVC, video data may be coded using ITU-T H.265/HEVC or other video coding standards. With respect to the example of H.265, example log files are described at x265.readthedocs.io/en/default/cli.html#input-output-file-options. Such log files may include parameters as follows:
In still other examples, other video codecs, such as ITU-T H.266/Versatile Video Coding (VVC) may be used. Similar sets of data may be generated for analysis using VVC or other such video codecs.
Trace combination unit 208 may generate v-trace data 216 according to certain v-trace generation principles. Certain v-trace generation principles are described at slhck.info/video/2017/02/24/crf-guide.html. Trace combination unit 208 may use H.264/AVC with x264 in FFMPEG and H.265/HEVC with x265 in FFMPEG. Trace combination unit 208 may use a near lossless constant quality mode, with options including:
Game engine 202, predictive model encoding unit 204, intra model encoding unit 206, and trace combination unit 208 may be implemented using one or more processors implemented in circuitry, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. The functions attributed to these various components may be implemented in hardware, software, or firmware. When implemented in software or firmware, it should be understood that instructions for the software or firmware may be stored on a computer-readable medium and executed by requisite hardware.
In this example, content encoding and delivery model 222 receives v-trace data 216, which includes data representing encoding of frames of video data, from trace combination unit 208 (
Slice to IP/RCL mapping unit 224 may use real-time transport protocol (RTP) to form IP packets and RLC fragments from the slices of s-trace data 232. Slice to IP/RLC mapping unit 224 may be configured with IP packet and payload size information for forming the packets and fragments. In general, if a packet is lost, the entire slice to which the packet corresponds is also lost. Slice to IP/RLC mapping unit 224 produces p-trace data 234, which slice to IP/RLC mapping unit 224 provides to RAN simulation unit 226. RAN simulation unit 226 may be configured with delay requirements and loss requirements for each slice/packet and determine, through a RAN simulation, whether each slice is received or lost at the end of the simulation, as well as latency values or received time stamp values for each slice.
RAN simulation unit 226 may provide P′-trace data 236 including data representing whether slices are received or lost and the latencies/receive times for the received slices to IP/RLC to slice mapping unit 228. IP/RLC to slice mapping unit 228 provides s′-trace data 238 to content encoding and delivery model 222. Content encoding and delivery model 222 may determine overall quality value 240 by comparing video data decoded prior to the RAN simulation to decoded video data following the RAN simulation.
RAN simulation unit 226 may be configured according to a maximum delay requirement for each slice downlink. For example, there may be a MAC to MAC 10 ms deadline and a no loss requirement from outside. For an uplink, there may be tracking, sensor, and pose information, content delivery uplink information, and pose update traffic (e.g., at a periodicity of 1.25 ms or 2 ms). The system may determine how pose frequency impacts quality, which may depend on XR service type. For gaming, pose frequency may have a large impact. Pose and game action may also be synchronized.
RAN simulation options may include open loop and closed loop options. For open loop, for an entire V-trace, the system may generate s-trace data, p-trace data, RAN simulations one way, p′-trace data, and s′-trace data, and then perform a quality evaluation, in the form of, e.g., overall quality value 240. For closed loop, for every v-trace entry, the system may generate s-trace data, p-trace data, RAN simulations one way, p′-trace data, and s′-trace data, and then feedback the resulting data into a next s-trace, and generate quality evaluations in the form of overall quality value 240.
Content encoding and delivery model 222, slice to IP/RLC mapping unit 224, RAN simulation unit 226, and IP/RCL to slice mapping unit 228 may be implemented using one or more processors implemented in circuitry, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. The functions attributed to these various components may be implemented in hardware, software, or firmware. When implemented in software or firmware, it should be understood that instructions for the software or firmware may be stored on a computer-readable medium and executed by requisite hardware.
In this example, decoded media data is compared before and after having passed through RAN simulation unit 256. Video data from content delivery and decoding model 258, which has passed through RAN simulation unit 256, is assessed to determine values for quality per frame 262. Decoded media data from content delivery and decoding model 260 and the values for quality per frame 262 are used to determine overall quality data 264.
In particular, video content model 252 provides v-trace data 268 and encoded video data to be transmitted to content encoding and delivery model 254. Content delivery model 254 receives global configuration data 266 and uses global configuration data 266 to encode and deliver the video data corresponding to v-trace data 268. Global configuration data 266 may include, for example, bitrate control information (e.g., to select one or more of constant quality, constant bitrate, feedback-based variable bitrate, or constant rate factor), slice settings (number of slices, maximum slice size), error resilience (frame-based, slice-based, regular intra refresh, feedback-based intra refresh, or feedback-based prediction), and feedback data (off, statistical, or operational). The maximum slice size setting may depend on various statistics, including the number of slices, which may vary.
Content encoding and delivery model 254 outputs s-trace data 270 to RAN simulation unit 256 and s′-trace data 270′ to content delivery and decoding model 260. RAN simulation unit 256 performs a RAN simulation on s-trace data 270 and outputs s′-trace data 272 to content delivery and decoding model 258. Content delivery and decoding model 260 outputs q-trace data 274, and content delivery and decoding model 258 outputs values for quality per frame 262, yielding q′-trace data 276. Overall quality data 264 may be one or more quality measures for an entire video sequence (e.g., an aggregation of qualities for individual video frames).
Various content encoding considerations may influence encoding of video data corresponding to v-trace data 268 to form s-trace data 270. In one example, an objective is to have consistent quality, use ITU-T H.265/HEVC encoding, form a fixed number of slices, one slice being encoded using intra prediction in opportunistic mode, and no feedback. In this example, the bitrate control may be the use of a constant rate factor of 28, a number of slices of 10, a periodic intra refresh value of 10, period intra refresh and reference picture invalidation, and feedback being off. To create slice sizes, content encoding and delivery model 254 may generate 10 slices for every video frame, use a constant rate factor of 28 (where ±6 results in about half/double the frame size, 12% decrease/increase for each +/−), one intra slice is assumed, based on VI-trace (i.e., 10% of adjusted I-frame size), and 9 inter slices are generated based on VP-trace (where two aspects may be considered: size of frame (10%) and statistical variation).
RAN simulation unit 256 may maintain state data for each macroblock (ITU-T H.264/AVC) or coding tree unit (ITU-T H.265/HEVC), such as: damaged (lost or predicted from damaged area, whether temporal or spatial), or correct. RAN simulation unit 256 may provide feedback to content encoding and delivery model 254, including data representing numbers of lost or damaged blocks of video data. Content encoding and delivery model 254, in turn, may integrate the feedback for subsequent encoding. For example, whether statistical or operational, content encoding and delivery model 254 may react to a slice loss. In response, content encoding and delivery model 254 may adjust bitrate (e.g., get an encoding bitrate and adjust quantization parameters up or down, where +/−1 may yield a 12% impact). Content encoding and delivery model 254 may add an intra-predicted slice if a slice is lost (significantly more intra-predicted data may be added in the case of a reported loss. The intra-predicted slice may cover a large area, depending on motion vector activity. In some cases, content encoding and delivery model 254 may predict from acknowledged regions only, which may cause a statistical increase for frame size for lost slices, as the latest slice may not be used for prediction.
S-trace data 270 may be formatted to include data for a timestamp representing an associated frame, a size of a slice, a quality of the slice (which may include more information from the v-trace, for example, complexity), a number of macroblocks (for ITU-T H.264/AVC) or coding tree units (for ITU-T H.265/HEVC), and/or a timing of the slice (e.g., a deadline for receipt of the slice). S′-trace data 270′ and 272 may be formatted to include all information from the format for s-trace data 270 discussed above, and in addition, data representing slice loss and/or slice delay.
There may be a constant deadline for each slice, which can be set to a higher number. For example, for 60 fps video data staggered by each eye, i.e., 120 fps video data, there may be a desired latency set to 7 ms. A drop deadline can be set differently, and may be useful, but should not complicate the RAN simulation.
Content decoding considerations to produce Q-trace data 274 from s′-trace data 270′ may be used to take the results from RAN simulations performed by RAN simulation unit 256 and to map these simulation results to estimated values for quality per frame 262. When determining values for quality per frame 262, slice considerations may include, for example, quality of encoded slices that are received, numbers of lost slices, which slices are so degraded in quality as to be considered lost, and delayed slices, which may be considered as lost. Other considerations may include error propagation (e.g., prediction of a slice from a degraded quality slice or lost slice), complexity of frame/slice may determine error propagation (how fast incorrect data spreads over frames), new intra reset quality, and percent of correct/wrong data for each frame and for different configurations.
A metric for determining q-trace data 274 may be a percentage of incorrect data, which may impact lost/error propagation. Q-trace data 274 format may include data representative of encoding quality, resulting quality after delivery (percentage of degraded data), and/or data rate of frames.
System 250 may maintain a state for each macroblock (ITU-T H.264/AVC) or coding tree unit (CTU) (ITU-T H.265/HEVC) of either “damaged” or “correct.” If a macroblock/CTU is damaged, system 250 may determine whether the macroblock/CTU is part of a slice that is lost for this transmission or if the macroblock/CTU is correctly received by predicted from an incorrectly decoded/lost region of another slice/frame. If the macroblock/CTU is correct, system 250 determines that the macroblock/CTU was received correctly and that it is predicted from a non-damaged region of another slice/frame. Predicting from a non-damaged region of another slice/frame means that spatial prediction is correct, temporal prediction is correct, or macroblocks/CTUs are recovered correctly using intra-refresh and prediction from correct regions of other slices/frames following the intra-refresh.
System 250 may determine overall quality value 264 by averaging encoding quality (e.g., average over quantization parameters, and possibly convert to peak signal to noise ratio (PSNR)), average incorrect video data (e.g., an average incorrect number of video frames and/or slices), and/or if needed, a multiplication/combination/aggregation. 3GPP SA4 defines a constant rate factor (CRF) based encoding model with a specific quality factor (e.g., using FFMPEG default of 28). Different configurations for error resilience may be applied. Criteria for simulations may be the percentage of incorrect video area such that at most one macroblock/CTU for every X (e.g., 60) seconds is wrong. For example, if 4096×4096 is used at 60 fps, then this results in 10e-6 damaged area in average.
The models discussed above may work for pixel-based split rendering and/or cloud gaming. Conversational applications, video streaming applications (e.g., Twitch.tv), and others may also be used.
Video content model 252, content encoding and delivery model 254, RAN simulation unit 256, content delivery and decoding model 258, and content delivery and decoding model 260 may be implemented using one or more processors implemented in circuitry, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. The functions attributed to these various components may be implemented in hardware, software, or firmware. When implemented in software or firmware, it should be understood that instructions for the software or firmware may be stored on a computer-readable medium and executed by requisite hardware.
System 250 represents an example content modeling based on discussion in S4-200771. This modeling includes a v-model input, global configuration for an encoder, statistical or dynamic feedback from content delivery receiver, a decoding model and a quality model.
Content encoding by content encoding and delivery model 254 may be modeled as follows:
Configuration parameters of global configuration data 266 may include:
Aspects of model encoding may include:
Decoding emulation by content delivery and decoding model 258 may be based on the delay and/or loss of slices. Late and lost slices may be considered unavailable and cause errors (e.g., in decoding of subsequent frames/slices that refer to the lost or late slices for inter-prediction).
RAN simulation unit 256 may emulate a RAN simulation based on existing 5Qis.
Quality evaluation may be based on two aspects, including encoding quality and quality degradation due to lost slices. The following simulation may be used to identify damaged macroblocks (or coding tree units or coding units, in ITU-T H.265/HEVC):
Depending on the configuration and the setting of the delivered video quality, different results may be obtained. A quality threshold may for example be to have at most 0.1% of damaged video area. Also, the quality of the original content may be a threshold.
Game engine 282 receives input data 296 including pose model and trace data, game data, and game configuration data. Game engine 282 generates video frames 298 from input data 296. Encoder 284 (which may be an NVIDIA NVENC encoder) encodes video frames 298 using configuration data 302 to generate video bitstream 300. Encoder 284 may operate at 600 GByte per hour, and may run over several hours. Decoder 286 may be an FFMPEG decoder, and decodes video bitstream 300 to generate decoded video data stored as raw data 288 in a computer-readable medium, such as a hard disk, flash drive, or the like. Predictive encoder 290 and intra encoder 292 receive configuration data 304 and encode raw data 288 to form respective VP-trace and VI-trace data. V-trace to 3GPP unit 294 generates v-trace data from the VP-trace and VI-trace data.
As shown in
Based on these 180 encoding runs, a good encoder modeling is expected such that only a subset of runs are necessary for longer game sequences.
Parameters for generation of the V-trace data in this example may include a duration of a P-trace (e.g., several hours), various games being executed by game engine 282, pose traces, settings for encoder 284 (with the goal being, e.g., simple, high quality FFMPEG decodable video data), and FFMPEG configuration data. Four FFMPEG decoder configurations may be tested, and in some examples, pipelined in parallel:
x265.readthedocsio/en/default/cli.html# describes possible configuration parameters. These parameters may be used to configure rvrplugin.ini with 2048×2048 resolution, 5 ps, and generated bitstream using ITU-T H.264/AVC encoder at server side, which may run at 60 or 120 fps. This data may be re-encoded with crf-24 and updated PSNR and SSIM values. Initial command-line arguments may include the following:
Game engine 282, encoder 284, decoder 286, predictive encoder 290, intra encoder 292, and v-trace to 3GPP unit 294 may be implemented using one or more processors implemented in circuitry, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. The functions attributed to these various components may be implemented in hardware, software, or firmware. When implemented in software or firmware, it should be understood that instructions for the software or firmware may be stored on a computer-readable medium and executed by requisite hardware.
Encoder 314 may encode video data using ITU-T H.265/HEVC, e.g., “x265.exe.” Encoder 314 may operate with the following command line options:
Encoder 314 need not use “--frame-dup disabled,” “--constrained-intra,” and “--no-deblock.”
Renderer 312 and encoder 314 may be implemented using one or more processors implemented in circuitry, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. The functions attributed to these various components may be implemented in hardware, software, or firmware. When implemented in software or firmware, it should be understood that instructions for the software or firmware may be stored on a computer-readable medium and executed by requisite hardware.
Error propagation may also be modeled. As a basic principle, error in a slice may destroy the slice in the current picture (e.g., X % damaged). As the next frame references this frame, error may propagate both temporally as well as spatially, until an intra frame is received or an intra slice for the region, the region may be damaged, and the error may propagate spatially to the next frame. This may depend on the amount of motion vectors, e.g., in the next frame, if more than X % are damaged, unless intra is applied. In the next frame, the size of the damaged area may grow. Intra prediction may remove the error propagation, but with multiple reference frames, the damaged area may be brought back.
Initially, XR server device 110 may receive tracking and sensor information 132 from XR client device 140 (350). For example, XR client device 140 may determine an orientation in which a user is looking using tracking/XR sensors 146. XR client device 140 may send tracking and sensor information 132 representing the orientation in which the user is looking. XR server device 110 may receive tracking and sensor information 132 and provide this information to XR scene generation unit 112. XR scene generation unit 112 and XR viewport pre-rendering rasterization unit 114 may then generate video frames of scene data (352). In some examples, a video game engine may generate the scene data using tracking and sensor information 132, video game data, and video game configuration data.
2D media encoding unit 116 may then encode the generated video frames of the scene data (354). In some examples, as shown in and described with respect to
XR media content delivery unit 118 may then packetize slices of the encoded video frames (356). In general, 2D media encoding unit 116 may be configured with a particular maximum transmission unit (MTU) size for packets of a radio access network (RAN) and produce slices having amounts of data less than or equal to the MTU size. Thus, each slice may be capable of transmission within a single packet, in some examples.
XR server device 110 may then perform a RAN simulation of the RAN for transferring the packets (358), e.g., using configuration of network 130. Examples of performing a RAN simulation in this manner are described above with respect to
XR server device 110 may then assemble simulation-received packets into encoded video frames (360) and decode the video frames (362). XR server device 110 may then calculate individual frame qualities (364). The individual frame qualities may represent differences between the video frames prior to or after encoding and the decoded video frames, e.g., as discussed with respect to
XR server device 110 may then determine an overall quality for the system from the individual frame qualities (366). For example, XR server device 110 may determine the overall quality as an average encoding quality and/or an average number of incorrect video frames (e.g., frames for which data was lost or corrupted). XR server device 110 may perform this method for a variety of different types of configurations, e.g., to determine a number of users that can be supported for a given configuration. For example, XR server device 110 may calculate a percentage of corrupted video frames as a function of a number of supported users for one or more sets of configurations. Additionally or alternatively, XR server device 110 may calculate a percentage of corrupted video frames as a function of a quantization configuration for one or more numbers of supported users.
In this manner, the method of
The following clauses represent certain examples of the techniques of this disclosure:
Clause 1: A method of sending media data, the method comprising: receiving tracking and sensor information from an extended reality (XR) client device; generating scene data using the tracking and sensor information; rendering an XR viewport to form video data from the scene data; encoding the video data; and sending the encoded video data to the XR client device via a 5G network.
Clause 2: A method of retrieving media data, the method comprising: sending tracking and sensor information to an extended reality (XR) server device; receiving, from the XR server device, encoded video data corresponding to the tracking and sensor information; decoding the encoded video data; and rendering an XR viewport using the decoded video data.
Clause 3: A method comprising a combination of the methods of clauses 1 and 2.
Clause 4: A method of processing media data, the method comprising: receiving tracking and sensor information from an extended reality (XR) client device; generating scene data using the tracking and sensor information, the scene data comprising one or more video frames; encoding the video data to form v-trace data; performing a radio access network (RAN) simulation of delivering the encoded video data via a 5G network; decoding the delivered encoded video data to form decoded video data; calculating values representing quality for each of the video frames from the generated one or more video frames and the decoded video data; and determining an overall quality value from the values representing the quality for each of the video frames.
Clause 5: The method of clause 4, wherein generating the scene data comprises executing a video game engine using the tracking and sensor information, video game data, and video game configuration data.
Clause 6: The method of any of clauses 4 and 5, wherein encoding the video data comprises encoding the video data without changing a quantization configuration.
Clause 7: The method of any of clauses 4-6, wherein determining the overall quality value comprises generating a graph representing a percentage of corrupted video frames as a function of a number of supported users for one or more sets of configurations.
Clause 8: The method of any of clauses 4 and 5, wherein determining the overall quality value comprises generating a graph representing a percentage of corrupted video frames as a function of a quantization configuration for one or more numbers of supported users.
Clause 9: The method of any of clauses 4-8, wherein encoding the video data to form the v-trace data comprises: encoding at least some portions of the video data using inter-prediction to form vp-trace data; encoding at least some portions of the video data using intra-prediction to form vi-trace data; and combining the vp-trace data and the vi-trace data to form the v-trace data.
Clause 10: The method of any of clauses 4-9, wherein the v-trace data is formatted according to a passlogfile format of FFMPEG.
Clause 11: The method of any of clauses 4-10, wherein the v-trace data includes one or more of a display picture number value, a coded picture number value, a picture type value, a quality value, an intra-texture-bits value, an inter-texture-bits value, a motion vector bits value, a miscellaneous bits value, an f-code value, a b-code value, a mc_mb_var_sum value, an mb_var_sum value, an i_count value, a skip count value, or a header bits value.
Clause 12: The method of any of clauses 4-11, wherein performing the RAN simulation comprises: receiving one or more slices of the encoded video data corresponding to the v-trace data; packetizing and fragmenting the slices to form packets in p-trace data using real-time transport protocol (RTP) according to IP packet and payload size configuration data; simulating transfer of the packets; determining that one of the slices is lost when a packet for the one of the slices is lost during the simulation of the transfer of the packets; and outputting data representing, for each slice, whether the slice was received or lost during the simulation of the transfer and a latency for transfer of the received packets.
Clause 13: The method of any of clauses 4-12, wherein performing the RAN simulation comprises: receiving one or more slices of the encoded video data corresponding to the v-trace data; and forming s-trace data from the slices according to configuration data indicating a bitrate control technique, slice settings, an error resilience technique, and a feedback technique.
Clause 14: The method of clause 13, wherein the bitrate control technique comprises one of constant quality, constant bitrate, feedback-based variable bitrate, or constant rate factor.
Clause 15: The method of any of clauses 13 and 14, wherein the slice settings comprise one or more of a number of slices or a maximum slice size.
Clause 16: The method of any of clauses 13-15, wherein the error resilience technique comprises one of frame-based resilience, slice-based resilience, regular intra refresh, feedback-based intra refresh, or feedback based prediction.
Clause 17: The method of any of clauses 13-16, wherein the feedback technique comprises one of no feedback, statistical feedback, or operational feedback.
Clause 18: The method of any of clauses 4-17, wherein calculating the values representing the quality for each of the video frames comprises: determining, for each received encoded slice, a quality of the slice; and determining, for non-received slices, whether the non-received slices were lost or of degraded quality.
Clause 19: The method of any of clauses 4-18, wherein determining the overall quality value comprises determining the overall quality value using one or more of an average encoding quality or an average incorrect video data.
Clause 20: A device for processing media data, the device comprising one or more means for performing the method of any of clauses 1-19.
Clause 21: The device of clause 20, wherein the apparatus comprises at least one of: an integrated circuit; a microprocessor; and a wireless communication device.
Clause 22: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor perform to the method of any of clauses 1-19.
Clause 23: A device for sending media data, the device comprising: means for receiving tracking and sensor information from an extended reality (XR) client device; means for generating scene data using the tracking and sensor information; means for rendering an XR viewport to form video data from the scene data; means for encoding the video data; and means for sending the encoded video data to the XR client device via a 5G network.
Clause 24: A device for retrieving media data, the device comprising: means for sending tracking and sensor information to an extended reality (XR) server device; means for receiving, from the XR server device, encoded video data corresponding to the tracking and sensor information; means for decoding the encoded video data; and means for rendering an XR viewport using the decoded video data.
Clause 25: A device for processing media data, the device comprising: means for receiving tracking and sensor information from an extended reality (XR) client device; means for generating scene data using the tracking and sensor information, the scene data comprising one or more video frames; means for encoding the video data to form v-trace data; means for performing a radio access network (RAN) simulation of delivering the encoded video data via a 5G network; means for decoding the delivered encoded video data to form decoded video data; means for calculating values representing quality for each of the video frames from the generated one or more video frames and the decoded video data; and means for determining an overall quality value from the values representing the quality for each of the video frames.
Clause 26: A method of processing media data, the method comprising: receiving tracking and sensor information from an extended reality (XR) client device; generating scene data using the tracking and sensor information, the scene data comprising one or more video frames; encoding the video frames to form encoded video frames; performing a radio access network (RAN) simulation of delivering the encoded video frames via a radio access network; decoding the encoded video frames delivered according to the RAN simulation to form decoded video frames; calculating values representing individual frame quality for each of the video frames from the generated one or more video frames and the decoded video frames; and determining an overall quality value from the values representing the individual frame quality for each of the video frames.
Clause 27: The method of clause 26, wherein generating the scene data comprises executing a video game engine using the tracking and sensor information, video game data, and video game configuration data.
Clause 28: The method of clause 26, wherein encoding the video frames comprises encoding the video frames without changing a quantization configuration.
Clause 29: The method of clause 26, wherein determining the overall quality value comprises calculating a percentage of corrupted video frames as a function of a number of supported users for one or more sets of configurations.
Clause 30: The method of clause 26, wherein determining the overall quality value comprises calculating a percentage of corrupted video frames as a function of a quantization configuration for one or more numbers of supported users.
Clause 31: The method of clause 26, wherein encoding the video frames further comprises forming v-trace data, the v-trace data representing complexity of the encoding of the video frames, comprising: encoding at least some portions of the video frames using inter-prediction to form vp-trace data; encoding the at least some portions of the video frames using intra-prediction to form vi-trace data; and combining the vp-trace data and the vi-trace data to form the v-trace data.
Clause 32: The method of clause 26, wherein encoding the video frames further comprises forming v-trace data, the v-trace data representing complexity of the encoding of the video frames, and wherein the v-trace data includes one or more of a display picture number value, a coded picture number value, a picture type value, a quality value, an intra-texture-bits value, an inter-texture-bits value, a motion vector bits value, a miscellaneous bits value, an f-code value, a b-code value, a mc_mb_var_sum value, an mb_var_sum value, an i_count value, a skip count value, or a header bits value.
Clause 33: The method of clause 26, wherein performing the RAN simulation comprises: receiving one or more slices of the encoded video frames; packetizing and fragmenting the slices to form packets in p-trace data using real-time transport protocol (RTP) according to IP packet and payload size configuration data; performing a simulation of transfer of the packets; determining that one of the slices is lost when a packet for the one of the slices is lost during the simulation of the transfer of the packets; and outputting data representing, for each slice, whether the slice was received or lost during the simulation of the transfer and a latency for transfer of the received packets.
Clause 34: The method of clause 26, wherein performing the RAN simulation comprises: receiving one or more slices of the encoded video frames; and forming s-trace data from the slices according to configuration data indicating a bitrate control technique, slice settings, an error resilience technique, and a feedback technique.
Clause 35: The method of clause 34, wherein forming the s-trace data comprises assigning, to each of the slices, data representing a frame associated with the slice, a size of the slice, a quality of the slice, an area of the frame covered by the slice, and timing information for the slice.
Clause 36: The method of clause 34, wherein the bitrate control technique comprises one of constant quality, constant bitrate, feedback-based variable bitrate, or constant rate factor.
Clause 37: The method of clause 34, wherein the slice settings comprise one or more of a number of slices or a maximum slice size.
Clause 38: The method of clause 34, wherein the error resilience technique comprises one of frame-based resilience, slice-based resilience, regular intra refresh, feedback-based intra refresh, or feedback-based prediction.
Clause 39: The method of clause 34, wherein the feedback technique comprises one of statistical feedback or operational feedback.
Clause 40: The method of clause 26, wherein calculating the values representing the individual frame quality for each of the video frames comprises: determining, for each received encoded slice of the video frame, a quality of the slice; and determining, for non-received slices of the video frame, whether the non-received slices were lost or of degraded quality.
Clause 41: The method of clause 26, wherein determining the overall quality value comprises determining the overall quality value using one or more of an average encoding quality or an average incorrect video frames.
Clause 42: The method of clause 26, wherein the radio access network comprises a 5G network.
Clause 43: The method of clause 26, wherein the RAN simulation comprises a first RAN simulation of a plurality of RAN simulations, the overall video quality comprises a first overall video quality, and performing the first RAN simulation comprises performing the first RAN simulation using first configuration parameters, the method further comprising: determining respective configuration parameters for each of the plurality of RAN simulations; performing each of the RAN simulations using the respective configuration parameters; determining overall video quality values for each of the RAN simulations; and configuring a network to deliver XR data using the configuration parameters corresponding to the best overall video quality values for the RAN simulations.
Clause 44: A device for processing media data, the device comprising: a memory configured to store video data; and one or more processors implemented in circuitry and configured to: receive tracking and sensor information from an extended reality (XR) client device; generate scene data using the tracking and sensor information, the scene data comprising one or more video frames of the video data; encode the video frames to form encoded video frames; perform a radio access network (RAN) simulation of delivering the encoded video frames via a radio access network; decode the encoded video frames delivered according to the RAN simulation to form decoded video frames; calculate values representing individual frame quality for each of the video frames from the generated one or more video frames and the decoded video frames; and determine an overall quality value from the values representing the individual frame quality for each of the video frames.
Clause 45: The device of clause 44, wherein the one or more processors are configured to execute a video game engine using the tracking and sensor information, video game data, and video game configuration data to generate the scene data.
Clause 46: The device of clause 44, wherein to determine the overall quality value, the one or more processors are configured to calculate a percentage of corrupted video frames as a function of a number of supported users for one or more sets of configurations.
Clause 47: The device of clause 44, wherein to determine the overall quality value, the one or more processors are configured to calculate a percentage of corrupted video frames as a function of a quantization configuration for one or more numbers of supported users.
Clause 48: The device of clause 44, wherein the one or more processors are further configured to form v-trace data, the v-trace data representing complexity of the encoding of the video frames, and wherein to form the v-trace data, the one or more processors are configured to: encode at least some portions of the video frames using inter-prediction to form vp-trace data; encode the at least some portions of the video frames using intra-prediction to form vi-trace data; and combine the vp-trace data and the vi-trace data to form the v-trace data.
Clause 49: The device of clause 44, wherein the one or more processors are further configured to form v-trace data, the v-trace data representing complexity of the encoding of the video frames, and wherein the v-trace data includes one or more of a display picture number value, a coded picture number value, a picture type value, a quality value, an intra-texture-bits value, an inter-texture-bits value, a motion vector bits value, a miscellaneous bits value, an f-code value, a b-code value, a mc_mb_var_sum value, an mb_var_sum value, an i_count value, a skip count value, or a header bits value.
Clause 50: The device of clause 44, wherein to perform the RAN simulation, the one or more processors are configured to: receive one or more slices of the encoded video frames; packetize and fragmenting the slices to form packets in p-trace data using real-time transport protocol (RTP) according to IP packet and payload size configuration data; perform a simulation of transfer of the packets; determine that one of the slices is lost when a packet for the one of the slices is lost during the simulation of the transfer of the packets; and output data representing, for each slice, whether the slice was received or lost during the simulation of the transfer and a latency for transfer of the received packets.
Clause 51: The device of clause 44, wherein to perform the RAN simulation, the one or more processors are configured to: receive one or more slices of the encoded video frames; and form s-trace data from the slices according to configuration data indicating a bitrate control technique, slice settings, an error resilience technique, and a feedback technique.
Clause 52: The device of clause 51, wherein to form the s-trace data, the one or more processors are configured to assign, to each of the slices, data representing a frame associated with the slice, a size of the slice, a quality of the slice, an area of the frame covered by the slice, and timing information for the slice.
Clause 53: The device of clause 44, wherein to calculate the values representing the individual frame quality for each of the video frames, the one or more processors are configured to: determine, for each received encoded slice of the video frame, a quality of the slice; and determine, for non-received slices of the video frame, whether the non-received slices were lost or of degraded quality.
Clause 54: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to: receive tracking and sensor information from an extended reality (XR) client device; generate scene data using the tracking and sensor information, the scene data comprising one or more video frames; encode the video frames to form encoded video frames; perform a radio access network (RAN) simulation of delivering the encoded video frames via a radio access network; decode the encoded video frames delivered according to the RAN simulation to form decoded video frames; calculate values representing individual frame quality for each of the video frames from the generated one or more video frames and the decoded video frames; and determine an overall quality value from the values representing the individual frame quality for each of the video frames.
Clause 55: The computer-readable storage medium of clause 54, wherein the instructions that cause the processor to generate the scene data comprise instructions that cause the processor to execute a video game engine using the tracking and sensor information, video game data, and video game configuration data.
Clause 56: The computer-readable storage medium of clause 54, wherein the instructions that cause the processor to determine the overall quality value comprise instructions that cause the processor to calculate a percentage of corrupted video frames as a function of a number of supported users for one or more sets of configurations.
Clause 57: The computer-readable storage medium of clause 54, wherein the instructions that cause the processor to determine the overall quality value comprise instructions that cause the processor to calculate a percentage of corrupted video frames as a function of a quantization configuration for one or more numbers of supported users.
Clause 58: The computer-readable storage medium of clause 54, wherein the instructions that cause the processor to encode the video frames further comprise instructions that cause the processor to form v-trace data, the v-trace data representing complexity of the encoding of the video frames, and wherein the instructions that cause the processor to form the v-trace data include instructions that cause the processor to: encode at least some portions of the video frames using inter-prediction to form vp-trace data; encode the at least some portions of the video frames using intra-prediction to form vi-trace data; and combine the vp-trace data and the vi-trace data to form the v-trace data.
Clause 59: The computer-readable storage medium of clause 54, wherein the instructions that cause the processor to encode the video frames further comprise instructions that cause the processor to form v-trace data, the v-trace data representing complexity of the encoding of the video frames, and wherein the v-trace data includes one or more of a display picture number value, a coded picture number value, a picture type value, a quality value, an intra-texture-bits value, an inter-texture-bits value, a motion vector bits value, a miscellaneous bits value, an f-code value, a b-code value, a mc_mb_var_sum value, an mb_var_sum value, an i_count value, a skip count value, or a header bits value.
Clause 60: The computer-readable storage medium of clause 54, wherein the instructions that cause the processor to perform the RAN simulation comprise instructions that cause the processor to: receive one or more slices of the encoded video frames; packetize and fragmenting the slices to form packets in p-trace data using real-time transport protocol (RTP) according to IP packet and payload size configuration data; perform a simulation of transfer of the packets; determine that one of the slices is lost when a packet for the one of the slices is lost during the simulation of the transfer of the packets; and output data representing, for each slice, whether the slice was received or lost during the simulation of the transfer and a latency for transfer of the received packets.
Clause 61: The computer-readable storage medium of clause 54, wherein the instructions that cause the processor to perform the RAN simulation comprise instructions that cause the processor to: receive one or more slices of the encoded video frames; and form s-trace data from the slices according to configuration data indicating a bitrate control technique, slice settings, an error resilience technique, and a feedback technique.
Clause 62: The computer-readable storage medium of clause 61, wherein the instructions that cause the processor to form the s-trace data comprise instructions that cause the processor to assign, to each of the slices, data representing a frame associated with the slice, a size of the slice, a quality of the slice, an area of the frame covered by the slice, and timing information for the slice.
Clause 63: The computer-readable storage medium of clause 54, wherein the instructions that cause the processor to calculate the values representing the individual frame quality for each of the video frames comprise instructions that cause the processor to: determine, for each received encoded slice of the video frame, a quality of the slice; and determine, for non-received slices of the video frame, whether the non-received slices were lost or of degraded quality.
Clause 64: A device for processing media data, the device comprising: means for receiving tracking and sensor information from an extended reality (XR) client device; means for generating scene data using the tracking and sensor information, the scene data comprising one or more video frames; means for encoding the video frames to form encoded video frames; means for performing a radio access network (RAN) simulation of delivering the encoded video frames via a radio access network; means for decoding the encoded video frames delivered according to the RAN simulation to form decoded video frames; means for calculating values representing individual frame quality for each of the video frames from the generated one or more video frames and the decoded video frames; and means for determining an overall quality value from the values representing the individual frame quality for each of the video frames.
Clause 65: A method of processing media data, the method comprising: receiving tracking and sensor information from an extended reality (XR) client device; generating scene data using the tracking and sensor information, the scene data comprising one or more video frames; encoding the video frames to form encoded video frames; performing a radio access network (RAN) simulation of delivering the encoded video frames via a radio access network; decoding the encoded video frames delivered according to the RAN simulation to form decoded video frames; calculating values representing individual frame quality for each of the video frames from the generated one or more video frames and the decoded video frames; and determining an overall quality value from the values representing the individual frame quality for each of the video frames.
Clause 66: The method of clause 65, wherein generating the scene data comprises executing a video game engine using the tracking and sensor information, video game data, and video game configuration data.
Clause 67: The method of any of clauses 65-66, wherein encoding the video frames comprises encoding the video frames without changing a quantization configuration.
Clause 68: The method of any of clauses 65-67, wherein determining the overall quality value comprises calculating a percentage of corrupted video frames as a function of a number of supported users for one or more sets of configurations.
Clause 69: The method any of clauses 65-68, wherein determining the overall quality value comprises calculating a percentage of corrupted video frames as a function of a quantization configuration for one or more numbers of supported users.
Clause 70: The method of any of clauses 65-69, wherein encoding the video frames further comprises forming v-trace data, the v-trace data representing complexity of the encoding of the video frames, comprising: encoding at least some portions of the video frames using inter-prediction to form vp-trace data; encoding the at least some portions of the video frames using intra-prediction to form vi-trace data; and combining the vp-trace data and the vi-trace data to form the v-trace data.
Clause 71: The method any of clauses 65-70, wherein encoding the video frames further comprises forming v-trace data, the v-trace data representing complexity of the encoding of the video frames, and wherein the v-trace data includes one or more of a display picture number value, a coded picture number value, a picture type value, a quality value, an intra-texture-bits value, an inter-texture-bits value, a motion vector bits value, a miscellaneous bits value, an f-code value, a b-code value, a mc_mb_var_sum value, an mb_var_sum value, an i_count value, a skip count value, or a header bits value.
Clause 72: The method of any of clauses 65-71, wherein performing the RAN simulation comprises: receiving one or more slices of the encoded video frames; packetizing and fragmenting the slices to form packets in p-trace data using real-time transport protocol (RTP) according to IP packet and payload size configuration data; performing a simulation of transfer of the packets; determining that one of the slices is lost when a packet for the one of the slices is lost during the simulation of the transfer of the packets; and outputting data representing, for each slice, whether the slice was received or lost during the simulation of the transfer and a latency for transfer of the received packets.
Clause 73: The method of any of clauses 65-72, wherein performing the RAN simulation comprises: receiving one or more slices of the encoded video frames; and forming s-trace data from the slices according to configuration data indicating a bitrate control technique, slice settings, an error resilience technique, and a feedback technique.
Clause 74: The method of clause 73, wherein forming the s-trace data comprises assigning, to each of the slices, data representing a frame associated with the slice, a size of the slice, a quality of the slice, an area of the frame covered by the slice, and timing information for the slice.
Clause 75: The method of any of clauses 73 and 74, wherein the bitrate control technique comprises one of constant quality, constant bitrate, feedback-based variable bitrate, or constant rate factor.
Clause 76: The method of any of clauses 73-75, wherein the slice settings comprise one or more of a number of slices or a maximum slice size.
Clause 77: The method of any of clauses 73-76, wherein the error resilience technique comprises one of frame-based resilience, slice-based resilience, regular intra refresh, feedback-based intra refresh, or feedback based prediction.
Clause 78: The method of any of clauses 73-77, wherein the feedback technique comprises one of statistical feedback or operational feedback.
Clause 79: The method of any of clauses 65-78, wherein calculating the values representing the individual frame quality for each of the video frames comprises: determining, for each received encoded slice of the video frame, a quality of the slice; and determining, for non-received slices of the video frame, whether the non-received slices were lost or of degraded quality.
Clause 80: The method of any of clauses 65-79, wherein determining the overall quality value comprises determining the overall quality value using one or more of an average encoding quality or an average incorrect video frames.
Clause 81: The method of any of clauses 65-80, wherein the radio access network comprises a 5G network.
Clause 82: The method of any of clauses 65-81, wherein the RAN simulation comprises a first RAN simulation of a plurality of RAN simulations, the overall video quality comprises a first overall video quality, and performing the first RAN simulation comprises performing the first RAN simulation using first configuration parameters, the method further comprising: determining respective configuration parameters for each of the plurality of RAN simulations; performing each of the RAN simulations using the respective configuration parameters; determining overall video quality values for each of the RAN simulations; and configuring a network to deliver XR data using the configuration parameters corresponding to the best overall video quality values for the RAN simulations.
Clause 83: A device for processing media data, the device comprising: a memory configured to store video data; and one or more processors implemented in circuitry and configured to: receive tracking and sensor information from an extended reality (XR) client device; generate scene data using the tracking and sensor information, the scene data comprising one or more video frames of the video data; encode the video frames to form encoded video frames; perform a radio access network (RAN) simulation of delivering the encoded video frames via a radio access network; decode the encoded video frames delivered according to the RAN simulation to form decoded video frames; calculate values representing individual frame quality for each of the video frames from the generated one or more video frames and the decoded video frames; and determine an overall quality value from the values representing the individual frame quality for each of the video frames.
Clause 84: The device of clause 83, wherein the one or more processors are configured to execute a video game engine using the tracking and sensor information, video game data, and video game configuration data to generate the scene data.
Clause 85: The device of any of clauses 83 and 84, wherein to determine the overall quality value, the one or more processors are configured to calculate a percentage of corrupted video frames as a function of a number of supported users for one or more sets of configurations.
Clause 86: The device of any of clauses 83-85, wherein to determine the overall quality value, the one or more processors are configured to calculate a percentage of corrupted video frames as a function of a quantization configuration for one or more numbers of supported users.
Clause 87: The device of any of clauses 83-86, wherein the one or more processors are further configured to form v-trace data, the v-trace data representing complexity of the encoding of the video frames, and wherein to form the v-trace data, the one or more processors are configured to: encode at least some portions of the video frames using inter-prediction to form vp-trace data; encode the at least some portions of the video frames using intra-prediction to form vi-trace data; and combine the vp-trace data and the vi-trace data to form the v-trace data.
Clause 88: The device of any of clauses 83-87, wherein the one or more processors are further configured to form v-trace data, the v-trace data representing complexity of the encoding of the video frames, and wherein the v-trace data includes one or more of a display picture number value, a coded picture number value, a picture type value, a quality value, an intra-texture-bits value, an inter-texture-bits value, a motion vector bits value, a miscellaneous bits value, an f-code value, a b-code value, a mc_mb_var_sum value, an mb_var_sum value, an i_count value, a skip count value, or a header bits value.
Clause 89: The device of any of clauses 83-88, wherein to perform the RAN simulation, the one or more processors are configured to: receive one or more slices of the encoded video frames; packetize and fragmenting the slices to form packets in p-trace data using real-time transport protocol (RTP) according to IP packet and payload size configuration data; perform a simulation of transfer of the packets; determine that one of the slices is lost when a packet for the one of the slices is lost during the simulation of the transfer of the packets; and output data representing, for each slice, whether the slice was received or lost during the simulation of the transfer and a latency for transfer of the received packets.
Clause 90: The device of any of clauses 83-89, wherein to perform the RAN simulation, the one or more processors are configured to: receive one or more slices of the encoded video frames; and form s-trace data from the slices according to configuration data indicating a bitrate control technique, slice settings, an error resilience technique, and a feedback technique.
Clause 91: The device of clause 90, wherein to form the s-trace data, the one or more processors are configured to assign, to each of the slices, data representing a frame associated with the slice, a size of the slice, a quality of the slice, an area of the frame covered by the slice, and timing information for the slice.
Clause 92: The device of any of clauses 83-91, wherein to calculate the values representing the individual frame quality for each of the video frames, the one or more processors are configured to: determine, for each received encoded slice of the video frame, a quality of the slice; and determine, for non-received slices of the video frame, whether the non-received slices were lost or of degraded quality.
Clause 93: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to: receive tracking and sensor information from an extended reality (XR) client device; generate scene data using the tracking and sensor information, the scene data comprising one or more video frames; encode the video frames to form encoded video frames; perform a radio access network (RAN) simulation of delivering the encoded video frames via a radio access network; decode the encoded video frames delivered according to the RAN simulation to form decoded video frames; calculate values representing individual frame quality for each of the video frames from the generated one or more video frames and the decoded video frames; and determine an overall quality value from the values representing the individual frame quality for each of the video frames.
Clause 94: The computer-readable storage medium of clause 93, wherein the instructions that cause the processor to generate the scene data comprise instructions that cause the processor to execute a video game engine using the tracking and sensor information, video game data, and video game configuration data.
Clause 95: The computer-readable storage medium of any of clauses 93 and 94, wherein the instructions that cause the processor to determine the overall quality value comprise instructions that cause the processor to calculate a percentage of corrupted video frames as a function of a number of supported users for one or more sets of configurations.
Clause 96: The computer-readable storage medium of any of clauses 93-95, wherein the instructions that cause the processor to determine the overall quality value comprise instructions that cause the processor to calculate a percentage of corrupted video frames as a function of a quantization configuration for one or more numbers of supported users.
Clause 97: The computer-readable storage medium of any of clauses 93-96, wherein the instructions that cause the processor to encode the video frames further comprise instructions that cause the processor to form v-trace data, the v-trace data representing complexity of the encoding of the video frames, and wherein the instructions that cause the processor to form the v-trace data include instructions that cause the processor to: encode at least some portions of the video frames using inter-prediction to form vp-trace data; encode the at least some portions of the video frames using intra-prediction to form vi-trace data; and combine the vp-trace data and the vi-trace data to form the v-trace data.
Clause 98: The computer-readable storage medium of any of clauses 93-97, wherein the instructions that cause the processor to encode the video frames further comprise instructions that cause the processor to form v-trace data, the v-trace data representing complexity of the encoding of the video frames, and wherein the v-trace data includes one or more of a display picture number value, a coded picture number value, a picture type value, a quality value, an intra-texture-bits value, an inter-texture-bits value, a motion vector bits value, a miscellaneous bits value, an f-code value, a b-code value, a mc_mb_var_sum value, an mb_var_sum value, an i_count value, a skip count value, or a header bits value.
Clause 99: The computer-readable storage medium of any of clauses 93-98, wherein the instructions that cause the processor to perform the RAN simulation comprise instructions that cause the processor to: receive one or more slices of the encoded video frames; packetize and fragmenting the slices to form packets in p-trace data using real-time transport protocol (RTP) according to IP packet and payload size configuration data; perform a simulation of transfer of the packets; determine that one of the slices is lost when a packet for the one of the slices is lost during the simulation of the transfer of the packets; and output data representing, for each slice, whether the slice was received or lost during the simulation of the transfer and a latency for transfer of the received packets.
Clause 100: The computer-readable storage medium of any of clauses 93-99, wherein the instructions that cause the processor to perform the RAN simulation comprise instructions that cause the processor to: receive one or more slices of the encoded video frames; and form s-trace data from the slices according to configuration data indicating a bitrate control technique, slice settings, an error resilience technique, and a feedback technique.
Clause 101: The computer-readable storage medium of clause 100, wherein the instructions that cause the processor to form the s-trace data comprise instructions that cause the processor to assign, to each of the slices, data representing a frame associated with the slice, a size of the slice, a quality of the slice, an area of the frame covered by the slice, and timing information for the slice.
Clause 102: The computer-readable storage medium of any of clauses 93-101, wherein the instructions that cause the processor to calculate the values representing the individual frame quality for each of the video frames comprise instructions that cause the processor to: determine, for each received encoded slice of the video frame, a quality of the slice; and determine, for non-received slices of the video frame, whether the non-received slices were lost or of degraded quality.
Clause 103: A device for processing media data, the device comprising: means for receiving tracking and sensor information from an extended reality (XR) client device; means for generating scene data using the tracking and sensor information, the scene data comprising one or more video frames; means for encoding the video frames to form encoded video frames; means for performing a radio access network (RAN) simulation of delivering the encoded video frames via a radio access network; means for decoding the encoded video frames delivered according to the RAN simulation to form decoded video frames; means for calculating values representing individual frame quality for each of the video frames from the generated one or more video frames and the decoded video frames; and means for determining an overall quality value from the values representing the individual frame quality for each of the video frames.
In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.
By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.
The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and/or firmware.
Various examples have been described. These and other examples are within the scope of the following claims.
This application claims the benefit of U.S. Provisional Application No. 63/026,498, filed May 18, 2020, the entire contents of which are hereby incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 63026498 | May 2020 | US |
