Patent Application
20200137134
A multimedia application generates data representative of pictures in a multimedia stream, e.g., a multimedia stream that has been requested by a user. An encoder encodes the data for each picture and uses the encoded data to form a bitstream that is transmitted over a network to a decoder. The decoder decodes the bitstream and provides the decoded video information to a multimedia application or any other application for display to the user. In certain low latency multimedia encoding applications, multiple encoding sessions are active to encode data for different bitstreams at the same time. Encoded output from each session is required to be ready at a certain rate to meet latency requirements. For example, in some video applications, encoded output data is required to meet latency requirements to ensure a satisfactory user experience.
The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings. The use of the same reference symbols in different drawings indicates similar or identical items.
To illustrate, the encoder of a multimedia system encodes a first portion of a frame at a first session until a preemption requirement is met. The preemption requirement is configurable to include any preemption condition suitable to application or system requirements, such as a predetermined portion size of a frame (e.g., number of coding blocks, such as macroblocks in the H.264 encoding standard), a predetermined encoded bitstream output size, or an encoding time limit for each encoding session. In response to the preemption requirement being met, the encoder outputs the encoded portion, stores session information and context for the encoded portion at a buffer, and sends feedback to the first session that the requested portion of the bitstream is ready so the multimedia application can start processing the bitstream with low latency. The encoder then switches to the next active session and encodes a first portion of a frame at the second session until a preemption requirement is met. In response to the preemption requirement, the encoder outputs the encoded portion of the frame of the second session, stores session information and context for the encoded portion at the buffer, and sends feedback to the second session that the requested portion of the bitstream is ready so the multimedia application can start processing the bitstream. Once all active encoding sessions are complete, the encoder returns to the first preempted session, loads the stored session information and context for the first session and encodes a second portion of the frame at the first session. By loading the stored session information and context, the encoder maintains continuity between encoding the first portion of the frame of the first session and encoding the second portion of the frame of the first session. By switching between active encoding sessions in response to a preemption requirement, and loading stored context information in response to returning to a previously preempted encoding session, the encoder seamlessly encodes frames at multiple encoding sessions at low latency without introducing artifacts, such that the output bitstream is the same as if it were encoded at once as a single frame. Target bitrate and other encoder tunings also remain the same as if the entire frame were encoded at once.
The encoder 120 is configured to receive digital information that represents a stream or sequence of image frames (e.g., frame 104) in a multimedia stream. The term “multimedia” refers to either video only or a combination of video and audio. The encoder 120 encodes the digital information for transmission over a network 140 such as a wide area network (WAN), an intranet, an Internet, a wireless network, and the like. For example, in some embodiments the encoder 120 is used to encode the digital information according to an encoding standard such as Moving Picture Expert Group (“MPEG”)-2, MPEG-4, Advanced Video Coding (“AVC”), and the like.
The encoder 120 is configured to switch between encoding portions of frames from multiple active sessions at low latency in response to a preemption requirement 122 being met. Thus, the encoder 120 temporarily suspends the encoding of a current frame of an active session and encodes a portion of a current frame of another active session. Each active session is associated with a client or an application. For example, in some embodiments, the encoder 120 switches between encoding bitstreams received from two or more clients, with each client bitstream representing an active session. In some embodiments, the encoder 120 switches between encoding bitstreams received from two or more applications, such as two or more video games being encoded by a single encoder, with each application representing an active session. In some embodiments, the encoder 120 switches between encoding bitstreams received from two or more virtual machines (VMs), which each VM representing an active session.
In the depicted example, the encoder 120 switches between encoding frames at three active encoding sessions: a first session 102, a second session 112, and a third session 122. The encoder 120 begins by encoding portion 0105 of frame 104 at the first session 102. When the preemption requirement 122 is met, the encoder 120 outputs the encoded portion 0105 to the application 130, stores session information and context 127 for the encoded portion 0105 at the buffer 135, sends feedback to the first session 102 that requested portion of the bitstream is ready so the application 130 can start processing the bitstream, and preempts the first session 102. The session information and context 127 includes identification of the first session 102 and identification of information such as the frame, row, macroblocks, codec information, pixel values, configuration, and settings.
The encoder 120 then switches to the next active session to encode portion 0115 of frame 0114 at the second session 112 until the next preemption requirement 122 is met. In response to the preemption requirement 122 being met, the encoder 120 outputs the encoded portion 0115 to the application 130, stores session information and context 127 for the encoded portion 0115 at the buffer 135, sends feedback to the second session 112 that requested portion of the bitstream is ready so the application 130 can start processing the bitstream, and preempts the second session 112. The encoder 120 then switches to the next active session to encode portion 0125 of frame 0124 at the third session 122 until the next preemption requirement 122 is met. In response to the preemption requirement 122 being met, the encoder 120 outputs the encoded portion 0125 to the application 130, stores session information and context 127 for the encoded portion 0125 at the buffer 135, sends feedback to the third session 122 that requested portion of the bitstream is ready so the application 130 can start processing the bitstream, and preempts the third session 122. For ease of illustration, a single application 130 is illustrated as executing all three sessions 102, 112, 122, however, it is understood that in some embodiments different applications are executing at some or all of the sessions 102, 112, 122.
In response to determining that the encoder 120 has encoded a portion from each of the active sessions, the encoder 120 returns to the first session 102. The encoder loads the stored session information and context 127 for the previously encoded portion 0105 from the buffer 135 and encodes the next portion 1106 of frame 0104 at the first session 102 based on the stored context 127. The encoder 120 continues switching between sessions in this fashion, such that, in the depicted example, the encoder encodes the portions of the frames of the first, second, and third sessions 102, 121, 122 in the following order: 1. Portion 0105 (first session 102), 2. Portion 0115 (second session 112), 3. Portion 0125 (third session 122), 4. Portion 1106 (first session 102), 5. Portion 1116 (second session 112), 6. Portion 1126 (third session 122), 7. Portion 2107 (first session 102), 8. Portion 2117 (second session 112), 9. Portion 2127 (third portion 122), 10. Portion 3108 (first session 102), 11. Portion 3118 (second session 112), 12. Portion 3128 (third session 122). After completing encoding all the portions of each of frame 0104, 114, and 124, the encoder 120 proceeds to encode the first portion of the next frame of each session.
The preemption requirement 122 is configurable based on the latency and throughput requirements of the application 130 and the throughput rate of the encoder 120. For example, some applications have an end-to-end latency requirement from video capture to display of one to two frames. The preemption requirement 122 is programmably configured to meet the requirements of the application 130 and the capabilities of the encoder 120. In some embodiments, the preemption requirement 122 is based on a portion size of a frame. For example, in some embodiments, the preemption requirement 122 is met when a predetermined fraction (e.g., an eighth) of a frame has been encoded or when a predetermined number of macroblocks have been encoded. In some embodiments, the preemption requirement 122 is based on the encoded output of the encoder 122 reaching a predetermined size. For example, in some embodiments, the preemption requirement 122 is met when the size of the encoded portion output by the encoder 122 reaches the packet size for the network 140. Thus, once a predetermined number of bytes, based on the network packet size, is encoded, the preemption requirement 122 is met. In some embodiments, the preemption requirement 122 is based on a time limit, e.g., 1 ms, for each portion of a frame per encoding session.
By encoding a portion of each frame of each session at a time, the encoder 120 maintains a high throughput, resulting in low latency. Further, by storing and subsequently accessing session and context information 127 for each encoded portion, and basing the encoding of each successive portion of the frame for each session on the session and context information 127 of the previously encoded portion, the encoder 120 incrementally encodes all the portions of the frame such that, once the entire frame has been encoded, the encoded frame does not differ from a frame that was encoded in a single encoding session. Thus, by switching between sessions to encode portions of frames at each session in this manner, the encoder 120 reduces latency while retaining encoding quality.
The encoder 120 loads the context information 135 for portion 0 of the current frame at session 0 from the buffer 125. Based on the context information 135, the encoder 120 encodes the next portion (portion 1206) of the current frame at Session 0 until the preemption requirement 122 is met. In response to the preemption requirement 122 being met, the encoder 120 outputs the encoded portion to the application 130 for processing and stores context information 136 for portion 1 of the current frame at Session 0 at the buffer 125. The encoder 120 preempts the current session and switches to the next active session, Session 1. The encoder loads the context information 145 for portion 0 of the current frame at Session 1 from the buffer 125. Based on the context information 145, the encoder 120 encodes the next portion (portion 1216) of the current frame at Session 1 until the preemption requirement is met. In response to the preemption requirement 122 being met, the encoder 120 outputs the encoded portion to the application 130 for processing and stores context information 146 for portion 1 of the current frame at Session 1 at the buffer 125. The encoder 120 preempts the current session and, in response to detecting that there are no more active sessions, the encoder 120 returns to the first active session, Session 0 to repeat the process for the next portion.
In operation, the macroblock limit 324 is set to N macroblocks such that the preemption requirement 322 is met when the encoder 320 has completed encoding N macroblocks. The encoder 320 encodes macroblocks MB0302, MB1303, and so on until MBN 304, at which point the macroblock limit 324 has been reached and the preemption requirement 322 is met. The encoder 320 outputs the encoded macroblocks MB0302, MB1303, . . . , MBN 304 to an application (not shown) for processing, and stores session information and context (not shown) at a buffer (not shown). The encoder 320 preempts the current encoding session and switches to the next active session.
In operation, the bitstream calculator 424 is set to a threshold size of M bytes before the preemption requirement 422 is met. The encoder 420 encodes macroblocks MB0402, MB1403, and so on until MBN 404, at which point the bitstream calculator 424 determines that the threshold of M bytes has been reached and the preemption requirement 422 is met. The encoder 420 outputs the encoded macroblocks MB0402, MB1403, . . . , MBN 404 to an application (not shown) for processing, and stores session information and context (not shown) at a buffer (not shown). The encoder 420 preempts the current encoding session and switches to the next active session.
In operation, the time limit 524 is set to a threshold time limit of X ms before the preemption requirement 522 is met. The encoder 520 encodes macroblocks MB0502, MB1503, and so on until MBN 504, until the time limit 524 of X ms has been reached and the preemption requirement 522 is met. The encoder 520 outputs the encoded macroblocks MB0502, MB1503, MBN 504 to an application (not shown) for processing, and stores session information and context (not shown) at a buffer (not shown). The encoder 520 preempts the current encoding session and switches to the next active session.
If, at block 606, the preemption requirement 122 has been met, the method flow continues to block 608. At block 608, the encoder 120 outputs the encoded portion of frame N at the current session to the application 130 for transmission across the network 140. At block 610, the encoder 120 stores session information and context 127 for the current portion of frame N of the current session at the buffer 125. At block 612, the encoder 120 preempts the current encoding session. At block 614, the encoder 120 switches to the next active encoding session, and the method flow continues back to block 602.
In some embodiments, the apparatus and techniques described above are implemented in a system having one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips), such as the multimedia system described above with reference to
A computer readable storage medium includes any non-transitory storage medium, or combination of non-transitory storage media, accessible by a computer system during use to provide instructions and/or data to the computer system. Such storage media includes, but is not limited to, optical media (e.g., compact disc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media (e.g., floppy disc, magnetic tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or Flash memory), or microelectromechanical systems (MEMS)-based storage media. The computer readable storage medium may be embedded in the computing system (e.g., system RAM or ROM), fixedly attached to the computing system (e.g., a magnetic hard drive), removably attached to the computing system (e.g., an optical disc or Universal Serial Bus (USB)-based Flash memory), or coupled to the computer system via a wired or wireless network (e.g., network accessible storage (NAS)).
In some embodiments, certain aspects of the techniques described above are implemented by one or more processors of a processing system executing software. The software comprises one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer readable storage medium. The software includes the instructions and certain data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer readable storage medium includes, for example, a magnetic or optical disk storage device, solid state storage devices such as Flash memory, a cache, random access memory (RAM) or other non-volatile memory device or devices, and the like. The executable instructions stored on the non-transitory computer readable storage medium is in source code, assembly language code, object code, or other instruction format that is interpreted or otherwise executable by one or more processors.
Note that not all of the activities or elements described above in the general description are required, that a portion of a specific activity or device is not necessarily required, and that one or more further activities may be performed, or elements included, in addition to those described. Still further, the order in which activities are listed are not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.
Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature of any or all the claims. Moreover, the particular embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the claims below. It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the claims below.
