Patent Application
20180018999
The disclosed embodiments of the present invention relate to video data processing, and more particularly, to a video processing system using a ring buffer and a racing-mode ring buffer access control scheme.
One conventional video system design may include a video transmitting system (which may serve as a video recording system) and a video receiving system (which may serve as a video playback system). Regarding the video transmitting system, it may be composed of a plurality of processing stages, including a video encoder, an audio/video multiplexing circuit, a transmitting circuit, etc. Regarding the video receiving system, it may be composed of a plurality of processing stages, including a receiving circuit, an audio/video demultiplexing circuit, a video decoder, a display engine, etc. However, the conventional video system design may fail to meet the requirements of some ultra-low latency applications due to long playback latency at the video receiving system. Hence, there is a need for an innovative low-latency and high-performance video receiving system.
In accordance with exemplary embodiments of the present invention, a video processing system using a ring buffer and a racing-mode ring buffer access control scheme is proposed to solve the above-mentioned problem.
According to a first aspect of the present invention, an exemplary video processing system is disclosed. The exemplary video processing system includes a storage device, an audio/video demultiplexing circuit, and a video decoder. The storage device has a bitstream buffer that is a ring buffer. The audio/video demultiplexing circuit is arranged to receive an input data, and perform an audio/video demultiplexing operation upon the input data to write data of a video bitstream into the ring buffer. The video decoder is arranged to fetch data of the video bitstream from the ring buffer, and perform a video decoding operation upon the fetched data of the video bitstream.
According to a second aspect of the present invention, an exemplary video processing system is disclosed. The exemplary video processing system includes a storage device, a video decoder, and a display engine. The storage device includes a display buffer that is a ring buffer. The video decoder is arranged to perform a video decoding operation upon data of a video bitstream to write pixel data of a reconstructed video frame into the ring buffer. The display engine is arranged to fetch pixel data of the reconstructed video frame from the ring buffer, and drive a display device according to the fetched pixel data of the reconstructed video frame.
According to a third aspect of the present invention, an exemplary video processing system is disclosed. The exemplary video processing system includes a storage device, a receiving circuit, and an audio/video demultiplexing circuit. The storage device includes a data buffer that is a ring buffer. The receiving circuit is arranged to receive packets and unpack the packets to write payload data of the packets into the ring buffer. The audio/video demultiplexing circuit is arranged to fetch an input data from the ring buffer, and perform an audio/video demultiplexing operation upon the fetched input data, wherein the fetched input data comprise payload data of at least one of the packets.
These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.
Certain terms are used throughout the description and following claims to refer to particular components. As one skilled in the art will appreciate, manufacturers may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not function. In the following description and in the claims, the terms “include” and “comprise” are used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to . . . ”. Also, the term “couple” is intended to mean either an indirect or direct electrical connection. Accordingly, if one device is electrically connected to another device, that connection may be through a direct electrical connection, or through an indirect electrical connection via other devices and connections.
The video transmitting system 100 may serve as a video recording system that is used to encode video frames provided from one or more video sources (not shown) and then transmit encoded video data to the video receiving system 120 via the communication link 110. The video receiving system 120 may serve as a video playback system that is used to receive encode video frame data from the communication link 110 and then decode the encoded video frame data to generate reconstructed video frames to a display device 140 for video playback. For example, the display device 140 may be a display screen of a VR headset. In addition, the communication link 110 may be implemented using a wired communication link or a wireless communication link.
The receiving circuit 122 receives packets from the communication link 110, and unpacks the packets to write payload data of the packets into the data buffer 134. The payload data of the packets may include encoded video data, encoded audio data and other user defined data. The audio/video demultiplexing circuit 126 fetches an input data from the data buffer 134, and performs an audio/video demultiplexing operation upon the fetched input data, where the fetched input data may include payload data of at least one of the packets received by the receiving circuit 122. Due to audio/video demultiplexing, a video bitstream and an audio bitstream are separated and forwarded to the bitstream buffer 136 and the audio data path 133, respectively. In other words, the audio/video demultiplexing circuit 126 performs the audio/video demultiplexing operation upon the fetched input data to write data of a video bitstream into the bitstream buffer 136 and supply data of an audio bitstream to the audio data path 133. The audio bitstream is decoded by the audio data path 134 to obtain audio data for audio playback. Regarding the video processing and playback, the video decoder 128 fetches data of the video bitstream from the bitstream buffer 136, and performs a video decoding operation upon the fetched data of the video bitstream to write pixel data of a reconstructed video frame into the display buffer 138.
A coding block in a video frame may be encoded using an intra prediction mode or an inter prediction mode. When a coding block in a current video frame is encoded using the inter prediction mode, a predicted block used to reconstruct the coding block of the current video frame is found in a reference frame which is a previously reconstructed video frame and stored in the reference frame buffer 139. Hence, a reconstructed video frame generated from decoding a previous video frame by the video decoder 128 is further stored in the reference frame buffer 139 to serve as a reference frame that may be used to decode a current video frame.
The display engine 130 is a driving circuit controlled by the display control circuit 132. The display engine 130 fetches pixel data of the reconstructed video frame from the display buffer 138, and drives the display device 140 according to the fetched pixel data of the reconstructed video frame.
In this embodiment, the storage device 124 may be implemented using an internal storage device, an external storage device, or a combination of an internal storage device and an external storage device. For example, the internal storage device may be a static random access memory (SRAM) or may be flip-flops; and the external storage device may be a dynamic random access memory (DRAM) or may be a flash memory.
Since different processing stages in the video receiving system 120 may have different data processing rates, one buffer is coupled between different processing stages. As shown in
As shown in
During audio/video demultiplexing of an input data (which contains encoded video data and encoded audio data) of one video frame, the audio/video demultiplexing circuit 126 writes data of a video bitstream into the bitstream buffer 136, such that a write pointer wptr (which is indicative of a current write address of writing data of the video bitstream into the bitstream buffer 136) moves downwards. Specifically, the audio/video demultiplexing circuit 126 analyzes the input data (which contains encoded video data and encoded audio data), and writes data of a video bitstream into the bitstream buffer 136. Hence, the audio/video demultiplexing circuit 126 updates the write pointer wptr according to the size of video data stored into the bitstream buffer 136. When the write pointer wptr reaches the bottom address v_end, the write pointer wptr will roll back to the top address v_start, such that writing data of the video bitstream into the bitstream buffer 136 continues seamlessly.
During video decoding of a video bitstream of one video frame, the video decoder 128 reads data of the video bitstream from the bitstream buffer 136, such that a read pointer rptr (which is indicative of a current read address of reading data of the video bitstream from the bitstream buffer 136) moves downwards. When the read pointer rptr reaches the bottom address v_end, the read pointer wptr will roll back to the top address v_start, such that reading data of the video bitstream from the bitstream buffer 136 continues seamlessly.
After an old data segment stored in the bitstream buffer 136 is read and processed by the video decoder 128, a storage area which buffers the old data segment can be reused to buffer a new data segment generated from the audio/video demultiplexing circuit 126. In addition, after the old data segment (which has been read and processed by the video decoder 128) is overwritten by the new data segment generated from the audio/video demultiplexing circuit 126, the storage area can be read by the video decoder 128 again to obtain the new data segment. Due to inherent characteristics of the ring buffer, the write pointer wptr chases the read pointer rptr, and the read pointer rptr also chases the write pointer wptr. A racing mode between the read pointer rptr and the write pointer wptr may be employed to control access (read/write) of the bitstream buffer 136 that is a ring buffer. In accordance with the racing-mode ring buffer access control scheme, the audio/video demultiplexing circuit 126 is configured to include a write controller 202 which updates the write pointer wptr to the video decoder 128, and the video decoder 128 is configured to include a read controller 204 which updates the read pointer rptr to the audio/video demultiplexing circuit 126.
At step 404, the write controller 202 compares its write pointer wptr with the received read pointer rptr to check if a distance between the write pointer wptr and the read pointer rptr does not reach a threshold (e.g., wptr !=rptr−1). When the distance between the write pointer wptr and the read pointer rptr does not reach the threshold (e.g., wptr !=rptr−1), the write controller 202 can continue or resume writing data of the video bitstream into the bitstream buffer 136 (step 408). In this example, the threshold is equal to a step size of incrementing the write pointer wptr. However, this is for illustrative purposes only, and is not meant to be a limitation of the present invention.
When the distance between the write pointer wptr and the read pointer rptr reaches the threshold (e.g., wptr==rptr−1), meaning that the write pointer wptr will catch up the read pointer rptr soon, the write controller 202 stops writing data of the video bitstream into the bitstream buffer 136 (step 406). In this way, the racing-mode ring buffer access control scheme prevents the audio/video demultiplexing circuit 126 from overwriting encoded video data that is not processed by the video decoder 128 yet. The write controller 202 does not resume writing data of the video bitstream into the bitstream buffer 136 until the video decoder 128 updates the read pointer rptr to a new value. Hence, when the distance between the write pointer wptr and the read pointer rptr does not reach the threshold (e.g., wptr !=rptr−1) due to the read pointer rptr updated to a new value under a condition that writing data of the video bitstream into the bitstream buffer 136 is paused, the write controller 202 resumes writing data of the video bitstream into the bitstream buffer 136 (steps 404 and 408).
At step 408, the write controller 202 updates the write pointer wptr to move the write pointer wptr for writing data of the video bitstream into a next write address of the bitstream buffer 136. For example, the write controller 202 updates the write pointer wptr according to the following pseudo code.
After data of the video bitstream is written into the bitstream buffer 136 according to the updated write pointer wptr, the write controller 202 checks if there are more data needed to be stored into the bitstream buffer 136 (step 410). When there is no data needed to be stored into the bitstream buffer 136, the write operation is ended. Otherwise, the control flow proceeds with step 404.
At step 504, the read controller 204 compares its read pointer rptr with the received write pointer wptr to check if the read pointer rptr does not catch up the write pointer wptr (e.g., rptr !=wptr). When the read pointer rptr does not catch up the write pointer wptr (e.g., rptr !=wptr), the read controller 204 can continue or resume reading data of the video bitstream from the bitstream buffer 136 (step 508).
However, when the read point rptr catches up the write pointer wptr (e.g., rptr==wptr), the read controller 204 stops reading data of the video bitstream from the bitstream buffer 136 (step 506). In this way, the racing-mode ring buffer access control scheme prevents the video decoder 128 from retrieving wrong encoded video data from the bitstream buffer 136. The read controller 204 does not resume reading data of the video bitstream from the bitstream buffer 136 until the audio/video demultiplexing circuit 126 updates the write pointer wptr to a new value. Hence, when the read pointer rptr does not catch up the write pointer wptr (e.g., rptr !=wptr) due to the write pointer wptr updated to a new value under a condition that reading data of the video bitstream from the bitstream buffer 136 is paused, the read controller 204 resumes reading data of the video bitstream from the bitstream buffer 136 (steps 504 and 508).
At step 508, the read controller 204 calculates a read length which indicates an amount of data still waiting to be fetched from the bitstream buffer 136 to the video decoder 128 for further processing. For example, the read controller 204 calculates the read length according to the following pseudo code.
At step 510, the read controller 204 updates the read pointer rptr to move the read pointer rptr for reading data of the video bitstream from a next read address of the bitstream buffer 136. For example, the read controller 204 updates the read pointer rptr according to the following pseudo code.
After data of the video bitstream is read from the bitstream buffer 136 according to the updated read pointer rptr, the read controller 204 checks if the frame decode operation is finished (step 512). When the frame decode operation is finished, the read operation is ended. Otherwise, the control flow proceeds with step 504.
As shown in
Since the display buffer 138 is a ring buffer, the display buffer 138 can be accessed (read/written) in a direction from the top address Buf_start to the bottom address Buf_end and then rolling back to the top address Buf_start from the bottom address Buf_end.
During video decoding of one video frame, the video decoder 128 writes pixel data of a reconstructed video frame into the display buffer 138, such that a write address of the pixel data moves downwards. When the write address reaches the bottom address Buf_end, the write address will roll back to the top address Buf_start, such that writing pixel data of the reconstructed video frame into the display buffer 138 continues seamlessly.
During video playback of a reconstructed video frame, the display engine 130 reads pixel data of the reconstructed video frame from the display buffer 138, such that a read address of the pixel data moves downwards. When the read address reaches the bottom address Buf_end, the read address will roll back to the top address Buf_start, such that reading pixel data of the reconstructed video frame from the display buffer 138 continues seamlessly.
After an old data segment stored in the display buffer 138 is read and processed by the display engine 130, a storage area which buffers the old data segment can be reused to buffer a new data segment generated from the video decoder 128. In addition, after the old data segment (which has been read and processed by the display engine 130) is overwritten by the new data segment generated from the video decoder 128, the storage area can be read by the display engine 130 again to obtain the new data segment. The video playback of a video frame is not started unless some pixel data of the video frame are already available in the display buffer 138. Due to inherent characteristics of the ring buffer, the write address chases the read address. To achieve smooth video playback, a read operation of the display buffer 138 cannot be paused. Hence, a racing mode between the read operation and the write operation may be employed to control the write operation of the display buffer 138 that is a ring buffer. In accordance with the racing-mode ring buffer access control scheme, the video decoder 128 is configured to include a reconstructed line counter 602 which maintains a reconstructed line count value R_Line indicative of the number of pixel lines that are reconstructed by the video decoder 128, and the display engine 130 is configured to include a displayed line counter 604 which maintains a displayed line count value D_Line indicative of the number of pixel lines that are displayed on the display device 140 through the display engine 130.
In this embodiment, the video decoder 126 is arranged to perform block-based decoding, and the display engine 130 is arranged to perform scan line based displaying. Hence, the reconstructed line counter 602 increases the reconstructed line count value R_Line by a first increment value (e.g., a value equal to a height of one coding block) each time one coding block row (e.g., MB row, SB row, or CTU row) is completely decoded. For example, assuming that one coding block has N pixels in a column direction, the reconstructed line count value R_Line is updated by R_Line+N when one coding block row is completely decoded. In addition, the displayed line counter 604 increases the displayed line count value D_Line by a second increment value (e.g., “1”) each time one pixel line is completely displayed. For example, the displayed line count value D_Line is updated by D_Line+1 when one pixel line is completely displayed.
At step 804, the display control circuit 132 compares the reconstructed line count value R_Line and the displayed line count value D_Line to determine if the write operation of the display buffer 138 should be stopped. In this example, the display control circuit 132 checks if a distance between the reconstructed line count value R_Line and the displayed line count value D_Line does not reach a threshold (e.g., R_Line !=D_Line+k), where the threshold k may be set by any positive integer value, depending upon the actual design considerations. When the distance between the reconstructed line count value R_Line and the displayed line count value D_Line does not reach the threshold (e.g., R_Line !=D_Line+k), the display control circuit 132 does not assert a control signal Sctrl for instructing the video decoder 128 to stop writing pixel data of the reconstructed video frame into the display buffer 138. Next, the flow proceeds with step 810, such that the display engine 130 continues reading pixel data of the reconstructed video frame from the display buffer 138 and driving the display device 140 to display the pixel data of the reconstructed video frame. Hence, at step 810, one pixel line is displayed on the display device 140 through the display engine 130, and the displayed line counter 604 updates the displayed line count value D_Line (e.g., D_Line=D_Line+1) correspondingly.
However, when the distance between the reconstructed line count value R_Line and the displayed line count value D_Line reaches the threshold (e.g., R_Line==D_Line+k), meaning that the video decoding speed is much faster than the video playback speed, the display control circuit 132 asserts the control signal Sctrl for instructing the video decoder 128 to stop writing pixel data of the reconstructed video frame into the display buffer 138 (step 806). In this way, the racing-mode ring buffer access control scheme prevents the video decoder 128 from overwriting pixel data that is not processed by the display engine 130 for video playback yet. As mentioned above, a read operation of the display buffer 138 cannot be paused for achieving smooth video playback. Next, the flow proceeds with step 808, such that the display engine 130 continues reading pixel data of the reconstructed video frame from the display buffer 138 and driving the display device 140 to display the pixel data of the reconstructed video frame. Similarly, at step 808, the displayed line counter 604 updates the displayed line count value D_Line (e.g., D_Line=D_Line+1) correspondingly.
The display control circuit 132 does not instruct the video decoder 128 to resume writing pixel data of the reconstructed video frame into the display buffer 138 until the display engine 130 (particularly, the displayed line counter 604) updates the displayed line count value D_Line to a new value. Hence, when the distance between the reconstructed line count value R_Line and the displayed line count value D_Line does not reach the threshold (e.g., R_Line !=D_Line+k) due to the displayed line count value D_Line updated to a new value under a condition that writing pixel data of the reconstructed video frame into the display buffer 138 is paused, the display control circuit 132 de-asserts the control signal Sctrl for instructing the video decoder 128 to resume writing pixel data of the reconstructed video frame into the display buffer 138 (steps 804 and 810).
At step 812, the display control circuit 132 checks if there are more pixel lines needed to be displayed. When no pixel line is needed to be displayed, the display control operation is ended. Otherwise, the control flow proceeds with step 804.
In one exemplary design, the communication link 110 shown in
In another exemplary design, the communication link 110 shown in
As shown in
During packet receiving of one video frame, the receiving circuit 122 writes payload data of the packets into the data buffer 134, such that a write pointer wptr (which is indicative of a current write address of writing payload data of the packets into the data buffer 134) moves downwards. Specifically, the receiving circuit 122 receives a packet from the communication link 110, and unpacks the received packet to write payload data included in the received packet into the data buffer 134. Hence, the receiving circuit 122 updates the write pointer wptr according to the number of packets received and the size of payload data included in each packet received. When the write pointer wptr reaches the bottom address v_end, the write pointer wptr will roll back to the top address v_start, such that writing payload data of the packets into the data buffer 134 continues seamlessly.
During audio/video demultiplexing of an input data (which includes payload data of at least one of the packets) of one video frame, the audio/video demultiplexing circuit 126 reads the input data from the data buffer 134, such that a read pointer rptr (which is indicative of a current read address of reading the input data from the data buffer 134) moves downwards. When the read pointer rptr reaches the bottom address v_end, the read pointer wptr will roll back to the top address v_start, such that reading the input data from the data buffer 134 continues seamlessly.
After an old data segment stored in the data buffer 134 is read and processed by the audio/video demultiplexing circuit 126, a storage area which buffers the old data segment can be reused to buffer a new data segment generated from the receiving circuit 122. In addition, after the old data segment (which has been read and processed by the audio/video demultiplexing circuit 126) is overwritten by the new data segment generated from the receiving circuit 122, the storage area can be read by the audio/video demultiplexing circuit 126 again to obtain the new data segment. Due to inherent characteristics of the ring buffer, the write pointer wptr chases the read pointer rptr, and the read pointer rptr also chases the write pointer wptr. A racing mode between the read pointer rptr and the write pointer wptr may be employed to control access (read/write) of the data buffer 134 that is a ring buffer. In accordance with the racing-mode ring buffer access control scheme, the audio/video demultiplexing circuit 126 is configured to include a write controller 902 which updates the write pointer wptr to the audio/video demultiplexing circuit 126, and the audio/video demultiplexing circuit 126 is configured to include a read controller 904 which updates the read pointer rptr to the receiving circuit 122.
At step 1104, the write controller 902 compares its write pointer wptr with the received read pointer rptr to check if a distance between the write pointer wptr and the read pointer rptr does not reach a threshold (e.g., wptr !=rptr−1). When the distance between the write pointer wptr and the read pointer rptr does not reach the threshold (e.g., wptr !=rptr−1), the write controller 902 can continue or resume writing payload data of the packets into the data buffer 134 (step 1108). In this example, the threshold is equal to a step size of incrementing the write pointer wptr. However, this is for illustrative purposes only, and is not meant to be a limitation of the present invention.
However, when the distance between the write pointer wptr and the read pointer rptr reaches the threshold (e.g., wptr==rptr−1), meaning that the write pointer wptr will catch up the read pointer rptr soon, the write controller 902 stops writing payload data of the packets into the data buffer 134 (step 1106). In this way, the racing-mode ring buffer access control scheme prevents the receiving circuit 122 from overwriting payload data that is not processed by the audio/video demultiplexing circuit 126 yet. The write controller 902 does not resume writing payload data of the packets into the data buffer 134 until the audio/video demultiplexing circuit 126 updates the read pointer rptr to a new value. Hence, when the distance between the write pointer wptr and the read pointer rptr does not reach the threshold (e.g., wptr !=rptr−1) due to the read pointer rptr updated to a new value under a condition that writing payload data of the packets into the data buffer 134 is paused, the write controller 902 resumes writing data of the video bitstream into the data buffer 134 (steps 1104 and 1108).
At step 1108, the write controller 902 updates the write pointer wptr to move the write pointer wptr for writing data of the video bitstream into a next write address of the data buffer 134. For example, the write controller 902 updates the write pointer wptr according to the following pseudo code.
After payload data is written into the bitstream buffer 136 according to the updated write pointer wptr, the write controller 902 checks if there are more data needed to be stored into the data buffer 134 (step 1110). When there is no data needed to be stored into the data buffer 134, the write operation is ended. Otherwise, the control flow proceeds with step 1104.
At step 1204, the read controller 904 compares its read pointer rptr with the received write pointer wptr to check if the read pointer rptr does not catch up the write pointer wptr (e.g., rptr !=wptr). When the read pointer rptr does not catch up the write pointer wptr (e.g., rptr !=wptr), the read controller 204 can continue or resume reading payload data of the packets from the data buffer 134 (step 1208).
However, when the read point rptr catches up the write pointer wptr (e.g., rptr==wptr), the read controller 904 stops reading payload data of the packets from the data buffer 134 (step 1206). In this way, the racing-mode ring buffer access control scheme prevents the audio/video demultiplexing circuit 126 from retrieving wrong payload data from the data buffer 134. The read controller 904 does not resume reading payload data of the packets from the data buffer 134 until the receiving circuit 122 updates the write pointer wptr to a new value. Hence, when the read pointer rptr does not catch up the write pointer wptr (e.g., rptr !=wptr) due to the write pointer wptr updated to a new value under a condition that reading payload data of the packets from the data buffer 134 is paused, the read controller 904 resumes reading payload data of the packets from the data buffer 134 (steps 1204 and 1208).
At step 1208, the read controller 204 calculates a read length which indicates an amount of data still waiting to be fetched from the data buffer 134 to the audio/video demultiplexing circuit 126 for further processing. For example, the read controller 904 calculates the read length according to the following pseudo code.
At step 1210, the read controller 904 updates the read pointer rptr to move the read pointer rptr for reading payload data of the packets from a next read address of the data buffer 134. For example, the read controller 904 updates the read pointer rptr according to the following pseudo code.
After payload data is read from the data buffer 134 according to the updated read pointer rptr, the read controller 904 checks if the frame audio/video demultiplexing operation is finished (step 1212). When the frame audio/video demultiplexing operation is finished, the read operation is ended. Otherwise, the control flow proceeds with step 1204.
To achieve smooth video playback, a read operation of the display buffer 138 cannot be paused. However, due to certain factors, it is possible that the video decoder 128 is unable to provide some pixel data of one reconstructed video frame in time. For example, the decoding speed becomes unstable because of the bandwidth variation of the communication link (e.g., a wired communication link or a wireless communication link) 110 between the video transmission system. 100 and the video receiving system 120. When the reconstructed pixel data of a portion of a reconstructed video frame are not ready at the time the display engine 130 starts driving the display device 140 to display the portion of the reconstructed video frame, a broken picture may be displayed on the display device 140, thus resulting in a poor video quality. To prevent the display device 140 from displaying a broken picture, the present invention further proposes a frame base address switching scheme.
Please refer to
The reconstructed frame base address is indicative of a start address of a reconstructed video frame (which is a current frame being decoded) stored in the display buffer 138. The reference frame base address is indicative of a start address of a reference video frame (which is a previous frame already decoded) stored in the reference frame buffer 139. As shown in
When the decoding speed is stable during video playback of the whole of a current video frame, the display device 140 will display the current video frame completely. However, when the decoding speed is only stable during video playback of a portion of a current video frame, the display device 140 displays the portion of the current video frame and further displays a portion of a previous video frame.
Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.
This application claims the benefit of U.S. provisional application No. 62/361,091, filed on Jul. 12, 2016 and incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62361091 | Jul 2016 | US |
