Digital video systems, including wireless systems, are becoming increasingly common. In such systems each picture or frame of a moving image is encoded in a transmitter, transmitted to a receiver, decoded in the receiver, and displayed on a display device. In some such systems a user may perform some action at the transmitter and expect to see an essentially instant response on a display visible to that user or to another user at a different location. An example of this is a user moving a cursor on a hand-held device and expecting to see essentially instantaneous corresponding movement of a cursor on a display located in the same room with the user. Any observable delay, or latency, between the user's action and the response on the display is undesirable. Another example is a video conferencing system, where two users in different locations are exchanging images. Observable latency may occur when buffers cannot keep up with a large incoming bit rate or a bit rate that fluctuates over a large range from frame to frame. To reduce latency, bit rates must be carefully controlled and not allowed to fluctuate outside a restricted range.
The terms compression and quantization refer to reducing a number of bits needed to encode a frame without noticeable degradation of image quality.
A method and system are described for low-latency video. In the method a frame, selected from a group of frames, is divided into P-regions and an I-region based on an assigned refresh pattern in a refresh loop. An I-region bit budget and a P-region bit budget are determined. Quantization parameters are determined using the I-region bit budget and the P-region bit budget. Macroblocks of the selected frame are encoded based on the quantization parameters. An I-complexity and P-complexity are updated and a new frame bit budget is determined. The dividing, determining of the I-region bit budget, determining of the P-region bit budget, determining of quantization parameters, and encoding are repeated for each remaining frame in the group of frames.
A video compression system configured to reduce latency and prevent buffer overflow includes an encoder configured to encode a received frame, a refresh system configured to provide a predetermined refresh pattern to the encoder, and a rate control system, configured to control an encoding rate of the encoder.
To reduce latency, pixels of a given frame in a video transmission may be divided into regions of different designated types. Each region may be further divided into blocks of pixels called macroblocks. Regions of each type may be encoded in different ways, having different amounts of compression, or quantization. For example a region designated as an I-region (“intra” region) may be encoded as if it were a single still image. Encoding of an I-region uses only the bits contained in that region itself. All macroblocks in an I-region are encoded as I-macroblocks. By contrast, a region designated as a P-region (“progressive” region) may be encoded by making use of a region in a previous frame. A macroblock in a P-region may be encoded as either an I-macroblock or a P-macroblock. As an example, a current P-region may be compared with a previously processed region, either I or P, and only those pixels in the current region that have changed are encoded. The current region and the previously processed region may or may not have corresponding locations in a current frame and a previous frame, respectively. An example is provided by a video scene in which an object is moving in front of a stationary background. In a P-region, only those pixels representing the moving object may require encoding. It follows that an I-region will likely require more bits for encoding than does a P-region. It may be desirable, therefore, to reduce the number of I-regions as much as possible. On the other hand, at least one I-region may be necessary to maintain image quality and stop errors propagating from one frame to another. In particular, use of at least one I-region may be a necessity following a scene change, since the first frame of a new scene may have no pixels in common with the last frame of the previous scene. Scene changes are addressed further below.
An example of part of a method for reducing latency is shown in
The refresh pattern may be described as the pattern of movement of the I-region from frame to frame during the refresh loop. The particular movement of the I-region shown in
Referring to
In an embodiment, an initial value of bit rate budget, Br0,0 may be set as
where BPP is a number of bits per picture, or bits per frame, Rb denotes a nominal bit rate and Rf denotes a frame rate. As an example, initial values for the I-complexity and P-complexity for all N frames in the initial refresh loop may be set as:
Once the initial values are set, the method operates in a cycle, running over consecutive frames. A refresh loop having a pre-defined refresh pattern is applied to a first group of N frames. The refresh pattern assigns a type—either I or P—to each region of each frame, in the manner of
Referring once again to
In an embodiment shown in
In Equation 2, K is a balancing constant, defined in order to constrain the relative values of the I-region quantization parameters and the P-region quantization parameters as a further guard against excessive latency and buffer overflow. For example, the quantization parameters may be constrained according to Equation 3:
where QIi,j and QPi,j are averages of quantization parameters over macroblocks of I-regions and P-regions, respectively, and K may be between 1.0 and 2.0, inclusive.
Corresponding to I-region bit budget TIi,j, a P-region bit budget TPi,j may be determined 215. In an embodiment, TPi,j may determined by subtracting I-region bit budget from frame bit budget, as in Equation 4:
T
P
i,j
=B
r
i,j
−T
I
i,j. Equation (4)
Once bit budgets TIi,j and TPi,j have been determined, they may be used to determine current quantization parameters 220. These current quantization parameters may then be used to encode the current frame 225, as described above.
Once the current frame is encoded, the complexities CIi,j and CPi,j may be updated for use in determining bit budgets for a corresponding frame j in the next refresh loop 230. According to the index definitions above, these complexities are denoted as CIi+1,j and CPi+1,j. These updated complexities may be determined as follows. Let Qa,Ii,j and Qa,Pi,j be determined averages of quantization parameters over all macroblocks in the I- and P-regions, respectively, of the current frame j. These averages may be weighted averages. Let Bu,Ii,j and Bu,Pi,j be bit quantities generated by encoding of the I- and P-regions, respectively. Determine the quantities ĈIi,j and ĈPi,j using Equations 5 and 6, respectively.
Ĉ
I
i,j
=Q
a,I
i,j
B
u,I
i,j Equation (5)
Ĉ
P
i,j
=Q
a,P
i,j
B
u,P
i,j Equation (6)
Finally, determine updated I- and P-complexities using Equations 7 and 8, respectively.
Once the current frame is encoded, a new frame bit budget is determined 240, 245 in a manner depending on whether or not the current frame is the last frame in the current refresh loop 235. If the current frame j is the last frame in current refresh loop i 235, then a new frame bit budget for a first frame in a new refresh loop may be determined 245 using Equation 9:
B
r
i+1,0=BPP+max(LIMIT_VALUE,Bri,N−1−Bu,Ii,N−1−Bu,Pi,N−1). Equation (9)
If the current frame j is not the last frame in current refresh loop i 235, then a new frame bit budget for the next frame in the current refresh loop may be determined 240 using Equation 10:
B
r
i,j+1=BPP+max(LIMIT_VALUE,Bri,j−Bu,Ii,j−Bu,Pi,j). Equation (10)
In equations 9 and 10, the quantity LIMIT_VALUE is introduced to restrict the bit-budget for each single frame to avoid overflow of a coded picture buffer in a rate control system such as the embodiment described below in connection with
Method 200 may be adapted to handle bit rate requirements of a scene change, as described above. The selected frame may be classified as either a low texture frame or a high texture frame based on a number of bits generated in encoding the selected frame. An initial value of one or more quantization parameters for the selected frame is set to an average of quantization parameters of previous frames, in which quantization parameters of previous low texture frames are excluded from this average.
As an example criterion for classifying a frame, a frame may be classified as low texture if a number of bits generated when the frame is encoded, b(t), satisfies Equation 11:
b(t)<0.2bT(t) Equation (11)
where bT(t) is a target bit number. A frame not satisfying Equation 11 is classified as high texture.
System 300 may also include a scene change handling system 325. Scene change handling system 325 includes classification circuitry, configured to classify the selected frame as either low texture or high texture based on a number of bits generated in the encoding of the selected frame; and averaging circuitry configured to determine an average of quantization parameters of previous frames, such that quantization parameters of previous low texture frames are excluded from the determining of the average.
As will be appreciated by a person of ordinary skill in the art, certain embodiments disclosed herein may result in reduced latency video. One example of such embodiments is a video display system as shown in
Another example embodiment of low-latency video compression may be implemented as a video conferencing system, which may be one-way or two-way. Users in two or more locations transmit and receive video images. Audio may be transmitted and received as well. Reduced latency (avoiding delays) may be essential for, for example, creating a more realistic experience for users participating in a video conference.
Embodiments of the present embodiments may be represented as instructions and data stored in a non-transitory computer-readable storage medium. For example, aspects of the present embodiments may be implemented using Verilog, which is a hardware description language (HDL). When processed, Verilog data instructions may generate other intermediary data (e.g., netlists, GDS data, or the like) that may be used to perform a manufacturing process implemented in a semiconductor fabrication facility. The manufacturing process may be adapted to manufacture semiconductor devices (e.g., processors) that embody various aspects of the present embodiments.
Suitable processors include, by way of example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, a graphics processing unit (GPU), a DSP core, a controller, a microcontroller, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), any other type of integrated circuit (IC), and/or a state machine, or combinations thereof.
Other embodiments, uses, and advantages of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. The specification and drawings should be considered exemplary only, and the scope of the disclosure is accordingly intended to be limited only by the following claims and equivalents thereof.