Method and System for Encoding Order and Frame Type Selection Optimization

Abstract

A method for resource allocation for video encoder to achieve the minimum sequence cost within given resource budgets. Optimal video encoder design by deriving the optimal sequence order and frame type selection is invented. In order to achieve computationally practical resource allocation, the current invention utilizes various encoder model and buffer model. The models allow the optimization procedure to assess the best encoding design without actually performing the computationally expensive encoding. Efficient optimization algorithm is also derived to substantially reduce the computations required to search for the optimal action sequence.

Description

BACKGROUND

The present invention relates to the field of encoder control more specifically, the present invention relates to frame encoding decision for video sequence to achieve a minimum sequence cost within a sequence resource budget.

Video compression takes advantage of these redundancies, with the intention to reduce the size of the video data while maintaining the quality as much as possible. Such compression is referred to as lossy (as opposed to lossless), since the original data cannot be recovered exactly. Most modern codecs take advantage of correlation among video frames and transmit the differences between a current frame and a predicted frame compactly represented by prediction parameters. The predicted frame is usually similar to the original one. The residual information, i.e., prediction error, is transmitted together with the prediction parameters. The error signal usually is much smaller than the original signal and can be compactly represented using spatial transform, such as the discrete cosine transform (DCT) with subsequent quantization of the transform coefficients. The quantized coefficients and the prediction parameters are entropy-encoded to reduce furthermore their redundancy.

Inter-prediction exploits the temporal redundancy by using temporal prediction. Due to high level similarity among consecutive frames, temporal prediction can largely reduce the information required to present the video data. The efficiency of temporal prediction can be further improved by taking into account the object motion in the video sequence. The motion parameters associated with temporal prediction are transmitted to the decoder for reconstructing the temporal prediction. This type of prediction is used in MPEG-like codecs. The process of producing intra-prediction is called motion estimation. Typically, it is performed in a block-wise manner, when many modern codec support motion estimation with blocks of adaptive size.

At the other end, intra-prediction takes advantage of spatial redundancy and predicts portions of a frame (blocks) from neighbor blocks within the same frame. Such prediction is usually aware of spatial structures that may occur in a frame, namely smooth regions and edges. In general, larger block size is more coding efficient for smooth areas and smaller block size is more coding efficient for areas with more texture variations. In the latter case, the prediction based on neighboring blocks can improve coding efficiency and a directional prediction can even further improve the efficiency. Such intra-prediction is used in the recent H.264 Advances Video Codec (AVC) standard.

The actual reduction in the amount of transmitted information is performed in the transmission of the residual. The residual frame is divided into blocks, each of which undergoes a DCT-like transform. The transform coefficients undergo quantization, usually performed by scaling and rounding. Quantization allows represent the coefficients by using less precision, thus reducing the amount of information required. Quantized transform coefficients are transmitted using entropy coding. This type of coding exploits the statistical characteristics of the underlying data.

For color video, often they are represented in the RGB color coordinate. However, in most video coding systems, the encoder usually uses the YCbCr color space because it is a more compact representation. Y is the luminance component (luma) and Cb and Cr are chrominance components (chroma) of the color video. The chroma is typically down-sampled to half frame size in each direction because human eyes are less sensitive to the chroma signals and such format is referred to as the 4:2:0 format.

The performance of a lossy video codec is measured as the tradeoff between the amount of bits required to describe the data and the distortion introduced by the compression, referred to as the rate-distortion (RD) curve. As the distortion criterion, the mean squared error (MSE) is usually used. The MSE is often converted into logarithmic units and represented as the peak signal-to-noise ratio (PSNR),

$\begin{array}{cc}p=10{\log }_{10}\left(\frac{y_{\max }^2}{d}\right)& \left(1\right)\end{array}$

where d is the MSE and y_max is the maximum allowed value of the luma pixels, typically 255 if the luma data has an 8-bit precision and is represented in the range 0, . . . , 255.

The H.264 AVC is one of the most recent standards in video compression, offering significantly better compression rate and quality compared to the previous MPEG-2 and MPEG-4 standards and targeted to high definition (HD) content. For example, H.264 delivers the same quality as MPEG-2 at a third to half the data rate.

The encoding process can be briefly described as follows: a frame undergoing encoding is divided into non-overlapping macroblocks, each containing 16×16 luma pixels and 8×8 chroma pixels. Within each frame, macroblocks are arranged into slices, where a slice is a continuous raster scan of macroblocks. Each slice can be encoded independently of the other. Main slice types are P and I. An I slice may contain only I macroblocks; a P slice may contain P or I macroblocks. The macroblock type determines the way it is predicted. P refers to inter-predicted macroblocks; such macroblocks are subdivided into smaller blocks. I refers to intra-predicted macroblocks; such macroblocks are divided into 4×4 blocks (the luma component is divided into 16 blocks; each chroma component is divided into 4 blocks). In I mode (intra-prediction), the prediction macroblock is formed from pixels in the neighbor blocks in the current slice that have been previously encoded, decoded and reconstructed, prior to applying the in-loop deblocking filter. The reconstructed macroblock is formed by imitating the decoder operation in the encoder loop. In P mode (inter-prediction), the prediction macroblock is formed by motion compensation from reference frames. The prediction macroblock is subtracted from the current macroblock. The error undergoes transform, quantization and entropy coding. According to the length of the entropy code, the best prediction mode is selected (i.e., the choice between an I or a P macroblock, the motion vectors in case of a P macroblock and the prediction mode in case of an I macroblock). The encoded residual for the macroblock in the selected best mode is sent to the bitstream.

A special type of frame referred to as instantaneous data refresh (IDR) is used as a synchronization mechanism, in which the reference buffers are reset as if the decoder started “freshly” from the beginning of the sequence. IDR frame is always an I-frame. The use of IDR allows, for example, starting decoding a bitstream not necessarily from its beginning. The set of P and I frames between two IDRs is called a group of pictures (GOP). A GOP always starts with an IDR frame. The maximum GOP size is limited by the standard.

The operating point on the RD curve is controlled by the quantization parameter, determining the “aggressiveness” of the residual quantization and the resulting distortion. In the H.264 standard, the quantization parameter is an integer in the range 0, . . . , 51, denoted here by q′. The quantization step doubles for every increment of 6 in q′. Sometimes, it is more convenient to use the quantization step rather than the quantization parameter, computed according to

$\begin{array}{cc}q=0.85·2^{\frac{q^{\prime }-12}{6}}.& \left(2\right)\end{array}$

In the following, the q and q′ are used interchangeably.

Theoretically, optimal resource allocation for a video sequence requires encoding the sequence with different sets of parameters and selecting one achieving the best result. However, such an approach is impossible due to a very large number of possible combinations of parameters, which leads to a prohibitive computational complexity. Suboptimal resource allocation approaches usually try to model some typical behavior of the encoder as function of the encoding parameters. If the model has an analytical expression which can be efficiently computed, the optimization problem can be practically solved using mathematical optimization. However, since the model is only an approximate behavior of the encoder, the parameters selected using it may be suboptimal. The main difference between existing encoders is the decision process carried out by the bitrate controller that produces the encoder control parameters. Usually, the encoder parameters are selected in a way to achieve the best tradeoff between video quality and bitrate of the produced stream. Controllers of this type are referred to as RDO.

Parameters controlled by the bitrate controller typically include: frame type and reference frame or frames if the frame is a P-frame on the sequence level, and macroblock type and quantization parameter for each macroblock on the frame level. For this reason, it is natural and common to distinguish between two levels of bitrate control: sequence-and frame-level. The sequence-level controller is usually responsible for the frame type selection and allocation of the bit budget for the frame, and the frame-level controller is responsible for selection of the quantization parameter for each macroblock within the frame.

In a conventional coding system, the sequence level control is very limited. The frame type of a frame in a sequence is usually based on its order in a GOP according to a pre-determined pattern. For example, the IBBPBBP . . . pattern is often used in MPEG standard where B is a bi-directional predicted frame. The GOP may be formed by partitioning the sequence into GOP of fixed size. However, when a scene change is detected, it may trigger the start of a new GOP for better coding efficiency. The bit rate allocated to each frame usually is based on a rate control strategy. Such resource allocation for sequence encoding only exercises limited resource allocation and there is more room to improve. Based on the discussion presented here, there is a need for an optimal resource allocation for video sequence as well as for video frame. The current invention addresses the resource allocation for video sequence.

SUMMARY

The present invention provides an optimal resource allocation for video sequence to achieve the best possible quality within the sequence resource budget.

In an exemplary embodiment of this invention, the encoder provides as the output an action sequence of encoder parameters by selecting the frame type, quantization scale, and encoding complexity of picture to produce an encoded version of the sequence to achieve a minimum sequence cost within the sequence resource budget. Due to the high computational complexity of calculating the resource budget and video quality by actually performing the encoding process, the resource budget and the video quality are all estimated based on respective models. The resource budget used in this invention includes bit rate and encoding time.

The optimal encoding process by selecting all possible combinations of frame type for individual frames of the sequence would require extremely high complexity if exhaustive search is used. The current invention uses a branch and bound algorithm to simplify the search procedure for an optimal decision on frame type of the sequence.

The benefits of the current intention can be easily appreciated by any person skilled in the art of video compression.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a shows a block diagram of the optimal resource allocation for video content.

FIG. 1 b shows a general optimal resource allocation process.

FIG. 2 a. Example of buffer model: raw frame buffer.

FIG. 2 b. Example of buffer model: encoder bit buffer.

FIG. 2 c. Example of buffer model: decoder bit buffer.

FIG. 3. Sequence-level controller flow diagram.

FIG. 4. Flow diagram of branch and bound frame type decision algorithm.

FIG. 5. Flow diagram of encoding order optimization

DETAILED DESCRIPTION

The invention is directed to a system and method of encoding video sequence, that generally includes providing as the input a video sequence containing pictures as the input, a resource budget for the sequence, and a sequence cost function. The output is an action sequence of encoder parameters configured to produce an encoded version of the sequence to achieve a minimum sequence cost within the sequence resource budget.

In one embodiment, the resource allocation and control mechanism are generic and in principle can be used with any MPEG-type encoder or a similar mechanism. For illustration, in the examples of embodiments below, it is assumed that the underlying encoding pipeline implements the H.264 AVC encoder, with independent slice encoders working simultaneously. Such an implementation is currently typical of real-time hardware encoders.

A system-level block diagram of exemplary controllers configured according to the invention is shown in FIG. 1. On preferred resource allocation process, illustrated as an Optimal Resource Allocation System is shown in FIG. 1a, where a video content and resource/quality constraints are inputs to the system. The video content may be a video sequence, a video frame, a video field, or a partial frame. The video content is fed to Perceptual Quality Model 20 and Visual Clues Computation 40. Perceptual Quality Model derives the parameters of the perceptual quality model based on the input video content. On the other hand, the Visual Clues Computation also derives the visual clues based on input video content. The information from Perceptual Quality Model and the information from the Visual Clues Computation are used to guide partitioning the input video content. The partitioning is based on a unit smaller than the input video content, such as a macroblock, macroblock pair, or block. The partitioning is performed by the Regions Partitioning 30. The smaller unit for the partitioning is named sub-frame. The information from the Visual Clues Computation is used by both the Bit Production Model 50 and PSNR Model 60. Both the Bit Production Model and the PSNR Model are useful for the encoder to evaluate the performance (bit rate and PSNR) without actually going through the computationally heavy encoding process. While the Bit Production is used as a measure of resource, any other resource measurement such as processing time can be used. Similarly, other quality model such as distortion can be used. The Optimal Resource Allocation 10 then exercises optimal resource allocation among the regions using the Bit Production Model, PSNR Model, and Perceptual Quality Model.

The detailed process of the Optimal Resource Allocation 10 is further explained in FIG. 1b. The Video Quality Computation 80 block estimates the video quality of the video data of regions for the set of given encoder parameters instead of actually compress the video data using encoder parameters. The particular video quality computation in this exemplary case is PSNR shaping. The quality estimation is based on the perceptual quality model and PSNR model. The Resource Computation 82 block estimates the resource consumed for the given set of encoder parameters instead of actually compressing the video data. The bit production model is used to estimate the resource consumed. The estimated resource is compared in the compare process “≦Budget?” block 86. If the result is true, the Switch 84 will close to allow the corresponding estimated quality computation pass to the Select Best Quality and Parameters block 88. The Select Best Quality and Parameters block will compare all the video quality results entering the block and will select the best quality value and output the corresponding encoder parameters.

In one embodiment, the system consists of the sequence-level controller, which is responsible for frame type and reference selection and allocation of the bit budget and initial quantization parameter for the currently encoded frame. The frame-level sequence controller allocates the resources on sub-frame level, in order to utilize the total frame bit budget and achieve the highest perceptual visual quality. For this purpose, a perceptual quality model and a PSNR shaping mechanism are employed. Both controllers are based on the encoder model, which predicts the behavior of the encoder given a set of encoding parameters. The model, in turn, utilizes visual clues, a set of descriptors of video data.

The main focuses of the current invention are the encoder models and the sequence-level controller. The most important features of the sequence-level controller include:

• • A. Content-aware encoder model—an accurate encoder model based on visual clues, allowing optimizing the resource allocation prior to encoding,
  • B. Optimal frame type decision—including IDR insertion at scene cuts, optimization of frame dropping capable of detecting and removing redundant frames, e.g. due to telecine conversion, and optimal reference frame selection based on frame predictability, allowing to achieve compression ratios close to those of codecs using multiple reference frames at significantly lower computational complexity,
  • C. Optimal encoding order decision—allowing out-of-order encoding to achieve better predictability,
  • D. Optimal frame bit allocation, and
  • E. Optimal encoding time allocation, allowing achieving better encoding quality by distributing encoding time between frames according to their coding complexity.

Content-Aware Encoder Model

The purpose of encoder model is to estimate the amount of resources used by the encoder without resorting to the expensive process of encoding itself. Specifically, the amount of bits produced by the encoder, the distortion as the result of encoding and the time complexity of the encoding process are predicted according to the encoder model. The encoder model is disclosed in U.S. patent application Ser. No. 12/040,788, filed Feb. 29, 2008.

In general, if x is the quantity to be predicted (amount of bits, distortion, or time) for the current frame, the predicted value according to the encoder model is computed by

$\begin{array}{cc}\hat{x}=K\sum _{i=1}^{N_{\mathrm{MB}}}{\hat{x}}_i\left(O_i;z_i\right)& \left(3\right)\end{array}$

where {circumflex over (x)}_i is a macroblock-wise predictor, θ_i are the encoding parameters and z_i are the visual clues in macroblock i, N_MB is the number of macroblocks in the frame and K is some normalization factor. Specifically, the following models are used.

Bit Production Model

In each macroblock, the amount of bits produced is proportional to the quantization step q_i⁻¹ and to the macroblock complexity z_i. Depending on the frame type, either texture (in I-frames) of motion complexity (in P-frames) are used. The macroblock-wise predictor is given by

{circumflex over (b)} _i(q_i,z_i)=α₁+α₂z_iq_i⁻¹   (4)

where α1,α2≦0 are the model parameters obtained by training. Assuming that q is constant for the entire frame, the frame-wise predictor of the amount of bits can be therefore expressed as

$\begin{array}{cc}\begin{array}{c}\hat{b}=\sum _{i=1}^{N_{\mathrm{MB}}}{\hat{b}}_i\left(q,z_i\right)\\ =N_{\mathrm{MB}\alpha 1}+\sum _{i=1}^{N_{\mathrm{MB}}}{\alpha }_2z_iq_i^{-1}\\ ={\alpha }^b+{\beta }^bq^{-1}\end{array}& \left(5\right)\end{array}$

where the coefficients α^b=N_MBα1 and β^b=Σ_i=1^NMBα₂z_i depend on the frame content and frame type.

Distortion Model

The luma PSNR is used as an objective distortion criterion. In each macroblock, the PSNR is inversely proportional to the quantization parameter q_i⁻¹ and to the macroblock complexity z_i (texture complexity is used for both P- and I-frames).

{circumflex over (p)} _i(q_i,z_i)=β₁−β₂z_i−β₃q′_i   (6)

where β₁,β₂,β₃≧0 are the model parameters obtained by training. Assuming that q′_i is constant for the entire frame, the predictor of the frame-wise average PSNR can be therefore expressed as

$\begin{array}{cc}\begin{array}{c}\hat{p}=\frac{1}{N_{\mathrm{MB}}}\sum _{i=1}^{N_{\mathrm{MB}}}{\hat{p}}_i\left(q^{\prime },z_i\right)\\ ={\beta }_1-\frac{1}{N_{\mathrm{MB}}}\sum _{i=1}^{N_{\mathrm{MB}}}{\beta }_2z_i-\frac{1}{N_{\mathrm{MB}}}\sum _{i=1}^{N_{\mathrm{MB}}}{\beta }_3q^{\prime }{\alpha }^P+B^pq^{\prime }\end{array}& \left(7\right)\end{array}$

where the coefficients

${\alpha }^P={\beta }_1-\frac{1}{N_{\mathrm{MB}}}\sum _{i=1}^{N_{\mathrm{MB}}}{\beta }_2z_i$

and

${\beta }_p=-\frac{1}{N_{\mathrm{MB}}}\sum _{i=1}^{N_{\mathrm{MB}}}{\beta }_3$

depend on the frame content and frame type.

Distortion Model for Dropped Frames

In some cases, the sequence-level controller may decide to drop a frame. In such a case, it is assumed that the decoder delays the previous frame for a longer duration, equal to the duration of the previous and current (dropped frame). The distortion due to the drop is computed as the average PSNR between a downsampled (16 times in each axis) version the dropped frame and the previous frame. As an exemplary implementation, the average PSNR is computed by first calculating the average MSE and then converting it into logarithmic scale (rather than averaging the PSNR values directly). For example, dropping a frame at the beginning of a new shot would result in a low PSNR, while dropping a frame in a sequence of frames with slow motion will result in high PSNR. It may appear that though the average PSNR is high, in some small portions of the image the distortion due to frame drop is very large (such a situation is typical for a frame with abrupt motion of a foreground object, while the background is static). In order to take into consideration such situations, together with the average PSNR, we measure the 0:1% lowest quantile of the PSNR values, denoted by {circumflex over (p)}^min.

Time Complexity Model

The underlying assumption of the time complexity model is that the encoding pipeline used to encode a macroblock consists of multiple stages, some of which are bypassed according to the encoded macroblock type using some decision tree. Consequently, it can be simplistically assumed that encoding a macroblock of certain type takes a fixed amount of time. Let c_T denote the maximum amount of time it takes to encode a macroblock (nominal encoding time), which depends on the frame type and on the complexity scale c′. Let's denote τ′ the fraction of c_T is takes to encoder macroblock i, the time used to encode a macroblock is τ_i=cTτ′_i. According to the model, the normalized encoding time for macroblock i is given by

{circumflex over (τ)}′_i(q_i,z_i)=max{min{γ₁+γ₂z_i−γ₃q′_i,1},0}T   (8)

where γ1,γ2,γ3≧0 are the model parameters obtained by training, and z_i and q′ are the texture complexity (for I-frames) and motion complexity (for P-frames) respectively. The minimum and maximum is taken in order to ensure that the predicted normalized time value is within a valid range of [0,T] (for a reasonable range of z_i and q′, this assumption will hold). Assuming that q′ is constant for the entire frame, the predictor of the encoding time for the entire frame is approximated (neglecting the possible nonlinearity due to the minimum and maximum functions) as

$\begin{array}{cc}\hat{\tau }\approx \sum _{i=1}^{N_{\mathrm{MB}}}{\hat{\tau }}_i^{\prime }\left(q^{\prime },z_i\right)c=N_{\mathrm{MB}\gamma 1\mathrm{cT}}-\sum _{i=1}^{N_{\mathrm{MB}}}{\gamma }_2z_i\mathrm{cT}-N_{\mathrm{MB}{\beta }_3{\mathrm{cTq}}^{\prime }}& \left(9\right)\end{array}$

where the coefficients α¹=N_MBγ1cT−Σ_I=1^NMBγ₂z_iT and β¹=−N_MBβ3T depend on the frame content and frame type.

Buffer Models

The purpose of the buffer models is to provide a simple means of modeling the encoder input and output as well as the hypothetic decoder input behavior, imposing constrains in bit and time allocation. The encoder model comprises a raw frame buffer constituting the input of the system to which raw frames are written, and the encoder bit buffer, constituting the output of the system, to which the encoded bitstream is written. The hypothetic decoder model comprises a decoder bit buffer, connected to the encoder bit buffer by a channel responsible for the transport of the coded stream.

Raw Frame Buffer Model

The e fullness of the raw frame buffer at time t is denoted by l^raw(t). We assume that initially l^raw(0)=0, and raw frames are being filled at constant time intervals of 1/F, where F denotes the input sequence frame rate. Though l^raw may assume integer values only, in most parts of our treatment we will relax this restriction treating l^raw as a continuous quantity. The encoder starts reading the first frame from the raw buffer as soon as the raw buffer level reaches. l^raw≧l_min^raw where l_min^raw≧1 denotes the minimum amount of look-ahead required by the encoder in the current implementation. The first frame in the sequence starts being encoded at the time

$\begin{array}{cc}{t}_0=\frac{l_{\min }^{\mathrm{raw}}}{F}& \left(10\right)\end{array}$

The encoding of the first frame takes τ₁ seconds, after which the frame is removed from the raw buffer, decreasing its level by 1. The moment this happens is the encoding end time, denoted by

t ₁ ^enc =t ₀+τ₁   (11)

If l^raw(t_n^enc+∈)<l_min^raw, for infinitesimal ∈>0, encoder raw buffer underflow occurs; in this case the encoder will stall until the minimum buffer level is reached. We denote this idle time by

$\begin{array}{cc}{\tau }_1^{\mathrm{idle}}=\frac{l_{\min }^{\mathrm{raw}}-l^{\mathrm{raw}}\left(t_1^{\mathrm{enc}}\right)}{F}& \left(12\right)\end{array}$

In this notation, the encoding of the next frame will start at time t₁^enc+t₁^idle=t₀+τ₁+τ₁^idle. The raw frame buffer is assumed to be capable of holding up to l_max^raw raw frames; when the buffer capacity reaches this limit, an encoder raw buffer overflow occurs, and the input process stalls. Whenever the duration of such a stall is not negligible, input data may be lost.

The buffer level has to be checked for overflow immediately before the encoder finishes encoding a frame, and for underflow immediately before the encoder starts encoding a frame. Combining the contributions of frame production by the input subsystem and consumption by the encoder, the raw frame buffer levels are given by

l ^raw(t_n^enc−∈)=t_n^encF−n+1.   (13)

The raw frame buffer model is shown in FIG. 2a, where the horizontal axis 110 is the encoding time index and the vertical axis 120 is the raw frame buffer level. The minimum buffer level 115 and the maximum buffer level 125 are the upper and lower bounds of the raw frame buffer levels that the system desires to maintain. The raw frame buffer profile 135 is the achieved dynamic raw frame buffer level in this exemplary case.

Encoder Bit Buffer Model

The fullness of the encoder bit buffer at time t is denoted by l^enc(t) We assume that initially l^enc(0)=0 and at the time t_n^enc when the encoder completes encoding frame n, b_n bits corresponding to the access unit of the coded frame are added to the buffer. After the first time the buffer level exceeds l_init^enc the buffer starts being drained at a rate r(t)≦r_max(t), determined by the transport subsystem. If at a given time the buffer is empty, the instantaneous draining rate drops to r(t)=0. This situation is referred to as encoder bit buffer underflow. Except for inefficiency of utilizing the full channel capacity, encoder buffer underflow poses no danger to the system. The complement situation of encoder bit buffer overflow occurs when l^enc(t_n^enc)+b_n≧l_tot^enc. As a consequence, the encoder will stall until the bit buffer contains enough space to accommodate the encoded bits. If the stall lasts for a non-negligible amount of time, unpredictable results (including input raw frame buffer overflow) may occur.

The buffer level has to be checked for undertow immediately before the encoder finishes encoding a frame, and for overflow immediately after the encoding is finished. Combining the contributions of bit production by the encoder and consumption by the transport subsystem, the encoder bit buffer levels are given by

l ^enc(t_n^enc−∈)=l^enc(t_n^enc+∈)−∫_tn+1enc^tnencr(t)dt   (14)

l ^enc(t_n^enc+∈)=l^enc(t_n^enc−∈)+b_n.   (15)

The encoder bit buffer model is shown in FIG. 2b, where the horizontal axis 140 is the encoding time index and the vertical axis 150 is the encoder bit buffer level. The minimum buffer level 145 and the maximum buffer level 155 are the upper and lower bounds of the encoder bit buffer levels that the system desires to maintain. The encoder bit buffer profile 165 is the achieved encoder bit buffer level in this exemplary case. For convenience, let us denote by

$\begin{array}{cc}{r}_n=\frac{1}{t_n^{\mathrm{enc}}-t_{n-1}^{\mathrm{enc}}}{\int }_{t_{n-1}^{\mathrm{enc}}}^{t_n^{\mathrm{enc}}}r\left(t\right)t& \left(16\right)\end{array}$

the average rate while the frame n being encoded. Assuming that r(t)=r_max(t) if l^enc(t)>0, and r(t)=0 otherwise, we obtain

$\begin{array}{cc}{r}_n=\frac{1}{t_n^{\mathrm{enc}}-t_{n-1}^{\mathrm{enc}}}\min \left\{l^{\mathrm{enc}}\left(t_{n-1}^{\mathrm{enc}}+\in \right),{\int }_{t_{n-1}^{\mathrm{enc}}}^{t_n^{\mathrm{enc}}}{\gamma }_{\max }\left(t\right)t\right\}.& \left(17\right)\end{array}$

In these terms, we may write

l ^enc(t_n^enc−∈)=l^enc(t_n−1^enc+∈)−(t_n^enc−t_n−1^enc)r_n   (18)

Decoder Bit Buffer Model

Bits drained from the encoder bit buffer at time t at rate r(t) are transported by the channel and are appended to the decoder bit buffer at time t=δ_chan at the same rate. The channel delay δ_chan needs not to be constant; in this case, maximum channel delay may be assumed. The fullness of the decoder bit buffer at time t is denoted by l^dec(t). Initially, l^dec(0)=0; the buffer level remains zero until t=t₁^enc+δ_chan, where the first bits of the first frame start arriving. The decoder remains idle until l^dec≧l_init^dec; once the bit buffer is sufficiently full, the decoder removes b1 bits from the buffer and starts decoding the first frame of the sequence. The time lapsing between the arrivals of the first bit until the first access unit is removed from the bit buffer is given by the smallest δ_dec satisfying

∫_t1enc^t1enc+δdecr(t′)dt′≧l_init^dec.   (19)

We denote by

t ₁ ^dec =t ₁ ^enc+δ_chan+δ_dec   (20)

the time when the first frame starts being decoded The delay passing between production and consumption of the access unit corresponding to a frame is denoted by

δ=δ_chan+δ_dec   (21)

The decoder removes access units corresponding to the encoded frames at a constant rate F, resulting in the following decoding schedule

$\begin{array}{cc}{t}_n^{\mathrm{dec}}=t_1^{\mathrm{dec}}+\frac{n-1}{F}.& \left(22\right)\end{array}$

The decoder bit buffer level assumes highest and lowest values immediately before frame decoding start, and immediately after the decoding is started, respectively. Combining the contributions of bit production by the transport subsystem and consumption by the decoder, the decoder bit buffer levels are given by

l ^dec(t_n^dec−∈)=l^dec(t_n−1^dec+∈)+∫_tn−1dec+δ^tndec+δr(t)dt   (23)

l ^dec(t_n^dec−∈)=l^dec(t_n−1^dec−∈)−b_n.   (24)

The decoder bit buffer model is shown in FIG. 2c, where the horizontal axis is the encoding time index 170 and the vertical axis is the decoder bit buffer level 180. The minimum buffer level 175 and the maximum buffer level 185 are the upper and lower bounds of the decoder bit buffer levels that the system desires to maintain. The decoder bit buffer profile 195 is the achieved decoder bit buffer level in this exemplary case.

Sequence Resource Allocation

Assuming that the encoder is encoding an i-th frame in the current GOP, and is capable of observing n−1 additional future frames i+1, . . . , i+n−1 queued in the raw input buffer, the problem of sequence resource allocation consists of establishing a resource budget for the latter sequence of n frames. Here, allocation of bit and encoding time budgets, denoted by b_T and τ_T, respectively are considered. The display time stamp relative to the beginning of the current GOP of the i-th frame is denoted by t_i. Assuming an estimate {circumflex over (δ)} of the decoder delay is available and denoted by

$\begin{array}{cc}{t}_{\max }=\max \left\{t_{\max }^{\mathrm{gop}},\hat{\delta }\right\}& \left(25\right)\end{array}$

The maximum GOP duration (if the duration is smaller than the estimated delay, t_max={circumflex over (δ)} is used). The minimum GOP duration is set to the minimum temporal distance between two consecutive IDR frames,

t_min=t_min^idr.   (26)

The maximum amount of time remaining till the GOP end is denoted by

t _rem =t _max −t _i,   (27)

from where the number of remaining frames,

n _rem=t_remF   (28)

is obtained.

The sequence-level controller flow diagram is shown in FIG. 3 where the Frame Type Decision 270 and Encoding Order Optimization 280 are the system design parameters that the sequence-level controller will produce to achieve the minimum sequence cost within the sequence budget. The Buffer Model 210 will extract buffer model for the raw frame buffer, encoder bit buffer and decoder bit buffer. On the other hand, the Encoder Model 260 acts on the frame decision f from Frame Type Decision 270 and the underlying sequence to produce parameters for encoder model. The encoder model with the associated parameters allow the system to evaluate the corresponding system cost and quality without actually performing the computationally expensive encoding process. The system costs described in FIG. 3 are bit budget and time budget. Nevertheless, other system costs can also be used by people skilled in the field. The Sequence Bit Budget Allocation 250 allocates the frame bit rate budget using the results from Encoder Model 260 and Buffer Model 210 to avoid the complexity associated with actually encoding process. Similarly, the Sequence Time Budget Allocation 230 allocates the frame time budget using the results from Encoder Model 260 and Buffer Model 210 to avoid the complexity associated with actually encoding process. Based on the frame bit budget by, the Frame Bit Budget Allocation 240 performs optimal bit budget allocation within the frame. The frame bit budget allocation is typically done on a smaller picture unit such as macroblock or block by assigning a proper quantization level. The Frame Time Budget Allocation 220 works in a similar way to that as Frame Bit Budget Allocation 240.

Effective Bit Rate Estimation

The estimated rate, {circumflex over (r)}^dec at the decoder at the time when the i-th frame will be decoded is approximated by

{circumflex over (r)}^dec≈r(t^enc+δ),   (29)

where r is the current rate. Assuming the simplistic model of the rate remaining constant r in the interval [t^en,t^enc+δ], and then remaining constant {circumflex over (r)}^dec, the average bit rate of the encoder buffer drain in the interval [t^enc,t^enc+t_max] is given by

$\begin{array}{cc}{r}^{\mathrm{enc}}=\hat{\delta }r+\frac{\left(t_{\max }-\hat{\delta }\right){\hat{r}}^{\mathrm{dec}}}{t_{\max }}.& \left(30\right)\end{array}$

Under the assumption of nearly constant channel rate, the encoder and decoder bit buffers remain synchronized (up to an initial transient), connected by the relation

l ^enc(t)+l^dec(t+δ)=l_tot^enc.   (31)

However, bit rate fluctuations result in a loss of such synchronization. For example, in the case of a future decrease in the channel capacity ({circumflex over (r)}^dec<r), encoding the next frames at the rate r will cause the decoder buffer level to drop and, eventually, to underflow, while keeping the encoder buffer level balanced. In the converse case of a future increase in the channel capacity ({circumflex over (r)}^dec>r) keeping the video stream rate at r will result in a decoder bit buffer overflow.

A solution proposed here consists of modifying the rate at which the stream is encoded such that both buffers remain as balanced as possible. We refer to such a modified rate as to the effective bit rate,

r _eff=max{min{0.5(r^enc+{circumflex over (r)}_dec)+Δr, max{r^enc,{circumflex over (r)}^dec}},min{r^enc,{circumflex over (r)}^dec}},   (32)

where

$\begin{array}{cc}\Delta r=\frac{1}{2t_{\max }}\{\frac{{\hat{l}}^{\mathrm{dec}}-l^{\mathrm{enc}}+l_{\min }^{\mathrm{enc}}-l_{\min }^{\mathrm{dec}}:{\hat{r}}^{\mathrm{dec}}\ge r}{l_{\max }^{\mathrm{dec}}-l_{\max }^{\mathrm{enc}}-l^{\mathrm{enc}}+{\hat{l}}^{\mathrm{dec}}:{\hat{r}}^{\mathrm{dec}}<r}& \left(33\right)\end{array}$

Using r_eff instead of r in the first example will cause both encoder and decoder buffer levels to drop by a less extent

Bit Budget Allocation

Given an estimated effective encoding bit rate r_eff, the amount of bit budget remaining till the end of the GOP is given by

b_rem=r_efft_rem.   (34)

According to the bit production model (6), the estimated amount of bits generated by the encoder for an i-th frame given a quantizer q_i is given by

$\begin{array}{cc}{\hat{b}}_i={\alpha }_i^b+\frac{{\beta }_i^b}{q_i}& \left(35\right)\end{array}$

where the coefficients α^b=(α₁^b, . . . , α^b)^T and β^b=(β₁^b, . . . , β^b)^T depend on the frame content and frame type fi. The latter is assumed to be assigned by a higher-level frame type decision algorithm and

$\begin{array}{cc}{\alpha }^b=\sum _{i=1}^n{\alpha }_i^b{\beta }^b=\sum _{i=1}^n{\beta }_i^b& \left(36\right)\end{array}$

denote the sum of the model coefficients of the observed n frames. Substituting α^b and β^b into the bit production model yields the total estimate of bits produced by encoding the observed frames i, . . . , i+n−1 with a constant quantizer. Similarly, we denote by

α_rem^b=(n_rem−n) α^b

β_rem^b=(n_rem−n) β^b   (37)

the model coefficients for estimating the amount of bits of the remaining frames in the current GOP; α^b and β^b are decaying moving averages of α_i^b and β_i^b respectively, of the previously encoded frames.

The goal of sequence bit budget allocation is to maintain a fair distribution of the encoding bits throughout the sequence, considering the long-term effect of the decided allocation, as well as reacting to changes in channel bit rate and frame texture and motion complexity. Under the simplifying assumption that at the sequence level the visual quality is a function of the quantizer only, not depending on the frame content, the problem of sequence bit budget allocation can be translated to finding a single quantization scale q, which assigned to the remainder of the GOP frames produces b_rem bits. Substituting the appropriate bit production models.

$\begin{array}{cc}{\alpha }^b+{\alpha }_{\mathrm{rem}}^b+\left({\beta }^b+{\beta }_{\mathrm{rem}}^b\right)\frac{1}{q}=b_{\mathrm{rem}}& \left(38\right)\end{array}$

and solving for 1/q yields,

$\begin{array}{cc}\frac{1}{q}=\frac{b_{\mathrm{rem}}-{\alpha }^b+{\alpha }_{\mathrm{rem}}^b}{{\beta }^b+{\beta }_{\mathrm{rem}}^b}& \left(39\right)\end{array}$

Substituting the latter result into the bit production model yields the following bit budget for the sequence of the observed n frames.

$\begin{array}{cc}{b}_T^{\prime }=s·\{\begin{array}{ccc}{\alpha }^b+{\beta }^b·\frac{{\hspace{0.17em}}^b\mathrm{rem}-{\alpha }^b+{\alpha }_{\mathrm{rem}}^b}{{\beta }^b+{\beta }_{\mathrm{rem}}^b}& :& {\beta }^b+{\beta }_{\mathrm{rem}}^b>0\\ \frac{b_{\mathrm{rem}}}{n_{\mathrm{rem}}}& :& \mathrm{else},\end{array}\mathrm{Where},& \left(40\right)\\ s=\frac{l_{\mathrm{tot}}^{\mathrm{dec}}+\left(\frac{{\in }_1}{h}-{\in }_1-1\right){\hat{l}}^{\mathrm{dec}}}{l_{\mathrm{tot}}^{\mathrm{dec}}+\left(\frac{1}{h}-{\in }_1-1\right){\hat{l}}^{\mathrm{dec}}}& \left(41\right)\end{array}$

is a scaling factor decreasing the bit budget for higher, and increasing for lower decoder bit buffer levels, respectively; {circumflex over (l)}dec={circumflex over (l)}(t_i^dec−∈) is the estimated decoder bit buffer fullness immediately prior to decoding the i-th frame, ∈₁₌₃, and h is the target relative decoder bit buffer level, defined as

$\begin{array}{cc}h=\max \left\{\min \left\{0.5+\mathrm{.01}\frac{t_i-t_{\min }}{t_{\max }-t_{\min }^{\mathrm{idr}}}+t_{\max }\frac{{\hat{r}}^{\mathrm{dec}}-r_{\mathrm{eff}}}{l_{\mathrm{tot}}^{\mathrm{dec}}},0.6\right\},0.4\right\}& \left(42\right)\end{array}$

To guarantee some level of bit allocation fairness, the budget is constrained to be within 25% to 200% of the average bit budget,

$\begin{array}{cc}{\stackrel{\_}{b}}_T=\frac{\mathrm{nreff}}{F}& \left(43\right)\end{array}$

resulting in

b′T =min{max{b∵_T,0.25 b_T},2 b_T}  (44)

Encoder and decoder bit buffer constrains are further imposed, yielding the following final expression for the sequence bit budget

$\begin{array}{cc}{b}_T=\min \left\{\min \left\{b_T^n,l_{\max }^{\mathrm{enc}}-l^{\mathrm{enc}}+\frac{\mathrm{nr}}{F}\right\},{\hat{l}}^{\mathrm{dec}}+\frac{n{\hat{r}}^{\mathrm{dec}}}{F}-l_{\min }^{\mathrm{dec}}\right\}& \left(45\right)\end{array}$

where l^enc enc is the encoder bit buffer level immediately prior to encoding the i-th frame.

Encoding Time Budget Allocation

Similar to the bit budget allocation, the time budget allocation problem consists of assigning the target encoding time τ_T for the observed frames n. However, due to the stricter constrains imposed by the encoder raw frame buffer, encoding time allocation operates with shorter terms. Since frames are added to the encoder input buffer at the rate of F frames per second, the average encoding time of a frame is 1/F, resulting in the average budget

$\begin{array}{cc}{\stackrel{\_}{\tau }}_T=\frac{n}{F}& \left(46\right)\end{array}$

for the sequence of n observed frames. Using this budget and assuming the encoder has encoded n_enc frames so far (including the dropped ones), the time at the encoder should be n_enc/F. However, since the actual encoding time may differ from the allocated budget, the time at the encoder immediately prior to encoding the i-th frame (relative to the time t₀ when the first frame in the sequence starts being encoded) usually differs from the ideal value. We denote by

$\begin{array}{cc}{t}_{\mathrm{dif}}=\frac{n_{\mathrm{enc}}}{F}-t^{\mathrm{enc}}-t_0& \left(47\right)\end{array}$

the time difference between the ideal and actual encoding time; if t_dif>0 the encoder is faster than the input raw frame rate and the encoding time budget has to be increased in order to avoid raw buffer underflow. Similarly, if t_dif<0, the encoder is lagging behind the input and the time budget has to be decreased in order to prevent an overflow. Demanding the encoder to close the time gap in n_resp frames (n_res/F seconds), yields the following encoding time budget

$\begin{array}{cc}{\tau }^{\prime }={\stackrel{\_}{\tau }}_T+\frac{{\mathrm{nt}}_{\mathrm{dif}}}{n_{\mathrm{resp}}}& \left(48\right)\end{array}$

Typically, n_resp≈5, depending on l_max^raw. To guarantee some level of fairness, the budget is constrained by

τ_Tⁿ=max{min{τ′_T,1.5 τ_T},0.75 τ_T}  (49)

Encoder bit buffer constrains are further imposed yielding the final encoding time budget

$\begin{array}{cc}{\tau }_T=\max \left\{\min \left\{{\tau }_T^{″},{\tau }_{\max }\right\},{\tau }_{\min }\right\},\mathrm{where}& \left(50\right)\\ {\tau }_{\min }=\frac{l^{\mathrm{enc}}+\sum _{i=1}^{n-1}{\hat{b}}_i-l_{\max }^{\mathrm{enc}}}{r}{\tau }_{\max }=\max \left\{\frac{l^{\mathrm{enc}}+\sum _{i=1}^{n-1}{\hat{b}}_i-l_{\min }^{\mathrm{enc}}}{r},{\tau }_{\min }\right\}& \left(51\right)\end{array}$

and {circumflex over (b)}_i are the expected bits produced by encoding the i-th frame. Due to the dependence on {circumflex over (b)}_i, the time allocation must be preceded by bit allocation.

Frame Resource Allocation

The problem of frame resource allocation consists of distributing a given budget of resources between n frames in a sequence. More formally, we say that the vector x=(x1, . . . , xn)^T is an allocation of a resource x, where each xi quantifies the amount of that resource allocated to the frame i. For example, x can represent the amount of coded bits or encoding time. Ideally, we would like to maximize the xi for each frame; however, the allocation has to satisfy some constrains, one of which is

$\begin{array}{cc}\sum _{i=1}^nx_i=x_T& \left(52\right)\end{array}$

where xT is the resource budget assigned to the sequence by a higher-level sequence resource allocation algorithm. Other constrains stemming, for example, from the encoder buffer conditions apply as well. Formally, we say that an allocation vector x is feasible if it satisfies that set of conditions, and infeasible otherwise. Resource allocation can be therefore thought of as finding a feasible solution to the maximization problem.

$\begin{array}{cc}\underset{x}{\max }xs.t.x\mathrm{is}\mathrm{feasible}& \left(53\right)\end{array}$

Since usually the encoder is controlled using a set of parameters different from the allocated resource itself (e.g., though coded bits are allocated as a resource, the amount of produced bits is controlled through the quantization parameter), it is more convenient to express x as a function of some vector of parameters θ=(θ1, . . . , θm) . This yields the following optimization problem

$\begin{array}{cc}\underset{\theta }{\max }x\left(\theta \right)s.t.x\left(\theta \right)\mathrm{is}\mathrm{feasible}.& \left(54\right)\end{array}$

Note, however, that since the maximized objective is a vector rather than a scalar quantity, there exists no unique way to define what is meant by “maximum x”. Here, we adopt the following notion of vector objective optimality.

Definition 1 (Max-min optimality) A vector of resources x=(x1, . . . , xn)^T is said to be max-min optimal if it is feasible, and for any 1≦i≦n and a feasible y=(y1, . . . , yn) for which xp<yp, there is some j with xi≧yj and xj>yj.

Informally, this means that it is impossible to increase the resource allocated to a frame i without decreasing the resources allocated to frames that have already a smaller resource allocation than xi, and without violating the feasibility constrains. For this reason, a max-min optimal resource allocation can be thought of as fair. As to notation, given the vector of resources x as a function of some parameters θ and feasibility constrains θ ∈ Ω, we will henceforth interpret the vector objective maximization problem.

$\begin{array}{cc}{\theta }^{*}=\underset{\theta \in \Omega }{\mathrm{argmax}}x\left(\theta \right),& \left(55\right)\end{array}$

as finding a max-min optimal resource allocation x*=x(θ*). In the remainder of this section, we are going to explore and formulate allocation problems for two types of resources considered here, namely coded bits and encoding time.

Bit Allocation

The first problem we are going to address is the distribution of the budget of bT bits between the frames, which can be denoted as a bit allocation vector b=(b1, . . . , bn). However, since the encoder is not controlled directly by b, but rather by the quantizer value, we reformulate the problem as allocating a vector q=(q1, . . . , qn)^T of quantizer scales (or, equivalently, a vector q′ of quantization parameters). Though q and q′ may assume only discrete values, we relax this restriction by making them continuous variables. In these terms, the quantizer allocation problem can be formulated as max-min allocation of coding quality, in our case, average frame PSNR values p=(p1, . . . , pn)^T), as function of q and subject to feasibility constrains,

$\begin{array}{cc}\underset{q}{\max }p\left(q\right)s.t.b\left(q\right)\mathrm{is}\mathrm{feasible}.& \left(56\right)\end{array}$

Note that the feasibility constrains are imposed on the amount of bits produced by the encoder with the control sequence q. Since in practice the exact amount of bits produced for a given quantizer is unknown a priori, b(q) has to be substituted with the estimated coded bits {circumflex over (b)}(q). Such an estimate depends on the frame types, which are assumed to be decided by a higher-level frame type decision algorithm, detailed in the sequel. One of the feasibility constrains is, clearly, {circumflex over (b)}(q)=bT. However, this condition is insufficient, as it may happen that the produced bits violate the buffer constrains for a frame i<n. In order to formulate the buffer constrains on the allocated bits, let us denote by t^dec=(t₁^dec, . . . t_n^dec)^T the decoding timestamps of the frames, assuming without loss of generality that t1=0 and that the frames are numbered in the increasing order of ti. We denote the estimated decoder bit buffer fullness immediately before and immediately after frame i is decoded by {circumflex over (l)}₋^dec(q)=l₀^dec(t_i^dec−∈) and {circumflex over (l)}₊^dec(q)=l₀^dec(t_i^dec+∈), respectively. We assume that the initial decoder buffer fullness {circumflex over (l)}₋^dec={circumflex over (l)}₀^dec is given as the input. In vector notation, we can write

$\begin{array}{cc}{\hat{l}}_{\_}^{\mathrm{dec}}\left(q\right)=l_0^{\mathrm{dec}}-K\hat{b}\left(q\right)+{\hat{r}}^{\mathrm{dec}}t^{\mathrm{dec}}& \left(57\right)\\ {\hat{l}}_{+}^{\mathrm{dec}}\left(q\right)=l_0^{\mathrm{dec}}-J\hat{b}\left(q\right)+{\hat{r}}^{\mathrm{dec}}t^{\mathrm{dec}}\mathrm{where}& \left(58\right)\\ J=\left(\begin{array}{cccc}1& 0& \dots & 0\\ 1& 1& \dots & 0\\ ⋮& ⋮& ⋰& ⋮\\ 1& 1& \dots & 1\end{array}\right)& \left(59\right)\end{array}$

K=J−I, and r̂^dec is the estimated average decoder bit buffer filling rate on the time interval

$\left[t_1,t_n+\frac{1}{F}\right].$

The constrained optimal allocation problem becomes

$\begin{array}{cc}\underset{q_{\min }\le q\le q_{\max }}{\max }p\left(q\right)s.t.\{\begin{array}{c}{1}^T\hat{b}\left(q\right)=b_T\\ {\hat{l}}_{\_}^{\mathrm{dec}}\left(q\right)\le l_{\max }^{\mathrm{dec}}\\ {\hat{l}}_{+}^{\mathrm{dec}}\left(q\right)\ge l_{\min }^{\mathrm{dec}},\end{array}& \left(60\right)\end{array}$

where l_min^dec and l_max^dec are the minimum and the maximum decoder big buffer levels, respectively. Since it is reasonable to assume a monotonically decreasing dependence of the PSNR in the quantization parameter (or the quantizer scale), the maximizer of (60) coincides with the minimizer of

$\begin{array}{cc}\underset{q_{\min }\le q\le q_{\max }}{\max }qs.t.\{\begin{array}{c}{1}^T\hat{b}\left(q\right)=b_T\\ {\hat{l}}_{\_}^{\mathrm{dec}}\left(q\right)\le l_{\max }^{\mathrm{dec}}\\ {\hat{l}}_{+}^{\mathrm{dec}}\left(q\right)\ge l_{\min }^{\mathrm{dec}},\end{array}& \left(61\right)\end{array}$

A numerical solution of the max-min allocation problem (61) can be carried out using a variant of the bottleneck link algorithm, summarized in Algorithm 1. In the algorithm, we assume that the amount of coded bits produced for a frame i as a function of qi is given by the model

$\begin{array}{cc}{\hat{b}}_i={\alpha }_i^b+\frac{{\beta }_i^b}{q_i}& \left(62\right)\end{array}$

where the coefficients α^b=(α₁^b, . . . , α^b)^T and β^b=(β₁^b, . . . , β_n^b)^T depend on the frame content and frame types. In vector notation, this yields

$\begin{array}{cc}\hat{b}={\alpha }^b+\frac{{\beta }^b}{q}& \left(63\right)\end{array}$

where vector division is interpreted as an element-wise operation. The algorithm can be easily adopted to other models as well by replacing the closed-form solution in Steps 1,8, and 9 with the more general equation

$\begin{array}{cc}{b}_T=\sum _{i=1}^n{\hat{b}}_i\left(q\right)& \left(64\right)\end{array}$

with respect to the scalar q.

Algorithm 1: Max-min frame bit allocation
	input:	n × 1 vectors αb, βb of bit model parameters; n × 1 vector of decoding time stamps tdec
		starting with t1dec = 0; initial decoder bit buffer level l0dec bit budget bT;
		estimated average decoder buffer rate {circumflex over (r)}dec
	output:	n × 1 optimal max-min quantizer allocation vector q*.
1	Find optimal equal quantizer allocation
	$q=\max \left\{\min \left\{\frac{1^T{\beta }^B}{b_T-1^T{\alpha }^b},q_{\max }\right\},q_{\min }\right\}$
	and set q = q · 1
2	if {circumflex over (l)}—dec(q) ≦ lmaxdec and	{circumflex over (l)}+dec(q) ≧ lmaxdec then
3	set q* = q.
4	else
5	Find the smallest 1 ≦ m ≦ n for which one of the constrains is violated.
6	if {circumflex over (l)}min−dec(q) ≦ lmaxdec then set bT0 = l0dec + {circumflex over (r)}dectm+1dec − lmaxdec else set bT0 = l0dec + {circumflex over (r)}dectmdec − lmindec.
7	Set bT0 = max {bT0, 0}
8	Find
	$q=\max \left\{\min \left\{\frac{1^T{\left({\beta }_1^b,\dots {\beta }_m^b\right)}^T}{b_T^0-1^T{\left({\alpha }_1^b,\dots ,{\alpha }_m^b\right)}^T},q_{\max }\right\},q\min \right\}$
	and set q1*= . . . = qm* = q
9	Compute remaining bit budget
	$b_T=\max \left\{b_T-1^T{\left({\alpha }_1^b,\dots ,{\alpha }_m^b\right)}^T-\frac{1}{q}1^T{\left({\beta }_1^b,\dots {\beta }_m^b\right)}^T,0\right\}$
10	Recursively invoke the algorithm with α = (αm+1b, . . . , αn+1b)T, tdec = (tm+1dec, . . . , tndec)T − tm+1dec,
	and l0dec = l0dec − bT0{circumflex over (r)}dectm+1dec to fix the remaining quantizer values (qm+1*, . . . , qn*)
11 end

Encoding Time Allocation

The second problem we are going to address is the allocation of the encoding time budget τT between the frames, which can be denoted as an encoding time allocation vector τ=(τ1, . . . , τn)^T. As in the case of bit allocation, since the encoder is not controlled directly by τ, but rather by the complexity scale, we reformulate the problem as allocating a vector c=(c1, . . . , cn) of complexity scales. Again, we think of c as of a continuous variable, though in practice it may be restricted to a set of discrete values. We denote by {circumflex over (t)}^enc=({circumflex over (t)}₁^enc, . . . , {circumflex over (t)}_n^enc)^T the time at which frame i's encoding is complete, assuming without loss of generality that the first frame starts being encoded at time 0. Furthermore, we assume that the encoding time for a frame i coincides with the encoding start time for the frame i+1. In this notation,

{circumflex over (t)} ^enc =J{circumflex over (τ)}(c)   (65)

where τ̂=(̂τ1, . . . , τ̂n)^T denotes the estimated encoding times.

Assuming the quantizer allocation has fixed the estimated amount of bits {circumflex over (b)} produced by encoding each frame, our goal is to maximize the coding quality p(c) subject to the encoder buffer constrains. The encoder buffer constrains include both the encoder bit buffer constrains, applying to the output bit buffer, and the encoder raw buffer constrains, applying to the input raw frame buffer. Similar to the bit allocation problem, we denote the estimated encoder bit buffer fullness immediately before and immediately after frame i's encoding is complete by {circumflex over (l)}_i−^enc={circumflex over (l)}^enc({circumflex over (t)}_i^enc−∈) and {circumflex over (l)}_i+^enc={circumflex over (l)}^enc({circumflex over (t)}_i^enc+∈), respectively. The initial buffer level l₀^enc=l^enc(0) is assumed to be known from a direct observation. Using this notation, the buffer levels are given by

{circumflex over (l)} ₋ ^enc(c)=l₀^enc+K{circumflex over (b)}−rJ{circumflex over (τ)}(c)   (66)

{circumflex over (l)} ₊ ^enc(c)=l₀^enc+J{circumflex over (b)}−rJ{circumflex over (τ)}(c)   (67)

In the same manner, {circumflex over (l)}_i−^raw and {circumflex over (l)}_i+^raw denote the estimated encoder raw frame buffer fullness immediately before and immediately after frame i's encoding is complete. The initial buffer level at time 0 is denoted by l₀^raw and is available from a direct observation. Since the filling rate of the input buffer is F, we have

{circumflex over (l)} ^raw({circumflex over (t)}_i^enc−∈)=l₀^raw+{circumflex over (t)}_i^encF−(i−1)   (68)

and {circumflex over (l)}^raw({circumflex over (t)}_i^enc+∈)={circumflex over (l)}^raw({circumflex over (t)}_i^enc−∈)−1. In vector form, this yields

{circumflex over (l)} ₋ ^raw(c)=l₀^raw+FJ{circumflex over (τ)}(c)−K1   (69)

{circumflex over (l)} ₊ ^raw(c)=l₀^raw+FJ{circumflex over (τ)}(c)−J1   (70)

The buffer-constrained encoding complexity allocation problem can be expressed as

$\begin{array}{cc}{c}_{\min }\stackrel{\max }{\le c\le }c_{\max }cs.t.\{\begin{array}{ccc}{1}^T\hat{t}\left(c\right)& =& {\tau }_T\\ {\hat{l}}_{+}^{\mathrm{enc}}\left(c\right)& \le & l_{\max }^{\mathrm{enc}}\\ {\hat{l}}_{-}^{\mathrm{enc}}\left(c\right)& \ge & l_{\min }^{\mathrm{enc}}\\ {\hat{l}}_{-}^{\mathrm{raw}}\left(c\right)& \le & l_{\max }^{\mathrm{raw}}\\ {\hat{l}}_{+}^{\mathrm{raw}}\left(c\right)& \ge & l_{\max }^{\mathrm{raw}}\end{array}& \left(71\right)\end{array}$

However, this formulation suffers from two potential problems. The first drawback stems from our assumption that encoding end time for a frame i coincides with the encoding start time for the frame i+1. For example, if the i frame is dropped and the input buffer level falls below l_min^enc, the encoder will stall until the minimum buffer level is reached. This will make τi non-negligible, which in turn will require ci to be very high (or even infinite if we assume that the nominal encoding time for a dropped frame is strictly zero). We overcome this problem by relaxing the constrain c≦cmax. If some elements of the optimal complexity allocation vector c exceed the maximum complexity scale, we will say that the encoder yields a portion of its CPU time to other processes potentially competing with it over the CPU resources. This fact can be quantified by introducing a vector of CPU utilization η=(η1, . . . , ηn)^T, where 0≦ηi≦1 expresses the fraction of the CPU time used by the encoder process in the time interval [{circumflex over (t)}_i−1^enc,{circumflex over (t)}_i^enc], {circumflex over (t)}₀^enc=0. Setting,

$\begin{array}{cc}{n}_i=\frac{c_{\max }}{c_i^{*}},& \left(72\right)\end{array}$

and c=min{c*,c_max} the encoding complexity scale of frames with c_i* exceeding cmax will be set to cmax, and the utilizable CPU time will be lower than 100%. The second difficulty in the allocation problem (71) stems from the fact that sometimes the input and output buffer constrains may be conflicting (or, more formally, the feasible region may be empty).

Definition 2 (Ordered constrains). Given an ordered m-tuple of indicator functions χ=(χ₁, . . . , χ_m)^m)^T, χ_i: Rⁿ→{0, 1}, a vector x is said to satisfy the ordered constrains χ(x) if there exists no y with χ(y)<χ(x), where < denotes the lexicographic order relation between binary strings.

Informally, this definition implies that in case of conflicting constrains, satisfying the first constrain is more important than satisfying the second one, and so on. Using this relaxed notion of constrained optimality, allocation problem (71) can be rewritten as

$\begin{array}{cc}\underset{c\ge \mathrm{cL}\min }{\max }cs.t\left(\begin{array}{ccc}{\hat{l}}_{+}^{\mathrm{raw}}\left(c\right)& \ge & l_{\min }^{\mathrm{raw}}\\ {\hat{l}}_{-}^{\mathrm{raw}}\left(c\right)& \le & l_{\max }^{\mathrm{raw}}\\ {\hat{l}}_{+}^{\mathrm{enc}}\left(c\right)& \le & l_{\max }^{\mathrm{enc}}\\ {\hat{l}}_{-}^{\mathrm{enc}}\left(c\right)& \ge & l_{\min }^{\mathrm{enc}}\\ 1^T\hat{\tau }\left(c\right)& =& {\tau }_T,\end{array}\right)& \left(73\right)\end{array}$

where the constrains are interpreted as ordered constrains.

The time complexity allocation (73) is solved using Algorithm 2 similar to Algorithm 1 for bit allocation. In the algorithm, we assume that the encoding time of a frame i as a function of ci is given by the model

$\begin{array}{cc}{\hat{\tau }}_i=\frac{{\alpha }_i^t+{\beta }_i^tq_i^{\prime }c_i}{n_i}=\frac{{\gamma }_i^tc_i}{n_i}& \left(74\right)\end{array}$

where the coefficients and α^t=(α₁^t, . . . , α_n^t)^T and β^t=(β₁^t, . . . , β_n^t)^T depend on the frame content and frame types, and γ^t=(α₁^t+β₁^tq′₁, . . . , α₁^t+β₁^tq′)^T.

Joint Bit and Encoding Time Allocation

Allocation problems (61) and (73) tacitly assume that allocation of quantization and encoder time complexity scales are independent. However, while it is reasonable to assume that the coding quality p is a function of q only (and, thus, is independent of c), the amount of produced bits b clearly depends on both parameters. This dependence couples together the two problems through the more accurate expression for the encoder bit buffer levels

{circumflex over (l)} ₋ ^enc(q,c)=l₀^enc+K{circumflex over (b)}(q,c)−rJ{circumflex over (τ)}(c)   (75)

{circumflex over (l)} ₊ ^enc(q,c)=l₀^enc+J{circumflex over (b)}(q,c)−rJ{circumflex over (τ)}(c)   (76)

and the decoder bit buffer levels

{circumflex over (l)} ₋ ^dec(q,c)=l₀^dec+K{circumflex over (b)}(q,c)+{circumflex over (r)}^dect^dec   (77)

{circumflex over (l)} ₊ ^dec(q,c)=l₀^dec−J{circumflex over (b)}(q,c)+{circumflex over (r)}^dect^dec   (78)

Algorithm 2: Max-min frame time complexity allocation.
   input  : n × 1 vectors γt of time model parameters; initial encoder bit buffer level l0ENC; initial
          encoder raw frame buffer level l0raw; time budget τT; measured average encoder buffer
          draining rate r; frame rate F
   output :n × 1 optimal max-min complexity allocation vector c*; n × 1 CPU time utilization
          vector η.
1. Find optimal equal complexity scale allocation
c                    =                                          max                                                                    {                                                                                                            τ                              T                                                                                                                      1                                T                                                                                                                          γ                                t                                                                                                              ,                                                      c                            min                                                                          }
   and set c = c · 1.
2. if {circumflex over (l)}+enc(c)≦lmaxenc and {circumflex over (l)}−enc(c)≧lmin enc and {circumflex over (l)}−raw(c)≦lmaxraw and {circumflex over (l)}+raw(c)≧lminraw then set c* = c
   else
3  Find the smallest 1 ≦ m ≦ n for which one of the constrains is violated.
4  Compute the complexity constrains
   c                              1                                                        =                                                                                                                            l                                  min                                  raw                                                                -                                                                  l                                  0                                  raw                                                                +                                m                                                                                                                              F                                  ·                                                                      (                                                                                                                  γ                                        1                                        t                                                                            ,                                      …                                                                                                                                                                                                ,                                                                              γ                                        m                                        t                                                                                                              )                                                                                                                                  1                                                                                                                                                                                                                                                                    c                              2                                                        =                                                                                                                            l                                  min                                  raw                                                                -                                                                  l                                  0                                  raw                                                                +                                m                                -                                1                                                                                                                              F                                  ·                                                                      (                                                                                                                  γ                                        1                                        t                                                                            ,                                      …                                                                                                                                                                                                ,                                                                              γ                                        m                                        t                                                                                                              )                                                                                                                                  1                                                                                                                                                                                                                                                                    c                              3                                                        =                                                          max                                                                                            {                                                                                                      min                                                                                                              {                                                                                                                                                                                                                  l                                              0                                              enc                                                                                        -                                                                                          l                                              max                                              enc                                                                                        +                                                                                                                                          (                                                                                                                                                                                                            b                                                      ^                                                                                                        1                                                                                                    ,                                                  …                                                                                                                                                                                                                                                            ,                                                                                                                                                            b                                                      ^                                                                                                        m                                                                                                                                                  )                                                                                                                                          1                                                                                                                                                                                                                          r                                              ·                                                                                              (                                                                                                                                                      γ                                                    1                                                    t                                                                                                    ,                                                  …                                                                                                                                                                                                                                                            ,                                                                                                      γ                                                    m                                                    t                                                                                                                                                  )                                                                                                                                                                                  1                                                                                                                          ,                                                                                  c                                          2                                                                                                                    }                                                                                                        ,                                                                      c                                    1                                                                                                  }                                                                                                                                                                                                                                                                    c                              4                                                        =                                                          min                                                                                            {                                                                                                      max                                                                                                              {                                                                                                                                                                                                                  l                                              0                                              enc                                                                                        -                                                                                          l                                              min                                              enc                                                                                        +                                                                                                                                          (                                                                                                                                                                                                            b                                                      ^                                                                                                        1                                                                                                    ,                                                  …                                                                                                                                                                                                                                                            ,                                                                                                                                                            b                                                      ^                                                                                                        m                                                                                                                                                  )                                                                                                                                          1                                                                                                                                                                                                                          r                                              ·                                                                                              (                                                                                                                                                      γ                                                    1                                                    t                                                                                                    ,                                                  …                                                                                                                                                                                                                                                            ,                                                                                                      γ                                                    m                                                    t                                                                                                                                                  )                                                                                                                                                                                  1                                                                                                                          ,                                        c1                                                                            }                                                                                                        ,                                  c2                                                                }
   and set c1 = max{max{c1, c3}, cmin} and c2 = min{c2, c4}.
5  if {circumflex over (l)}m+1enc(c)≦lmaxenc or {circumflex over (l)}m+raw(c)≧lminraw then set τT0 = cmin(γ1t, . . . , γmt)1 else set
   τT0 = cmax(γ1t, . . . , γmt)1
6  Find
   c                      =                                              max                                                                          {                                                                                    τ                              T                              0                                                                                                                      (                                                                                                      γ                                    1                                    t                                                                    ,                                  …                                                                                                                                                                            ,                                                                      γ                                    m                                    t                                                                                                  )                                                                                          1                                                                                }                                                                                      ,                                          c                      min
   and set c1* = . . . = Cm* = c.
7  Compute remaining time budget τT = max{τT −c(γ1t, . . . , γmt)1, 0}
8  Recursively invoke the algorithm with γt = (γm+1t, . . . , γnt)T, l0enc = l0enc +
   ({circumflex over (b)}1, . . . , {circumflex over (b)}m)1−rτT0 and l0raw = l0raw + F(γ1t, . . . , γmt)1−m to fix the remaining complexity
   scale values (cm+1*, . . . , cn*).
9  Set η = cmax/c* and c* = max{c*, cmax}
10 end

which now depend on both q and c. Unfortunately, a joint bit and time complexity allocation problem is not well-defined, since combining the two vector-valued objectives q and c can no more be treated using the max-min optimality framework, as the two vectors are non-commensurable.³ However, joint allocation can be performed by alternatively solving the bit and time allocation problems, as suggested by the following algorithm

Algorithm 3: Joint frame bit and encoding complexity allocation.
Initialization: Set c* = 1
1  repeat
2    Fix c = c* and find the optimal quantizer allocation
q* by solving (61).
3    Fix q = q* and find the optimal time complexity allocation
c*by solving (73).
4  until until convergence

The convergence condition can be a bound on the change in c* and q*, the number of iterations, or any combination thereof. Our practice shows that a single iteration of this algorithm usually produces acceptable allocation.

Frame Type Decision

The purpose of frame type decision is to associate with a sequence of n frames an optimal sequence of frame types. The frame type with which an i-th frame is encoded is denoted by f_i. To simplify notation, we assume that f_i also specifies whether the frame is used as reference, whether it is an IDR, and which frames are used for its temporal prediction (unless it is a spatially predicted frame). For example a frame i=1 can be IDR, I, P predicted from frame 0, or DROP. The space of possible frame type assignment for a frame i is denoted by F_i, and depends solely on the status of the reference buffer immediately before the frame is encoded, denoted by R_i (R_i is defined as the list of reference frame indices). Speaking more broadly, F_i is a function of the encoder state immediately prior to encoding a frame i, defined as the pseudo-vector

σ_i=(R_i,l_i−^enc,l_i−^raw,l_i−^dec,t_i^nd,t_i^idr)   (79)

where c l_i−^enc, l_i−^raw, and l_i−^dec denote the levels of the encoder bit buffer, raw buffer, and decoder bit buffer, respectively, denotes the index of the last non-dropped frame, and denotes the presentation time of the last IDR frame. σ_i fully defines the instantaneous encoder state. In practice, only estimated buffer levels are available. We will henceforth denote the estimated encoder state by

σ_i=(R_i,{circumflex over (l)}_i−^enc,{circumflex over (l)}_i−^raw,{circumflex over (l)}_i−^dec,ι_i^nd,t_i^idr)   (80)

Note that R_i, t_i^nd, and, t_i^idr in {circumflex over (σ)}_i are fully deterministic.

It is important to observe that f_i does not fully define the set of control parameters required by the encoder in order to encode the i-th frame, as it does not specify the quantization and complexity scales q_i and c_i. In order to separate the optimization of frame type and reference indices from the optimal quantizer and complexity allocation, we assume that given the sequence of frame types f=(f₁, . . . , f_n), the allocation algorithms described in the previous section are invoked to find q(f)=(q*₁, . . . , q*_n) and c(f)=(c*₁, . . . , c*_n) As consequence, the amount of bits produced and the amount of time consumed by encoding the i-th frame can be expressed as functions of f. To simplify notation, we will denote the latter quantities by bi and τi, respectively. Similarly, pi will denote the distortion of the i-th frame. In the case where f_k=DROP, the distortion is evaluated as the PSNR of the difference between the original frame i and the last displayed frame ι_i^nd.

We jointly refer to a_i=(f_i, q*_i, c*_i) as the encoder action for the frame i. Note that although a_i is defined for a single frame, the optimal values of q*_i and c*_i depend on the f_i's of the entire sequence of frames 1, . . . , n. As consequence, a can be determined only as a whole, i.e. a_i+1 is required in order to determine a_i.

Encoder State Update

Given the encoder state σ_i and the action σ_i+1 the next state σ_i+1 is unambiguously determined by the state update rule σ_i=σ(σ_i,a_i). In practice, the update rule is applied to the estimated state, σ_i+1=σ({circumflex over (σ)}i ,a_i). The update for the buffer levels is given by

{circumflex over (l)} _i+1 ^enc ={circumflex over (l)} _i− ^enc +{circumflex over (b)} _i −r _i{circumflex over (τ)}_i

{circumflex over (l)} _i+1 ^raw ={circumflex over (l)} _i− ^raw +F{circumflex over (τ)} _i−1

{circumflex over (l)} _i+1 ^dec ={circumflex over (l)} _i− ^dec +{circumflex over (b)} _i +{circumflex over (r)} _i ^dec(t_i+1^dec−t_i^dec),   (81)

where t_i^dec is the decoding time stamp of the i-th frame, r_i is the average encoder bit buffer draining rate at the time t_i^enc, and {circumflex over (r)}_i is the predicted decoder bit buffer filling rate at the time t_i^dec. The last displayed is updated according to

$\begin{array}{cc}{l}_{i+1}^{\mathrm{nd}}=\{\begin{array}{ccc}{l}_i^{\mathrm{nd}}& :& f_i=\mathrm{DROP}\\ i& :& \mathrm{else}.\end{array}& \left(82\right)\end{array}$

The last IDR presentation time stamp is updated according to

$\begin{array}{cc}{t}_{i+1}^{\mathrm{idr}}=\{\begin{array}{ccc}{t}_i& :& f_i=\mathrm{IDR}\\ t_i^{\mathrm{idr}}& :& \mathrm{else}.\end{array}& \left(83\right)\end{array}$

The reference buffer is updated according to the sliding window policy

$\begin{array}{cc}{R}_{i+1}=\{\begin{array}{ccc}\varnothing & :& f_i=\mathrm{IDR}\\ R_i& :& f_i\in \mathrm{NONREF}\\ \left\{i\right\}\bigcup R_i& :& f_i\in \mathrm{REF}\mathrm{and}R_i<R_{\max }\\ \left\{i\right\}\bigcup R_i\backslash \left\{\min R_i\right\}& :& \mathrm{else},\end{array}& \left(84\right)\end{array}$

where min R_i denotes the smallest frame index found in the reference buffer, |R_i| denotes the number of frames in the reference buffer, and R_max stands for the maximum reference buffer occupancy. It is important to emphasize that though the next state σ_i−1 depends only on σ_i and a_i, a_i itself depends on a₁, . . . , a_i−1,a_i+1, . . . , a_n. Formally, this can be expressed by saying that the update of the full encoder state is non-Markovian. However, some constituents of the encoder state do satisfy the Markovian property. We denote by

σ_i^M=(R_i,l_i^nd,t_i^idr)   (85)

the Markovian part of the state, whose update rule can be expressed as

σ_i+1^M=σ(σ_i^M, f_i)   (86)

(note the dependence on f_i only). On the other hand, the remaining constituents of the encoder state

{circumflex over (σ)}_i^NM=(l_i−^enc,l_i−^raw,l_i−^dec)   (87)

are non-Markovian, since their update rule requires and which, in turn, depend on the entire sequence f through q* and c*. The update rule for σ_i^NM is a function of the initial state σ_l^NM, and the entire f

σ_i^NM=σ(σ₁^NM,f)   (88)

Action Sequence Cost

Given a sequence of encoder actions α=(α₁, . . . ,α_n), we associate with it a sequence cost ρ(α), defined as

$\begin{array}{cc}p\left(a,\hat{\sigma }\right)={\lambda }_{\mathrm{buf}}\sum _{i=1}^nP_{\mathrm{buf}}\left({\hat{l}}_{i+}^{\mathrm{dec}}\right)+{\lambda }_{\mathrm{dis}}\sum _{i=1}^n\mathrm{Pdis}\left({\hat{P}}_i\right)+\lambda \mathrm{bit}\sum _{i=1}^nP_{\mathrm{bit}}\left({\hat{b}}_i\right)+{\lambda }_{\mathrm{drop}}\sum _{i=1}^nP_{\mathrm{drop}}\left(f_i,{\hat{p}}_i,{\hat{p}}_i^{\min }\right)+{\lambda }_{\mathrm{idr}}\sum _{i=1}^nP_{\mathrm{idr}}\left(f_i,t_i,t_i^{\mathrm{idr}}\right)+{\lambda }_{\mathrm{qp}}\sum _{i=1}^nP_{\mathrm{qp}}\left(q_i^{\prime }\right),& \left(89\right)\end{array}$

where typically λ_buf=10, λ_dis=1, λ_bit=100, λ_idr=0.5, λ_qp=0.01 The constituent terms of the cost function are defined as follows.

Buffer cost penalizing the estimated decoder bit buffer violation is defined as

$\begin{array}{cc}{p}_{\mathrm{buf}}\left({\hat{l}}_{k+}^{\mathrm{dec}}\right)=h_{\mathrm{os}}\left({\hat{l}}_{k+}^{\mathrm{dec}},l_{\min }^{\mathrm{dec}}\right)+h_{\mathrm{os}}\left(l_{\mathrm{tot}}^{\mathrm{dec}}-{\hat{l}}_{k+}^{\mathrm{dec}},l_{\mathrm{tot}}^{\mathrm{dec}}-l_{\max }^{\mathrm{dec}}\right)\mathrm{where}{\hat{l}}_{i+}^{\mathrm{dec}}={\hat{l}}_{i-}^{\mathrm{dec}}-{\hat{b}}_i,& \left(90\right)\\ h_{\mathrm{os}}\left(x,y\right)=\frac{y}{{\in }^2}·\max \left\{\in -\min \left\{x,\in \right\},0\right\}+\max \left\{\frac{y}{\max \left\{x,e\right\}},1\right\}-1& \left(91\right)\end{array}$

is a single-sided hyperbolic penalty function, and ∈ is a small number, typically ∈≈10⁻⁶

Distortion cost penalizing the frame distortion is given by

p _dis({circumflex over (p)}_i)=255²·10^{−0.1 {circumflex over (p)}i}.   (92)

Drop cost penalizing the dropped frame distortion is given by

$\begin{array}{cc}{P}_{\mathrm{drop}}\left(f_i,{\hat{p}}_i,{\hat{p}}_i^{\min }\right)=\{\begin{array}{ccc}{255}^2\left({10}^{-0.1{\hat{p}}_i^{\min }-{10}^{-0.1\hat{p}i}}\right)& :& f_i=\mathrm{DROP}\\ 0& :& \mathrm{else},\end{array}& \left(93\right)\end{array}$

where {circumflex over (p)}_i^min is the estimated minimum PSNR. For a dropped frame, {circumflex over (p)}_i^min is computed as the 0.1%-quantile of the PSNR of the difference between the original frame i and the last displayed frame l_i^nd For a non-dropped frame, {circumflex over (p)}_i^min={circumflex over (p)}_i.

IDR cost for penalizes for a too early IDR frame,

$\begin{array}{cc}{p}_{\mathrm{idr}}\left(t_i,t_i^{\mathrm{idr}}\right)=\{\begin{array}{ccc}\infty & :& t_i-t_i^{\mathrm{idr}}<t_{\min }^{\mathrm{idr}}\\ 0& :& \mathrm{else}.\end{array}& \left(94\right)\end{array}$

For f_i≠IDR,the cost is given by

$\begin{array}{cc}{p}_{\mathrm{idr}}\left(t_i,t_i^{\mathrm{idr}}\right)=\{\begin{array}{ccc}\infty & :& t_i-t_i^{\mathrm{idr}}>t_{\max }^{\mathrm{gop}}\\ 0& :& t_i-t_i^{\mathrm{idr}}<t_{\min }^{\mathrm{idr}}\\ t_i-t_i^{\mathrm{idr}}-t_i^{\mathrm{idr}}& :& \mathrm{else},\end{array}& \left(95\right)\end{array}$

penalizing for a too late IDR frame. The cost is constructed in such a way that an IDR is placed in the time interval [t_min^idr,t_max^gop].

Quantizer fluctuation cost penalizes on the deviation from the average sequence quantization parameter according to

P _idr(q′_i)=max{2^q′i−q′,2^q′−q′i}−1,   (96)

where q′_i is the average quantization parameter in the previous non-dropped frames, computed as a decaying weighted average

$\begin{array}{cc}{\stackrel{\_}{q}}_i^{\prime }=\frac{{\stackrel{\_}{q}}_i^{\prime \mathrm{num}}}{{\stackrel{\_}{q}}_i^{\prime \mathrm{den}}},\mathrm{where}& \left(97\right)\\ {\stackrel{\_}{q}}_i^{\prime \mathrm{num}}=\{\begin{array}{ccc}{\stackrel{\_}{q}}_{i-1}^{\prime }& :& f_i=\mathrm{DROP}\\ \lambda {\stackrel{\_}{q}}_{i-1}^{\prime \mathrm{num}}+\mathrm{qi}-1& :& \mathrm{else},\end{array}& \left(98\right)\\ {\stackrel{\_}{q}}_i^{\prime \mathrm{den}}=\{\begin{array}{ccc}{\stackrel{\_}{q}}_{i-1}^{\prime \mathrm{den}}& :& f_i=\mathrm{DROP}\\ \lambda {\stackrel{\_}{q}}_{i-1}^{\prime \mathrm{den}}+1& :& \mathrm{else}.\end{array}& \left(99\right)\end{array}$

The decay parameter is set λ=0.99, and may be adjusted according to the sequence frame rate.

Bit budget deviation cost penalizes for action sequences resulting in a deviation from the allocated bit budget b_T

$\begin{array}{cc}{P}_{\mathrm{bit}}\left(\hat{b}\right)=\left(\max \left\{\frac{{\stackrel{\_}{b}}_T}{b_T}-1,0\right\}+\max \left\{\frac{b_T}{{\stackrel{\_}{b}}_T}-1,0\right\}\right)·h_{\mathrm{oe}}\left(\frac{1^T\hat{b}}{b_T}\right),\mathrm{where}& \left(100\right)\\ h_{\mathrm{oe}}\left(x\right)=\max \left\{{}^{\frac{x-1}{\in }}-1,0\right\},& \left(101\right)\end{array}$

and ∈=0.1.

Action Sequence Optimization

Using the notion of the action sequence cost, the frame type optimization problem can be expressed as the minimization problem

$\begin{array}{cc}\underset{f}{\min }p\left(f,q^{*}\left(f\right),c^{*}\left(f\right),{\sigma }^{*}\right)& \left(102\right)\end{array}$

Since the search space F₁× . . . ×F_n is usually very large, the complexity of finding the best action sequence by exhaustive search is prohibitive. Due to the non-Markovian property of the cost function, no greedy Markov decision algorithms can be employed. However, one can construct a Markovian lower bound for ρ, depending on f_i only, and allowing to prune the search space significantly

We observe that though the estimated frame distortion {circumflex over (p)}_i depends on the selection of the quantizer, it can be bounded below by the distortion achieved if b_T bits were allocated to the i-th frame alone. The lower bound {circumflex over (p)}_−i on the frame distortion can be expressed as

$\begin{array}{cc}\underset{-i}{\hat{p}}=\{\begin{array}{cc}{\hat{p}}_i\text{:}& f_i=\mathrm{DROP}\\ {\hat{p}}_i\left(\max \left\{{\hat{q}}_i\left(b_T\right),q_{\min }\right\}\right)\text{:}& \mathrm{else},\end{array}& \left(103\right)\end{array}$

where {circumflex over (q)}(b) is the inverse function of {circumflex over (b)}(q) given by

$\begin{array}{cc}{\hat{q}}_i\left(b\right)=\frac{{\beta }_i^b}{b-{\alpha }_i^b},& \left(104\right)\end{array}$

(the model coefficients α_i^b and β_i^b depend on fi). For dropped frames, the bound is exact; moreover, {circumflex over (p)}_i^min can be estimated as well.

Aggregating the terms of p that do not depend on the quantizer, the following lower bound on the action cost is obtained.

$\begin{array}{cc}\underset{\_}{p}\left(f_i\right)={\lambda }_{\mathrm{dis}}P_{\mathrm{dis}}\left(f_i,\underset{\_}{\hat{p}}\right)+{\lambda }_{\mathrm{drop}}P_{\mathrm{drop}}\left(\underset{\_}{\hat{p}},{\hat{p}}_i^{\min }\right)+{\lambda }_{\mathrm{idr}}P_{\mathrm{idr}}\left(f_i,t_i,t_i^{\mathrm{idr}}\right)& \left(105\right)\end{array}$

(though a lower bound on the buffer penalty ρ_buf can also be devised, we do not use it here for simplicity). Note that unlike ρ, ρ is additive, that is

$\begin{array}{cc}\underset{\_}{p}\left(f\right)=\sum _{i=1}^n\underset{\_}{p}\left(a_i\right).& \left(106\right)\end{array}$

The lower bound ρ(α) is used in the branch and bound Algorithm 4 for solving the combinatorial minimization problem (102). The algorithm is first invoked with f=Ø, ρ=∞, and the current state of the encoder. The order of the loop searching over all feasible frame types F₁ should be selected to maximize the probability of decreasing the bound ρ as fast as possible. Typically, the best ordering of F is DROP, followed by P (if multiple reference frames are available, they are ordered by increasing display time difference relative to the current frame), followed by IDR. Though not considered here, non-IDR I frames and non-reference P or B frames can be straightforwardly allowed for as well.

In the case where complexity allocation is performed after bit allocation and the latter is not reiterated, complexity allocation may be removed from the frame type decision algorithm and performed after the optimal frame types are assigned.

Branch and Bound Frame Type Decision Algorithm

input :  sequence of n frames; best frame type decision f so far; initial non-Markovian state
         {circumflex over (σ)}1NM ; current Markovian state σ1M
output : optimal action sequence a*; optimal action sequence cost p* = p(a *) ;lower bound
         ρ.
1 if n ≦ 0 then
// Leaf reached
2  From f and {circumflex over (σ)}1NM , allocate the sequence bit budget bT
3  Find optimal frame quantizer allocation q* .
4  From f and {circumflex over (σ)}1NM , allocate the sequence encoding time budget τT
5  Find optimal frame encoding complexity allocation c*
6  Form action sequence a* = (f ,q* ,c*).
7  Compute sequence cost ρ* = ρ(a* ,{circumflex over (σ)})
8  if ρ* <ρ then update lower bound ρ = ρ*
9 else
10  Set ρ* = ∞.
11  for all f1 ε F1 do
12   if ρ(f1) >ρ then continue // Prune subtree
13   Compute the updated Markovian encoder state σ2M = σ(σ1M ,f1).
14   Add f1 to the current f and invoke the algorithm recursively for the remaining n − 1 frames
in the sequence, producing the action sequence a, its cost ρ, and an updated bound ρ.
15   if ρ<ρ* then ρ* = ρ,a* = a . // Update current best action sequence
16  end
17 end

A flow chart corresponding to the Branch and bound frame type decision algorithm is shown in FIG. 4, where the frame type decision function is shown as a recursive process. The Frame Type Decision (n) 300 with n frames is shown to call itself with (n−1) frames as the parameters, i.e., Frame Type Decision (n−1) 350. The function first checks whether the recursion reaches end, i.e., whether n≦0. If the recursion reaches its end, the algorithm outputs frame bit budget b_T at step 370 for the Frame Bit Budget Allocation to come up the optimal allocation q* at step 372. On the other hand, the frame complexity in term of processing time is computed at step 374 and the Frame Time Budget Allocation derives the optimal encoder complexity allocation c* according the frame processing time τ_T at step 376. The optimal actions sequence α* and associated sequence cost ρ* are updated at steps 378 and 380. The optimal sequence cost ρ* is then compared with the lower bound ρ at step 382 and the lower bound is updated if ρ* is smaller than ρ, at step 384. This processing is also described in the sequence-level controller flow diagram as shown in FIG. 3.

When the recursion does not reach its end, the system initializes the sequence cost function by assigning its value to infinity at step 312 and starts the iteration for every possible frame type f₁. The number of all frame type of F₁ is denoted by L(F₁) the index i used to process all possible frame types by initializing i to 1 at step 314 and checking the ending condition i≦L(F₁) at step 366. For each frame type of f₁, the corresponding sequence cost is computed and compared with the lower bound at ρ step 320. If the computed cost is higher than the lower bound, no further processing is needed as shown as the “yes” branch at step 320.

If the computed sequence cost is smaller than the lower bound, the Markovian encoder state is updated at step 322 and the frame type f₁ is added to the sequence frame type f at step 324. Both the frame index n and the frame type index i are updated at step 326. After the frame index is decremented by 1, the Frame Type Decision recursion occurs as the step 350 is within in main function at step 300. The corresponding action sequence, sequence cost, and lower bound on the sequence cost are updated at step 360. The sequence cost is then compared with the optimal cost at step 362 and the optimal action sequence and sequence cost are updated at step 364 if the newly computed cost is lower.

Encoding Order Optimization

Given a sequence of n frames indexed by incrementing display order, the purpose of encoding order optimization is to find their optimal encoding order. For notation convenience, we denote the frame reordering as a permutation π of {1, . . . , n}. We also denote by α*(π) and ρ*(π) the optimal action sequence and its cost, respectively, found using the frame type decision algorithm from the previous section applied to the ordered sequence of frames π₁, . . . , π_n. Using this formalism, encoding order optimization can be expressed as finding the permutation minimizing

$\begin{array}{cc}\underset{\pi \in \Pi }{\min }p*\left(\pi \right)& \left(107\right)\end{array}$

Since out-of-display-order encoding requires temporary storing the decoded frames until the next consecutive frame has been decoded, the search space of feasible permutations is constrained by the decoded picture buffer level. In H.264, the decoded picture buffer is shared with the reference buffer.

We augment the Markovian component of the encoder state with the last displayed frame index ι_i^disp specifying the largest frame index that has been consecutively decoded from the sequence start at the time when frame i is being encoded. The last displayed frame is updated according to

$\begin{array}{cc}{l}_{i+1}^{\mathrm{disp}}=\{\begin{array}{cc}{l}_i^{\mathrm{disp}}\text{:}& \min \left\{\min R_i,{\pi }_i\right\}>l_i^{\mathrm{disp}}+1\\ \max \mathrm{con}\left(R_i\right)\text{:}& \mathrm{else}\end{array}& \left(108\right)\end{array}$

where con(Ri) denotes the largest subsequence of consecutive frame indices in Ri starting with min Ri (e.g., if R_i={1, 2, 3, 5}, con(R_i)={1, 2, 3}; if R R_i={1, 3, 4, 5}, con(R_i)={1}, etc). We also modify the state update rule for the reference buffer as

$\begin{array}{cc}{R}_{i+1}=\{\begin{array}{cc}{\stackrel{\_}{R}}_{i+1}\text{:}& {\stackrel{\_}{R}}_{i+1}\le R_{\max }\\ {\stackrel{\_}{R}}_{i+1}\backslash \left\{k\in \mathrm{con}\left({\stackrel{\_}{R}}_{i+1},R_{\max }-{\stackrel{\_}{R}}_{i+1}\right)\text{:}k\le l_i^{\mathrm{disp}}\right\}\text{:}& \mathrm{else},\end{array}& \left(109\right)\\ \mathrm{where}{\stackrel{\_}{R}}_{i+1}=\{\begin{array}{cc}\varnothing \text{:}& f_i=\mathrm{IDR}\\ R_i\text{:}& f_i\in \mathrm{NONREF}\\ \left\{{\pi }_i\right\}\bigcup R_i\text{:}& f_i\in \mathrm{REF},\end{array}& \left(110\right)\end{array}$

and con(R_i,k) denotes the sequence of at most k smallest elements in con(R_i) (or less, if |con(R_i)|<k). Note that π_i replaces i, and the update scheme now allows frames to be locked in the reference buffer until a consecutive sequence is formed.

In order to satisfy the reference buffer size constrains, |R_i|≦R_max has to hold for every frame in the sequence. This condition can be verified prior to minimizing ρ*(π); permutations not satisfying the reference buffer constrains are discarded from the search space. In some applications, additional constrains may apply to Π in order to enforce further frame ordering regularity. For example Π may be restricted to few pre-defined ordering patterns providing regular single-or multi-level temporal scalability of the encoded sequence. The encoding order optimization procedure is summarized in following algorithm.

Encoding Order Optimization Algorithm

input :         sequence of n frames.
output:         optimal permutation π; optimal action sequence
                a* ; optimal action sequence cost
                ρ* = ρ(a*)
initialization: Set ρ = ρ* = ∞.
1 for all π do
2  Verify reference buffer conditions.
3  if |Ri| > Rmax for any i then continue
4  Invoke the frame type decision algorithm on the sequence of
frames π1, ..., πn, producing the action sequence a,
its cost ρ, and an updated bound ρ.
5 if ρ<ρ*, then ρ* = ρ, a* = a, π* = π.
6 end

The flow chart of the encoding order optimization algorithm is shown in FIG. 5. The process starts with n frames at step 400 and an initial sequence cost of infinity. The total number for possible sequence order is K. In order to go through all possible sequence order, iteration is used with index k with initial value of 1 at step 415 and the termination condition is checked at step 420. The reference buffer is verified at step 425 and the reference buffer is compared with the maximum buffer size for all i at step 430 and no further processing is the overflow condition occurs. If the buffer condition is satisfied, the optimal frame type decision is invoked at step 435 and the sequence cost corresponding to the action sequence is calculated at step 440. The lower bound is also compared with this new sequence cost and is updated as well. The new sequence cost is compared with the optimal cost at step 445. If the new cost is lower than the optimal cost, the optimal cost, optimal action sequence and optimal sequence order are all updated at step 450. When the process is complete at step 460, the final optimal sequence order, optimal action sequence and sequence cost are the outputs from the encoding order optimization algorithm.

Claims

1. A method of encoding video sequence, comprising: receiving an input video sequence containing pictures;receiving a resource budget for the sequence; andreceiving a sequence cost function; andgenerating an optimal action sequence to produce an encoded version of the sequence to achieve a minimum sequence cost within the sequence resource budget, further comprising of generating an optimal permutation of the sequence of pictures andgenerating an optimal set of encoder parameters for each picture.

2. A method according to claim 1, wherein generating an optimal action sequence is performed by solving the optimization problem

3. A method according to claim 2, wherein the optimization problem is solved simultaneously with respect to a and π.

4. A method according to claim 2, wherein the optimization problem is solved separately with respect to a and π.

5. A method according to claim 2, wherein the optimization problem is solved separately for each of the encoding parameters.

6. A method according to claim 2, wherein the optimal permutation π* and optimal action sequence a* are found by performing the following steps Start: set best cost ρ* to infinity.For each candidate permutation π of the indices {1, . . . ,N} performStep 1: Verify whether π is an admissible permutation. If yes, continue to next step; otherwise, skip to next candidate permutation.Step 2: Find the optimal action sequence a and its cost p on the sequence of pictures π1, . . . πn.Step 3: If ρ<ρ*, update a*=a, π*=π, and ρ*=ρ.End

7. A method according to claim 6, where a permutation is deemed admissible if it does not violate the encoder reference buffer constraints.

8. A method of claim 6, wherein optimal action sequence corresponding to a candidate permutation π is found by receiving as the input the permuted sequence of pictures π1, . . . ,πn and the current encoder state, and outputting the solution of the optimization problem

9. A method according to claim 2, wherein the vector of resources consists at least one of the following (a) encoding bits;(b) encoding time;(c) encoder buffer utilization; and(d) decoder buffer utilization.

10. A method according to claim 2, wherein the set of encoder parameters αi for the ith picture includes (a) picture type;(b) reference frames used for motion compensation;(c) quantization scale;(d) encoder complexity for the picture.

11. A method according to claim 2, wherein the action sequence cost f is an aggregate of at least one of the following cost functions: (a) decoder bit buffer violation cost;(b) encoder bit buffer violation cost;(c) distortion cost of the pictures as a result of their encoding;(d) distortion cost of the sequence as a result of picture drop;(e) fluctuation cost of the quantization parameter around the average sequence quantization parameter;(f) IDR cost;(g) bit budget deviation cost.

12. A method according to claim 12, wherein the decoder bit buffer violation cost is estimated as

13. A method according to claim 12, wherein distortion cost of the picture is given by. pdis({circumflex over (p)}i)=2552·10−0.1{circumflex over (p)}i.

14. A method according to claim 12, wherein the IDR cost for penalizing for a too early IDR frame is given by:

15. A method according to claim 12, wherein the Drop cost penalizing the dropped frame distortion is given by:

16. A method according to claim 16, wherein the estimated minimum PSNR, {circumflex over (p)}imin is computed as the 0.1%-quantile of the PSNR of the difference between the original frame i and the last displayed frame lind.

17. A method according to claim 16, wherein the estimated minimum PSNR, {circumflex over (p)}imin is equal to {circumflex over (p)}i for a non-dropped frame.

18. A method according to claim 12, wherein the fluctuation cost of the quantization parameter around the average sequence quantization parameter is: Pidr(q′i)=max{2q′i−q′,2q′−q′i}−1,

19. A method according to claim 19, wherein the decay parameter, λ is 0.99, and may be adjusted according to the sequence frame rate.

20. A method according to claim 12, wherein the Bit budget deviation cost penalizing for action sequences resulting in a deviation from the allocated bit budget bT is

21. A method according to claim 12, wherein the IDR cost for penalizing for a too late IDR frame is given by:

22. A method according to claim 12, wherein aggregate is a weighted sum.

23. A method according to claim 9, wherein a decision tree is constructed corresponding to all combinations of picture types in the action sequence; the cost function is evaluated for all possible paths in the tree corresponding to all possible action sequences; and the sequence represented by the path with the minimum cost is selected.

24. A method according to claim 24, wherein branch and bound technique is used to accelerate the exhaustive search in the decision tree.

25. A method according to claim 25, receiving as the input a sequence of n pictures {1, . . . ,n}, the current encoder state σ1, and applying the following steps recursively:

26. A method according to claim 25, wherein the lower bound of the action sequence cost is used by the branch and bound technique.

27. A method according to claim 26, wherein the lower bound of the action sequence cost is a function dependent on the picture type.

28. A method according to claim 2, wherein an estimate {circumflex over (r)} of the resources is used instead of the true vector of resources r.

29. A method according to claim 28, wherein an encoder model is used to compute {circumflex over (r)}.

30. A method according to claim 29, wherein {circumflex over (r)} is a function of the action sequence a and the encoder model parameters θ.

31. A method according to claim 12, wherein the distortion is estimated using an encoder model.

32. A method according to claim 31, wherein the distortion for the encoder model is the macroblock PSNR, which is inversely proportional to the quantization parameter qi−1 and to the macroblock complexity zi: {circumflex over (p)}i(qi, zi)=β1−β2zi−β3q′i

33. A method according to claim 31, wherein the distortion for the encoder model is the frame-wise average PSNR:

34. A method according to claim 10, wherein the encoder time is estimated using an encoder model.

35. A method according to claim 34, wherein the encoder time is estimated for macroblock i as: τ′1(qi,zi)=max{min{γ1+γ2zi−γ3q′i,1}, 0}T

36. A method according to claim 34, wherein the encoder time is estimated for an entire frame as:

37. A method according to claim 10, wherein the encoder bit is estimated using an encoder model.

38. A method according to claim 37, wherein the bit production model for a macroblock is predicted by: {circumflex over (b)}i(qi,zi)=α1+α2ziqi−1

39. A method according to claim 37, wherein the bit production model for a frame is predicted by:

40. A method according to claim 25a, wherein a sequence of n pictures {1, . . . ,n} and their types p={p1, . . . ,pn} is received as the input, the following steps are repeated at least once, Step 1: Fix c={c1, . . . ,cn} and find the optimal quantizer scale sequence q={q1, . . . ,qn};Step 2: Fix q={q1, . . . ,qn} and find the optimal encoding complexity sequence c={c1, . . . ,cn};

41. A method according to claim 40, wherein the optimal sequence of quantization scales, q=(q1, . . . , qn)T where n is the number of frames in the video sequence is calculated according to

42. A method according to claim 40, wherein generating the optimal allocation of a vector of quantization scales is based on a variant of bottleneck link algorithm comprising the steps: Start: initializing the quantization scales to a same quantity according to a bit production model;Step 1: computing {circumflex over (l)}−dec(q) and {circumflex over (l)}+dec(q);Step 2: branching on the test conditions for {circumflex over (l)}−dec(q)≦lmaxdec and {circumflex over (l)}+dec(q)≧lmaxdec,

43. A method according to claim 42, wherein initializing the quantization scales is according to

44. A method according to claim 40, wherein the optimal sequence of picture encoding complexity scales c=(c1, . . . , cn)T is computed according to

45. A method according to claim 40, wherein generating the optimal allocation of a vector of complexity scales is based on a variant of bottleneck link algorithm comprising the steps: Start: initializing the complexity scales to a same quantity according to an encoding time model;Step 1: computing {circumflex over (l)}+enc(c), {circumflex over (l)}−enc(c), {circumflex over (l)}−raw(c) and {circumflex over (l)}+raw(c);Step 2: branching on the test conditions for {circumflex over (l)}+enc(c)≦lmaxenc and {circumflex over (l)}−enc(c)≧lminenc and {circumflex over (l)}31 raw(c)≦lmaxraw and {circumflex over (l)}+raw(c)≧lminraw,

46. A method according to claim 45, wherein initializing the complexity scales is according to