Patent Application
20070036227
This invention relates to the field of video encoding, and in particular to using visual analysis tools and their application to the problem of rate control.
In digital video applications the overriding factor in the visual quality of the encoded video is the number of bits per second that can be transmitted over a common channel, also known as the bit-rate. Low bit-rates allow for only lower quality video, while the higher bit-rates allow for better spatial and temporal quality. Generally the number of bits generated by an encoder is inherently variable in nature and is very much content dependent. Motion, dynamic texture, occlusions, and lighting changes are among some of the things that alter the pixel statistics from one frame to the next. However, channel requirements and/or storage requirements govern the bit-rate regardless of content. In order to regulate the number of compressed bits generated for such varying pixel data, encoders were used employing rate control methods.
The rate control methods previously used matched the bit-rate/storage requirements by trading off spatial and temporal quality in the compressed bitstream. These rate control methods regulated the volume of compressed data by adjusting the appropriate encoding controls. Intelligent rate controls adeptly allocated bits amongst the entire video while striving to achieve the best possible tradeoff between spatial and temporal quality. The rate controls were an important part of an encoding system and were key differentiators between different video encoders.
Uncompressed, or raw, digital video required tremendous amounts of bandwidth to transmit and equally large amounts of storage space to archive. Video compression standards such as MPEG-1, MPEG-2, MPEG-4, H.263 and H.264 all take advantage of the naturally existing spatio-temporal redundancies and allow for distortions in order to achieve significant bandwidth reductions. The higher the compression rate, the more distortions the encoders yielded. However, not all encoders produced the same distortions. The type and severity of distortions vary from one encoder to another and were a function of the individual encoding techniques such as motion estimation, mode selection, and rate control. Among these encoder techniques, rate control had the most impact on the overall encoded video quality.
A typical video encoding system 100, such as used in the prior art, is shown in
In order to achieve a regulated output bit-rate or frame rate, the encoder 104 relied on a rate controller 106. The rate controller 106 operated by using encoding status data provided by the encoder 104 and outputted rate control adjustment data to the encoder 104. The rate control adjustment data contained parameters that affected how the current or future frames were encoded. The rate control adjustment data included information that could be provided at the beginning of a frame and continuously updated throughout the frame when the encoder 104 allowed for update information to be utilized. The information that was conveyed in the rate control adjustment data was a function of the specific encoding technique.
In the MC-DCT family of codecs, the rate controller 106 typically used frame dropping and a quantization step size, Qp, to regulate the bit-rate of the output video bitstream. Frame dropping told the encoder 104 to not code the current frame in the video frames being inputted. This reduced the number of frames in the resulting video at the expense of temporal fidelity. Qp controlled the fidelity with which a frame was coded. A larger Qp encoded a frame with less granularity resulting in fewer output bits with more distortions while a smaller Qp encoded a frame with more bits and better quality. Quantization is a lossy process that introduced distortions by reducing the fidelity of the coded data into a number of finite quantization bins. In cases where allowed, Qp could be adjusted multiple times within a frame for better control of the number of bits generated. Thus, the rate controller 106 had to be able to balance both temporal (via frame drops) and spatial quality (via the Qp) such that a fixed bit-rate budget was met.
The majority of rate control algorithms today do not give preferential treatment to the contents of the video. These rate control algorithms typically operate using only statistics, such as the number of bits generated, the average Qp from the previous frame, and the number of frames dropped to derive information for rate control adjustment. This type of rate controller 106 was described in a Sep. 1997 publication entitled “Video Codec Test Model, Near Term, Version 8 (TMN8), Test Model 8 (TM8)”. As such, the decision mechanism was agnostic to the contents of the video input. The contents of the video input did alter the encoded statistics, but the rate controller 106 did not have any more knowledge beyond this.
A rate control method that incorporated the visual properties (significance) of the video frame information provided better tradeoffs of visual quality while maintaining the desired bit-rate. As presented in an Oct. 1998 article by S. Daly, K. Matthews and J. Ribas-Corbera entitled “Face Based Visually Optimized Image Sequence Coding”, an encoding technique that used the human visual system's properties and a face detection method to allocate more bits to the facial areas is outlined. The investigators allowed for better quantizer control in the area of interest in the frame to produce video adapted to the face. This was an adaptive technique that is limited to video with facial objects. Furthermore, this technique did not take into account other aspects of the rate control such as frame dropping and I/P frame mode decisions. Another work presented in a May 2001 article by Wallace K-H Ho and Daniel PK Lun entitled “Content-based Scalable H.263 Video Coding for Road Traffic Monitoring Based on Regularity of Video Content”, focused on content adaptive scalable H.263 coding for use in traffic monitoring. In this method the investigators extracted and classified moving objects in the scene to enable the rate control to operate nearly 20% more efficiently. However, this technique involved segmenting vehicles and was tailored specifically to traffic scenes and could not be applied to other types of video. Another technique using video analysis to allow better encoding was also investigated by Liang-Jin Lin and A. Ortega in an Oct. 1997 article entitled “Perceptually Based Video Rate Control Using Pre-filtering and Predicted Rate-distortion Characteristics”. In this technique pre-filtering of the video to classify areas within a frame to assist in a rate-distortion-based rate control was used. This resulted in better video with fewer artifacts. However, this method focused heavily on blocking artifact reduction and worked within a rate-distortion-optimized framework that was not feasible for real-time and low power application scenarios. Furthermore, it did not address the issues of frame dropping or frame type selection.
Visual analysis techniques today allow detailed analysis of the properties of the video sequence data. These tools have recently become important in categorizing, indexing, and organizing the ever-increasing volumes of digital data. Built on the foundations of basic image processing techniques, these tools provide statistical parameters that describe motion, texture, lighting, and complexity of a video frame. Examples of these tools can be found in the MPEG-7 Multimedia Content Description Interface Standard described in Information Technology - Multimedia Content Description Interface Part 3. In particular, the MPEG-7 standard provide for the MPEG-7 Visual Description Tools that extract color, texture, shape, motion, localization, and face recognition features of a video segment.
The MPEG-7 visual metrics offered were the result of research in multimedia information processing and digital library spanning the past decade. A visual metric is comprised of a visual feature definition, known as D, or Descriptor in MPEG-7 and an associated metric function. MPEG-7 defines a set of visual metrics for Color, Shape, Texture and Motion, which were validated by experiments to be compatible to subjective human perception. Among these visual metrics are the Color Layout Descriptor, or CLD, described in a Jun. 2001 article by B. S. Manjunath, J. R. Ohm, V. V. Vasudevan and A. Yamada, entitled “Color and Texture Descriptors”, and the Motion Activity Descriptor, or MAD, also described in a Jun. 2001 article by Jeannin and A. Divakaran, entitled “MPEG-7 Visual Motion Descriptors”.
The Color Layout Descriptor is a color feature that describes the rough color layout of the image. The CLD is very useful in describing the difference, or distance, between two frames of a video. Applied to successive frames, the CLD metric is a good approximation of visual content changes throughout a video sequence. The CLD can be used in a variety of measurements and has been used as a one-pass frame selection mechanism for video summaries as described in published U.S. patent application Ser. No. 20040085483. The Motion Activity Descriptor captures the amount of object motion in a video frame. It is based on the variance of the magnitude of motion vectors (MV) in the frame rather than the mean of the magnitude of MVs, which can be easily distorted by global camera motion. The MAD has shown to be a good measure of motion activity in a frame.
What is needed is an encoder that uses knowledge of the contents of the video input in a rate control mechanism that can allow a rate controller to make more visually significant decisions.
What is also needed is a method of rate control that incorporates the visual properties of the video frame data to provide better visual quality in an encoded video sequence while adhering to a desired bit-rate.
What is also needed is a method of rate control that uses visual analysis tools such as those specified in MPEG-7 to assess the significance of video frames being encoded.
What is also needed is a rate controller that utilizes visual analysis tools to provide information in the form of a rate control adjustment signal to an encoder to encode video frames having fewer distortions and better compression efficiency.
While this invention is susceptible of embodiment in many different forms, there is shown in the drawings and will herein be described in detail one or more specific embodiments, with the understanding that the present disclosure is to be considered as exemplary of the principles of the invention and not intended to limit the invention to the specific embodiments shown and described. In the description below, like reference numerals are used to describe the same, similar or corresponding parts in the several views of the drawings.
The MPEG-7 visual metrics described above have not previously been utilized in implementing content adaptive rate control. Likewise, the earlier MPEG-1, MPEG-2, MPEG-4, and H.261 visual metrics, nor more recently proposed H.263, and H.264 standard visual metrics described above have not been previously utilized in implementing content adaptive rate control.
The Color Layout Descriptor tool 304 determines the rough color layout of the video image in the video frame, determining the difference, or distance, between two video frames. The Color Layout Descriptor tool 304 is applied to successive video frames and generates a CLD metric Cn, that provides an approximation of the visual content changes throughout a video sequence. When implemented, the Motion Activity Descriptor tool 306 determines the amount of object motion in a video frame. The Motion Activity Descriptor tool 306 is also applied to successive video frames and generates a MAD metric vn that provides a measure of motion activity in a frame. Also when implemented, the Texture Descriptor tool 308 determines the texture of the video image in the video frame. The Texture Descriptor tool 308 is also applied to successive video frames, to generate a TD metric tn that provides a measure of texture within a frame.
The visual analyzer 208, in accordance with the present invention associates frame m to be the most recently encoded frame and frame n to be the current source frame. In accordance with the present invention, it is assumed that the value of m is less than n although this need not be the case. The CLD metric Cn for frame n is calculated in the visual analyzer 208 and supplied to the rate controller 206 to decide frame drops. When implemented, the MAD metric vn and the TD metric tn for frame n are also calculated in the visual analyzer 208 and are supplied to the rate controller 206 to decide frame coding modes (I/P/B), and the quantization step size Qp, respectively, as will be described below.
The rate controller 206 in accordance with the present invention utilizes Boolean adders and multipliers, such as adder 312, and multiplier 314, and may additionally include adders 318 and 322, and multipliers 316 and 320 as will be described in detail below. The output of the Color Layout Descriptor tool 304 couples to a first input of adder 312. The input to a second input of adder 312, and the operation thereof will be described below. The output of adder 312 couples to a first input of multiplier 314. The input to a second input of multiplier 314, and the operation thereof will be described below. When implemented, the output of the Motion Activity Descriptor tool 306 is coupled to a first input of multiplier 316. The input to a second input of multiplier 316, and the operation thereof will be described below. Also when implemented, the output of the Texture Descriptor tool 308 is coupled to a first input of multiplier 320, The input to a second input of multiplier 320, and the operation thereof will be described below.
The output of multiplier 314 is coupled to a first input of adder 318 when the Motion Activity Descriptor tool 306 is implemented, otherwise it couples directly to an input of a frame drop decision element 326. When the Motion Activity Descriptor tool 306 is implemented, the second input of adder 318 is coupled from the output of multiplier 316. The output of adder 318 is the coupled to an input of the frame-drop decision element 326 and an input of an I/P frame evaluation element 328. An internal rate control status buffer 324 provides rate control status information that is coupled to a second input of the frame-drop decision element 326 and when the Motion Activity Descriptor tool 306 is implemented to a second input of the I/P frame evaluation element 328. The frame-drop decision element 326 processes the information generated by the internal rate control status buffer 324 and the output of adder 318 to determined when a frame n will be dropped, as will be described further below, and generates encode/skip decision data at the output.
The output of the frame-drop decision element 326 also couples to an input of the I/P frame evaluation element 328 when the Motion Activity Descriptor tool 306 is implemented. When the frame-drop decision element 326 determines frame n will not to be dropped the encode/skip decision data is coupled the I/P frame evaluation element 328 to enable its operation. The I/P frame evaluation element 328 determines whether frame n is to be defined as an intra frame or an inter frame, as will be described further below, and delivers I/P decision data at the output.
When the Texture Descriptor tool 308 is implemented, the output of multiplier 320 is coupled to an input of adder 322. A second input of adder 322 is coupled to a second output of multiplier 314. The output of adder 322, which will be described in detail below, is coupled to an input of Qp calculation element 330. A second input of Qp calculation element 330 is also coupled to the output of the internal rate control status buffer 324. The Qp calculation element 330 processes the inputs, as will be described below, and outputs a Qp signal defining the spatial quality of the video frame being processed.
In the present invention the rate controller 206 determines frame drops using frame-drop decision element 326. The rate controller 206 further determines frame coding modes using I/P frame evaluation element 328 by computing a distance metric d(m, n) between frames m, and n. This distance metric is defined as
d(m,n)=w1(n)·(cn−cm)
where cm is the CLD of frame m and w1(n) is a weighting factor for frame n. When the Motion Activity Descriptor tool 306 is implemented, as shown in
d(m,n)=w1(n)·(cn−cm)+w2(n)·mn
where mn is the MAD metric and w2 (n) is a weighting factor for frame n. This combination of visual analysis metrics links the rate control operation more tightly to the video and can allow better responsiveness.
Frame drop decisions generated in the frame-drop decision element 326 are made using the distance d(m, n) in conjunction with internal rate control status information such as provided by the internal rate control status buffer 324 contents, and the time elapsed since the last encoded frame m. In the present invention a frame drop decision function incorporating the various parameters is computed as
where ƒm(n) is the non-zero probability of encoding n given m has been encoded, s(m, n) is the temporal distance between the two frames, and η(n) and γ(n) are weighting factors for the current frame. An exemplary frame drop decision mechanism combining the frame drop decision and I/P/B frame decision is presented graphically in
It will be appreciated from the description provided above, the video encoding system 200 providing content adaptive rate control described in
It will be appreciated from the description provided above, that the encoding system 200 providing content adaptive rate control includes the visual analyzer 208 and the rate controller 206 described in
Other memory devices can be utilized as well.
The decision mechanism presented above uses the frame drop decision function Fm(n) to decide both the frame drop and whether to encode the frame as an INTRA (I) or an INTER (P/B) frame. In the present invention, the rate control algorithm compares the frame drop decision function Fm(n) to the FCODE threshold. When Fm(n) is less than FCODE, frame n is dropped and not coded. When Fm(n) is larger than FCODE but less than FINTRA, the frame coding parameter, frame n is selected for encoding as a P or B frame. If Fm(n) exceeds both FCODE and FINTRA, frame n is encoded as an INTRA frame. Additionally, Fm(n) can be used by the rate control to request more INTRA macroblocks in an INTER frame as Fm(n) approaches FINTRA. The frame drop and mode mechanism is used after the first frame has been encoded using predefined parameters.
The frame drop mechanism is important in regulating the encoded bit-rate. However, without associated visual information about the source video, it can cause important, or key, frames to be dropped. Using visual information data derived from the Color Location Descriptor tool 304 and Motion Activity Descriptor tool 306, the rate controller 206 can better estimate those frames to encode that may otherwise have been dropped. In cases where visual information for future frames is known the rate controller 206 can tailor its operation based upon knowledge that certain frames in the future will have to be encoded while others can be sacrificed.
The quantization parameter, Qp, is generated using the internal rate control status information located in the internal rate control status buffer 324 p(m, n) augmented with the visual information quantization metric p(m, n). In this embodiment Qp is a function of the CLD metric p(m, n) and defined as
p(m, n)=w1(n)·(cn−cm)
Alternative embodiments can define the visual information quantization metric as a function of both the CLD metric p(m, n) and the TD metric tn as
p(m, n)=w1(n)·(cn−cm)+w3(n)·tn
where w3(n) is a weighting factor or use the CLD metric cn, MAD metric vn, and TD metric tn, as
p(m,n)=w1(n)·(cn−cm)+w2(n)(mn−mm)+w3(n)·tt.
While all rate control techniques have specific Qp calculation algorithms, p(m,n) can be used to either offset the calculated Qp or as an integral part of the calculation. Let bm be the number of bits generated by encoding frame m using an average Qp,
where α and β are weighting coefficients that are predefined or dynamically obtained. qn is the initial quantization step size for the current frame. As statistics are obtained as the frame is being coded, the quantization step size can be adjusted multiple times throughout the frame to achieve the target number of bits. qn is set to a desired value for encoding of the first frame of the video sequence where
Utilizing the CLD metric and TD metric, the rate controller 206 is able to better derive the Qp value. A high texture region will require more bits during the encoding. To ensure that the bit-rate is evenly regulated, the Qp calculation element 330 in the rate controller 206 can respond with a higher Qp to balance the source video's high complexity characteristic. It should also ensure that too much detail is not lost because of the high Qp that results in blocking artifacts. In low texture regions, the Qp calculation element 330 in the rate controller 206 can reduce the Qp to adapt to the easy nature of the frame. This will also reduce annoying quantization artifacts that are visible in low texture regions and due to Qp variations.
In accordance with the present invention begins the encoding process at step 502. A sequence of video frames is sequentially inputted beginning with frame n at step 504. One or more visual analysis metrics are computed as described above for the video frame inputted at step 508. A distance metric is computed between the input frame n and a previously encoded frame m, at step 508. A decision function Fm(n)is computed at step 510. The computed decision function Fm(n) is compared to a first threshold, FCODE, at step 512, which is used to determine when a video frame should be dropped. When FCODE is less than Fm(n), at step 512, the frame-drop decision element 326 generates an encode/skip decision signal to drop video frame n, and the encoder 204 drops video frame n, at step 514. When Fm(n) is greater than FCODE at step 512, the frame-drop decision element 326 generates an encode/skip decision signal to code video frame n
Fm(n) is compared to a second threshold, FINTRA, the frame coding parameter, at step 516, which is used to determine the type of encoding to be performed. When Fm(n) is less than FINTRA, at step 516, the I/P frame evaluation element 328 generates an I/P decision signal to encode video frame n as an INTER frame, i.e. frame n data is encoded as processed. When Fm(n)is greater than FINTRA, at step 516, the I/P frame evaluation element 328 generates an I/P decision signal to encode video frame n as an INTRA frame, i.e. the difference between the current frame n and the previous frame m is calculated, and the difference is encoded.
The present invention offers a key benefit in a variety of applications. It provides a method by which video is rate controlled by analyzing the contents of the video. This method improves on the operation of existing rate controllers with the addition of visual analysis tools that provide key features about the video contents. The MPEG-7 visual descriptors are a set of tools, as described above, that can be utilized in the visual analyzer 208. The visual analyzer 208 data can also be embedded within the bitstream to avoiding regeneration at the receiving end. In pre-stored applications, and power-limited receivers such as streaming of video data to a mobile phone client, the client can utilize the pre-computed MPEG-7 data saving unnecessary computation complexity and power.
The present invention is applicable for use in a number of areas. First within the fast growing Internet applications market, the present invention offers the capability of encoding data in an adaptive manner and key in differentiating amongst other competitors. The present invention focuses on the video encoding, video database, video browsing, surveillance, public safety, storage, and video streaming applications.
The present invention is a video encoding system and method for providing content adaptive rate control that utilizes visual analysis tools in a pre-processing role to guide the encoding process. The present invention provides a decision mechanism that adjusts the rate control to adapt the encoding based upon the content as parameterized by the visual analysis tools. The visual analysis tools allow a rate control mechanism to better decide which frame to encode and which frame to drop based upon the frame's necessity in the encoded video. Furthermore, the visual analysis tools can be utilized to modify the quantization parameter (Qp) based upon the complexity of the scene and bit constraints. The present invention allows for video encoders and rate controllers that are tailored to the source content providing better coding efficiency and video quality.
While the invention has been described in conjunction with specific embodiments, it is evident that many alternatives, modifications, permutations and variations will become apparent to those of ordinary skill in the art in light of the foregoing description. Accordingly, it is intended that the present invention embraces all such alternatives, modifications and variations as fall within the scope of the appended claims.
