Method and system for a highly efficient low bit rate video codec

Abstract

A system and a method for wavelet-based low bitrate video coding is provided. The system and method of this invention processes each frame of a video sequence based on its content. The discrete wavelet transform of each frame is taken separately. If the difference between the contents of a first frame and a second frame is above a threshold, then the wavelet transformed second frame is quantized and lossless encoded. If the difference is less than a threshold, then motion estimation is employed in wavelet domain. Then the difference between the wavelet transformed second frame and the motion compensated wavelet transformed second frame is quantized and lossless encoded. The quantization is applied by using dynamic adjustment. The quantization tables are optimized based on the coded frame characteristics. In addition to the coded frames, the lossless coded motion vectors are also transmitted to the decoder unit. To improve the quality of the coded video sequence another decision criteria is used at the feedback loop of this invention. The coded second frame is decoded and compared with the original second frame. If the difference between the two frames is above a threshold, then the wavelet transformed second frame is quantized and lossless encoded instead of the motion compensated second frame.

Description

FIELD OF THE INVENTION

[0002] The present invention relates to a video compressing technique and, more particularly, to a subband coding method.

BACKGROUND OF THE INVENTION

[0003] It is well known that a wavelet transform provides superior performance when performing image and video processing. Wavelet transform avoids blocking effect and mosquito effect, since the transform is taken on the whole image. U.S. Pat. No. 5,563,960 shows that this approach is better than a DCT transform. However, although this patent describes taking spatial redundancy into account, an image and video order processing system which incorporates the description of this patent still requires processing power due to temporal redundancy. In previous systems, a variable-block size multi-resolution motion estimation scheme is proposed reducing the temporal redundancy. This also requires additional processing power. For real-time application the speed of the algorithm is much more important than the accuracy of the motion estimation.

[0004] Accordingly, what is needed is a system and method for providing a low bit rate video coding. The method and system should be compatible with existing video processing system, cost effective and easily implementable. The present invention addresses such a need.

SUMMARY OF THE INVENTION

[0005] A system and a method for wavelet-based low bit rate video coding is provided. The system and method in accordance with the present invention processes each frame of a video sequence based on its content. The discrete wavelet transform of each frame is taken separately. Then the difference between these two frames are calculated. If the difference between the contents of the first frame and the second frame is above a threshold, then the wavelet transformed second frame is quantized and coded with a lossless coding method (Such as a run-length coder followed by a Huffman encoder or an arithmetic coder). If the difference is less than a threshold, then motion estimation is employed in wavelet domain. Then the difference between the wavelet transformed second frame and the motion compensated wavelet transformed second frame is quantized and coded with a lossless coding method. The quantization is applied by using dynamic adjustment. The quantization tables are optimized based on the coded frame characteristics. In addition to the coded frames, the lossless coded motion vectors are also transmitted to the decoder unit. To improve the quality of the coded video sequence another decision criteria is used at the feedback loop. The coded second frame is decoded and compared with the original second frame. If the difference between the two frames is above a threshold, then the wavelet transformed second frame is quantized and lossless coded instead of the motion compensated second frame.

[0006] A system and method in accordance with the present invention uses fixed block sizes and fixed refinement of motion vectors, which inevitably reduces processing power requirements. Also in the scheme of video coding, quantization is very important especially in high-efficiency coding. Quantization affects the quality of the reconstructed frames in the feedback loop, which affects the motion estimation as well. The dynamic adjusted quantization scheme in accordance with the present invention also improves the quality and efficiency of coding. Previous art also shows that estimating motion in transform domain, can give us opportunity to make processing such as watermarking easily after coding of the video, without using inverse transforms. Since the motion estimation is within in transform domain, our scheme also has this capability.

[0007] Therefore, to utilize the advantages of the wavelet transform and reduce processing complexity, a system and method in accordance with the invention employs hierarchical motion estimation in the wavelet domain. The motion estimate of each scale is often used as a starting point for the higher scale. After the compensation, the energy of the diagonal quarter is used as a fall back decision mechanism. If the energy is above a threshold then the system automatically sends the original image instead of the difference image.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 is a block diagram of the complete coder design.

[0009] FIG. 2 is a motion estimator block.

[0010] FIG. 3 illustrates a 1-level wavelet transform of an image.

[0011] FIG. 4 illustrates a 3-level wavelet transform.

[0012] FIG. 5 illustrates a motion vector computations for a fixed size block at lowest level.

[0013] FIG. 6 illustrates a motion vector refinement with fixed refinement limits.

[0014] FIG. 7 illustrates an error computation of real and motion vector calculated block locations.

DETAILED DESCRIPTION

[0015] The present invention relates to a video compressing technique and, more particularly, to a subband coding method. The following description is presented to enable one of ordinary skill in the art to make and use the invention and is provided in the context of a patent application and its requirements. Various modifications to the preferred embodiment and the generic principles and features described herein will be readily apparent to those skilled in the art. Thus, the present invention is not intended to be limited to the embodiment shown but is to be accorded the widest scope consistent with the principles and features described herein.

[0016] A system and method in accordance with the present invention uses fixed block sizes and fixed refinement of motion vectors, which inevitably reduces processing power requirements. Also in the scheme of video coding, quantization is very important especially in high-efficiency coding. Quantization affects the quality of the reconstructed frames in the feedback loop, which affects the motion estimation as well. The dynamic adjusted quantization scheme in accordance with the present invention also improves the quality and efficiency of coding. Previous art also shows that estimating motion in transform domain, can give us opportunity to make processing such as watermarking easily after coding of the video, without using inverse transforms. Since the motion estimation is within in transform domain, our scheme also has this capability.

[0017] Therefore, to utilize the advantages of the wavelet transform and reduce processing complexity, a system and method in accordance with the invention employs hierarchical motion estimation in the wavelet domain. The motion estimate of each scale is often used as a starting point for the higher scale. After the compensation, the energy of the diagonal quarter is used as a fall back decision mechanism. If the energy is above a threshold then the system automatically sends the original image instead of the difference image.

[0018] To describe the features of the present invention in more detail, refer now to the following description in conjunction with the accompanying figures. FIG. 1 is a block diagram of the complete coder 100. The coder 100 processes each frame of a video sequence based on its content. First, in the motion estimator block 102, the discrete wavelet transform of each frame is taken separately. Then the difference between these two frames are calculated. If the difference between the contents of the first frame and the second frame is above a threshold based upon the block 102, then the wavelet transformed second frame is quantized and lossless encoding is applied by the lossless encoder 104. If the difference is less than a threshold, then motion estimation is employed in wavelet domain. Then the difference between the wavelet transformed second frame and the motion compensated wavelet transformed second frame is quantized and lossless encoded again by the lossless encoder 104. The quantization is applied by using dynamic adjustment. The quantization tables are optimized based on the coded frame characteristics. In addition to the coded frames, the lossless coded motion vectors are also transmitted to the decoder unit. To improve the quality of the coded video sequence another decision criteria is used at the feedback loop. The coded second frame is decoded and compared with the original second frame by the decoder 106 and then provided to buffer 108. If the difference between the two frames is above a threshold, then the wavelet transformed second frame is quantized, lossless encoded instead of the motion compensated second frame.

[0019] A key feature of the present invention is the motion estimator block 102. It will be described in detail herein below. FIG. 2 is a motion estimator block 102. The motion estimator block 102 operates in the following manner. The wavelet transformer blocks 202 and 204 take the wavelet transform of both the frames respectively. The motion estimation block 206 calculates the motion vectors as explained in detail in the next section. The motion compensation block 208 generates the image using the calculated motion vectors. The subtractor 210 calculates the difference between the wavelet transformed original image and the image created using motion compensation to calculate the motion estimation error. The decision block 212 finds if the error is above a threshold. If it is above a threshold the second input to AND1 block 214 is 1 while the first input to AND2 block 216 is 0, so wavelet transformed 2nd frame is selected for coding. Similarly, if it is below a threshold, motion estimation is used for coding.

[0020] FIG. 3 illustrates a 1-Level wavelet transform of an image. The wavelet transform subsamples the image. As shown in FIG. 3, a 1-level of wavelet transform creates 4 subsampled separate quarters, containing different sorts information about the image. Further transforms are performed on the LL section of the transforms.

[0021] FIG. 4 illustrates a 3-level wavelet transform. Motion vectors are calculated at each quarter of the lowest level for a fixed size block. In this example, block size at level 3 is taken to be 4×4. FIG. 5 illustrates a motion vector computation for a fixed size block at lowest level. Once a block is chosen for a frame at (t−1), then a fixed area of N×(4×4) where N is chosen by experimentation is searched for the motion of the block at time (t). In our implementation, a fixed area of 9×(4×4) has given good results. If the size of motion vector is smaller than a predefined size of MVT3 (a,b), this is taken to mean that there is insignificant difference between the blocks and this is realized as no motion. However, if the motion vector is bigger than the threshold, then it needs to be refined at higher levels. All search blocks are 4×4 in all hierarchies. So a 4×4 search block in HD3 corresponds to 4 blocks of size 4×4 in HD2.

[0022] The ideal block size selected should be proportional to the whole image size. In this particular case, for an image size of 72×576 pixels, the ideal block size is found to be 4×4 pixels in the lowest level. This corresponds to 32×32 at the highest level. Different images will require different block sizes. A satisfactory fixed size can be found for different image sizes.

[0023] Choice of a bigger block size such as 32×32 at the lowest level may not be able to catch individual motions. Choice of a smaller block size will have a different problem. Consider a block size of 2×2; motion detected in this small block size may not be the dominant motion. It is highly likely that it is noise.

[0024] Therefore, the selection of the appropriate block size is an important parameter. The ideal block size needs to be selected using the following parameters:

[0025] desired quality

[0026] required bandwidth

[0027] desired accuracy of motion estimation

[0028] required code size

[0029] required execution speed

[0030] required memory size

[0031] In our case, a 4×4 block was found to be satisfactory. The important point here is not the actual fixed size but the idea of using fixed block sizes. The advantages of using fixed block sizes for motion vector calculation are:

[0032] reduces computational complexity

[0033] increases execution speed

[0034] reduces code size and memory usage

[0035] FIG. 6 illustrates that the motion vector is refined at each level by fixed amounts of (±1) in each direction. By only fixing by the amount of refinement at each level, the computational speed of the algorithm is increased. However, in some cases, depending on the information in the scene and the speed of movement, the error corresponding to motion vectors also increases. Although this is a drawback this is acceptable to gain speed in both compression and decompression. Additionally, ±1 regularity of refinement increases compression because this approach decreases the motion vector information. Only the refinement factors are sent instead of entire motion vector data.

[0036] There are several check locations in the coding scheme to decide whether the motion is estimated correctly. After quantization, the frame is regenerated from quantized error coefficients and motion vectors. FIG. 7 illustrates an error computation of real and motion vector calculated block locations. In accordance with this figure, then the signal to noise ratio (SNR) is computed with the original image and the regenerated frame. If SNR is below the threshold, motion estimation is not successful; the frame will be coded independently.

[0037] A system and method in accordance with the present invention uses fixed block sizes and fixed refinement of motion vectors, which inevitably reduces processing power requirements. Also in the scheme of video coding, quantization is very important especially in high-efficiency coding. Quantization affects the quality of the reconstructed frames in the feedback loop, which affects the motion estimation as well. The dynamic adjusted quantization scheme in accordance with the present invention also improves the quality and efficiency of coding. Previous art also shows that estimating motion in transform domain, can give us opportunity to make processing such as watermarking easily after coding of the video, without using inverse transforms. Since the motion estimation is within in transform domain, our scheme also has this capability.

[0038] Therefore, to utilize the advantages of the wavelet transform and reduce processing complexity, a system and method in accordance with the invention employs hierarchical motion estimation in the wavelet domain. The motion estimate of each scale is often used as a starting point for the higher scale. After the compensation, the energy of the diagonal quarter is used as a fall back decision mechanism. If the energy is above a threshold then the system automatically sends the original image instead of the difference image.

[0039] Although the present invention has been described in accordance with the embodiments shown, one of ordinary skill in the art will readily recognize that there could be variations to the embodiments and those variations would be within the spirit and scope of the present invention. Accordingly, many modifications may be made by one of ordinary skill in the art without departing from the spirit and scope of the appended claims.

Claims

1. A method of compressing consecutive frames of a video sequence, comprising the steps of: (a) transforming each of the consecutive video frames into wavelet domain, where each frame becomes a collection of subunits of several resolutions; (b) dividing each subunit into fixed sized blocks, wherein the block size is smaller than the dimensions of the smallest subunit; (c) performing a motion estimation between corresponding subunits of a current and a next transformed frames of the video sequence; (d) applying motion compensated differencing to the corresponding subunits; (e) quantizing the fixed sized blocks within the corresponding subunits; (f) transmitting the difference of the original blocks of a subunit based on a decision step; (g) providing the motion information based on the difference transmitting step (f); and (h) reconstructing the original frames of the video sequence in a receiver.

2. The method of claim 1 wherein the motion estimation step (c) includes the steps of: (c1) finding the motion vectors for the blocks in the lowest resolution subunit; and (c2) refining the motion vectors by any fixed amount in the higher levels.

3. The method of claim 1 wherein the decision step includes the steps of: (1) thresholding the difference between the motion compensated and wavelet transformed original blocks of a subunit on each fixed size block; (2) if threshold is not exceeded, requesting a local repeat and not sending the data; (3) if threshold is exceeded, coding further and transmitting the data.

4. The method of claim 1 wherein the difference transmitting step (f) includes the steps of: (f1) applying differential coding to the motion vectors; and (f2) applying lossless coding to the overall data to be sent.

5. The method of claim 1 wherein the quantizing step (e) includes the steps of: (e1) generating minimum and maximum quantization tables for each frame type; inter or intra; and (e2) averaging these tables according to the bit rate, distance between a previous intra-frame and signal to noise ratio (SNR).

6. A method for compressing successive frames of a video sequence comprising the steps of: (a) transforming a current and a consecutive frame; (b) providing each of the frames as a collection of subunits; (c) dividing each of the collection of subunits into a fixed size block, wherein the size of the fixed size block is smaller than the dimensions of the smallest subunit; (d) determining a difference in motion between corresponding subunits of the current and the next consecutive frame; (e) transmitting the difference based upon a decision step; and (f) providing the appropriate motion information based upon transmitting step (e).

7. The method of claim 6 wherein the difference determining step (d) comprises the steps of: (d1) estimating the motion between corresponding subunits and current and a next transforming frame; (d2) applying motion compensated differencing to the corresponding subunits; and (d3) quantizing the fixed sized blocks within the corresponding subunits.

8. The method of claim 1 which includes the step of reconstructing the original frames of the video sequence in a receiver.

9. A system for estimating the motion of consecutive frames of data comprising: a first wavelet transform for receiving a first frame; a second wavelet transform for receiving a second frame; a motion estimator for receiving the transformed first frame and the transformed second frame for calculating the motion vectors of the first and second transformed frames; a motion compensation for generating an image based upon the calculated motion vectors; a subtractor for receiving the original image from the second transformed frame and the image from the motion detector and calculating the difference between the images to provide a motion estimation error; a decision block for determining whether the motion estimation error is above a predetermined threshold; and a threshold logic for determining which of the transformed frames is ready for coding based upon the predetermined threshold.

10. The system of claim 9 wherein the threshold logic comprises first and second AND gates.

11. A coding system for compressing consecutive frames of data comprising: a motion estimator; wherein the motion estimator further comprises a first wavelet transform for receiving a first frame; a second wavelet transform for receiving a second frame; a motion estimator for receiving the transformed first frame and the transformed second frame for calculating the motion vectors of the first and second transformed frames; a motion compensation for generating an image based upon the calculated motion vectors; a subtractor for receiving the original image from the second transformed frame and the image from the motion detector and calculating the difference between the images to provide a motion estimation error; a decision block for determining whether the motion estimation error is above a predetermined threshold; and a threshold logic for determining which of the transformed frames is ready for coding based upon the predetermined threshold; a first coder coupled to the motion estimator; a second coder coupled to the first coder which provides a bitstream output; a decoder coupled to the second coder in feedback relationship; and a buffer coupled to the decoder and to the motion estimator.

12. The coding system of claim 11 wherein the threshold logic comprises first and second AND gates.

13. A computer readable means containing program instructions of compressing consecutive frames of a video sequence, comprising the steps of: (a) transforming each of the consecutive video frames into wavelet domain, where each frame becomes a collection of subunits of several resolutions; (b) dividing each subunit into fixed sized blocks, wherein the block size is smaller than the dimensions of the smallest subunit; (c) performing a motion estimation between corresponding subunits of a current and a next transformed frames of the video sequence; (d) applying motion compensated differencing to the corresponding subunits; (e) quantizing the fixed sized blocks within the corresponding subunits; (f) transmitting the difference of the original blocks of a subunit based on a decision step; (g) providing the motion information based on the difference transmitting step (f); and (h) reconstructing the original frames of the video sequence in a receiver.

14. The computer readable means of claim 13 wherein the motion estimation step (c) includes the steps of: (c1) finding the motion vectors for the blocks in the lowest resolution subunit; and (c2) refining the motion vectors by any fixed amount in the higher levels.

15. The computer readable means of claim 13 wherein the decision step includes the steps of: (1) thresholding the difference between the motion compensated and wavelet transformed original blocks of a subunit on each fixed size block; (2) if threshold is not exceeded, requesting a local repeat and not sending the data; (3) if threshold is exceeded, coding further and transmitting the data.

16. The computer readable means of claim 13 wherein the difference transmitting step (f) includes the steps of: (f1) applying differential coding to the motion vectors; and (f2) applying lossless coding to the overall data to be sent.

17. The computer readable means of claim 13 wherein the quantizing step (e) includes the steps of: (e1) generating minimum and maximum quantization tables for each frame type; inter or intra; and (e2) averaging these tables according to the bit rate, distance between a previous intra-frame and a signal to noise ratio (SNR).

18. A computer readable means for compressing successive frames of a video sequence comprising the steps of: (a) transforming a current and a consecutive frame; (b) providing each of the frames as a collection of subunits; (c) dividing each of the collection of subunits into a fixed size block, wherein the size of the fixed size block is smaller than the dimensions of the smallest subunit; (d) determining a difference in motion between corresponding subunits of the current and the next consecutive frame; (e) transmitting the difference based upon a decision step; and (f) providing the appropriate motion information based upon transmitting step (e).

19. The computer readable means of claim 18 wherein the difference determining step (d) comprises the steps of: (d1) estimating the motion between corresponding subunits a and current and a next transforming frame; (d2) applying motion compensated differencing to the corresponding subunits; and (d3) quantizing the fixed sized blocks within the corresponding subunits.

20. The computer readable means of claim 18 which includes the step of reconstructing the original frames of the video sequence in a receiver.