In traditional video coding schemes, asymmetric complexity exists in the encoder and decoder operations. Motion estimation, which is generally very time-consuming due to correlation exploration operations, typically dominates operational complexity of the encoder. Correlation exploration operations include temporal, spatial, and statistical correlations. Conventional distributed video coding (DVC) systems use temporal correlation at the decoder, for example, by generating a side information frame from neighboring intra-coded frames. Spatial correlation within Wyner-Ziv frames is generally utilized by performing DCT or wavelet transforms. Some channel coding algorithms, for example, such as turbo codes for DVC use statistical correlations. However, these conventional systems do not utilize high-order statistical correlations among transform coefficients in DVC scenarios.
Wyner-Ziv and wavelet video coding is described. In one aspect, Wyner-Ziv frames from multiple frames of source video content are zero-tree entropy encoded to generate encoded Wyner-Ziv content. The zero-tree entropy encoding operations are based on high-order statistical correlations among wavelet transforms from the Wyner-Ziv frames. The encoded Wyner-Ziv content is communicated to a decoder for decoding to generate reconstructed Wyner-Ziv frames for presentation to a user.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
In the Figures, the left-most digit of a component reference number identifies the particular Figure in which the component first appears.
High-order correlations among transform coefficients indicate that distribution of a transform coefficient is correlated with some other transform coefficients. For example, a transform coefficient may have a large probability to be zero, when its neighboring transform coefficients are also zero. Since this probability is defined in the sense of statistics over a large amount of data, it is also denoted as high-order statistical correlations. High-order statistical correlations can be utilized in image and video coding by jointly encoding multiple transform coefficients. In a special case, each transform coefficient can be encoded individually. In this scenario, 0-order statistical correlations are utilized. Compared with the high-order statistical correlation, 0-order statistical correlations are considered low-order statistical correlations.
In hybrid video coding schemes such as MPEG-2, which is not a DVC scheme, run-length coding operations leverage high-order statistical correlations, wherein consecutive transform coefficients with zero value are jointly encoded as a single symbol. However, run-length coding is not typically used in DVC schemes. This is because run-length coding reorganizes transform coefficients and generates a number of new symbols that depend on distribution of the re-organized transform coefficients. In DVC, and prior to decoding, transform coefficients of a side information frame are reorganized to comply with symbols generated at the encoder. The decoding process pre-processes side information. This pre-processing uses information that can only be achieved after the decoding process. As a consequence, run-length coding and high-order statistical correlations among transform coefficients are not used in conventional DVC schemes. However, high-order statistical correlation plays a significant role in entropy coding of Wyner-Ziv frames. For example, utilization of high-order statistical correlation can lower the theory bound of entropy, which makes it possible to further lower encoding bit-rates.
In contrast to such conventional techniques, the systems and methods described below in reference to
These and other aspects of Wyner-Ziv and wavelet video coding are now described in greater detail.
An Exemplary System
Although not required, Wyner-Ziv and wavelet video coding is described in the general context of computer-executable instructions (program modules) being executed by computing devices such as a general-purpose computer or a mobile handheld device. Program modules generally include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. While Wyner-Ziv and wavelet video coding is described in the foregoing context, acts and operations described hereinafter may also be implemented in hardware.
For example, client 102 includes processor 108 coupled to system memory 110. System memory 110 includes program modules 112 and program data 114. In this implementation, program modules 112 include, for example, distributed video coding (DVC) encoding module 116 and other program modules 118 such as an operating system (OS) to provide a runtime environment, one or more encoded video transmission modules, a browser to search for media content, a decoder, a media player, and/or so on.
Encoding module (“encoder”) 116 encodes video media content 120 using two different encoding schemes to generate encoded media content 122. Video media content 120 (“media content”) is an input video sequence including multiple image frames classified using known techniques into two categories: Intra frames (intra frames) and Wyner-Ziv frames. Both intra frames and Wyner-Ziv frames are individually encoded. However, an intra-coded frame is also individually decoded at the decoder side, whereas a Wyner-Ziv coded frame is decoded with some side information frame generated from the neighboring decoded frames. Encoder 116 uses a first encoding algorithm to encode all intra-coded frames of media content 120 for communication and reconstruction (decoding) by decoder 128. In this implementation, the first encoding algorithm is a traditional discrete cosine transform (DCT) based intra coding method such as one based on H.264 encoding.
Encoder 116 uses a second encoding algorithm to encode the Wyner-Ziv frames of media content 120. Specifically, encoder 116 compresses each Wyner-Ziv (“WZ”) frame X using discrete wavelet transforms (DWTs) to exploit statistical correlations among transform coefficients, and therefrom, generate a significance map 124 that identifies significant ones of the coefficients. Encoder 116 entropy encodes the significance map and transmits the encoded map in intra-coding mode to decoder 128 for generating side information 130 to reconstruct (decode) X. (The encoded significance map 124 is represented as a respective portion of encoded media content 122). Encoder 116 also turbo codes significant coefficients 126 to generates punctured parity bits from significant coefficients 126. In this implementation, encoder 116 allocates appropriate bits of X for transmission of corresponding parity bits, using a standard request bits feedback channel with decoder 128. In addition to the significance map 124, encoder 116 also sends these parity bits to decoder 128 for reconstruction of X. Thus, the bit-stream of a coded Wyner-Ziv frame is composed of the bits for significance map Xm and the parity bits Xb from the Wyner-Ziv encoder.
In one implementation, Wyner-Ziv coder 210 implements a rate-compatible punctured turbo code (RCPT) to generate the punctured parity bits. Such an RCPT, for example, is described by D. Rowitch and L. Milstein, “On the performance of hybrid FEC/ARQ systems using rate compatible punctured turbo codes,” IEEE Transactions on Communications, vol. 48, no. 6, pp 948-959, June 2000.
Decoder 128, responsive to receiving encoded intra-coded frames from encoder 116, reconstructs the encoded intra-coded frames using a conventional decoding technique. In this implementation, these reconstructed frames are presented by media player 132 to a user via display device 134. For each encoded WZ frame received by decoder 128, decoder 128 reconstructs a corresponding WZ frame X as follows. In this implementation, decoder 128 generates side information 130 (“Y”) using frame interpolation by predicting X from reconstructed adjacent intra frames (i.e., previous and subsequent frames temporally adjacent in media content 120), although other side information generating techniques (e.g., use of a previously reconstructed frame, extrapolation, etc.) could also be used. In this implementation, it is assumed that most of the motions in three successive frames are linear and the current. Thus, motion vectors are derived from motion between adjacent two intra frames. In view of this assumption, and in one implementation, decoder 128 determines motion compensation for an X when X is absent. Decoder 128 decodes X's corresponding significance map 124 and applies DWTs to side information Y. In view of the significance map 124, decoder 128 extracts the identified significant coefficients of X from the side information to form coefficient set Ys. Decoder 128 turbo decodes Ys and the parity bits of X to decode X. In this implementation, these reconstructed WZ frames are presented by media player 132 to a user via display device 134.
An exemplary encoder 116 and decoder 128 architecture is now described.
Referring to
When encoder 116 scans a node (a quantized coefficient), the node is first put to the significant set X, (shown as significant coefficients 126). Then, encoder 116 inserts a “1” or “0” to the significance map 124 as follows. As shown in section 302, one node 308 corresponds to four children and sixteen grandchildren, etc. If all offspring of the current node are insignificant coefficients (i.e., zero after quantization), this will correspond to “0” in the significance map, and all of the current node's offspring coefficients are set to SKIP mode. SKIP mode indicates that a node is to be skipped (not subsequently transmitted) during the set partitioning process. Otherwise, if all offspring of the current node are significant coefficients (i.e., not zero after quantization), the current node's respective value in the significance map is equal to “one”.
Referring to
Responsive to receiving an encoded Wyner-Ziv frame and bits for significant map Xm, Wyner-Ziv decoder 212 decodes the Wyner-Ziv frame using side information frame Y, which is generated at prediction logic 220 of decoder 128. Side information frame Y is the prediction of X generated from adjacent intra frames. The received bits for significant map Xm are first decoded at logic 222, wherein a DWT with the same number of stages as that used in the encoder 116 (please see component 202) is applied on the side information frame Y. Using Xm, decoder 128 extracts the transform coefficients corresponding to the significant set Xs. The extracted transform coefficients form the coefficient set Ys. Thus, decoder 128 sends Ys to Wyner-Ziv decoder 212 to decode the Wyner-Ziv frame together with the received parity bits Xb.
Wyner-Ziv decoder 212 successively decodes the coefficients of a sub-band until an acceptable probability of bit error rate is achieved. In one implementation, decoder 128 uses two known soft-input soft-output (SISO) constituent decoders to decode coefficients of a sub-band. Each SISO decoder uses a priori probabilities for Xb and the probabilities calculated from side information Ys and the corresponding parity bits to calculate extrinsic probabilities and a posteriori probabilities for Xs. The iterative decoding is executed by passing the extrinsic probability results of one SISO decoder as the a priori probabilities of the other SISO decoder. Iterations between the two constituent decoders are performed until a satisfactory convergence (e.g., the bit error rate below 10−3) is reached.
Exemplary Procedure
The operations of procedure 400 for Wyner-Ziv and wavelet video coding use high-order statistical motion estimation correlations among transform coefficients in a DVC architecture to zero-tree entropy (ZTE) separately encode Wyner-Ziv frames. More particularly, the encoder codes Wyner-Ziv frames using turbo codes. The transform coefficients are quantized using scalar quantization and reorganized into wavelet trees to exploit statistical correlations. Then, significant coefficients are identified and coded for transmission of punctured parity bits to the decoder. The decoder extracts corresponding coefficients from side information, which is generated from surrounding intra frames with the motion-compensated prediction, according to a significance map. The decoder uses these extracted coefficients to reconstruct (decode) the Wyner-Ziv frame for presentation to a user.
More particularly, operations of block 402 code intra frames of input video media content with DCT-based intra-coding techniques independent of any motion compensation correlation operations between frames. Operations of block 404, independent of any motion compensation correlation operations between frames at the encoder 116, use high-order statistical correlations among wavelet transforms to zero-tree entropy code Wyner-Ziv frames and identify corresponding punctured parity bits. At block 406, encoder 116 generates a significance map from reordered and significant ones of quantized coefficients of the wavelet transforms used to encode the Wyner-Ziv frames. At block 408, encoder 116 (or a different computer-program module of computing device 102) transmits encoded intra-frames, entropy encodings of the significance maps, and punctured parity bits to decoder 128 for reconstruction and presentation of the encoded media content to a user.
At block 410, decoder 128 reconstructs the encoded intra-frames and extracts transform coefficients from respective ones of the significant maps to generates side information for decoding respective ones of the Wyner-Ziv frames. At block 412, decoder 128 decodes the Wyner-Ziv frames using respective ones of the extracted transform coefficients, received punctured parity bits, and motion compensation correlation information derived from adjacent intra-frames. Operations of block 414 present the decoded/reconstructed video media content to a user.
Although Wyner-Ziv and wavelet video coding has been described in language specific to structural features and/or methodological operations or actions, it is understood that the implementations defined in the appended claims are not necessarily limited to the specific features or actions described. Rather, the specific features and operations discussed above with respect to
Number | Name | Date | Kind |
---|---|---|---|
5144423 | Knauer et al. | Sep 1992 | A |
5412741 | Shapiro | May 1995 | A |
5719632 | Hoang et al. | Feb 1998 | A |
5818536 | Morris et al. | Oct 1998 | A |
5854658 | Uz et al. | Dec 1998 | A |
5862450 | Mandal et al. | Jan 1999 | A |
6018303 | Sadeh | Jan 2000 | A |
6122320 | Bellifemine et al. | Sep 2000 | A |
6157746 | Sodagar et al. | Dec 2000 | A |
6275532 | Hibi et al. | Aug 2001 | B1 |
6289049 | Kim et al. | Sep 2001 | B1 |
6498810 | Kim et al. | Dec 2002 | B1 |
6526097 | Sethuraman et al. | Feb 2003 | B1 |
6542549 | Tan et al. | Apr 2003 | B1 |
6628712 | Le Maguet | Sep 2003 | B1 |
6717990 | Abousleman | Apr 2004 | B1 |
6735249 | Karczewicz et al. | May 2004 | B1 |
6920175 | Karczewicz et al. | Jul 2005 | B2 |
6947486 | Akhan et al. | Sep 2005 | B2 |
7003035 | Tourapis et al. | Feb 2006 | B2 |
7031389 | Guevorkian et al. | Apr 2006 | B2 |
7233269 | Chen et al. | Jun 2007 | B1 |
7256716 | Liu et al. | Aug 2007 | B2 |
7418037 | Nie et al. | Aug 2008 | B1 |
20030123540 | Zhong et al. | Jul 2003 | A1 |
20030151753 | Li et al. | Aug 2003 | A1 |
20030156644 | Song et al. | Aug 2003 | A1 |
20040039571 | Bruekers et al. | Feb 2004 | A1 |
20040114684 | Karczewicz et al. | Jun 2004 | A1 |
20040172478 | Jacobs et al. | Sep 2004 | A1 |
20050013371 | Yim et al. | Jan 2005 | A1 |
20050024487 | Chen | Feb 2005 | A1 |
20050036550 | Koba et al. | Feb 2005 | A1 |
20050047508 | Ha et al. | Mar 2005 | A1 |
20050084014 | Wang et al. | Apr 2005 | A1 |
20050105815 | Zhang et al. | May 2005 | A1 |
20050201468 | Tsai et al. | Sep 2005 | A1 |
20050232360 | Byun | Oct 2005 | A1 |
20050268200 | Garudadri et al. | Dec 2005 | A1 |
20060008006 | Cha et al. | Jan 2006 | A1 |
20060062299 | Park et al. | Mar 2006 | A1 |
20060072662 | Tourapis et al. | Apr 2006 | A1 |
20060197686 | Liu et al. | Sep 2006 | A1 |
20060197690 | Liu et al. | Sep 2006 | A1 |
20060200733 | Stankovic et al. | Sep 2006 | A1 |
20070013561 | Xu et al. | Jan 2007 | A1 |
20070014356 | Han et al. | Jan 2007 | A1 |
20070041652 | Budge et al. | Feb 2007 | A1 |
20070133690 | Xin et al. | Jun 2007 | A1 |
20070165717 | Ye | Jul 2007 | A1 |
20070217541 | Liu et al. | Sep 2007 | A1 |
20080189073 | Jagmohan et al. | Aug 2008 | A1 |
20080219351 | Kim et al. | Sep 2008 | A1 |
Number | Date | Country |
---|---|---|
0971542 | Jan 2000 | EP |
WO2005043882 | May 2005 | WO |
Number | Date | Country | |
---|---|---|---|
20080031344 A1 | Feb 2008 | US |