System and method for extracting spatially reduced image sequences in a motion compensated compressed format

FIELD OF THE INVENTION
This invention relates to the field of extracting information from a compressed video sequence. More specifically, the invention relates to the field of extracting spatially reduced image sequences from video information in an MPEG-2 compression format without the need for decompression.
BACKGROUND OF THE INVENTION
The MPEG-1 video standard defines three different types of frames (see FIG. 1A): Intra-coded frames (I frames), Predicted frames (P frames), and Bidirectionally predicted frames (B frames). P and B frames are also said to be inter-coded because they are coded based on prior anchor or reference frames. Details of the MPEG-1 standard can be found in ISO/IEC JTC1 CD 11172 Coding of moving pictures and associated audio for digital storage media up to 1.5 Mbits/s, 1992. This reference is incorporated by reference in its entirety. I frames are compressed using intraframe coding, i.e., they do not reference any other frames in the coded streams. Each I frame is divided into 8.times.8 blocks (typically 112) and a Discrete Cosine Transform (DCT) is taken. Of the 64 DCT coefficients, the DC, AC01 and AC10 are of particular interests. They are illustrated in FIG. 1B. Predicted frames are coded using a motion compensation information from past I frames or P frames. B frames are coded using motion compensation information from either past and/or future I or P frames. P and B frames can also contain intra-coded blocks, which are also coded using DCT as are the blocks in the I frames.
A MPEG-1 bit stream has a hierarchical representation with six layers: a sequence layer, a group of pictures layer, a picture layer, a slice layer, a macroblock layer, and a block layer. The sequence layer is the top coding layer. A group of pictures layer (GOP) is a set of pictures in contiguous display order. It contains at least one I frame. Each picture corresponds to a frame. Each picture is divided into slices. A slice is divided into macroblocks. A macroblock is composed of 6 blocks 112, four blocks for luminance, and two for chrominances. Further details of MPEG-1 standard can be found in ISO/IEC JTC1 CD 11172 Coding of moving pictures and associated audio for digital storage media up to 1.5 Mbits/s, 1992.
The details of a process of reduced image generation can be found in "On the extraction of DC sequences from MPEG compressed video", (by B. L. Yeo and B. Liu, International Conference on Image Processing, Vol. II, pp. 260-263, 1995). This reference is incorporated by reference in its entirety. A DC image is composed of block-wise averages of 8.times.8 blocks. For the I-frame of an MPEG-1 coded video, each pixel in the DC image corresponds to a scaled version of the DC coefficient of each DCT block.
A generic extraction of the DC images from P and B frames is shown in FIG. 1C. Here, P.sub.ref 131D is the target block of interest and is inter-coded. It is desired to convert it into an intra-coded DCT format so that the DC coefficient can be extracted. P.sub.0, . . . , P.sub.3 (134,135,136 and 137) are the four original intra-coded neighboring blocks (called anchor blocks) from which P.sub.ref is derived and the motion vector is (.DELTA.x, .DELTA.y). The motion vector (.DELTA.x, .DELTA.y) 150 is part of the motion compensation information associated with the block P.sub.ref. The shaded regions 134P, 135P, 136P and 137P in the anchor blocks P.sub.0, . . . , P.sub.3, called the contributing subblocks, are moved by (.DELTA.x, .DELTA.y). Thus, the DC coefficients of P.sub.ref are derived for DC image sequences.
The relation of inter-coded P.sub.ref with respect to P.sub.i is derived in "Manipulation and compositing of MC-DCT compressed video", (by S. F. Chang and D. G. Messerschmitt, IEEE Journal on Selected Areas in Communications: Special Issue on Intelligent Signal Processing, vol. 13, pp. 1-11, January 1995). This reference is incorporated by reference in its entirety. If we represent each block as an 8.times.8 matrix, then we can describe in the spatial domain through matrix multiplications: ##EQU1## where S.sub.ij are matrices like ##EQU2## Each I.sub.n is an identity matrix of size n. An example of such an effect of S.sub.ij on P is demonstrated in FIG. 1D. The goal is to move the subblock from the upper left corner to the lower right corner, and also to set the remaining values in the blocks to be zeros. The pre-multiplication shifts the sub-block of interest vertically while post-multiplication shifts the sub-block horizontally. There are four possible locations of the subblock of interest: upper-left, upper-right, lower-right and lower-left. The actions in terms of matrices are tabulated in Table 1.
While the value of S.sub.ij is clear from Table 1 with the given values of h.sub.i and w.sub.i, we will sometime write S.sub.ij as a function of h.sub.i and w.sub.i. For example, S.sub.01 =S.sub.01 (h.sub.0, w.sub.0)=S.sub.01 (h.sub.0).
Denoting the 2D DCT of an 8.times.8 block P as DCT(P), we can express the DC coefficient of DCT(P.sub.ref) as: ##EQU3## for some weighting coefficients w.sub.ml.sup.i.
A key result of "On the extraction of DC sequences from MPEG compressed video", (by B. L. Yeo and B. Liu, International Conference on Image Processing, Vol. II, pp. 260-263, 1995) is that ##EQU4##
TABLE 1______________________________________Matrices S.sub.i1 and S.sub.i2Subblock Position S.sub.i1 S.sub.i2______________________________________P.sub.0 lower right ##STR1##P.sub.1 lower left ##STR2## ##STR3##P.sub.2 upper right ##STR4## ##STR5##P.sub.3 upper left ##STR6## ##STR7##______________________________________
i.e., the weight w.sub.00.sup.i is the ratio of overlaps of the block P.sub.ref with block P.sub.i. An approximation, called the first-order approximation, approximates (DCT(P.sub.ref)).sub.00 by ##EQU5##
This approximation requires only the motion vector information and the DC values in the reference frames. It is shown in "On the extraction of DC sequences from MPEG compressed video", (by B. L. Yeo and B. Liu, International Conference on Image Processing, Vol. II, pp.260-263, 1995) that such approximation, when applied to B and P frames, yields good results in practice.
The use of DC images extracted directly from MPEG-1 video has led to efficient algorithms for processing MPEG-1 video. Example enabling applications are documented in "Rapid scene analysis on compressed videos" (by B. L. Yeo and B. Liu, IEEE Transactions on Circuits and Systems For Video Technology, vol.5, pp.533-544, December 1995) and "Efficient Matching and Clustering of Video Shots" (by M. M. Yeung and B. Liu, International Conference on Image Processing, Vol. I, pp. 338-341, 1995). These references are incorporated by reference in their entirety.
For the I frame of a MPEG-1 video, we define a DC+2AC as follows: for each 8.times.8 DCT block, the DC and two lower order AC coefficients, AC.sub.0,1, and AC.sub.1,0 are retained, and a 2.times.2 inverse DCT is taken to obtain 4 pixels. For B and P frames, equation (2) is used for constructing the DC and two lower order AC coefficients of P.sub.ref from the DC and two order AC coefficients of DCT blocks P.sub.i 's. ##EQU6## for a+b.ltoreq.1. Here, h.sub.i and w.sub.i are the height and width of block P.sub.i (FIG. 1C) respectively. Further details can be found in "On Fast Microscopic Browsing of MPEG compressed video" (by B. L. Yeo, IBM T. J. Watson Research Center, Technical Report RC20841, May 1997). This reference is incorporated by reference in its entirety.
MPEG-2 video is intended for higher data rates and has uses for broadcasting with high-quality video, whereas MPEG-1 is intended for data rates on the order of 1.5 Mbit/s. MPEG-2 supports much broader range of applications and modes of operations. It maintains all of the MPEG-1 video syntax, and uses extensions for additional flexibility and functions. MPEG-2 also supports the coding of interlaced video and scalable data streams. MPEG-1 does not support interlaced video, only non-interlaced (progressive) video.
In interlaced video, each frame is comprised of two fields, a top field and a bottom field. In MPEG-2, each field could be coded as a separate picture (field picture) or as a complete frame (frame picture). Frame pictures and field pictures could be mixed in a MPEG-2 data stream. The DCT coefficients for field pictures are always organized as independent fields. However, in a frame picture, DCT can be done either on fields or frames on a macroblock basis. That is, a frame picture could have both frame-coded macroblocks and field-coded macroblocks.
FIGS. 1E, 1F and 1G show different formats of DCT coded macroblocks possible in an MPEG-2 video. FIG. 1E is a drawing showing a representation of prior art field DCT coding in a frame picture in the MPEG-2 video standard. FIG. 1F is a drawing showing a representation of prior art frame DCT coding in a frame picture in the MPEG-2 video standard. FIG. 1G is a drawing showing a representation of prior art a coded macroblock in a field picture in the MPEG-2 video standard.
Motion compensation (MC) in MPEG-2 is also more general than those in MPEG-1. There are two basic modes of motion compensations: field-based motion compensation and frame-based motion compensation. In field-based MC, two motion vectors are used to specify the translations of the top and bottom fields (the mode field-based MC and the two motion vectors constitute the motion compensation information for field-based MC). In frame-based MC, only one motion vector is used (the mode frame-based MC and the motion vector constitutes the motion compensation information for field-based MC). Frame-based MC is similar to what is done in MPEG-1. In field pictures, only field-based MC can be used, whereas in frame pictures, both frame-based and field-based MC are allowed.
The two cases are illustrated in FIG. 1H and FIG. 1I. FIG. 1H is a drawing showing prior art frame prediction in a frame picture or prediction in a field picture in the MPEG-2 video standard. FIG. 1I is a drawing showing prior art field prediction in a frame picture in the MPEG-2 video standard.
In summary, one difference between MPEG-2 and MPEG-1 is the support of coding of interlaced video through the use of DCT and/or motion compensation of the interlaced frames through either frame or field encoding.
To improve prediction efficiency, MPEG-2 further supports the 16.times.8 motion-compensation mode. This mode allows a macroblock to be treated as an upper 16.times.8 region and a lower 16.times.8 region. Each region is then independently motion-compensated. Another mode is the dual-prime motion prediction.
It averages the predictions from two adjacent fields of opposite parity when coding a given field or frame. Dual-primal motion prediction tends to reduce the noise in the data.
Details of MPEG-2 standards can be found in ISO/IEC JTC1 CD 13818 Generic Coding of moving pictures and associated audio, 1994, MPEG Video Compression Standard (by J. L. Mitchell, W. B. Pennebaker, C. E. Foog and D. J. Le Gall, Chapman and Hall, 1996) and Digital Video: An Introduction to MPEG-2 (by B. G. Haskell, A.Puri, and A. N. Netravali, Chapman and Hall, 1997). These references are incorporated by reference in their entirety.
Notations
In this disclosure, the following notational conventions are used. We represent an 8.times.8 (16.times.16) data block (macroblock) as an 8.times.8 matrix (16.times.16 matrix), or vice versa, when there is no confusion. We denote spatial domain blocks or macroblocks (or matrices) with capital letters, (e.g., D, D', D.sub.i, P, etc) and the corresponding DCT domain blocks (or matrices) by the same capital letters with hats (e.g., D.sub.i , P.sub.i , etc), that is, for a 8.times.8 block represented by the matrix D, we have
D=DCT(D)=TDT.sup.t, (3)
where T is the 8.times.8 DCT matrix with entries t(i,j) (i denotes the i th row and j denotes the j th column) given by ##EQU7##
We also explicitly use DCT to denote the DCT domain value of a block when clearer notation is required. For example, DCT(AB) denotes the 2D-DCT applied to the matrix product AB. We will also make constant use of the distributive property of 2D DCT:
DCT(AB)=DCT(A)DCT(B)=AB. (6)
Thus, we can write the 2D-DCT of ##EQU8## for describing the movement of subblock in the spatial domain as ##EQU9## in the DCT domain. The conversion from inter-coded block P.sub.ref to intra-coded DCT block DCT(P.sub.ref) directly in the DCT domain using only the DCT values of P.sub.i 's and the motion vector (.DELTA.x, .DELTA.y) 150 is called DCT domain inverse motion compensation. For simplicity of notation, we will describe the movement of subblocks in the spatial domain in this disclosure; the DCT domain values of the blocks can be similarly deduced from equation (6).
A non-interlaced 16.times.16 macroblock is denoted by a simple capital letter (e.g., D) whereas the corresponding interlaced macroblock is denoted by a primed capital letter(e.g., D'). Given a non-interlaced macroblock D, we denote the four component 8.times.8 blocks by D.sub.0, D.sub.1, D.sub.2, and D.sub.3, i.e., D= ##EQU10## Similarly for the corresponding interlaced macroblock D', its four component blocks are denoted by D'.sub.0, D'.sub.1, D'.sub.2, and D'.sub.3, i.e., D'= ##EQU11## Each component in a block (matrix) is referenced by two indices, i.e., A.sub.ij represents the component at row i and column j in a block (matrix) A. To represent a row or column vector of a matrix, we use a--, i.e., A.sub.i-- represents the ith row vector and A.sub.--i the ith column vector.
Consider a non-interaced 16.times.16 macroblock D= ##EQU12## where D.sub.i, i=0,1, 2, 3, is an 8.times.8 block. Similarly, let D'= ##EQU13## be the corresponding interlaced macroblock, i.e., D'.sub.0 and D'.sub.1 correspond to the top field of D, and D'.sub.2 and D'.sub.3 correspond to the bottom field of D. The relationship between D and D' can be described using a 16.times.16 permutation matrix P as
D=PD' (7)
where P= ##EQU14## and ##EQU15##
STATEMENT OF PROBLEMS WITH THE PRIOR ART
The prior art fails to process interlaced video frames in the DCT domain.
The prior art does not provide a convenient way to process inter-coded frames in MPEG-2 compressed video without decompression.
The prior art does not teach the way to construct an intra-frame DCT coded frame from an inter-frame motion compensated coded frame for an MPEG-2 compressed video (this process is called DCT domain inverse motion compensation).
The prior art fails to provide a way to extract spatially reduced images sequence from MPEG-2 compressed video sequences without the need for decompression.
The prior art does not teach the use of spatially reduced images extracted from interlaced video frames coded using motion compensated and DCT for video processing tasks.
The prior art prohibits analysis of large video databases with video coded using schemes which include coding of interlaced video using DCT and motion compensation as in MPEG-2.
The prior art prohibits indexing and retrieval in large video databases which includes MPEG-2 video.
The prior art fails to provide a random access to frames in MPEG-2 video.
OBJECTS OF THE INVENTION
An object of this invention is a system and method for constructing spatially reduced image sequences in motion compensated compressed format.
An object of this invention is a system and method for efficient construction of spatially reduced image sequences in motion compensated compressed format without full decompression.
An object of this invention is a system and method for indexing and/or searching video information.
An object of this invention is a system and method for efficient browsing of video information.
An object of this invention is a system and method for efficient browsing of video information which allows random access to individual frames in video sequence.
SUMMARY OF THE INVENTION
Given compressed video which supports the coding of interlaced frames through DCT and motion compensation, the present invention applies different construction methods based on select DCT domain coefficients and/or motion vector information to produce spatially reduced images without decompressing. The spatially reduced images can then be used for video browsing, authoring and processing purposes.

BRIEF DESCRIPTION OF THE DRAWINGS
The foregoing and other objects, aspects and advantages will be better understood from the following detailed description of preferred embodiments of the invention with reference to the drawings that are include the following:
FIG. 1A is a drawing showing three different prior art frame types in the MPEG video standard.
FIG. 1B is a drawing of a prior art 8.times.8 Discrete Cosine Transform (DCT) block in the MPEG video standard.
FIG. 1C is a drawing showing a prior art reference block, motion vectors, and original blocks in the MPEG-1 video standard.
FIG. 1D is a drawing showing a representation of prior art matrix multiplications used to move MPEG-1 video contributing subblocks.
FIG. 1E is a drawing showing a representation of prior art field DCT coding in a frame picture in the MPEG-2 video standard.
FIG. 1F is a drawing showing a representation of prior art frame DCT coding in a frame picture in the MPEG-2 video standard.
FIG. 1G is a drawing showing a representation of prior art a coded macroblock in a field picture in the MPEG-2 video standard.
FIG. 1H is a drawing showing prior art frame prediction in a frame picture or prediction in a field picture in the MPEG-2 video standard.
FIG. 1I is a drawing showing prior art field prediction in a frame picture in the MPEG-2 video standard.
FIG. 2 is a block diagram showing one preferred embodiment of a computer system executing the present invention.
FIG. 3 is a flow chart showing the steps of the constructor for a reduced image sequence from an MPEG-2 video.
FIG. 4 is a flow chart showing the steps of constructing reduced image sequences from I frames.
FIG. 5 is a flow chart showing the steps of constructing reduced image sequences from P and/or B frames.
FIG. 6 is a flow chart showing the steps of applying DCT domain inverse motion compensation.
FIG. 6A is a flow chart showing the steps for computing contribution of each anchor blocks in the prediction of a block.
FIG. 7 is a block diagram showing field prediction.
FIG. 8 is a block diagram showing field-based motion compensation and an anchor block that is frame coded.
FIG. 9A is a block diagram showing field-based motion compensation where the achor block is field coded and the contributing sub-block comes from an upper region.
FIG. 9B is a block diagram showing field-based motion compensation where the achor block is field coded and the contributing sub-block comes from an lower region.
FIG. 10 is block diagram showing the predication of a 16.times.16 macroblock.
FIG. 11 is a block diagram showing prediction of two horizontally neighboring regions.
FIG. 12 is a block diagram showing the brute force computation of DCT domain inverse motion compensation of two adjacent blocks with same motion vectors.
FIG. 13 is a block diagram showing a preferred method for fast DCT domain inverse motion compensation of two adjacent blocks with same motion vectors using shared information in the two blocks.
FIG. 14 is a flow chart for a preferred method for fast DCT domain inverse motion compensation using shared information across two adjacent blocks.
FIG. 15 is a labeling of contributing subblocks of FIG. 10 and notations for use in FIGS. 16 to 23.
FIGS. 16 and 17 are block diagrams showing a preferred method for fast DCT domain inverse motion compensation of four adjacent blocks with same motion vectors using shared information in the four blocks to construct the block Q.sup.M.
FIGS. 18 and 19 are block diagrams showing a preferred method for fast DCT domain inverse motion compensation of four adjacent blocks with same motion vectors using shared information in the four blocks to construct the block Q.sup.N.
FIGS. 20 and 21 are block diagrams showing a preferred method for fast DCT domain inverse motion compensation of four adjacent blocks with same motion vectors using shared information in the four blocks to construct the block Q.sup.T.
FIGS. 22 and 23 are block diagrams showing a preferred method for fast DCT domain inverse motion compensation of four adjacent blocks with same motion vectors using shared information in the four blocks to construct the block Q.sup.U.
FIG. 24 is a TV screen which displays reduced image sequences.

DETAILED DESCRIPTION OF THE INVENTION
FIG. 2 is a block diagram showing a computer system 200, e.g. an IBM RISC System/6000 (Trademark of the IBM corporation) on which a preferred embodiment of the present invention operates. Components in the computer system 200 are connected by and communicate over a system bus 212. A central Processing Unit (CPU) 210, Read Only Memory (ROM) 216, and Random Access Memory (RAM) 214, typically are connected to the system bus 212. The preferred embodiment includes one or more application programs 202. One type of application program 202 is a process 300 further described in FIG. 3. Additional memory, e.g., diskettes 219, disks 220, and CDROM 221, can be connected to the system bus 212 by appropriate memory adaptors 218. Input/output adaptors (222, 236) connect devices to the system bus 212 that interact with a user. For example, a keyboard 224 and/or mouse 226 input are connected to the system bus 212 through an appropriate user input adapter 222 as a display (e.g. a Graphical User Interface, GUI) 238 and/or printer 239 are connected to the system bus 212 through an appropriate User Output Adapter 236. For example, a User Output Adapter 236 used to control a GUI would be an IBM RISC System/6000 Color Graphics Display Adaptor 236. Multimedia functions can be added to the system 200 by using an appropriate Multimedia Adapter 234 like the ActionMedia II Display Adapter 234 (described in the IBM ActionMedia II Technical Reference). Adapters like this are used to input and output audio signals through audio input devices (e.g. a microphone ) 232 and audio output devices (e.g. one or more loud speakers) 228. Video input (e.g. through a camera) 230 and video output 238 is also enabled by adaptors like these. Speech recognition can be accomplished by using an IBM VoiceType Dictation Adapter 234. User interfaces (222, 236) for other functions, e.g., virtual reality systems, can be connected to the system bus 212. Appropriate connectors 275 can be used to connect the computer system 200 to a general network (e.g., the internet, wide area network (WAN), and/or local area network (LAN)) 280 on which communicate servers 290 and/or clients. Some of these computers 290 are capable of performing the same functions of system 200. Hardware for computer systems like this 200 is well known.
FIGS. 3 to 6 are flow charts describing various processes performed by the invention to extract spatially image sequences from MPEG-2 video. FIGS. 7 to 11 are block diagrams that illustrate changes to the video as a result of performing these processes. Given an MPEG-2 video, or more generally, any compressed video which supports the coding of interlaced frames through DCT and motion compensation as in MPEG-2, our invention extracts a spatially reduced image sequence from different frames with different encoding frame types (i.e., I, P and B) without the need for decompression. (In this disclosure, any compressed video which supports the coding of interlaced frames through DCT and motion compensation will be referred to as MPEG-2 video, without loss of generality.) Depending on the frame type and the specifics of the encoding modes, our invention applies different extraction methods based on select DCT domain coefficients and/or motion vector information. The spatially reduced images can then be used for video browsing, authoring and processing purposes.
FIG. 3 is a flow chart which describes the overall process 300 of constructing a reduced image sequence from MPEG-2 video. In Step 310, a MPEG-2 video sequence is taken as input. This video sequence has headers with descriptions of the type of frames contained in data packets associated with the header. This is well known. In Step 320, header information is examined using known techniques to determined if there are more frames to process. If not, the reduced image sequence generated so far is output 330 and the process is finished. If there are more frames to process, the next frame is taken as the current frame which is named as F in Step 340. Step 350 tests if the current frame F is I frame, P frame, or B frame. Information in the headers is provided to classify these frames this way. If it is I frame, current frame F is processed in Step 360. Otherwise, the current frame F is processed in Step 370. After the current frame F is processed in Block 360 or 370, the process goes to block 320 and continues.
FIG. 4 is a flow chart which shows details of Step 360 in FIG. 3 and describes the steps of processing I frames to construct reduced images. Step 361 determines if the current frame has more macroblocks to process. If not, as shown in Step 366, the process is returned to Step 320 in FIG. 3. If there are more macroblocks to process, the next macroblock is taken as the current macroblock (named M) in Step 362. Then, Step 363 determines if the current macroblock M is frame-DCT coded or field-DCT coded based on the header information in the MPEG-2 bitstream. If it is frame-DCT coded, it is given to Step 365. If it is field-DCT coded, a DCT domain deinterlacing operation is applied to M in Step 364 (the detailed preferred embodiment will be presented later in the disclosure) and successively given to Step 365. In Step 365, DC, AC10 and AC01 values are taken from the current macroblock M, and the process 360 returns to Step 361. The DC component represents the average intensity value of its respective block and the AC10 and AC01 values represent various lowest frequency components of intensity variation in the block. Note that other frequency coefficients, e.g., 111 can be used in alternative embodiments. However, in a preferred embodiment these two lowest frequency components are chosen because along with the DC component they give enough rapidly and efficiently create a reduced size block an adequate amount of information contained in the original block.
The purpose of process 360 (in a preferred embodiment) is to identify the DC, AC10, and AC01 components of a frame coded macroblock. This is straight forward if the macroblock is frame coded 365. However, if the macroblock is field coded 364, the macroblock needs to be converted into to a frame coded format in order to identify the required coefficients. A preferred embodiment describing how to do this is given below.
A more rigorous description of FIG. 4 is now disclosed. At step 362, suppose M is a field-coded macroblock consisting of 4, 8.times.8 DCT blocks DCT(D'.sub.i). To obtain the DC coefficients of the deinterlaced version of DCT(D.sub.i).sub.00 has to be computed.
One preferred method for computing the DC coefficients of DCT(D.sub.i)'s for the macroblock is based on a weighted linear combination of DCT coefficients in the field-coded blocks: ##EQU16## where, the weights used for the linear combinations is:
(P.sub.0 ).sub.0-- =(0.5,0.453,0,-0.159,0,0.106,0,-0.090). (10)
Only 5 multiplications are needed. In an alternative embodiment, only 3 multiplications are used if the last two non-zero coefficients are ignored. Ignoring these last two non-zero coefficients creates enough resolution in the reduced size block for some applications such as browsing, shot boundary detection, etc.
A preferred embodiment of computing (D.sub.2 ).sub.00 without using any multiplications uses the results of (D.sub.0 ).sub.00 :
(D.sub.2 ).sub.00 =((D'.sub.0 ).sub.00 +(D'.sub.2 ).sub.00)-(D.sub.0 ).sub.00 (11)
Likewise, the DC values for D.sub.1 and D.sub.3 can be computed by the following equations:
(D.sub.1 ).sub.00 =(P.sub.0 ).sub.0-- ((D'.sub.1 ).sub.--0 +(D'.sub.3 ).sub.--0) (12)
(D.sub.3 ).sub.00 =((D'.sub.1 ).sub.00 +(D'.sub.3 ).sub.00)-(D.sub.1 ).sub.00 (13)
Therefore, the construction of a DC value for a non-interlaced block requires at most 2.5 multiplications (5/2=2.5). If the last two non-zero coefficients of Equation (10) is also ignored, only 1.5 multiplications (3/2=1.5) are needed.
Again, at step 362, if M is a field-coded macroblock, it may be necessary to generate low order AC coefficients (e.g., AC01 and AC10) of a frame-coded macro-block from a corresponding field-coded macro-block. A preferred embodiment to compute the weighted linear combination performs step 364 as follows: ##EQU17##
FIG. 5 shows a flow chart which details the Step 370 in FIG. 3 and describes the steps of constructing reduced image sequences from P or B frames. Here again the DC, AC10,and AC01 coefficients are identified for frame coded macroblocks. However, for the motion compensated macroblocks in P and B frame format, the macroblock must be a converted to intracoded and frame coded macroblocks.
Step 371 determines if the current frame has more macroblocks to process. If not, the process continues at Step 320 in FIG. 3. If there are more macroblocks to process, the next macroblock is taken as the current macroblock (named as F) in Step 372. Step 373 tests if the current macroblock F is inter-coded. If not, step 375 tests if the current macroblock M is frame-DCT coded or field-DCT coded. If frame-DCT coded, M is directly processed by Step 377. If field-DCT coded, DCT domain deinterlacing as described by equations (9), (11), (12), (13) and (14)-(16) are applied to M in Step 376 and the deinterlaced macroblock is processed in Step 377. In Step 377, DC, AC01, and AC10 values are taken from the current macroblock M, and the process continues in Step 371. Note that blocks 375, 376, and 377 are identical to blocks 363, 364, and 365, and their descriptions, respectively. If the current macroblock is inter-coded in Block 373, DCT domain inverse motion compensation below is applied to M in Block 374. After step 375, the error term is computed and added to each DCT coefficients generated from step 375 in step 378 and process continues in Block 371. This computation of the error terms can be done in the same way that I frames were processed in FIG. 4.
FIG. 6 shows the block diagram which details 374 in FIG. 5 and describes a preferred embodiment for the steps of applying the DCT domain inverse motion compensation for converting the interceded macroblocks to intracoded frame coded macroblocks so that the DC, AC10, and AC01 coefficients can be identified.
In Step 640, the number of blocks in the current macroblock is set to N and each block is set to B.sub.1, B.sub.2, . . . , B.sub.N. In Step 641, an index variable i is set to 0. In Step 642, the index variable i is incremented by 1. Step 643, determines if the current value of the index variable i is less than equal to N. If not, the DCT domain inverse motion compensation is finished and process continues in Step 371 in FIG. 5. Otherwise, in Step 644, block B.sub.i is taken. We want to obtain the reduced image representation associated with B.sub.i. In a more general sense, we want to obtain the intra-frame DCT coded representation of B.sub.i, which is coded using inter-frame motion compensation. Based on the motion vector associated with block B.sub.i, the number of contributing anchor blocks K is determined. Call the contributing anchor blocks A.sub.j, where j=1, 2, . . . K. In Step 645 to 647, the index j is initialized and used to compute the contribution of anchor block A.sub.j to the DCT representation of B.sub.i. The contribution of the current anchor block A.sub.j is computed and denoted by C (step 648) and then added to C(B.sub.i) (step 649). The process continues iterates from Step 646. The resulting C(B.sub.i) then gives the DCT domain representation of block B.sub.i or the reduced image representation of block B.sub.i, which is equivalent to the DC and low order AC coefficients of DCT domain values of block B.sub.i.
Referring to FIGS. 1C and 1D it is noted that a current block 131 is created from portions 134P-137P of the respective anchor blocks 134-137 of the reference frame. Therefore, contribution of each of these portions 134P-137P to the current block 131 must be determined in order to create the current block 131. These contributions are weighted in a linear fashion to create the current block, the weights being determined by the modes of motion compensation (field or frame-based motion compensation), types of encoding of the anchor blocks (field or frame coded) and the motion vector(s). Note that while this is done in the prior art for MPEG-1, i.e., frame coded blocks and frame based motion compensation only, there is no apparatus or method disclosed or suggested in the prior art for determining these contributions for frame and/or field coded blocks in combination with frame and/or field based motion compensation, i.e., MPEG-2.
To determine the contribution of each portion 134P-137P of the respective blocks (134-137) to the current block 13 ID of the macroblock 132, the header of each macroblock 132 is used to determine if the macroblock 132 is generated by: 1. field based or 2. frame based motion compensation. Then it is determined if each of the respective blocks (134-137) is field or frame coded. Each of the following four cases are converted into an intracoded and frame coded DCT block:
a. the current block is generated from field based motion compensation and the contributing archor block (e.g. 134) is field coded.
b. the current block is generated from field based motion compensation and the contributing archor block (e.g. 134) is frame coded.
c. the current block is generated from frame based motion compensation and the contributing archor block (e.g. 134) is field coded.
d. the current block is generated from frame based motion compensation and the contributing archor block (e.g. 134) is frame coded.
For the details of how this is done see the description of FIG. 6A below which shows the details of step 648.
FIG. 6A is a flowchart showing the step used to compute the contribution of an anchor block, i.e., the details of step 648 of FIG. 6. In step 6A1, the anchor block is set to block A.sub.j and motion type is set to MT. In block 6A2, it is tested if the motion type MT is frame prediction or field prediction. If it is frame prediction, it is further tested if the block A.sub.j is frame coded or field coded. If it is frame coded, process goes to step 684 and if it is field coded, the process goes to 683. On the other hand, if motion type MT is field based prediction, it is tested in step 6A4, if the anchor block is frame coded or field coded. If frame coded, the process continues to 682 and if not to 681. The steps 681, 682, 683, and 684 handle the cases marked by a, b, c, and d in the above paragraph respectively.
One preferred method of computing the contribution of anchor blocks A.sub.j, j=1, 2, . . . , K to current block of interest B.sub.i is now described. We have the DCT domain values of anchor blocks A.sub.j and we want to compute the DCT domain values of B.sub.i. Furthermore, motion vector informations are used. We will describe the construction of the entire set of DCT coefficients of B.sub.i and also the DC coefficient of the frame coded version of B.sub.i (i.e., for generating DC images).
In the following description, we call the current block which is to be constructed Q and the anchor blocks by M.sub.i or N.sub.i, where 0.ltoreq.i.ltoreq.3 for the ease of notation. We first consider the steps 681 and 682 (corresponding cases a and b) in which the motion compensation is field based prediction.
(1) Field-based Prediction of Target Block: step 681 and 682, i.e., cases a and b
In a field-based MC, two motion vectors are used to predict the top and bottom field of a macroblock. Field-based MC in the spatial domain is illustrated in FIG. 7. The goal here is to construct the DCT domain representation of the 8.times.8 block Q from the DCT domain representation of anchor blocks M.sub.i, N.sub.i and motion vectors (w.sub.0.sup.t, h.sub.0.sup.t) and (w.sub.0.sup.b, h.sub.0.sup.b). Note that the anchor blocks can either be coded in interlaced or noninterlaced formats. We now describe details of the preferred embodiment.
In FIG. 7, the top field of the target block Q, 710 is predicted from four anchor 8.times.8 blocks M.sub.0, M.sub.1, M.sub.2, M.sub.3 and the bottom field 720 is predicted from another set of four 8.times.8 anchor blocks N.sub.0, N.sub.1, N.sub.2, N.sub.3. Without loss of generality, we assume the motion vector for generating the top and bottom fields of Q are (w.sub.0.sup.t, h.sub.0.sup.t) and (w.sub.0.sup.b, h.sub.0.sup.b) respectively. Note that in FIG. 7, the anchor blocks and the target block are non-interlaced spatial domain blocks. However, in DCT domain, these anchor blocks can belong to different anchor macroblocks, which can be coded in different format, i.e., either frame or field coded. The prediction of block Q 720 in spatial domain can be described by:
Q=.SIGMA..sub.i=0.sup.3 Q.sup.M.sbsp.i +.SIGMA..sub.i=0.sup.3 Q.sup.N.sbsp.i(17)
Q.sup.M.sbsp.i =P.sub.i.sup.t S.sub.i1.sup.t M.sub.i S.sub.i2.sup.t and Q.sup.N.sbsp.i =P.sub.i.sup.b S.sub.i1.sup.b N.sub.i S.sub.i2.sup.b.(18)
M.sub.i and N.sub.i represent the anchor blocks in either interlaced or non-interlaced format. For each anchor block M.sub.i or N.sub.i, Q.sup.M.sbsp.i or Q.sup.N.sbsp.i denotes the contribution of M.sub.i or N.sub.i to the prediction of the target Q and is computed by P.sub.i.sup.t S.sub.i1.sup.t M.sub.i S.sub.i2.sup.t or P.sub.i.sup.b S.sub.i1.sup.b N.sub.i S.sub.i2.sup.t, where P.sub.i.sup.t, P.sub.i.sup.b, S.sub.ij.sup.t, and S.sub.ij.sup.b, 0.ltoreq.i.ltoreq.3 and 1.ltoreq.j.ltoreq.2, are matrices which we will describe below. The target block Q is formed by the summation of the contribution from each anchor block.
The DCT domain representation of Q is given by ##EQU18##
To compute the contributions Q.sup.M.sbsp.i or Q.sup.N.sbsp.i, 0.ltoreq.i.ltoreq.3, three steps are involved. First, from each anchor block M.sub.i or N.sub.i, the sub-block which contributes to the prediction of Q, called contributing sub-block, hereinafter, is identified. Second, the identified contributing sub-block is shifted vertically and horizontally to the appropriate corner. This is done by multiplying the matrices S.sub.ij.sup.t and S.sub.ij.sup.b, 0.ltoreq.i.ltoreq.3 and 1.ltoreq.j.ltoreq.2. Third, each row of the shifted contributing sub-block is put to the correct row position corresponding to Q by multiplying the matrices P.sub.t.sup.i and P.sub.b.sup.i. The matrices S.sub.ij.sup.t, S.sub.ij.sup.b, P.sub.i.sup.t, and P.sub.i.sup.b are decided depending on the representation of M.sub.i and N.sub.i.
We now considers two separate cases for the anchor blocks: (i) anchor non-interlaced and (ii) anchor interlaced.
(i) Anchor Noninterlaced: step 681 i.e., for case a
When M.sub.i (N.sub.i) represents a non-interlaced block, we write M.sub.i =M.sub.i (N.sub.i =N.sub.i). In this case, we want to compute the contribution of M.sub.i, which is coded in non-interlaced formats in the DCT domain, to Q. The mode of motion compensation is however, still, field-based motion compensation.
We first shift the contributing sub-block of M.sub.i (N.sub.i) and construct the contributing block after appropriate field selection (in this case, the top (bottom) field following FIG. 7). FIG. 8 illustrates the procedure to construct Q.sup.M.sbsp.2.
The sub-block is simply identified from the horizontal and vertical displacement (w.sub.i.sup.t, h.sub.i.sup.t) or (w.sub.i.sup.b, h.sub.i.sup.b). The location of the sub-block within the anchor block and its horizontal and vertical sizes decide the amount and direction of shifting in the same way as in MPEG-1. Therefore, the matrices for shifting in DCT domain motion prediction for MPEG-1, denoted by S.sub.ij (from Table 1), are used for S.sub.ij.sup.t and S.sub.ij.sup.t, i.e.,
S.sub.ij.sup.t =S.sub.ij (h.sub.i.sup.t,w.sub.i.sup.t) (20)
S.sub.ij.sup.b =S.sub.ij (h.sub.i.sup.b,w.sub.i.sup.b) (21)
In FIG. 8, the contributing sub-block is a h.sub.i.sup.t .times.w.sub.i.sup.t block in the upper right corner of the anchor block M.sub.2 and is shifted to the lower left corner in the contributing block Q.sup.M.sbsp.2.
The field selection in the contributing block is different for the top field prediction versus the bottom field prediction. In the top field prediction, every odd row is selected. However, in the bottom field selection, the selected odd rows are mapped to the even rows, i.e., ##EQU19##
TABLE 2______________________________________Selection of S.sub.i1.sup.t, S.sub.i1.sup.b, S.sub.i2.sup.t, andS.sub.i2.sup.b for verticaland horizontal shifting when an anchor block is interlaced: h' =h.sub.i.sup.t or h' = h.sub.i.sup.b, w' = w.sub.i.sup.t or w' = w.sub.i.sup.b, and0.sub.4 a 4 .times. 4 null matrix. S.sub.i1.sup.t (S.sub.i1.sup.b) S.sub.i2.sup.tAnchor block S.sub.i1.sup.u S.sub.i1.sup.t (S.sub.i2.sup.b)______________________________________ ##STR8## ##STR9## ##STR10## ##STR11## ##STR12## ##STR13## ##STR14##______________________________________
(ii) Anchor Interlaced: step 682 i.e., handling case b
When M.sub.i (N.sub.i) represents an interlaced block, M.sub.i (N.sub.i)=D'.sub.3, in some interlaced macroblock D', D'= ##EQU20## In this case, we want to compute the contribution of M.sub.i, which is coded in interlaced formats in the DCT domain, to Q. The mode of motion compensation is still field-based motion compensation.
Given an interlaced 8.times.8 block, call the upper half of the block (i.e., the first four rows) the upper region and the lower half of the block (i.e., the last four rows) the lower region. Then, the contributing sub-block of each interlaced anchor block belongs to either the upper region or the lower region. Within each region, the contributing sub-block is displaced by (w.sub.i.sup.t, .left brkt-top.h.sub.i.sup.t /2.right brkt-top.) or (w.sub.i.sup.b, .left brkt-top.h.sub.i.sup.b /2.right brkt-top.) for M.sub.i and for N.sub.i, when i=0, 1 and displaced by (w.sub.i.sup.t, .left brkt-bot.h.sub.i /2.right brkt-bot.) or (w.sub.i.sup.b, .left brkt-bot.h.sub.i.sup.b /2.right brkt-bot.) for M.sub.i and for N.sub.i, when i=2, 3.
FIG. 9 illustrates a scenario of field-based MC and field-coded anchor block M.sub.2. In FIG. 9, the motion vector 150 is (h.sub.2.sup.t, w.sub.2.sup.t) and it is desired that the contributing sub-block in M.sub.2 contributes to the top field of Q. In FIG. 9(a), the contributing sub-block comes from the upper region while in 9(b), the contributing sub-block comes from the lower region.
The direction and amount of horizontal shifting is decided by w.sub.i.sup.t or w.sub.i.sup.b, in the same way as in the anchor non-interlaced case, i.e., S.sub.i2.sup.t =S.sub.i2 (w.sub.i.sup.t) and S.sub.i2.sup.b =S.sub.i2 (w.sub.i.sup.b). The vertical shifting, however, is different for each region because the blocks are coded in interlaced format. The choices of the matrices S.sub.i1.sup.t and S.sub.i1.sup.b for vertical shifting is shown in Table 2. When the contributing sub-block belongs to the upper region of an interlaced anchor block, S.sub.i1.sup.u is selected for both S.sub.i1.sup.t and S.sub.i1.sup.b whereas S.sub.i1.sup.l is selected when the sub-block belongs to lower region.
Thus shifted contributing sub-block in S.sub.i1.sup.t M.sub.i S.sub.i2.sup.t or S.sub.i1.sup.b M.sub.i S.sub.i2.sup.b is still interlaced. Therefore, each row of it needs to be mapped to the right row position to construct the non-interlaced contributing block, i.e., Q.sup.M.sbsp.i or Q.sup.N.sbsp.i. This can be done by using the two component matrices P.sub.0 and P.sub.1 of the permutation matrix P used for DCT domain de-interlacing. For the top field prediction, each row of the subblock is expanded to every other rows starting from the first row of the target block using P.sub.0, i.e., P.sub.i.sup.t =P.sub.0, 0.ltoreq.i.ltoreq.3. For the bottom field prediction, each row is expanded to every other rows starting from the second row of the target block using P.sub.1, P.sub.i.sup.b =P.sub.1, 0.ltoreq.i.ltoreq.3.
(2) Frame-Based Prediction of Target Block Q: steps 683 and 684 for cases c and d
In a frame prediction, a motion vector is used to predict both top and bottom field of a target block. The prediction of a spatial domain block can be generally described by: ##EQU21##
If an anchor block M.sub.i is non-interlaced, it contributes both to the top and bottom fields of the target block. The contribution can be written as:
Q.sup.M.sbsp.i =S.sub.i1 M.sub.i S.sub.i2,
where S.sub.ij, for i=0, 1, 2, 3 and j=1, 2, are the matrices for shifting contributing sub-blocks previously described. When the anchor block M.sub.i is interlaced, M.sub.i contributes either to the top field and the bottom field of the target block and there could be more than four anchor blocks. In this case, the contribution Q.sup.M.sbsp.i can be computed as in field based prediction (anchor interlaced case), i.e.,
Q.sup.M.sbsp.i P.sub.i.sup.t S.sub.i1.sup.t M.sub.i S.sub.i2.sup.2 or(24)
Q.sup.M.sbsp.i =P.sub.i.sup.b S.sub.i1.sup.b M.sub.i S.sub.i2.sup.b.(25)
Computing Select DCT Coefficients for Reduced Image Generation
To generate reduced images like the DC or DC+2AC images from inter-coded P and B frames, only select DCT coefficients of Q S.sub.ij.sup.t , S.sub.ij.sup.b , P.sub.i.sup.t , and P.sub.i.sup.b , 0.ltoreq.i.ltoreq.3 and 1.ltoreq.j.ltoreq.2, can be precomputed. The contribution of each anchor block to each component in Q can be written as follows a weighted linear combinations of the DCT coefficients of the contributing anchor blocks as follows: ##EQU22## where the weights
w.sub.kl (i,m,n)=DCT(P.sub.i S.sub.i1).sub.km (S.sub.i2 ).sub.nl(28)
depends on (a) the mode of motion compensation (field-based or frame-based), (b) the mode of encoding of the contributing anchor blocks (field or frame coded), and (c) the motion vectors. Then, ##EQU23##
From Equation (27), the prediction of a component DCT value of a contributing block requires 64 multiplications resulting in a maximum of 64.times.8 multiplications for the prediction of a target block, since there a maximum of 8 possible contributing anchor blocks.
In addition, we will need to construct the DC and select AC values of the target block, which could potentially be used as future anchor blocks. Unlike MPEG-1, blocks in B frame could be used as anchor block (field-based motion compensation for prediction of the other field). This means that we need to maintain the DC and select AC values of most predicted blocks in P and B frames.
Fast computation of the weighted linear combination of equation (27) can be achieved by using only select set of coefficients in equation (28). One preferred embodiment is to use restrict the computation of equation (27) to 0.ltoreq.m+n.ltoreq.1, i.e., only use the DC, AC01 and AC10 coefficients of the intra-coded anchor blocks. We call this the DC+2AC approximation. In the DC+2AC approximation, equation (27) becomes: ##EQU24## where w.sub.kl (i, m, n) is defined in equation (28). Of particular interest is the retention of only DCT(Q).sub.00, DCT(Q).sub.01 and DCT(Q).sub.10.
Fast Algorithms
The following describes one preferred embodiment of the above invention. This is claimed in U.S. patent application Ser. No. 08/873,490 entitled "SYSTEM AND METHOD FOR DCT DOMAIN INVERSE MOTION COMPENSATION USING SHARED INFORMATION" of Song and Yeo, which was filed on the same day as this application and is herein incorporated by reference.
We will now describe preferred algorithms for fast DCT domain inverse motion compensation and fast extraction of spatially reduced images based on shared information. More rigorously, we will describe preferred algorithms for fast evaluation of equations (18) and (31). The speedup is based on two classes of methods: (i) neighboring anchoring blocks with the same coding format, and (ii) the use of shared information across anchor blocks in the same macroblock.
A common theme in the speedup is to re-order the movement of contributing subblocks in the inverse motion compensation process such that permutation matrices are involved. The use of permutation matrices will allow maximal exploitation of neighboring anchor blocks with same coding format and use of shared information across anchor blocks in the same macroblock. We will first describe how permutation matrices allow speedup in the reconstruction of spatially reduced images, i.e., the speedup in the evaluation of equation (31) using neighboring anchoring blocks with the same coding formations and shared information across anchor blocks in the same macroblocks.
First, a preferred embodiment of approximate computation of DC, AC01 and AC10 coefficients of the form DCT(ABC), where A, B and C are 8.times.8 matrices, is as follows: ##EQU25## where v.sub.kl (m, n)=A.sub.km C.sub.nl, and 0.ltoreq.k+l.ltoreq.1. When either A and/or C is a permutation matrix P, we obtain matrix P, we obtain ##EQU26## where k+l.ltoreq.1 and m+n.ltoreq.1. Table 3, Table 4, and Table 5 show the values of 9 coefficients used in the approximation, i.e., v.sub.kl.sup.0 (m, n) (Table 3), v.sub.kl.sup.1 (m, n) (Table 4), and v.sub.kl.sup.2 (m, n) (Table 5), for each (k, l) and (m, n) pairs. Table 3 and Table 4 state that three values (DC, AC01, and AC10) of DCBA) or DCBP) can be computed with 5 multiplications, saving 4 multiplications over the general case, as four entries are zero. Similarly, Table 5 states that three values of DCBP') can be computed only with 2 multiplications as one entry is a 1 and 6 entries are 0s.
When vertically or horizontally neighboring anchor blocks have the same coding format, the equations for DCT domain inverse motion compensation can be rearranged so that they include permutation matrices and thereby the computational cost for the same computation can be reduced as suggested by Table 3, 4 and 5. To proceed, we note that the matrices for horizontal shifting S.sub.i2 have the following properties: ##EQU27##
TABLE 3______________________________________P.sub.km A.sub.nl - Pre-multiplicationby a permutation matrix: 5 multiplications(k, l).backslash.(m, n) (0, 0) (0,1 ) (1, 0)______________________________________(0, 0) 0(0, 1) 0(1, 0) 0 0______________________________________
TABLE 4______________________________________A.sub.km P.sub.nl - Post-multiplicationby a permutation matrix: 5 multiplications(k, l).backslash.(m, n) (0, 0) (0,1 ) (1, 0)______________________________________(0, 0) 0(0, 1) 0(1, 0) 0 0______________________________________
TABLE 5______________________________________P.sub.km P.sup.1.sub.nl - Pre- and Post-multiplicationby a permutation matrices: 2 multiplications(k, l).backslash.(m, n) (0, 0) (0,1 ) (1, 0)______________________________________(0, 0) 1 0 0(0, 1) 0 0(1, 0) 0 0______________________________________ and P.sup.0 is a 8.times.8 permutation matrix.
We now proceed to describe the preferred embodiment of rewriting equation (31) to take advantage of equations (36) and (37), and the speedup as suggested by Table 3, 4 and 5.
Neighboring Anchor Blocks with the Same Coding Format
There are two cases: (i) two neighboring anchor blocks with the same coding format and (ii) all four neighboring anchor blocks with the same coding format.
(i) Two Neighboring Anchor Blocks with the Same Coding Format
Suppose that two upper anchor blocks M.sub.0 and M.sub.1 belong to macroblocks coded in the same format. In this case, regardless of coding format or motion prediction mode,
S.sub.01 =S.sub.11, P.sub.0 =P.sub.1. (38)
Using these relationships, a preferred way of writing the prediction of Q.sup.M.sbsp.0 +Q.sup.M.sbsp.1 is:
Q.sup.M.sbsp.0 +Q.sup.M.sbsp.1 =P.sub.0 S.sub.01 M.sub.0 S.sub.02 +P.sub.1 S.sub.11 M.sub.1 S.sub.12 (39)
=P.sub.0 S.sub.01 (M.sub.0 -M.sub.1)S.sub.02 +P.sub.0 S.sub.01 M.sub.1 P.sup.0 (40)
In the second term of equation (40), there is a post-multiplication by a permutation matrix. While the DC+2AC approximation to the first term in equation (40) requires 9 multiplications, a preferred method of computing the DC+2AC approximation to the second term can be done with 5 multiplications, from Table 4. Compared to equation (39), three extra additions are needed to compute M.sub.0 -M.sub.1. However, as four terms are zero (from Table 4), four additions can also be saved, resulting in saving 4 multiplication and 1 addition operation. This preferred embodiment computes the contribution of Q.sup.M.sbsp.0 +Q.sup.M.sbsp.1 together in a single step versus computing the contribution in two separate steps (i.e., computing Q.sup.M.sbsp.0 and Q.sup.M.sbsp.1 independently). The gain comes from expressing Q.sup.M.sbsp.0 +Q.sup.M.sbsp.1 in terms of a permutation matrix and thus the reduction of number of multiplications as suggested by Table 4.
Likewise, when the two bottom anchor blocks M.sub.2 and M.sub.3 are coded in the same format, the equation for the prediction of Q.sup.M.sbsp.2 +Q.sup.M.sbsp.3 can be rewritten in a similar fashion saving the same amount of computation. In this case, the fact that S.sub.21 =S.sub.31 and P.sub.2 =P.sub.3 will be used.
When vertically neighboring anchor blocks are coded with the same formats, similar saving can be obtained. However, the reduction in the computation depends on the prediction mode and coding formats of the anchor blocks. For example, when the two anchor blocks M.sub.0 and M.sub.2 are coded in the same format, we have P.sub.0 =P.sub.2, and
Q.sup.M.sbsp.0 +Q.sup.M.sbsp.2 =P.sub.0 S.sub.01 M.sub.0 S.sub.02 +P.sub.2 S.sub.21 M.sub.2 S.sub.22
=P.sub.0 S.sub.01 (M.sub.0 -M.sub.2)S.sub.02 +P.sub.0 XM.sub.2 S.sub.02,(41)
where X=S.sub.01 +S.sub.21. Note in the case of frame-based MC and non-interlaced anchor blocks, P.sub.0 =I and X is a permutation matrix, resulting in the same saving as in the horizontally neighboring case. In general, if we have interlaced anchor blocks, X will not be a permutation matrix.
(ii) All Four Anchor Blocks with the Same Coding Format
When all four anchor blocks are coded in the same format, a preferred way of expressing Q is ##EQU28##
In equation (42), two terms include the permutation matrix P.sup.0. Therefore, the DC+2AC approximation of Q can be computed with 28 (=9+9+5+5) multiplications. It also requires 9 extra addition operations to compute R.sub.1 =M.sub.1 -M.sub.3, R.sub.2 =M.sub.2 -M.sub.3, and M.sub.0 -R.sub.1 -R.sub.2 ; 8 additions are saved by zero coefficients. It saves 8 multiplication operations and requires 1 more addition operation. In the case of frame-based MC and non-interlaced anchors, P.sub.0 X is also a permutation matrix. In this case, the number of multiplications is reduced to 21 (=9+5+5+2), saving 15 multiplication an also 5 addition operations. We also label the terms in equation (42) as Z.sub.1, Z.sub.2, Z.sub.3 and Z.sub.4. They will be referred to later in equation (47), equation (48) and equation (49), and also FIGS. 16 to 23 for fast method to perform the DCT domain inverse motion compensation of 4 blocks within the same macroblock.
Macroblock Based Prediction: Reusing Intermediate Computation
In many cases, a set of neighboring target blocks are predicted using the same motion vector. For example, four 8.times.8 luminance data blocks (or chrominance data blocks in CHROMA 444) within a macroblock share the same motion vector in both the field and frame prediction. In CHROMA 422, two vertically neighboring blocks of chrominance share the same motion vector. In such cases, the same anchor block can be shared across the multiple target blocks. This means that there are possibly shared information in the predictions of multiple target blocks. For the description that follows, we will describe the preferred embodiment for computing the DCT inverse motion compensation of a 16.times.16 macroblock consisting of four 8.times.8 blocks by taking into account shared information among the blocks and demonstrate the speedup obtained. Other similar cases can be deduced.
FIG. 10 shows the prediction of a 16.times.16 macroblock Q. Each of the four 8.times.8 target blocks Q.sup.M 1005, Q.sup.N 1010, Q.sup.T 1015, and Q.sup.U 1020 is predicted from four anchor blocks, M.sub.i, N.sub.i, T.sub.i, and U.sub.i, i=0, 1, 2, 3, respectively. For the prediction of each target block (1005, 1010, 1015, 1020), the DCT domain values of the four contributing anchor blocks need to be computed and added. However, note that the predictions of four target blocks are strongly correlated to each other and do not have to be computed independently. Consider the following two observations.
First, although there are totally 16 contributing anchor blocks, there are only 9 different anchor blocks and 5 of them are shared among multiple target blocks, since the target blocks belong to the same macroblock and are predicted using the same motion vector 1050. For example, the target blocks Q.sup.M 1005 and Q.sup.N 1010 share two anchor blocks, i.e., M.sub.i =N.sub.0 and M.sub.3 =N.sub.2. Similarly, Q.sup.M 1005 and Q.sup.T 1015 share M.sub.2 =T.sub.0 and M.sub.3 =T.sub.1, etc. Second, the vertical and horizontal displacements of each contributing sub-block within the anchor block are identical across the four target blocks, Q.sup.M 1005, Q.sup.N 1010, Q.sup.T 1015, and Q.sup.U 1020. That is, w.sup.M.sbsp.i =w.sup.N.sbsp.i =w.sup.T.sbsp.i =w.sup.U.sbsp.i and h.sup.M.sbsp.i =h.sup.N.sbsp.i h.sup.T.sbsp.i =h.sup.U.sbsp.i for 0.ltoreq.i.ltoreq.3, where w's and h's superscribed by the anchor block (such as w.sup.M.sbsp.i h.sup.M.sbsp.i) denote the horizontal and vertical displacements caused by the motion vectors associated with each anchor block.
(i) Computation of Contribution of Two Neighboring Regions
We first consider the prediction of upper regions 1115 of two horizontally neighboring target blocks Q.sup.M 1005 and Q.sup.N 1010 as shown in FIGS. 10 and 11, when the respective anchor blocks have the same coding formats. From FIG. 12 onward, we will only illustrate the movement and additions of contributing sub-blocks assuming they are coded in non-interlaced formats, i.e., for each movement of contributing sub-blocks described generically by P.sub.0 YLZ, where Y is of the form S.sub.i1 or X, L is either an anchor block (e.g., M.sub.0) or a combination of anchor blocks (e.g., M.sub.0-M.sub.1), and Z is of the form S.sub.i2 or P.sup.0, we will ignore the effect of P.sub.0 in the figure for simplicity. The situation when the contributing sub-blocks are coded in interlaced formats is similar.
The brute-force approach (prior art) is shown in FIG. 12. Each of the subblocks A, B, C and D are treated separately--in the DCT domain, A and C are moved from the lower right corner of M.sub.0 and M.sub.1 respectively to the upper left corners of intermediate blocks 1220 and 1225, and B and D are moved from the lower left corners of M.sub.1 (=N.sub.0) and N.sub.1 respectively to the upper right corners of intermediate blocks 1230 and 1235. The remaining coefficients of the intermediate blocks are set to zeros (the white space in intermediate blocks 1220, 1225, 1230 and 1235). The S.sub.0,j matrices used for pre and post multiplication to move the blocks are also shown in FIG. 12. After the movements, the intermediate blocks containing A and B are added together, and the blocks containing C and D are added together to yield the upper regions (1105 and 1110) of the intra-coded target blocks 1005A and 1010A. In this method, 4 pairs of matrix pre and post multiplications are needed.
However, our invention shows a preferred embodiment for computing the contribution of the upper regions 1105 and 1110 of the target blocks takes into account the shared information (B and C) across the three contributing intra-coded anchor blocks M.sub.0, M.sub.1 (=N.sub.0) and N.sub.1. The sharing of information as processed by the invention is illustrated in FIG. 13 and the flowchart of this process is shown in FIG. 14. In the preferred embodiment, instead of moving each of the subblocks A, B, C and D separately, each pair of adjacent intra-coded anchor blocks (the pair M.sub.0, M.sub.1 and the pair M.sub.1, N.sub.1) are first subtracted 1302 from each other (1402) as specified in FIG. 14. In 1403, the lower right subblock A-C resulting from the block subtraction in 1402 is moved to the upper left corner, and in 1404, the lower left subblock D-B resulting from the block subtraction in 1402 is moved to the upper right corner. The matrices for this movement are shown in FIG. 13 and are the same as the one used in FIG. 11. A key difference (1404) is to simultaneously move the subblock B at the lower left corner and C at the lower right corner to the upper right and upper left respectively. Such a move can be described by one pre and post multiplication: S.sub.01 (h)M.sub.1 P.sup.0 (w), where P.sup.0 (w) is defined in equation (37). The results are again added (1306,1406) to yield the results: L.sub.1 (1305A) and L.sub.2 (1310A). In the preferred embodiment, only 3 pairs of matrix pre and post multiplications are needed versus using 4 pairs in the brute-force approach of FIG. 12.
Rather than separating the shared information (i.e., contributing sub-blocks B and C) of the intra-coded shared anchor blocks into four intermediate blocks, as in the prior art, the invention accounts for the shared information (subblocks B and C) in each of the "non-shared" intra-coded anchor blocks. Then the positions of the contributing sub-blocks (B and C) are reversed in the intra-coded shared anchor block so that the addition of the respective pair of intermediate blocks results (1305A and 1310A) in the same upper regions (1105 and 1110 respectively) of the intra-coded target blocks (e.g., 1305A=1005A and 1310A=1010A). However, this is accomplished with 25% fewer matrix multiplications that move the sub-blocks (A, B, C and D). Further, only three intermediate blocks are used.
A more rigorous explanation of the flow chart in FIG. 14 follows. Note that equation (40) defines steps 1402, 1404 and 1406. We can rewrite equation (40) as
Q.sup.M.sbsp.0 +Q.sup.M.sbsp.1 =P.sub.0 S.sub.01 (M.sub.1 -M.sub.0)S.sub.12 +P.sub.0 S.sub.01 M.sub.0 P.sup.0, (43)
and thus write the corresponding equation for the target block Q.sup.N as:
Q.sup.N.sbsp.0 +Q.sup.N.sbsp.1 =P.sub.0 S.sub.01 (N.sub.1 -N.sub.0)S.sub.12 +P.sub.0 S.sub.01 N.sub.0 P.sup.0, (44)
which defines steps 1402, 1404 and 1406. Note that the DCT domain value for the second term of equation (44), i.e., P.sub.0 S.sub.01 N.sub.0 P.sup.0 has already been computed as P.sub.0 S.sub.01 M.sub.1 P.sup.0 using equation (40). This is the shift 1404 and swap 1404 of the shared subblocks B and C. Only P.sub.0 S.sub.01 (N.sub.1 -N.sub.0 )S.sub.12 (pair subtraction and move) needs to be additionally computed to get Q.sup.N.sbsp.0 +Q.sup.N.sbsp.1 . Therefore, in the context of generating the DC+2AC approximation, 9 additional multiplications are needed. Since equation (43) needs 14 multiplications, a total of 23 (=14+9) multiplications are needed to construct the DC+2AC approximation of Q.sup.m.sbsp.0 +Q.sup.M.sbsp.1 and Q.sup.N.sbsp.0 +Q.sup.N.sbsp.1 .
Given this disclosure, the same novel steps can be applied to any three adjacent (i.e., horizontally, vertically and diagonally adjacent intra-coded anchor blocks). More rigorously, the prediction of Q.sup.M.sbsp.0 +Q.sup.M.sbsp.2 and Q.sup.T.sbsp.0 +Q.sup.T.sbsp.2 ,
Q.sup.M.sbsp.0 +Q.sup.M.sbsp.2 =P.sub.0 S.sub.01 M.sub.0 S.sub.02 +P.sub.0 S.sub.21 M.sub.2 S.sub.22 =P.sub.0 (S.sub.01 M.sub.0 +S.sub.21 M.sub.2)S.sub.02
=P.sub.0 (S.sub.01 M.sub.0 +(X-S.sub.01)M.sub.2)S.sub.02 =P.sub.0 S.sub.01 (M.sub.0 -M.sub.2)S.sub.02+P.sub.0 XM.sub.2 S.sub.02 (45)
Similarly
Q.sup.T.sbsp.0 +Q.sup.T.sbsp.2 =P.sub.0 S.sub.21 (T.sub.2 -T.sub.0)S.sub.02 +P.sub.0 XT.sub.0 S.sub.02. (46)
The second terms of the two equations are the same since M.sub.2 =T.sub.0. Note that this is true, whether or not X is a permutation matrix. In this case the total number of multiplications to predict two regions is 27 (=9+9+9). In the frame prediction and anchor non-interlaced case, it becomes 23 (=9+5+9).
ii) Prediction of Two Neighboring Blocks
Here we will describe the use of the invention to convert two adjacent inter-coded target blocks (e.g., 1005 and 1010) to two adjacent intra-coded target blocks (1005A and 1010A) using three or more (e.g., 6) anchor blocks (here, M.sub.0, M.sub.1, N.sub.1, M.sub.2, M.sub.3 and N.sub.3).
In FIG. 15, we first set up the notation for use in FIG. 16. It depicts the same setup of blocks as in FIG. 10 with contributing sub-blocks A-P as shown. Anchor blocks M.sub.0 and M.sub.1 are labeled and are combined 1510 as M.sub.0 -M.sub.1. The notation is that a full arrow drawn from M.sub.0 means that M.sub.0 is taken without any modification, whereas a dashed arrow drawn from M.sub.1 means that M.sub.1 is taken as -M.sub.1, i.e., the values in block M.sub.1 are negated. As before, the effect of matrix pre and post multiplication to shift the summed blocks M.sub.0 -M.sub.1 is described by the two matrices S.sub.01 and S.sub.02. FIG. 16 illustrates the effect of the anchor block combinations and movement as described in equation (42). The subblocks A, B, C, E, F, G, I, J and K contribute to the value of Q.sup.M. The effect of block combination (1605-1607) and movements (1615-1618) results in the intermediate blocks Z.sub.1, Z.sub.2, Z.sub.3 and Z.sub.4, as labeled in FIG. 16 and equation (42). The combination and movement of subblock is similar to that described in FIG. 13. More specifically, the contributing subblocks A, C, I and K from the respective adjacent anchor blocks (M.sub.0, M.sub.1, M.sub.2 and M.sub.3) are combined 1605 and moved 1615 so that the intermediate block Z.sub.1 contains shared information A-C-I+K that is shifted 1615 to the top left corner of intermediate block Z.sub.1. The contributing subblocks B, C, J and K from the respective adjacent anchor blocks (M.sub.0 and M.sub.3) are combined 1606 and moved 1616 so that the intermediate block Z.sub.2 contains shared information C-K that is shifted 1616 to top left corner and B-J shifted 1616 simultaneously to top right corner of Z.sub.2. Similarly, the contributing subblocks E, G, I and K from the respective adjacent anchor blocks (M.sub.2 and M.sub.3) are combined 1607 and moved 1617 so that the intermediate block Z.sub.3 contains shared information I-K that is shifted 1617 to top left corner and E-G shifted 1617 simultaneously to bottom left corner of Z.sub.3. Lastly, The contributing subblocks G, F, K and J from anchor block M.sub.3 are combined 1618 so that the intermediate block Z.sub.4 contains the subblock G, F, K and J but shifted from left to right, and bottom to top from M.sub.3 to Z.sub.4.
In FIG. 17, Z.sub.1 through Z.sub.4 are added together to yield Q.sup.M. In FIG. 18, the combination and shifting of blocks to yield the adjacent target block Q.sup.N is shown. Here, the combination and movement of blocks are similar to those in FIG. 16. Furthermore, Z.sub.2 and Z.sub.4 can be reused from FIG. 16 and the two blocks are further marked with a circle. This form of sharing is again analogous to the one shown in FIG. 12. In FIG. 19, the results are added to obtain Q.sup.N. From the combination and addition of subblocks in FIGS. 16 and 18 to yield the new intra-coded target blocks Q.sup.m 1005 and Q.sup.N 1010, only 6 pairs of pre and post matrix multiplications are needed as possible due to the use of shared information as opposed to 8 pairs of pre and post matrix multiplications if each of the contributing sub-blocks are treated separately.
A more rigorous explanation of FIGS. 16 through 19 is equation (42) and equation (47), which is shown below: ##EQU29##
The second (Z.sub.2) and the fourth terms (Z.sub.4) of equation (47) are the same as those of equation (42) and can be reused, since N.sub.0 =M.sub.l and N.sub.2 =M.sub.3. Therefore, Z.sub.1 =P.sub.0 S.sub.01 (N.sub.1 -N.sub.0 -N.sub.3 +N.sub.2 )S.sub.12 and Z.sub.3 =P.sub.0 X(N.sub.3 -N.sub.2 )S.sub.12 corresponding to the first and the third terms need to be additionally computed. The sharing of information is analogous to that shown in FIG. 13 except that instead of considering only 2 subblocks in each block, this case involves all 4 subblocks. The total number of multiplications to construct the DC+2AC approximation of Q.sup.M and Q.sup.N becomes 46 (=9+5+9+5+9+9). For the frame prediction, it becomes 35 (=9+5+5+2+9+5).
(iii) Prediction of the Whole Macroblock
When all the anchor blocks are of the same formats for the four target blocks in a macroblock, we can apply similar ideas to save the computation. Assuming the intra-coded DCT representation of Q.sup.M 1005 and Q.sup.N 1010 have already been computed based on sharing as described by equation (42) and equation (47) and FIGS. 16 to 19, we can further use the shared information to compute Q.sup.T 1015 and Q.sup.U 1020.
In FIG. 20, the combination and shifting of blocks to yield Q.sup.T is shown. Again, two of the four intermediate blocks can be reused (Z.sub.3 and Z.sub.4 from FIG. 16). In FIG. 22, the combination and shifting of blocks to yield Q.sup.U is shown. Here, three of the four intermediate blocks can be reused (Z.sub.8 from FIG. 20, Z.sub.6 from FIG. 18 and Z.sub.4 from FIG. 16). The net effect is that only 9 pairs of pre and post matrix multiplications are needed for this preferred embodiment compared to 16 if each of the blocks Q.sup.M, Q.sup.N, Q.sup.T and Q.sup.U are treated separately.
A more rigorous explanation of FIGS. 20 through 23 is as follows. We write Q.sup.T and Q.sup.U as: ##EQU30##
In DC+2AC approximation, we have shown that Q.sup.M and Q.sup.N can be computed with 46 multiplications (or 35 for frame prediction and anchor non-interlaced case). Since one of the two new terms of equation (48) includes a permutation matrix, the DC+2AC approximation of Q.sup.T can be computed with only 14 (=9+5) multiplications. For the DC+2AC approximation of Q.sup.U , we need only 9 additional multiplications for the first term. Therefore, for the DC+2AC approximation of the whole macroblock, 69 multiplications are used (58 multiplications in frame prediction and anchor non-interlaced), resulting in 17.25 (14.5) multiplications per block. The preferred embodiment for fast computation of equation (31) thus yields about 50% (or 60%) improvement over the number of the multiplications over the method which requires 36 multiplications.
Applications of Reduced Images
The set of reduced images extracted from compressed video using this invention can be further used for video browsing. An example browsing technique using reduced images can be found in "On Fast Microscopic Browsing of MPEG compressed video" (by B. L. Yeo, IBM T. J. Watson Research Center, Technical Report RC20841, May 1997). Since the size of each reduced image is small compared to the full frame and to the compressed video itself (for example, a DC image of an MPEG-2 video at 4.0 Mbits/sec and encoded at a spatial resolution of 708 by 480 is of the size 88 by 60 and takens up only about 0.2 Mbits/sec. The set of DC images or a subset of the DC images can be further transmitted without using as much bandwidth from a server to a client for quick browsing purposes. The DC images can be further compressed to reduce the bit rate.
At browsing time, the reduced images can be played back one after the other on a graphical display (e.g., 200 in FIG. 2). The display 238 could either be a television or a computer monitor 238. Because there is no need to perform the computationally expensive decompression tasks, the DC images can be played back faster-than-realtime, i.e., for a 30 frames/sec video, the set of DC images can be played back faster than 30 frames/sec. This functionality allows a user to quickly skim through the video content. Furthermore, random access to other part of the video can be offered. With compressed video, random access is much more difficult and time-consuming. The reduced images can also serve as indices into the respective video segment. The indices could be visual in the sense that a user, by looking at selected reduced images (for example, one for each shot), can easily navigate to the respective shots. The indices could also be used for image-based matching.
FIG. 24 shows a block diagram for a TV which displays reduced image sequences. The screen composes of display of main TV program (2405) along with one or more windows to display reduced image sequences (2402, 2403). In the settop box 2401, there could be a regular MPEG-2 decoder for decoding the full size display of the main TV program. Furthermore, the settop box 2401 could also contain one or more reduced image extractor (processes 300, 360 and 370) from MPEG-2 video. The reduced image extractors can then be used to display multiple views of the same TV program (e.g., different camera views of the same sporting event) or to simultaneously show different programs, possibly from different channels. The control buttons 2404 can be used (via a remote control for example) can be used to turn the display of the reduced images from the other programs on or off or to switch to different programs for display in the addition windows (e.g., 2402 and 2403). The reduced image extractor in the settop is of much lower cost compared to a full MPEG-2 coder, and thus adding a few of the extractor does not significantly increase the cost of the settop boxes, yet provides better TV viewing and channel-surfing experience for the viewers.
Given this disclosure alternative equivalent embodiments will become apparent to those skilled in the art. These embodiments are also within the contemplation of the inventors.

Number	Name	Date
5737019	Kim	Apr 1998
5832120	Prabhakar et al.	Nov 1998
5845015	Martucci	Dec 1998
5875039	Ohsawa et al.	Feb 1999
5933537	Hajjahmad et al.	Aug 1999

System and method for extracting spatially reduced image sequences in a motion compensated compressed format

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

US Classifications

Field of Search

US

International Classifications

Abstract

Description

Claims

US Referenced Citations (5)

Non-Patent Literature Citations (7)