APPARATUS FOR DIGITAL VIDEO FORMAT DOWN-CONVERSION WITH ARBITRARY CONVERSION RATIO AND METHOD THEREFOR

Abstract

The efficient motion compensation apparatus for digital video format down-conversion with variable conversion ratio is disclosed. The apparatus is characterized by an interpolation and decimation filters derived using a number of orthogonal transforms with variable transform sizes and implemented using efficient computation architectures. The computation architecture comprises the orthogonal transform kernel selection means, frequency component computing means, coefficient weighting means and pixel reconstruction means. A simple architecture for both interpolation and decimation filtering processes has been invented. The result is the dramatic reduction of the shifting and adding/subtracting operations, making them suitable for implementation in LSI realization of the video format down-conversion of digital video systems.

Description

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an apparatus for digital video format down-conversion with arbitrary conversion ration, and to a method therefor. The invention is applicable to the implementation of a digital video format down-conversion for use in digital video decoder. Typical applications of this invention include HDTV decoding, DVD decoder, video conferencing and picture-in-picture systems.

2. Description of the Related Art

Low-resolution digital video decoders have received considerably attention lately in academia and industry. In a digital video decoding system, the format down-conversion can be achieved by decimating the decoded full-resolution video sequences. Reconstructed video with good quality can be obtained by using this method. However, the decimation of decoded video sequences adds complexity to the full-resolution video decoding. In order to reduce the amount of computation, the memory size and other constrains such as memory bandwidth and clock rates incurred by this approach, image decimation has to be realized in the earlier stage of the decoder, for example, inside the decoding loop.

In European patent application EP0707426, a digital video decoder that provides format down-conversion with motion-compensation is disclosed. Motion compensation is achieved by first interpolating, then performing full-resolution motion compensation, and finally, decimation of the compensated output.

European patent application EP0786902A discusses a technique for changing image resolution using a direct discrete cosine transformation (DCT) mapping, whereby DCT coefficient values of an original resolution are mapped to converted coefficient values of a new resolution, without having to convert the original DCT coefficient values into pixels first.

An effective method for the digital video format down-conversion has been invented and filed in Japan on Jun. 8, 1999, entitled “A generalized orthogonal transform method for low-resolution video decoding” with application No. H11-160876, published as JP 2000-350207 and assigned to Matsushita Electric Industrial Co. Ltd. FIG. 1 shows a block diagram of this video format down-conversion method. The details of the system operation and the orthogonal kernels were discussed in the above-mentioned patent application. In this architecture, the low-resolution pixels stored in the frame buffer are interpolated and decimated using orthogonal transform basis functions before and after the full-resolution motion compensation. The interpolation and decimation filters play a very important role in controlling the error propagation introduced by picture decimation of the format down-conversion system of digital video. In the format down-conversion system of digital video shown in FIG. 1, these filters are realized using a number of orthogonal transform kernels. One example for the orthogonal transform kernels used for video down-conversion with the decimation ratio of 8:3 is illustrated in FIGS. 2A to 2G. The direct computation architecture of the interpolation and decimation filtering operations based on these kernels are shown in FIGS. 3A and 3B. Since the coefficients of the kernels are simpler the implementation of the system is relatively easy compared to the conventional digital video format down-conversion methods. Simulation results show that this method is also very effective in error propagation control.

The digital video format down-conversion method using orthogonal transform described in the prior art generates high quality down-converted video. The conversion ratio is however fixed in the methods described in the prior art. Due to the expansion and diversity of multimedia applications and present communication devices, especially the mobile terminals equipped with various resolution screens, there has been growing need for variable resolution digital video format down-conversion. The in-loop variable size video format down-decoding algorithms are required to efficiently decode high resolution encoded bitstreams and display the decoded down-sized pictures on various communication terminals with different resolutions. The problem to be solved by the current invention is to derive a set of interpolation and decimation filters using orthogonal transform with different transform sizes and establish efficient computation architectures for the interpolation and decimation filtering processes to achieve effective motion compensation for the digital video format down-conversion system with variable conversion ratio.

SUMMARY OF THE INVENTION

U.S. Pat. No. 4,768,159 discloses an efficient computation method for discrete Fourier transform. In order to solve the above-described problem, efficient computation architecture for implementing interpolation and decimation filters used by the digital video format down-conversion system is invented.

The original resolutions for encoded videos may differ from target resolution of video displayer with various ratios. Orthogonal kernels used for all the possible integer resolution ratios are invented. The orthogonal transform kernels are defined in the invention, and the selection of proper kernels for a particular resolution change is defined also. The computation architecture comprises three apparatus, namely frequency component computing means, coefficient weighting means and pixel reconstruction means. Less computational operations are required compared to the direct implementation of the orthogonal transform kernels described in the prior art.

The frequency component computing means is used to transform the input into frequency domain to generate the transform coefficients The coefficient weighting means is used for receiving transform coefficients and generating weighted transform coefficients. The weighted transform coefficients are finally transformed into spatial domain to generate the filtered pixels having different resolution from the original pixels. Said decimation/interpolation parameter generator is used to determine the integer resolution conversion ratio, 8:r, select the appropriate orthogonal kernels and generate and provide decimation/interpolation parameters to said frequency component computing means, coefficient weighting means and pixel reconstruction means.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects and features of the present invention will become clear from the following description taken in conjunction with the preferred embodiments thereof with reference to the accompanying drawings throughout which like parts are designates by like reference numerals, and in which:

FIG. 1 illustrates a block diagram for low-resolution video decoder described in the prior art.

FIG. 2A illustrates the kernels, K₁ and K₂, for low-resolution video decoding for down-conversion ratio of 8:7 to 8:2.

FIG. 2B illustrates the kernels, K₃ and K₄, for low-resolution video decoding with the down-conversion ratio of 8:7.

FIG. 2C illustrates the kernels, K₃ and K₄, for low-resolution video decoding with the down-conversion ratio of 8:6.

FIG. 2D illustrates the kernels, K₃ and K₄, for low-resolution video decoding with the down-conversion ratio of 8:5.

FIG. 2E illustrates the kernels, K₃ and K₄, for low-resolution video decoding with the down-conversion ratio of 8:4.

FIG. 2F illustrates the kernels, K₃ and K₄, for low-resolution video decoding with the down-conversion ratio of 8:3.

FIG. 2G illustrates the kernels, K₃ and K₄, for low-resolution video decoding with the down-conversion ratio of 8:2.

FIG. 3A illustrates the direct computation architecture of transform kernels for 8:3 digital video down-conversion with computation architecture for interpolation filtering.

FIG. 3B illustrates the direct computation architecture of transform kernels for 8.3 digital video down-conversion with computation architecture for decimation filtering.

FIG. 4 illustrates a block diagram of an efficient motion compensation apparatus for low-resolution digital video format down-conversion system.

FIG. 5 illustrates a block diagram for pixel interpolation and decimation filtering processes with various interpolation and decimation ratios 8:r, r=2, 3, . . . , 7.

FIG. 6 illustrates a block diagram of the frequency component computing means.

FIG. 7 illustrates a block diagram of the coefficient weighting means.

FIG. 8 illustrates a block diagram of the pixel reconstruction means.

FIG. 9 illustrates a block diagram for interpolation and decimation filtering processing using cascaded arithmetic units.

FIG. 10 illustrates a block diagram of the pre-processing means.

FIG. 11 illustrates a block diagram of cascaded arithmetic units.

FIG. 12 illustrates the transform kernel indicator (integer value r) generation.

FIG. 13A illustrates the computation architectures for interpolation filter used for digital video format down-conversion with the ratio of 8:3.

FIG. 13B illustrates the computation architectures for decimation filter used for digital video format down-conversion with the ratio of 8:3.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention is an apparatus for performing efficient motion compensation for digital video format down-conversion for motion compensation in digital video format down-conversion, which comprises:

a frequency component computing means having an input terminal for receiving a block of original pixels, transforming said original pixels into frequency domain and providing transform coefficients;

a coefficient weighting means for receiving said transform coefficient, multiplying each said transform coefficient by one of the pre-determined constant values to generate weighted transform coefficients;

a pixel reconstruction means having an input terminal for receiving said weighted transform coefficients and having an output terminal, for generating filtered pixels which have different resolution from said original pixels,

a decimation/interpolation parameter generator having a first input terminal for receiving original resolution (Ro), having a second input terminal for receiving target resolution (Rt) and having two output terminals, said decimation/interpolation parameter generator for deriving a transform kernel indicator (an integer value r), by identifying the integer value r from integer set {2, 3, 4, 5, 6, 7} such that the ratio 8:r is the most close to the resolution ratio Ro:Rt, and providing said transform kernel indicator (said integer value r) and decimation/interpolation parameters through its two output terminals;

transform kernels K1 and K2 generator having an input terminal for receiving said transform kernel indicator (said integer value r) and having two output terminals, said transform kernels K1 and K2 generator for generating orthogonal transform kernels K1[r], K2[r] from pre-determined transform kernels K1 and K2. by extracting the first r rows from K1 and first r columns from K2, respectively, characterized in that the transform kernels K1 and K2 are provided in accordance with a generalized orthogonal transformation having kernels defined as follows: $K_1=\left(\begin{array}{cccccccc}\alpha & \alpha & \alpha & \alpha & \alpha & \alpha & \alpha & \alpha \\ 5\beta & 4\beta & 3\beta & \beta & -\beta & -3\beta & -4\beta & -5\beta \\ 2\gamma & \gamma & -\gamma & -2\gamma & -2\gamma & -\gamma & \gamma & 2\gamma \\ 4\beta & -\beta & -5\beta & -3\beta & 3\beta & 5\beta & \beta & -4\beta \\ \alpha & -\alpha & -\alpha & \alpha & \alpha & -\alpha & -\alpha & \alpha \\ 3\beta & -5\beta & \beta & 4\beta & -4\beta & -\beta & 5\beta & -3\beta \\ \gamma & -2\gamma & 2\gamma & -\gamma & -\gamma & 2\gamma & -2\gamma & \gamma \end{array}\right)$$K_2=\left(\begin{array}{ccccccc}1& 5& 2& 4& 1& 3& 1\\ 1& 4& 1& -1& -1& -5& -2\\ 1& 3& -1& -5& -1& 1& 2\\ 1& 1& -2& -3& 1& 4& -1\\ 1& -1& -2& 3& 1& -4& -1\\ 1& -3& -1& 5& -1& -1& 2\\ 1& -4& 1& 1& -1& 5& -2\\ 1& -5& 2& -4& 1& -3& 1\end{array}\right)$

transform kernels K3 and K4 generator having an input terminal for receiving said transform kernel indicator (said integer value r) and having two output terminals, said transform kernels K3 and K4 generator for selecting orthogonal transform kernels K3[r] and K4[r] from a pool of pre-determined transform kernels K3 and K4 candidates, by choosing the transform kernels defined for resolution ratio 8:r from the pre-determined candidate kernels, characterized in that the transform kernels K3[r] and K4[r] candidates are provided in accordance with a generalized orthogonal transformation having kernels defined as follows: $r=7$$K_3\left[7\right]=\left(\begin{array}{ccccccc}{\sigma }_7& {\sigma }_7& {\sigma }_7& {\sigma }_7& {\sigma }_7& {\sigma }_7& {\sigma }_7\\ 3{\mu }_7& 2{\mu }_7& {\mu }_7& 0& -{\mu }_7& -2{\mu }_7& -3{\mu }_7\\ 3{\nu }_7& {\nu }_7& -2{\nu }_7& -4{\nu }_7& -2{\nu }_7& {\nu }_7& 3{\nu }_7\\ 2{\mu }_7& -{\mu }_7& -3{\mu }_7& 0& 3{\mu }_7& {\mu }_7& -{\mu }_7\\ 2{\nu }_7& -3{\nu }_7& -{\nu }_7& 4{\nu }_7& -{\nu }_7& -3{\nu }_7& 2{\nu }_7\\ {\mu }_7& -3{\mu }_7& 2{\mu }_7& 0& -2{\mu }_7& 3{\mu }_7& -{\mu }_7\\ {\nu }_7& -2{\nu }_7& 3{\nu }_7& -4{\nu }_7& 3{\nu }_7& -2{\nu }_7& {\nu }_7\end{array}\right)$$K_4\left[7\right]=\left(\begin{array}{ccccccc}1& 3& 3& 2& 2& 1& 1\\ 1& 2& 1& -1& -3& -3& -2\\ 1& 1& -2& -3& -1& 2& 3\\ 1& 0& -4& 0& 4& 0& -4\\ 1& -1& -& 3& -1& -2& 3\\ 1& -2& 1& 1& -3& 3& -2\\ 1& -3& 3& -1& 2& -1& 1\end{array}\right)$$r=6$$K_3\left[6\right]=\left(\begin{array}{cccccc}{\sigma }_6& {\sigma }_6& {\sigma }_6& {\sigma }_6& {\sigma }_6& {\sigma }_6\\ 4{\mu }_6& 3{\mu }_6& {\mu }_6& -{\mu }_6& -3{\mu }_6& -4{\mu }_6\\ {\nu }_6& 0& -{\nu }_6& -{\nu }_6& 0& {\nu }_6\\ {\sigma }_6& -{\sigma }_6& -{\sigma }_6& {\sigma }_6& {\sigma }_6& -{\sigma }_6\\ {\nu }_6& -2{\nu }_6& {\nu }_6& {\nu }_6& -2{\nu }_6& {\nu }_6\\ {\mu }_6& -3{\mu }_6& 4{\mu }_6& -4{\mu }_6& 3{\mu }_6& -{\mu }_6\end{array}\right)$$K_4\left[6\right]=\left(\begin{array}{cccccc}1& 4& 1& 1& 1& 1\\ 1& 3& 0& -1& -2& -3\\ 1& 1& -1& -1& 1& 4\\ 1& -1& -1& 1& 1& -4\\ 1& -3& 0& 1& -2& 3\\ 1& -4& 1& -1& 1& -1\end{array}\right)$$r=5$$K_3\left[5\right]=\left(\begin{array}{ccccc}{\sigma }_5& {\sigma }_5& {\sigma }_5& {\sigma }_5& {\sigma }_5\\ 2{\mu }_5& {\mu }_5& 0& -{\mu }_5& -2{\mu }_5\\ 3{\nu }_5& -{\nu }_5& -4{\nu }_5& -{\nu }_5& 3{\nu }_5\\ {\mu }_5& -2{\mu }_5& 0& 2{\mu }_5& -{\mu }_5\\ {\nu }_5& -3{\nu }_5& 4{\nu }_5& -3{\nu }_5& {\nu }_5\end{array}\right)$$K_4\left[5\right]=\left(\begin{array}{ccccc}1& 2& 3& 1& 1\\ 1& 1& -1& -2& -3\\ 1& 0& -4& 0& 4\\ 1& -1& -1& 2& -3\\ 1& -2& 3& -1& 1\end{array}\right)$$r=4$$K_3\left[4\right]=\left(\begin{array}{cccc}{\sigma }_4& {\sigma }_4& {\sigma }_4& {\sigma }_4\\ 2{\mu }_4& {\mu }_4& -{\mu }_4& -2{\mu }_4\\ {\sigma }_4& -{\sigma }_4& -{\sigma }_4& {\sigma }_4\\ {\mu }_4& -2{\mu }_4& 2{\mu }_4& -{\mu }_4\end{array}\right)$$K_4\left[4\right]=\left(\begin{array}{cccc}1& 2& 1& 1\\ 1& 1& -1& -2\\ 1& -1& -1& 2\\ 1& -2& 1& -1\end{array}\right)$$r=3$$K_3\left[3\right]=\left(\begin{array}{ccc}{\sigma }_3& {\sigma }_3& {\sigma }_3\\ {\mu }_3& 0& -{\mu }_3\\ {\nu }_3& -2{\nu }_3& {\nu }_3\end{array}\right)$$K_4\left[3\right]=\left(\begin{array}{ccc}1& 1& 1\\ 1& 0& -2\\ 1& -1& 1\end{array}\right)$$r=2$$K_3\left[2\right]=\left(\begin{array}{cc}{\sigma }_2& {\sigma }_2\\ {\mu }_2& -{\mu }_2\end{array}\right)$$K_4\left[2\right]=\left(\begin{array}{cc}1& 1\\ 1& -1\end{array}\right)$

The operation of the apparatus of the invention is explained.

First the operation of the computation architecture for the interpolation and decimation filtering processes is explained. The decimation/interpolation parameter generator receives the original resolution of decoded video, Ro, and target resolution of video displayer, Rt. The integer resolution conversion ratio r (8:r) is derived by identifying the integer value r from integer set {2, 3, 4, 5, 6, 7} such that the ratio 8:r is the most close to the resolution ratio Ro:Rt. The orthogonal transform kernels (K1[r], K2[r], K3[r] and K4[r]) are retrieved from a pool of pre-determined orthogonal transform kernels. The decimation/interpolation parameters are then generated and provided to the frequency component computing means, the coefficient weighting means and pixel reconstruction means. The original pixels are transformed into frequency domain by said frequency component computing means to generate the transform coefficients. Said transform coefficients are multiplied by a set of pre-determined constants by said coefficient weighting means to generate the weighted transform coefficients. The weighted transform coefficients are transformed from frequency domain into spatial domain by said pixel reconstruction means to provide filtered pixels which have different resolution from said original pixels.

Next, the operations of the frequency component computing means are explained. A reversed sequence of a block of the original pixels is generated in upper or lower address reversed order. A pair of selected pixel sequences is selected from said pixel sequence, the reversed sequence, the transform coefficients and the bit-shifted coefficient sequence by a pixel selecting means. An operation indication sequence is generated by the pixel selecting means to indicate the adding or subtracting operation. The sum or difference of said pair of selected pixel sequences is computed based on said operation indication sequence to generate said transform coefficients. Each transform coefficient is shifted by one or more bits to generate said bit-shifted coefficient sequence.

The frequency component computing means can also be operated using another method described here. The data address reversing means provides a reversed data set of a block of said original pixels in upper or lower address reversed order. A data selecting means receives said original pixels and said reversed data set to provide an operation indication set and two selected data sets. The calculator computes sum or difference of each pair of said selected data to generate processed data. One or more cascaded arithmetic units receives said processed data, manipulates them algebraically to provide said transform coefficients.

The operations of the coefficient weighting means are explained. Each transform coefficient is multiplied by one of said pre-determined constant values stored in said coefficient memory. The output of said multiplying means or said transform coefficients are switched based on a coefficient bypass control signal to provide said weighted transform coefficients. Said coefficient bypass control signal is determined based on the transform kernels used for the format down-conversion system of digital video.

The operations of said pixel reconstruction means are explained. The weighted transform coefficients are shifted by one or more bits to generate said bit-shifted vector. A pair of selected coefficient vectors is selected from said coefficient vectors said bit-shifted vector, filtered pixels and reversed pixel vector by a coefficient selecting means. An operation indication vector is generated by said coefficient selecting means to indicate the adding or subtracting operation. The sum or difference of said pair of coefficient samples is computed based on said operation indication vector to generate said filtered pixels. The reversed pixel vector of a block of filtered coefficients is generated by an address reversing means in upper or lower address reversed order.

The pixel reconstruction means can also be realized using one or More cascaded arithmetic units. The operations of the arithmetic units used for said frequency component computation means and pixel reconstruction means are now explained. The shifter shifts the input data by one or more bits to generate bit-shifted data set. The data selector receives said input data and said bit-shifted data set to provide an operation indication set and two selected data sets. A calculator adds or subtracts two selected data sets based on said operation indication.

The input terminal of the frequency component computing means can be coupled to the output terminal of the frame buffer, and the output terminal of the pixel reconstruction means can provide the interpolated pixels to the motion compensation means.

The input terminal of the frequency component computing means can be coupled to the output terminal of the motion compensation means, and the output terminal of the pixel reconstruction means can provide the decimated pixels to the adding means.

An embodiment shown in FIG. 4 illustrates the block diagram of an efficient motion compensation system for digital video format down-conversion. The system comprises an syntax parser and variable-length decoding means 210, an interpolation means 220, an inverse motion compensation means 230, a decimation means 240 and a frame buffer 250. The interpolation means 220 and the decimation means 240 are used before and after the inverse motion compensation means 230.

The video bitstream 201 is first decoded by the syntax parser and variable-length decoding means 210 to obtain the decoded motion parameters 211. The frame buffer 250 stores low-resolution video pictures. The low-resolution reference pixels 251 are retrieved from the frame buffer 250 by the interpolation means 220 and interpolated to generate the interpolated pixels 221 for inverse motion compensation means 230. The inverse motion compensation means 230 performs motion compensation based on the interpolated pixels 221 and the decoded motion parameters 211 to obtain the motion-compensated pixels 231. The motion-compensated pixels 231 are then decimated by the decimation means 240 to generate decimated pixels 241.

The effect of this embodiment is that the accuracy of inverse motion compensation for down-converted video can be improved by introducing the interpolation and decimation means. Since the format down-conversion processing of each video frame introduces error, it is extremely important to control the propagation of decoding errors. The properly designed interpolation and decimation means are efficient error control engines for minimizing the error of each decoded frame.

Another embodiment shown in FIG. 5 explains the method used in the interpolation and decimation means illustrated in FIG. 4. It comprises six components, namely, frequency component computing means 300, coefficient weighting means 310, pixel reconstruction means 320, decimation/interpolation parameter generator 330, transform kernel K1 and K2340, transform kernels K3 and K4 candidates 350.

The operation of this embodiment is now explained. The operation of said computation architecture for the interpolation and decimation filtering processes is now explained. The decimation/interpolation parameter generator 330 receives the original resolution of decoded video 333, Ro, and target resolution of video displayer 334, Rt. The integer resolution conversion ratio 332 r (8:r) is derived by identifying the integer value r 332 from integer set {2, 3, 4, 5, 6, 7} such that the ratio 8:r is the most close to the resolution ratio Ro:Rt. The orthogonal transform kernels (K1[r], K2[r], K3[r] and K4[r]) are retrieved from pre-determined transform kernels K1 and K2340 and a pool of pre-determined transform kernels K3 and K4 candidates 350. K1[r] and K2[r] are derived from transform kernels K1 and K2, defined in FIG. 2A, by extracting the first r rows from K1 and first r columns from K2. The K3[r] and K4[r] are generated by choosing the transform kernels defined for resolution ratio 8:r from the candidate kernels defined in FIG. 2B through FIG. 2G. The decimation/interpolation parameters 331 are then generated and provided to the frequency component computing means 300, the coefficient weighting means 310 and pixel reconstruction means 320. The original pixels 301 retrieved from the frame buffer 250 are transformed into transform coefficients 302 by frequency component computing means 300. The transform coefficients 302 are multiplied by the pre-determined values to generate weighted transform coefficients 311 using the coefficient weighting means 310. The weighted transform coefficients 311 are transformed, by the pixel reconstruction means 320, into spatial domain to generate the filtered pixels 321 having different resolution from the original pixels 301.

Another embodiment shown in FIG. 12 explains the generation of transform kernel indicator mentioned in the embodiment in FIG. 5. At first, the values r_cuur and r_past are set to be 7 and 8, respectively. The values rdiffcurr and rdiffpast are then computed by $\mathrm{rdiffcurr}=\frac{r_{\mathrm{curr}}}{8}-\frac{R_t}{R_o}$ and $\mathrm{rdiffpast}=\frac{r_{\mathrm{past}}}{8}-\frac{R_t}{R_o},$ respectively. If rdiffcurr is smaller than rdiffpast, r_past and r_cuur will be assigned to r_past=r_cuur and r_cuur=r_cuur−1, Otherwise, r_curr will be outputted as the transform kernel indicator. After assignment of r_past=r_cuur and r_cuur=r_cuur−1 are completed, the value of r_cuur is examined. If r_cuur is 2, the r_curr will be outputted as the transform kernel indicator, otherwise, the rdiffcurr and rdiffpast will be re-calculated by using updated r_cuur and r_past. The above process will be repeated until the transform kernel indicator (an integer value r) is obtained and outputted.

Another embodiment shown in FIG. 6 explains the realization of the frequency component computing means 300 illustrated in FIG. 5. This apparatus comprises an address reversing means 400, a pixel selecting means 410, an adder/subtracter 420 and a bit shifting means 430.

The operation of this embodiment is now explained. The reversed sequence 402 of a block of the original pixels 401 is generated in upper/lower address reversed order by the address reversing means 400. A pair of selected pixel sequences 412, 413 is selected from the original pixels 401, reversed sequence 402, transform coefficients 421 and bit-shifted coefficient sequence 431 by a pixel selecting means 410. An operation indication sequence 411 is also generated by the pixel selecting means 410 to indicate the adding or subtracting operation. The sum or difference of the pair of selected pixel sequences 412. 413 is computed based on the operation indication sequence 411 to generate the transform coefficients 421. Each transform coefficient 421 is shifted by one or more bits by the bit shifting means 430 to generate the bit-shifted coefficient sequence 431.

Another embodiment shown in FIG. 7 explains the details of the coefficient weighting means 310 shown in FIG. 5. This apparatus comprises a coefficient memory 500, a multiplying means 510 and a multiplexer 520.

The operation of this embodiment is now explained. Each transform coefficient 511 is multiplied by one of the pre-determined constant values stored in the coefficient memory 500. The output of multiplying means 510 and the transform coefficients 511 are multiplexed based on a coefficient bypass control signal 522 to provide the weighted transform coefficients 521. The coefficient bypass control signal is determined based on the transform kernels used for the format down-conversion system of digital video.

Another embodiment shown in FIG. 8 explains the details of the pixel reconstruction means 320 shown in FIG. 5. This apparatus comprises a bit shifting means 600, a coefficient selecting means 610 and an adder/subtracter 620.

The operation of this embodiment is now explained. The weighted transform coefficients 601 are shifted by one or more bits, by the bit shifting means 600 to generate the bit-shifted vector 602. A pair of selected coefficient vectors 612, 613 is selected from the weighted transform coefficients 601, bit-slifted vector 602 and filtered pixels 621 by the signal selecting means 610. An operation indication vector 611 is also generated by the coefficient selecting means 610 to indicate the adding or subtracting operation. The sum or difference of the selected coefficient vectors 612, 613 is computed based on the operation indication vector 611 to generate the filtered pixels 621

The immediate effect of the embodiments shown in FIG. 5 through FIG. 8 is that an image interpolation and decimation apparatus can be realized using efficient computation architecture derived according to the properties of generalized orthogonal transforms. Same apparatus can be used for both interpolation and decimation filtering processes derived based on orthogonal transforms. The intermediate computation results are fed back to a signal selecting means for further processing using same circuit. Thus, another effect of the embodiment shown in FIG. 5 through FIG. 8 is that it is possible to reduce the scale of the circuits required for format down-conversion system of digital video.

The embodiment shown in FIG. 9 explains another apparatus for implementation of the interpolation and decimation filtering processes. This apparatus comprises a pre-processing means 710, two sets of cascaded arithmetic units 720, 740 and coefficient weighting means 730.

The operation of this embodiment is now explained. The original pixels 701 are processed by the pre-processing means 710 to generate processed data 711. The processed data 711 is further processed by one set of cascaded arithmetic units 720 to generate the transform coefficients 721 which is the same as the transform coefficients 302 shown in FIG. 5. The coefficient weighting means 730 performs the same operation described in the embodiment shown in FIG. 5 on the transform coefficients 721 and provides the weighted transform coefficients 731 Another set of cascaded arithmetic units receives the weighted transform coefficients 731 and processes them to generate the filtered pixel 741

The embodiment shown in FIG. 10 explains the details of the pre-processing means used in the embodiment illustrated in FIG. 9. It comprises a data selector 810, a data address reversing means 820 and an adder/subtracter 830.

The operation of this embodiment is now explained. The reversed data set 821 of a block of original pixels 801 is generated in upper/lower address reversed order by the data address reversing means 820. The data selector 810 chooses a pair of data 812, 813, from the original pixels 801 and the reversed data set 821, and generates an operation indicator 811. The operation indicator 811 is a binary data with one value indicating adding operation and another value indicating subtracting operation. The adder/subtracter 830 computes the sum/difference of the selected pair of data 812, 813 based on the operation indicator 811 to generate the processed data 831.

Another embodiment shown in FIG. 11 explains the details of the cascaded arithmetic units. Arithmetic unit 1900 through arithmetic unit N 910, N≧1, are connected with each other in a cascaded way. The Nth arithmetic unit 910 comprises a shifter 920a data selector 930 and an adder/subtracter 940.

The operation of the nth (n≧1) arithmetic unit 910 is now explained. The input r_n−1, which is the output of the (n−1)th arithmetic unit (or the output of the pre-processing means 710 if n=1), is shifted by one or more bits by a shifter 920 to generate the bit-shifted data S_n. The data selector 930 chooses a pair of data (d_1n and d_2n), from r_n−1 and s_n, and an operation indicator (op_n) The operation indicator (op_n) is a binary data with one value indicating adding operation and another indicating subtracting operation. The adder/subtracter 940 computes the sum/difference of d_1n and d_2n based on the value of op_n to generate the output r_n of the nth arithmetic unit 910.

The effect of the embodiments shown in FIG. 9 through FIG. 11 is that it provides an alternative way to implement the interpolation and decimation filtering processing. Similar to the embodiments shown in FIG. 5 through FIG. 8, same architecture can be used for both interpolation and decimation filtering processing derived based on orthogonal transforms. However, there is no feedback loop in each embodiment. Thus, the latency introduced by the interpolation and decimation circuits can be minimized at the cost of more hardware requirements. A computation architecture, which is built based on the apparatus described in the embodiments shown in FIG. 9 through FIG. 11, for the purpose of video format down-conversion using the orthogonal transform kernels presented in FIGS. 2A and 2F of this patent specification are illustrated in FIGS. 13A and 13B, respectively,

This invention produces high-quality video format down-conversion solution. The computational requirement of the invention is much less intensive than that required for the conventional low-resolution video decoding methods or the direct implementation of the digital video format down-conversion method mentioned in the prior art. The apparatus designed for interpolation filter and decimation filter are of the same architecture. The number of shifting and adding operations required by the interpolation and decimation can be reduced by 46% and 21%, respectively, for the video format down-conversion at the down-conversion ratio of 8:3.

Claims

1. An apparatus for performing efficient motion compensation for digital video format down-conversion for motion compensation in digital video format down-conversion, comprising: a frequency component computing means having an input terminal for receiving a block of original pixels, transforming said original pixels into frequency domain and providing transform coefficients; a coefficient weighting means for receiving said transform coefficient, multiplying each said transform coefficient by one of the pre-determined constant values to generate weighted transform coefficients; a pixel reconstruction means having an input terminal for receiving said weighted transform coefficients and having an output terminal, for generating filtered pixels which have different resolution from said original pixels, a decimation/interpolation parameter generator having a first input terminal for receiving original resolution (Ro), having a second input terminal for receiving target resolution (Rt) and having two output terminals, said decimation/interpolation parameter generator for deriving a transform kernel indicator (an integer value r), by identifying the integer value r from integer set {2, 3, 4, 5, 6, 7} such that the ratio 8:r is the most close to the resolution ratio Ro:Rt, and providing said transform kernel indicator (said integer value r) and decimation/interpolation parameters through its two output terminals; transform kernels K1 and K2 generator having an input terminal for receiving said transform kernel indicator (said integer value r) and having two output terminals, said transform kernels K1 and K2 generator for generating orthogonal transform kernels K1[r], K2[r] from pre-determined ransform kernels K1 and K2, by extracting the first r rows from K1 and first r columns from K2, respectively, characterized in that the transform kernels K1 and K2 are provided in accordance with a generalized orthogonal transformation having kernels defined as follows: K1=(αααααααα5⁢ ⁢β4⁢ ⁢β3⁢ ⁢ββ-β-3⁢ ⁢β-4⁢ ⁢β-5⁢ ⁢β2⁢ ⁢γγ-γ-2⁢ ⁢γ-2⁢ ⁢γ-γγ2⁢ ⁢γ4⁢ ⁢β-β-5⁢ ⁢β-3⁢ ⁢β3⁢ ⁢β5⁢ ⁢ββ-4⁢ ⁢βα-α-ααα-α-αα3⁢ ⁢β-5⁢ ⁢ββ4⁢ ⁢β-4⁢ ⁢β-β5⁢ ⁢β-3⁢ ⁢βγ-2⁢ ⁢γ2⁢ ⁢γ-γ-γ2⁢ ⁢γ-2⁢ ⁢γγ)K2=(1524131141-1-1-5-213-1-5-11211-2-314-11-1-231-4-11-3-15-1-121-411-15-21-52-41-31)transform kernels K3 and K4 generator having an input terminal for receiving said transform kernel indicator (said integer value r) and having two output terminals, said transform kernels K3 and K4 generator for selecting orthogonal transform kernels K3[r] and K4[r] from a pool of pre-determined transform kernels K3 and K4 candidates) by choosing the transform kernels defined for resolution ratio 8:r from the pre-determined candidate kernel, characterized in that the transform kernels K3[r] and K4[r] candidates are provided in accordance with a generalized orthogonal transformation having kernels defined as follows: r=7K3⁡[7]=(σ7σ7σ7σ7σ7σ7σ73⁢μ72⁢μ7μ70-μ7-2⁢μ7-3⁢μ73⁢ν7ν7-2⁢ν7-4⁢ν7-2⁢ν7ν73⁢ν72⁢μ7-μ7-3⁢μ703⁢μ7μ7-μ72⁢ν7-3⁢ν7-ν74⁢ν7-ν7-3⁢ν72⁢ν7μ7-3⁢μ72⁢μ70-2⁢μ73⁢μ7-μ7ν7-2⁢ν73⁢ν7-4⁢ν73⁢ν7-2⁢ν7ν7)K4⁡[7]=(1332211121-1-3-3-211-2-3-12310-4040-41-1-3-1-231-211-33-21-33-12-11)r=6K3⁡[6]=(σ6σ6σ6σ6σ6σ64⁢μ63⁢μ6μ6-μ6-3⁢μ6-4⁢μ6ν60-ν6-ν60ν6σ6-σ6-σ6σ6σ6-σ6ν6-2⁢ν6ν6ν6-2⁢ν6ν6μ6-3⁢μ64⁢μ6-4⁢μ63⁢μ6-μ6)K4⁡[6]=(141111130-1-2-311-1-1141-1-111-41-301-231-41-11-1)r=5K3⁡[5]=(σ5σ5σ5σ5σ52⁢μ5μ50-μ5-2⁢μ53⁢ν5-ν5-4⁢ν5-ν53⁢ν5μ5-2⁢μ502⁢μ5-μ5ν5-3⁢ν54⁢ν5-3⁢ν5ν5)K4⁡[5]=(1231111-1-2-310-4041-1-12-31-23-11)r=4K3⁡[4]=(σ4σ4σ4σ42⁢μ4μ4-μ4-2⁢μ4σ4-σ4-σ4σ4μ4-2⁢μ42⁢μ4-μ4)K4⁡[4]=(121111-1-21-1-121-21-1)r=3K3⁡[3]=(σ3σ3σ3μ30-μ3ν3-2⁢ν3ν3)K4⁡[3]=(11110-21-11)r=2K3⁡[2]=(σ2σ2μ2-μ2)K4⁡[2]=(111-1)

2. The apparatus according to claim 1 wherein said orthogonal transform kernels K1[r], K2[r] are generated by extracting the first r rows from said K1 and first r columns from said K2.

3. The apparatus according to claim 1 wherein said input terminal of said frequency component computing means is coupled to said output terminal of said frame buffer, and said output terminal of said pixel reconstruction means provides said interpolated pixels to said motion compensation means.

4. The apparatus according to claim 1 wherein said input terminal of said frequency component computing means is coupled to an output terminal of an inverse motion compensation means, and said output terminal of said pixel reconstruction means provides said decimated pixels.

5. The apparatus according to claim 1 wherein said frequency component computing means further comprises: an address reversing means for providing reversed sequence of a block of said original pixels in upper address reversed order; a bit shifting means for shifting each of said transform coefficients by one or more bits to generate a bit-shifted coefficient sequence; a pixel selecting means for receiving said original pixels, said reversed sequence, said transform coefficients and bit-shifted coefficient sequence and providing an operation indication sequence, first selected pixel sequence, and second selected pixel sequence; and a calculator for receiving said operation indication sequence, said first selected pixel sequence, and second selected pixel sequence; and for computing at least one of sum and difference of each pair of pixel samples, one from said first selected pixel sequence and the other from said second selected pixel sequence, based on said operation indication sequence, to generate said transform coefficients.

6. The apparatus according to claim 1 wherein said frequency component computing means further comprises: an address reversing means for providing reversed sequence of a block of said original pixels in lower address reversed order; a bit shifting means for shifting each of said transform coefficients by one or more bits to generate a bit-shifted coefficient sequence; a pixel selecting means for receiving said original pixels, said reversed sequence, said transform coefficients and bit-shifted coefficient sequence and providing an operation indication sequence, first selected pixel sequence, and second selected pixel sequence; and a calculator for receiving said operation indication sequence, said first selected pixel sequence, and second selected pixel sequence; and for computing at least one of sum and difference of each pair of pixel samples, one from said first selected pixel sequence and the other from said second selected pixel sequence, based on said operation indication sequence, to generate said transform coefficients.

7. The apparatus according to claim 1 wherein said coefficient weighting means further comprises: a coefficient memory for storing pre-determined constant values; a multiplying means, having an input terminal for receiving said transform coefficients, multiplying one of said transform coefficients by one of said pre-determined constant values stored in said coefficient memory; and a multiplexer for choosing either the output of said multiplying means or said transform coefficients, based on a coefficient bypass control signal, to provide said weighted transform coefficients.

8. The apparatus according to claim 1 wherein said pixel reconstruction means further comprises: a bit shifting means for shifting each of said weighted transform coefficients by one or more bits to generate bit-shifted vector; a coefficient selecting means for receiving said weighted transform coefficients, said bit-shifted vector and said filtered pixels and providing an operation indicator vector and two selected coefficient vectors, first selected coefficient vector and second selected coefficient vector; and a calculator for receiving said operation indicator vector, said first selected coefficient vector, and said second selected coefficient vector; and for computing at least one of sum and difference of each pair of coefficient samples, one chosen from said first selected coefficient vector and the other from said second selected coefficient vector, based on said operation indication vector, to generate said filtered pixels.

9. The apparatus according to claim 1 wherein said frequency component computing means comprises: a pre-processing means for receiving said original pixels, manipulating them algebraically to provide processed data; and one or more cascaded arithmetic units having an input terminal and an output terminal.

10. The apparatus according to claim 9 wherein said input terminal of the first cascaded arithmetic unit is coupled to said pre-processing means.

11. The apparatus according to claim 9 where in said input terminal of the mth (m>1) cascaded arithmetic unit is coupled to the (m−1)th cascaded arithmetic unit.

12. The apparatus according to claim 9 wherein said output terminal of the last cascaded arithmetic unit provides said transform coefficients to said coefficient weighting means.

13. The apparatus according to claim 9 wherein said pre-processing means further comprises: a data address reversing means for providing reversed data set of a block of said original pixels in upper address reversed order; a data selecting means for receiving said original pixels and said reversed data set and providing an operation indication set, first selected data set and second selected data set; and a calculator for receiving said operation indication set, said first selected data set and said second selected data set; and for computing sum/difference of each pair of data, one from said first selected data set and the other from said second selected data set, based on said operation indication set to generate said processed data.

14. The apparatus according to claim 9 wherein said pre-processing means further comprises: a data address reversing means for providing reversed data set of a block of said original pixels in lower address reversed order; a data selecting means for receiving said original pixels and said reversed data set and providing an operation indication set, first selected data set and second selected data set; and a calculator for receiving said operation indication set, said first selected data set and said second selected data set; and for computing sum/difference of each pair of data, one from said first selected data set and the other from said second selected data set, based on said operation indication set to generate said processed data.

15. The apparatus according to claim 1 wherein said pixel reconstruction means further comprises one or more cascaded arithmetic units having an input terminal and an output terminal.

16. The apparatus according to claim 15 wherein said input terminal of the first cascaded arithmetic unit is coupled to said coefficient weighting means.

17. The apparatus according to claim 15 where in said input terminal of the mth (m>1) cascaded arithmetic unit is coupled to the (m−1)th cascaded arithmetic unit.

18. The apparatus according to claim 15 wherein said output terminal of the last cascaded arithmetic unit provides said filtered pixels.

19. The apparatus according to claim 9 wherein nth (n≧1) cascaded arithmetic unit comprises: a shifter for shifting the input data (rn−1) by one or more bits to generate bit-shifted data set (Sn); a data selector for receiving said input data (rn−1) and said bit-shifted data set (Sn) and providing an operation indication set (opn), first selected data set and second selected data set; and a calculator for receiving said operation indication set (opn), said first selected data set and said second selected data set; and for adding/subtracting two said selected data sets (d1n, d2n), one chosen from said first selected data set and the other from said second selected data set, based on said operation indication set (opn) and providing the output of said cascaded arithmetic unit (rn).

20. The apparatus according to claim 1 wherein said transform kernel indicator (said integer value r) is obtained by the following steps: setting rcuur=7 and rpast=8; computing rdiffcurr=rcurr8-RtRo and rdiffpast=rpast8-RtRo; comparing rdiffcurr with rdiffpast and outputting “yes” if rdiffcurr<rdiffpast and “no” otherwise; replacing rpast with rcuur and rcuur with (rcuur−1) if output of said step of comparing is “yes”; checking if the value of rcuur is 2 and outputting “yes” if rcuur=2 and outputting “no” otherwise; assigning said transform kernel indicator (said integer value r) to rcurr and outputting said transform kernel indicator; jumping to said step of assigning if output of said comparing is “no”; and jumping to said step of computing if output of said checking is “no”.

21. An apparatus for performing efficient motion compensation for digital video format down-conversion using generalized orthogonal transformation, comprising: a syntax parser and variable-length decoding means for decoding video bitstream, having an output terminal for providing decoded motion parameters; a frame buffer for storing reconstructed low-resolution pictures and having an output terminal for providing low-resolution reference pixels; an interpolation means for mapping said low-resolution reference pixels retrieved from said frame buffer into a high-resolution space and providing interpolated pixels for use in inverse motion compensation; an inverse motion compensation means for performing motion compensation and having a first input terminal for receiving said interpolated pixels, a second input terminal for receiving decoded motion parameters provided by said syntax parser and variable-length decoding means, and an output terminal for providing high-resolution motion compensated pixels; and a decimation means for mapping said high-resolution motion-compensated pixels into a low-resolution space and providing decimated pixels.

22. A method for performing efficient motion compensation for digital video format down-conversion for motion compensation in digital video format down-conversion, comprising: computing a frequency component and transforming, upon receiving a block of original pixels, said original pixels into frequency domain and providing transform coefficients; weighting a coefficient and multiplying, upon receiving said transform coefficient, each said transform coefficient by one of the pre-determined constant values to generate weighted transform coefficients; pixel reconstructing, upon receiving said weighted transform coefficients, and generating filtered pixels which have different resolution from said original pixels, generating a decimation/interpolation parameter, upon receiving an original resolution (Ro) and a receiving target resolution (Rt), and deriving a transform kernel indicator (an integer value r), by identifying the integer value r from integer set {2, 3, 4, 5, 6, 7} such that the ratio 8:r is the most close to the resolution ratio Ro:Rt, and providing said transform kernel indicator (said integer value r) and decimation/interpolation parameters; receiving said transform kernel indicator (said integer value r) and generating orthogonal transform kernels K1[r], K2[r] from pre-determined transform kernels K1 and K2, by extracting the first r rows from K1 and first r columns from K2, respectively, characterized in that the transform kernels K1 and K2 are provided in accordance with a generalized orthogonal transformation having kernels defined as follows: K1=(αααααααα5⁢ ⁢β4⁢ ⁢β3⁢ ⁢ββ-β-3⁢ ⁢β-4⁢ ⁢β-5⁢ ⁢β2⁢ ⁢γγ-γ-2⁢ ⁢γ-2⁢ ⁢γ-γγ2⁢ ⁢γ4⁢ ⁢β-β-5⁢ ⁢β-3⁢ ⁢β3⁢ ⁢β5⁢ ⁢ββ-4⁢ ⁢βα-α-ααα-α-αα3⁢ ⁢β-5⁢ ⁢ββ4⁢ ⁢β-4⁢ ⁢β-β5⁢ ⁢β-3⁢ ⁢βγ-2⁢ ⁢γ2⁢ ⁢γ-γ-γ2⁢ ⁢γ-2⁢ ⁢γγ)K2=(1524131141-1-1-5-213-1-5-11211-2-314-11-1-231-4-11-3-15-1-121-411-15-21-52-41-31)receiving said transform kernel indicator (said integer value r) and selecting orthogonal transform kernels K3[r] and K4[r] from a pool of pre-determined transform kernels K3 and K4 candidates, by choosing the transform kernels defined for resolution ratio 8:r from the pre-determined candidate kernels, characterized in that the transform kernels K3[r] and K4[r] candidates are provided in accordance with a generalized orthogonal transformation having kernels defined as follows: r=7K3⁡[7]=(σ7σ7σ7σ7σ7σ7σ73⁢μ72⁢μ7μ70-μ7-2⁢μ7-3⁢μ73⁢ν7ν7-2⁢ν7-4⁢ν7-2⁢ν7ν73⁢ν72⁢μ7-μ7-3⁢μ703⁢μ7μ7-μ72⁢ν7-3⁢ν7-ν74⁢ν7-ν7-3⁢ν72⁢ν7μ7-3⁢μ72⁢μ70-2⁢μ73⁢μ7-μ7ν7-2⁢ν73⁢ν7-4⁢ν73⁢ν7-2⁢ν7ν7)K4⁡[7]=(1332211121-1-3-3-211-2-3-12310-4040-41-1-3-1-231-211-33-21-33-12-11)r=6K3⁡[6]=(σ6σ6σ6σ6σ6σ64⁢μ63⁢μ6μ6-μ6-3⁢μ6-4⁢μ6ν60-ν6-ν60ν6σ6-σ6-σ6σ6σ6-σ6ν6-2⁢ν6ν6ν6-2⁢ν6ν6μ6-3⁢μ64⁢μ6-4⁢μ63⁢μ6-μ6)K4⁡[6]=(141111130-1-2-311-1-1141-1-111-41-301-231-41-11-1)r=5K3⁡[5]=(σ5σ5σ5σ5σ52⁢μ5μ50-μ5-2⁢μ53⁢ν5-ν5-4⁢ν5-ν53⁢ν5μ5-2⁢μ502⁢μ5-μ5ν5-3⁢ν54⁢ν5-3⁢ν5ν5)K4⁡[5]=(1231111-1-2-310-4041-1-12-31-23-11)r=4K3⁡[4]=(σ4σ4σ4σ42⁢μ4μ4-μ4-2⁢μ4σ4-σ4-σ4σ4μ4-2⁢μ42⁢μ4-μ4)K4⁡[4]=(121111-1-21-1-121-21-1)r=3K3⁡[3]=(σ3σ3σ3μ30-μ3ν3-2⁢ν3ν3)K4⁡[3]=(11110-21-11)r=2K3⁡[2]=(σ2σ2μ2-μ2)K4⁡[2]=(111-1)