The present invention relates to a motion estimation method for realizing motion compensation by performing image processing.
In H. 264 (also referred to as MPEG-4 Part 10, or AVC (advanced video coding)) as one of high efficiency coding systems, motion compensation is realized using seven search blocks of different block sizes and three interpolated images of different pixel accuracies.
According to a conventional full search method, search is performed using each of the search blocks of all types in the most densely interpolated image to obtain an optimal motion vector with regard to each search block.
Techniques relevant to the motion estimation using several kinds of search blocks of different block sizes are introduced for example in the following patent publications 1 to 5.
Patent Publication 1: Japanese Patent Application Laid-Open No. 2004-186897
Patent Publication 2: Japanese Patent Application Laid-Open No. 2004-48552
Patent Publication 3: United States Published Application No. 2004/0120440
Patent Publication 4: United States Published Application No. 2004/0190616
Patent Publication 5: United States Published Application No. 2004/0218675
Problems to be Solved by the Invention
According to the conventional full search method, an optimal motion vector for each search block has higher adaptability. This however results in a problem that a considerably large amount of calculation is required for motion estimation.
The present invention has been made to solve the above-discussed problem. It is an object of the present invention to provide a motion estimation method capable of reducing the amount of calculation as compared to the full search method.
Means for Solving Problems
A motion estimation method of a first invention comprises: (a) in a macroblock targeted for motion estimation, a step of defining a coarse search block, and several fine search blocks that are given by dividing the coarse search block into a plurality of blocks so that the several fine search blocks are contained in the coarse search block; (b) in a first interpolated image, a step of performing search using the coarse search block to obtain an optimal point with the highest degree of similarity to the coarse search block in the macroblock; (c) in a second interpolated image denser than the first interpolated image, a step of performing search of a surrounding region of the optimal point using the coarse search block to obtain an optimal motion vector with regard to the coarse search block; and (d) in the second interpolated image, a step of performing search of the surrounding region of the optimal point using each one of the several fine search blocks to obtain respective optimal motion vectors with regard to the fine search blocks, the step (d) being carried out simultaneously with the step (c). The motion estimation method of the first invention is characterized in that, in the step (c), the degree of similarity with respect to the coarse search block at each point in the surrounding region of the optimal point is calculated as a total sum of respective degrees of similarity that are obtained in the step (d) with respect to the several fine search blocks.
A motion estimation method of a second invention is characterized in that, especially in the motion estimation method of the first invention, the coarse search block includes a first coarse search block, and a second coarse search block and a third coarse search block that are given by dividing the first coarse search block into a plurality of blocks so that the second and the third coarse search blocks are contained in the first coarse search block. The step (b) comprises: (b-1) in the first interpolated image, a step of performing search using the first coarse search block to obtain a first optimal point with the highest degree of similarity to the first coarse search block in the macroblock; (b-2) in the first interpolated image, a step of performing search using the second coarse search block to obtain a second optimal point with the highest degree of similarity to the second coarse search block in the macroblock; and (b-3) in the first interpolated image, a step of performing search using the third coarse search block to obtain a third optimal point with the highest degree of similarity to the third coarse search block in the macroblock. The step (c) comprises: (c-1) in the second interpolated image, a step of performing search of a surrounding region of the first optimal point using the first coarse search block to obtain an optimal motion vector with regard to the first coarse search block; (c-2) in the second interpolated image, a step of performing search of a surrounding region of the second optimal point using the second coarse search block; (c-3) in the second interpolated image, a step of performing search of a surrounding region of the third optimal point using the third coarse search block; and (c-4) in the second interpolated image, a step of performing search of the surrounding region of the first optimal point using each of the second coarse search block and said third coarse search block, the step (c-4) being carried out simultaneously with the step (c-1). In the step (c-1), the degree of similarity with respect to the first coarse search block at each point in the surrounding region of the first optimal point is calculated as a total sum of respective degrees of similarity that are obtained in the step (c-4) with respect to the second coarse search block and the third coarse search block.
A motion estimation method of a third invention is characterized in that, especially in the motion estimation method of the second invention, the execution of the step (c-2) is omitted when the first optimal point and the second optimal point coincide with each other.
Effect of the Invention
According to the motion estimation method of the first invention, with regard to the fine search blocks, only the surrounding region of the optimal point obtained in step (b) is searched. This eliminates the search targeting the entire region of the second interpolated image by using each one of the several fine search blocks. Thus the amount of calculation required for motion estimation can be reduced.
Further, in step (a), the several fine search blocks are so defined that the several fine search blocks are contained in the coarse search block. Thus the degree of similarity with respect to the coarse search block is calculated as a total sum of respective degrees of similarity that are obtained with respect to the several fine search blocks. As a result, the degree of similarity with respect to the coarse search block is not required to be obtained independently, so the amount of calculation required for motion estimation can be reduced to a greater degree.
According to the motion estimation method of the second invention, in step (c-4), the surrounding region of the first optimal point is searched using the second coarse search block and the third coarse search block. This widens the scopes of search with regard to the second coarse search block and the third coarse search block.
Further, the second coarse search block and the third coarse search block are so defined that the second and the third coarse search blocks are contained in the first coarse search block. Thus the degree of similarity with respect to the first coarse search block is calculated as a total sum of respective degrees of similarity that are obtained with respect to the second coarse search block and the third coarse search block. As a result, the degree of similarity with respect to the first coarse search block is not required to be obtained independently, so the amount of calculation required for motion estimation can be reduced.
According to the motion estimation method of the third invention, the execution of search of the surrounding region of the second optimal point is omitted when the first optimal point and the second optimal point coincide with each other. Thus the amount of calculation required for motion estimation can be reduced to a greater degree.
These and other objects, features, aspects and advantages will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.
An embodiment of the present invention is discussed below that is applied to H. 264 as one of high efficiency coding systems. The application of the present invention is not limited to H. 264. The present invention is applicable to all coding systems that allow motion compensation using several kinds of search blocks of different block sizes and several kinds of interpolated images of different pixel accuracies.
First, search blocks are defined in step SP1. As shown in
As shown for example in
In place of
The present embodiment is discussed below when the coarse search block G1 and the fine search block G2 as shown in
Next, interpolated images are defined in step SP2 shown in
As shown for example in
Next, in step SP3 shown in
Next, in step SP4 shown in
With regard to the search block BL1, for example, only a surrounding region of the optimal point P1 (about plus or minus some pixels in both the horizontal direction and the vertical direction) in the densely interpolated image I2 is subjected to the search using the search block BL1. More specifically, the SAD between each point in the surrounding region of the optimal point P1 and the search block BL1 in the macroblock (hereinafter referred to as an “SAD1”) is obtained. Then, a motion vector representing a point at which the SAD1 scores lowest is defined as an optimal motion vector with regard to the search block BL1 (hereinafter referred to as an “optimal motion vector MV1”).
Here, in order to obtain the SAD1, the SADs between each point in the surrounding region of the optimal point P1 and the search blocks BL4 to BL7 that constitute the search block BL1 (hereinafter referred to as “SAD4” to “SAD7”) are respectively obtained. Then, the value of the SAD1 is calculated as a total sum of the values of the SAD4 to SAD7. Namely, the formula “SAD1=SAD4+SAD5+SAD6+SAD7” is established.
Likewise, with regard to the search block BL2, a surrounding region of the optimal point P2 in the densely interpolated image I2 is subjected to the search using the search block BL2. More specifically, the SAD between each point in the surrounding region of the optimal point P2 and the search block BL2 in the macroblock (hereinafter referred to as an “SAD2”) is obtained. Then, a motion vector representing a point at which the SAD2 scores lowest is defined as an optimal motion vector with regard to the search block BL2 (hereinafter referred to as an “optimal motion vector MV2”).
Here, in order to obtain the SAD2, the SAD4 and the SAD5 between each point in the surrounding region of the optimal point P2 and the search blocks BL4 and BL5 that constitute the search block BL2 are respectively obtained. Then, the value of the SAD2 is calculated as a total sum of the values of the SAD4 and the SAD5. Namely, the formula “SAD2=SAD4+SAD5” is established.
Likewise, with regard to the search block BL3, a surrounding region of the optimal point P3 in the densely interpolated image I2 is subjected to the search using the search block BL3. More specifically, the SAD between each point in the surrounding region of the optimal point P3 and the search block BL3 in the macroblock (hereinafter referred to as an “SAD3”) is obtained. Then, a motion vector representing a point at which the SAD3 scores lowest is defined as an optimal motion vector with regard to the search block BL3 (hereinafter referred to as an “optimal motion vector MV3”).
Here, in order to obtain the SAD3, the SAD6 and the SAD7 between each point in the surrounding region of the optimal point P3 and the search blocks BL6 and BL7 that constitute the search block BL3 are respectively obtained. Then, the value of the SAD3 is calculated as a total sum of the values of the SAD6 and the SAD7. Namely, the formula “SAD3=SAD6+SAD7” is established.
Several SADs4 are required to define the optimal motion vectors MV1 and MV2. A motion vector representing a point that is given the lowest value of these several SADs4 is defined as an optimal motion vector with regard to the search block BL4 (hereinafter referred to as an “optimal motion vector MV4”). This is also applicable to an optimal motion vector with regard to the search block BL5 (hereinafter referred to as an “optimal motion vector MV5”).
Several SADs6 are required to define the optimal motion vectors MV1 and MV3. A motion vector representing a point given the lowest value of these several SADs6 is defined as an optimal motion vector with regard to the search block BL6 (hereinafter referred to as an “optimal motion vector MV6”). This is also applicable to an optimal motion vector with regard to the search block BL7 (hereinafter referred to as an “optimal motion vector MV7”).
In the example shown in
Thus, like the above-mentioned algorithm in which the search of the fine search block G2 is performed along with the search of the coarse search block G1, the search of the second coarse search block G1b may be performed along with the search of the first coarse search block G1a. More specifically, when the surrounding region of the optimal point P1 is searched using the search block BL1 in step SP4, the search using the search blocks BL2 and BL3 may be performed simultaneously. That is, the SAD2 with respect to the search block BL2 is obtained by SAD4+SAD5. The SAD3 with respect to the search block BL3 is obtained by SAD6+SAD7. The SAD1 with respect to the search block BL1 is obtained by SAD2+SAD3=SAD4+SAD5+SAD6+SAD7. Thus, the search using the search blocks BL2 and BL3 can be performed simultaneously with the search using the search block BL1.
In this case, based on the search result with regard to the surrounding region of the optimal point P2 and the search result with regard to the surrounding region of the optimal point P1, the optimal motion vector MV2 with regard to the search block BL2 is obtained as a motion vector representing a point at which the SAD2 scores lowest. Likewise, based on the search result with regard to the surrounding region of the optimal point P3 and the search result with regard to the surrounding region of the optimal point P1, the optimal motion vector MV3 with regard to the search block BL3 is obtained as a motion vector representing a point at which the SAD3 scores lowest. This widens the scopes of search with regard to the search blocks BL2 and BL3, so the motion vectors MV2 and MV3 can be obtained more accurately.
When the optimal point P1 and the optimal point P2 obtained in step SP3 coincide with each other, for example, the search using the search block BL2 is not required in step SP4. This is because, as the optimal point P1 and the optimal point P2 coincide with each other, the search using the search block BL2 can be performed simultaneously with the search using the search block BL1. The elimination of the search using the search block BL2 results in reduction of the amount of calculation required for motion estimation.
Next, in step SP5 shown in
According to the motion estimation method of the present embodiment, with regard to the search blocks BL4 to BL7 belonging to the fine search block G2, the surrounding regions of the optimal points P1 to P3 obtained in step SP3 are searched. This eliminates the search targeting the entire region of the densely interpolated image I2 by using each one of the search blocks BL4 to BL7. Thus the amount of calculation required for motion estimation can be reduced.
Further, in step SP1, several fine search blocks G2 are defined so that the fine search blocks G2 are contained in the coarse search block G1. Thus, the SAD with respect to the coarse search block G1 is calculated as a total sum of the respective SADs with respect to the several fine search blocks G2. As a result, the SAD with respect to the coarse search block G1 is not required to be obtained independently, so the amount of calculation required for motion estimation can be reduced to a greater degree.
In the motion estimation method of the present embodiment, the motion estimation with regard to the fine search block G2 is simplified as compared to the full search method. It has been shown by a simulation conducted by the inventors that the degree of image degradation is considerably lower as compared to the full search method.
While the invention has been shown and described in detail, the foregoing description is in all aspects illustrative and not restrictive. It is therefore understood that numerous modifications and variations can be devised without departing from the scope of the invention.
Number | Date | Country | Kind |
---|---|---|---|
2005-284116 | Sep 2005 | JP | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/JP2006/312980 | 6/29/2006 | WO | 00 | 3/27/2008 |