Method for determining boundary strength

Abstract

A method for determining the boundary strengths of edges in a block-based digitally encoded image is disclosed. The method includes setting the boundary strength of two adjacent blocks in an Inter macroblock to a first strength value if any one of the two adjacent blocks contains non-zero prediction residual in the encoding data and setting the boundary strength thereof to a second strength value if the two adjacent blocks are located in the same motion compensation block. An edge with boundary strength equal to the second strength value will be skipped in a deblocking process.

Description

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to a digital image processing technology, and more particularly to a method for determining the boundary strength (or boundary filter strength) of edges in a block-based digital encoding image.

2. Description of the Prior Art

A block-based digital image encoding technology typically divides an image frame into many macroblocks (MBs) which are then encoded according to the luminance and chroma data of pixels (picture elements) thereof. For example, in the H.264 coding standard, a macroblock represents an image area containing 16×16 pixels. A macroblock can be encoded in Intra Prediction mode or Inter prediction mode. Intra Prediction is formed based on previously encoded and reconstructed blocks in the same video frame, while Inter Prediction creates a prediction model from one or more previously encoded video frames. Usually, a macroblock tends to be encoded in Inter prediction mode when it is correlated with content of another video frame(s).

Among available Inter prediction encoding techniques, motion compensation no doubt plays an important role. The motion compensation technology creates motion compensation blocks based on previously encoded picture(s), and represents the displacement or shift relative to block(s) of the previously encoded pictures with motion vector(s). The picture used to predict or create the motion compensation block is usually called the reference picture. In addition to motion vectors, encoding data may also include the difference between the reference picture block and the motion compensation block, which is referred to as the prediction residual. In the encoding technology such as H.264, the motion vector accuracy can be down to quarter pixel level.

In an encoding technology such as H.264, the motion compensation block mentioned above can be areas with block sizes of 16×16, 8×16, 16×8 or 8×8 pixels in a macroblock as shown in FIG. 9A. These areas are called partitions. For example, if a macroblock is divided into four 8×8 areas, then the macroblock will include four divided areas from D0 to D3 as shown in FIG. 9A. When the partition is the 8×8 mode, it can be further divided into areas with block sizes of 8×8, 4×8, 8×4, or 4×4 pixels as shown in FIG. 9B. These areas are called sub-partitions. The motion compensation block may also be a sub-partition. Such mechanism in which a macroblock may be further partitioned into motion compensation blocks of variant sizes is known as the tree structured motion compensation. Every motion compensation block may be encoded with one or two motion vectors. When a motion compensation block is encoded with two motion vectors, these two motion vectors can reference the same or different reference pictures. In the encoding technology using a partition or a sub-partition as a motion compensation prediction unit, such as H.264, the blocks located in the same partition (or sub-partition) will lie in the same motion compensation block and contain the same motion vector(s).

In order to improve the encoding efficiency of dynamic areas and static areas in a motion picture, the coding standard such as H.264 may encode each vertically adjacent macroblock pair in either frame mode or field mode. This special coding mechanism is generally called the macroblock adaptive frame/field coding which is hereinafter referred as “MBAFF” for short. Macroblocks encoded in frame mode and field mode are respectively referred to as the Frame MBs and the Field MBs in the following description. When two adjacent macroblocks (may belong to different macroblock pairs) are either both Frame MBs or both Field MBs, they are referred to as having the same parity; otherwise (i.e., one of them is a Frame MB, but the other is a Field MB), they are referred to as having different parities.

The decoding operations in the block-based digital image encoding/decoding system mainly include the inverse quantization and the inverse cosine transformation. The inverse quantization may not completely recover the original image data value, especially for data located in the edge between blocks used as units of the cosine transformation. This problem is called the block effect. The block effect usually causes visual defects on the decoded image, and thus the encoding module and decoding module of the coding technologies such as H.264 may include the function of deblocking process to alleviate or reduce the block effect. FIG. 1 shows a block-based digital image encoder or decoder 100 containing a deblocking module 120 which receives the image data output V of the decoding unit 110, performs deblocking process on the image data V, and then outputs the outcome D(V) after deblocking process to the frame buffer 130. The digital image encoder or decoder 100 generally includes other modules which are not shown in FIG. 1 to simplify the drawing.

The main operations of a conventional deblocking module may be shown by the flow chart of FIG. 2. It includes determining all the edges which need to be deblocked in the image frame (Step 22), determining the boundary strengths of all edges (Step 24), and performing the deblocking process in the image frame based on the boundary strengths (Step 26). Here the edges mean the boundary which locate in the top, bottom, left or right side of a block unit, which may be located inside a macroblock or inside an image frame. In the encoding and decoding technologies such as H.264, the boundary strength can be a value between 0 and 4. The deblocking module skips the deblocking process for any edge with boundary strength 0. The larger the boundary strength is, the stronger the deblocking process will be applied to the pixel data adjacent to the corresponding boundary. For example, the deblocking strength for an edge with boundary strength 4 is generally larger than that for an edge with boundary strength 3, 2, or 1. The deblocking process typically changes the pixel value (either luminance or chroma) lying in both sides of an edge (left and right sides or top and bottom sides), so as to make the difference smaller and accordingly reduce the artificial defects. The larger the boundary strength of an edge is, the stronger the deblocking process therefor is, and the larger the change of values of the pixels lying in both sides of the edge will be.

From above description, when a block-based digitally encoded image is processed with the encoding technology such as H.264, every video frame needs to determine all the boundary strengths efficiently to benefit the proceeding of the entire deblocking process. So there is a need to provide an efficient boundary strength determining method to improve the total performance of the deblocking process as well as the encoding/decoding operations.

SUMMARY OF THE INVENTION

One aspect of the present invention is to provide a method which can efficiently determine the boundary strength of edges in a block-based digitally encoding image so as to improve the entire performance of the deblocking process.

Another aspect of the present invention is to provide a method for determining the boundary strength of edges in a block-based digitally encoding image based on a macroblock. It can determine the boundary strength of edges of the digital image efficiently.

According to one embodiment of the present invention, the present invention provides a method for determining the boundary strength of edges in a block-based digitally encoding image, the method including determining the boundary strength of left edges, top edges, and internal edges of a macroblock according to the encoding mode and other encoding data of the macroblock, in which the encoding mode is Intra Prediction or Intra Prediction and the deblocking process for the block-based digitally encoding image applies operations with different strengths to edges with different boundary strengths.

The method for determining the boundary strength of edges in a block-based digitally encoding image of the present invention may further includes: setting the boundary strength of two adjacent blocks to a first strength value if any block of the two adjacent blocks contains a non-zero prediction residual in the encoding data; and setting the boundary strength thereof to a second strength value if the two adjacent blocks are located in the same motion compensation block, in which the two adjacent blocks are located in the same macroblock in the block-based digitally encoding image, and the macroblock is encoded in inter prediction mode.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block-based digital image encoder or decoder containing a deblocking module.

FIG. 2 shows the main operations of a conventional deblocking module.

FIG. 3A shows the spatial relationship between a macroblock and its adjacent macroblocks.

FIG. 3B shows the locations of left edges, top edges, and internal edges of a macroblock.

FIG. 4 shows the main flows of the digital image deblocking process according to the present invention.

FIG. 5 shows the method for determining the boundary strength of left edges, top edges and internal edges of an Intra MB according to an embodiment of the present invention.

FIGS. 6A and 6B show the flow chart of the method for determining boundary strength of an internal edge of an Inter MB according to an embodiment of the present invention.

FIG. 6C illustrates the internal edges respectively located between adjacent blocks lying in the same partition or different partitions.

FIG. 7 shows the flow chart of a method for determining the boundary strength of a top edge of an Inter MB according to an embodiment of the present invention.

FIG. 8 shows the flow chart of a method for determining the boundary strength of a left edge of an Inter MB according to one embodiment of the present invention.

FIG. 9A shows possible ways for a macroblock to be divided into partition(s).

FIG. 9B shows possible ways for an 8×8 partition to be divided into sub-partition(s).

DETAILED DESCRIPTION OF THE INVENTION

In the context of this specification, the boundary strength refers to the strength of the edge between two adjacent blocks (left and right or top and bottom). The boundary strength of the edge between two adjacent blocks may be alternatively referred to as the boundary strength of the two adjacent blocks hereinafter. Furthermore, it should be understood that only steps essential to the present invention will be detailed below. For example, the typical steps for reading the decoded image data (e.g. output from the decoding unit 110) from system memory and/or writing result boundary strength values back to specific locations in memory, although not explicitly described below, may be generally part of the disclosed method.

FIG. 3A shows the spatial relationship between a macroblock Cur MB and it adjacent macroblocks. In the following description, the macroblock which is exactly above the macroblock Cur MB is generally called the Top MB of Cur MB, and the one which is to the left of the macroblock Cur MB is generally called the Left MB of Cur MB. FIG. 3B shows the locations of left edges E0-E3, top edges E4-E7, and internal edges E8-E31 of a macroblock M. The macroblock M can be divided into 16 4×4 blocks B0-B15. For example, the edge between block B0 and block B1 is the internal edge E8, and the edge between block B8 and block B10 is the internal edge E28.

In contrast with the conventional boundary strength determining method based on edges, the method according to the present invention is based on macroblocks. It can rapidly determine boundary strengths of various edges in a block-based digitally encoding image. For each macroblock, the present invention first determines its encoding mode (Intra Prediction or Inter Prediction), and then determines all boundary strengths of left edges, top edges, and internal edges of the macroblock according to other encoding data. From embodiments illustrated below, it will be realized that the “other encoding data” may include the prediction residuals, numbers and contents of motion vectors, motion compensation blocks and reference pictures. Referring to FIG. 4, which shows the main flows of the digital image deblocking method according to the present invention. The deblocking method includes the method for determining boundary strength according to one embodiment of the present invention. The digital image deblocking method shown in FIG. 4 includes steps 42 and 44. Step 42 determines the boundary strength of left edges, top edges, and internal edges of each macroblock in a digital image frame according to the encoding mode and other encoding data of respective macroblock, in which the encoding mode is either Intra Prediction or Intra Prediction; and step 44 performs the deblocking process based on the boundary strengths determined for the image frame.

In the following description, a macroblock encoded in intra prediction mode is called an Intra MB, and a macroblock encoded in inter prediction mode is called an Inter MB. The method for determining the boundary strengths of left edges, top edges, and internal edges of Intra MB and Inter MB according to the present invention will be detailed below.

FIG. 5 shows the method for determining the boundary strengths of left edges, top edges and internal edges of an Intra MB according to one embodiment of the present invention. For each edge E of the Intra MB, if the edge E is a left edge, i.e. an edge lying between Intra MB and its Left MB, then the boundary strength of edge E is assigned to 4 (step 52j, step 52). If the edge E is a top edge, i.e. an edge lying between Intra MB and its Top MB, and Intra MB and its top MB are both Frame MBs (that is, macroblocks encoded in MBAFF frame mode), then the boundary strength of edge E is set to 4 (steps 54j, 55j, 54); otherwise, the boundary strength of edge E is set 3 (step 54j, 55j, 55). If the edge E is an internal edge, then its boundary strength is set to 3 (step 56). As described above, the larger the boundary strength is, the stronger the deblocking process will be applied to the pixel data adjacent to the corresponding boundary. In other words, the deblocking process may apply the strongest deblocking operation to the left edges and the top edges of an Intra MB.

FIGS. 6A and 6B collectively show the flow chart of the method for determining the boundary strength of an internal edge E of an Inter MB according to one embodiment of the present invention. For each internal edge E of the Inter MB, if the encoding data of any one of the blocks adjacent across the internal edge E contains non-zero prediction residual, then its boundary strength is assigned to 2 (step 61j, step 61). If the two blocks adjacent across the internal edge E both lie in the same motion compensation block, then the boundary strength of edge E is set to 0 (step 62j, step 62). If the two blocks adjacent across the internal edge E have different number of motion vectors, then the boundary strength of edge E is set to 1 (step 63j, step 63). If the two blocks adjacent across the internal edge E are both encoded with only one motion vector but refer to different reference pictures, or the difference between the horizontal components or the vertical components of the motion vectors of the two blocks is larger than or equals to one pixel, then the boundary strength of edge E is set to 1 (step 64j, step 64). If the two blocks adjacent across the internal edge E are both encoded with two motion vectors, and the two blocks refer to the same reference picture, which means the four motion vectors all refer to the same reference picture, and if the difference between the horizontal components or the vertical components of any two motion vectors in different block is larger than or equals to one pixel, then the boundary strength of edge E is set to 1 (step 65j, step 65). If the two blocks adjacent across the internal edge E are both encoded with two motion vectors, and the two blocks refer to different reference pictures, then the boundary strength of edge E is set to 1 (step 66j, step 66). If the two blocks adjacent across the internal edge E are both encoded with two motion vectors, and the two blocks refer to different reference pictures A and B, and if the difference between the horizontal components or the vertical components of the two motion vectors both referring to the reference picture A or the reference picture B is larger than or equals to one pixel, then the boundary strength of edge E is set to 1 (step 67j, step 67). In other cases, the boundary strength of the internal edge E is assigned to 0 (step 68).

For the encoding technology with quarter pixel motion vector, such as H.264, the one pixel difference usually means 4. But if the macroblock is a Field MB, the one pixel difference in vertical components means 2.

As described above, the blocks lying in the same partition or sub-partition will lie in the same motion compensation block. In step 62j of FIG. 6A, determining whether two blocks adjacent across the internal edge E lie in the same motion compensation block or not may be performed through examining if the two blocks are located in the same partition or sub-partition of a macroblock. FIG. 6C illustrates the internal edges E1 and E2 which are respectively located between adjacent blocks lying in the same partition and different partitions. As shown in FIG. 6C, blocks B0 and B1 adjacent across the internal edge E1 both lie in the partition D0, while blocks B9 and B12 adjacent across the internal edge E2 lie in different partitions D0 and D1 respectively. In this way, it may determine that the two blocks (B0 and B1) adjacent across the internal edge E1 lie in the same motion compensation block (D0), while the two blocks (B9 and B12) adjacent across the internal edge E2 lie in different motion compensation blocks (D0 and D1). From this illustration, those skilled in the art should be able to infer the detail of the determining step (62j) for the other partition or sub-partition mode shown in FIGS. 9A and 9B.

FIG. 7 shows the flow chart of a method for determining the boundary strength of a top edge of an Inter MB according to one embodiment of the present invention. For each top edge E of the Inter MB, if the Top MB of the Inter MB is an Intra MB, then the boundary strength of the top edge E is set to 3 (step 71j, step 71). If the Inter MB and its Top MB have different parities, and if the encoding data of any one of the two blocks adjacent across the top edge E has non-zero prediction residual, then setting the boundary strength of edge E to 2; otherwise setting the boundary strength of edge E to 1 (step 72j, step 72). If the Inter MB and its Top MB have the same parity, then determining the boundary strength of the top edge E with the method shown in FIGS. 6A and 6B (step 72j, step 73).

FIG. 8 shows the flow chart of a method for determining the boundary strength of a left edge of an Inter MB according to one embodiment of the present invention. For each left edge E of the Inter MB, if the Left MB of the Inter MB is an Intra MB, then the boundary strength of the left edge E is set to 4 (step 81j, step 81). If the Inter MB and its Left MB have the same parity, then determining the boundary strength of the left edge E with the method shown in FIGS. 6A and 6B (step 82j, step 82). If the Inter MB and its Left MB have different parities, and if the encoding data of any one of the two blocks adjacent across the left edge E has non-zero prediction residual, then setting the boundary strength of edge E to 2. Otherwise the boundary strength of edge E is set to 1 (step 82j, step 83).

The above examples and description have been provided only for the purpose of illustration, and are not intended to limit the invention in any way. A great variety of variations and modifications may carry out the present invention without exceeding the scope thereof. For example, although above illustrations are based on the encoding protocol of H.264, the application of the present invention should not be limit by this. Moreover, many variations such as the sizes of blocks, macroblocks, partitions and sub-partitions, as well as the accuracy unit of motion vectors, and possible values of the boundary strength (e.g. other than 0-4) may be made without departing from the scope of the present invention, which is intended to be limited solely by the appended claims.

Claims

1. A method for determining boundary strengths of edges in a block-based digitally encoded image, the method comprising the steps of: determining boundary strengths of left edges, top edges and internal edges of a macroblock according to encoding mode and encoding data of the macroblock, the encoding mode being either Intra Prediction or Inter Prediction; andapplying deblocking operations with different strengths to edges with different boundary strength values by a deblocking process for the block-based digitally encoded image;wherein if the encoding mode of the macroblock is Inter Prediction, and none of two adjacent blocks in the macroblock has non-zero residuals and the two adjacent blocks both lie in the same motion compensation block, then setting boundary strength of the two adjacent blocks to a first strength value, and if the encoding mode of the macroblock is Intra Prediction, then the boundary strengths of the left edges of the macroblock are set to a second strength value; if both the macroblock and its Top macroblock (MB), which is the macroblock above the current macroblock, are both Frame macroblocks (MBs), which are macroblocks encoded in frame mode, then the boundary strengths of the top edges of the macroblock are set to the second strength value, otherwise the boundary strengths of the top edges of the macroblock are set to a third strength value; and the boundary strengths of the internal edges of the macroblock are all set to the third strength value.

2. The method of claim 1, wherein the deblocking process skips any deblocking operation for any edge with boundary strength equal to the first strength value.

3. The method of claim 2, wherein whether the two adjacent blocks lie in the same motion compensation block or not is determined through examining if the two adjacent blocks are located in the same partition or sub-partition of the macroblock.

4. The method of claim 3, wherein the block is a 4×4 pixel area, and wherein the partition is a pixel area with size of 16×16, 16×8, 8×16 or 8×8 pixels.

5. The method of claim 3, wherein the block is a 4×4 pixel area, and wherein the sub-partition is a pixel area with size of 8×8, 8×4 or 4×8 pixel.

6. The method of claim 3, wherein: if the two adjacent blocks lie in different motion compensation blocks and have different number of motion vectors, then setting the boundary strength of the two adjacent blocks to a fourth strength value.

7. The method of claim 6, wherein the strength of the deblocking operation applied to an edge with boundary strength equal to the third strength value is stronger than the strength of the deblocking operation applied to an edge with boundary strength equal to the fourth strength value.

8. The method of claim 1, wherein the strength of the deblocking operation applied to an edge with boundary strength equal to the second strength value is stronger than the strength of the deblocking operation applied to an edge with boundary strength equal to the third strength value.

9. A method for determining boundary strengths of edges in a block-based digitally encoded image, the method comprising the steps of: determining boundary strengths of left edges, top edges and internal edges of a macroblock according to encoding mode and encoding data of the macroblock, the encoding mode being either Intra Prediction or Inter Prediction; andapplying deblocking operations with different strengths to edges with different boundary strength values by a deblocking process for the block-based digitally encoded image;wherein if the encoding mode of the macroblock is Intra Prediction, then the boundary strengths of the left edges of the macroblock are set to a first strength value; if both the macroblock and its Top macroblock (MB), which is the macroblock above the current macroblock, are both Frame macroblocks (MBs), which are macroblocks encoded in frame mode, then the boundary strengths of the top edges of the macroblock are set to the first strength value, otherwise the boundary strengths of the top edges of the macroblock are set to a second strength value; and the boundary strengths of the internal edges of the macroblock are all set to the second strength value.

10. The method of claim 9, wherein the strength of the deblocking operation applied to an edge with boundary strength equal to the first strength value is stronger than the strength of the deblocking operation applied to an edge with boundary strength equal to the second strength value, and the strength of the deblocking operation applied to an edge with boundary strength equal to the second strength value is stronger than the strength of the deblocking operation applied to an edge with boundary strength equal to a third strength value.

11. The method of claim 9, wherein the encoding data comprises prediction residuals, numbers and contents of motion vectors, motion compensation blocks and reference pictures.