Patent Application
20070223591
The present invention relates to information technology in general, and, more particularly, to video decoding and filtering.
A source video stream can be encoded to decrease the amount of resources that are required to process the video frames in the stream. Various video compression standards, such as the VC-1 standard, the ITU-H.264 standard (H.264), and the MPEG-4 standard can be used to encode the picture elements, or “pixels,” contained in the frames. The encoding may comprise one or more of spatial prediction, temporal prediction, discrete cosine transformation, quantization, and entropy coding to compress multiple regions in each video frame that are referred to as “macroblocks.” As a consequence of the video encoding process, some of the original source information might be lost, particularly as a result of the quantization process.
In order to reconstruct the original video frames, such as for the purpose of displaying the frames, the encoded video stream is eventually decoded.
The quality of the decoded video signal, however, is typically impaired as a result of some loss of information, and artifacts consequently might appear. An effect known as “transform blockiness” might be present in the decoded video frames, in the form of square grid artifacts; this occurs as a result of information loss during the encoding process. In addition, effects other than transform blockiness might appear in the decoded video frames, as the result of information loss that occurs during the transmission between the encoder and the decoder. Transform blockiness might be associated with missing information along one or more horizontal or vertical boundaries between adjacent macroblocks or between adjacent sub-blocks within a macroblock. The blockiness often appears as harsh, unintended visual transitions between the macroblocks or sub-blocks.
In video systems utilizing blocks of video information, a video deblocker may be utilized to smooth the visual transition between adjacent video blocks. During the deblocking process in the prior art, macroblocks within a video frame are fetched, row-by-row, starting with the top row, and left-to-right within each row. The pixels that are near the boundaries of each sub-block in the video frame are then filtered (i.e., “deblocked”) in an order that is either (i) horizontal boundaries first, followed by vertical boundaries or (ii) vertical boundaries first, followed by horizontal boundaries, depending on the video standard in use. Typically, the order in which the horizontal boundaries are filtered is from top to bottom, and the order in which the vertical boundaries are filtered is from left to right.
Depending on the video standard in use, the problem with the specified blocking order in some cases is that all of the boundaries across one dimension—for instance, all horizontal boundaries—must be filtered before the boundaries across the other dimension—for instance, the vertical boundaries—can be filtered. The order matters because some pixels (i.e., those in the corners of sub-blocks) are filtered twice and might be affected differently if the order were to be reversed. Deblocking across the entire video frame requires two memory read operations—the first for filtering across the first dimension, the second for filtering across the second dimension—which consumes memory and processing cycles.
The present invention seeks to mitigate resource utilization during the deblocking process and is based on the observation that as long as the standardized filter order is preserved for those individual pixels that are filtered twice, such as corner pixels, it is unnecessary to filter across the entire video frame, first across one dimension, then across another. The video deblocker of the illustrative embodiment of the present invention deblocks the video frame by considering the pixels to be filtered on a macroblock-by-macroblock basis. In some embodiments, the deblocker deblocks the macroblock by considering the pixels to be filtered on a sub-block-by-sub-block basis. To say that the video frame is deblocked on a block-by-block basis means that a row of pixels of a first block can be filtered after the column of a second block, even though the video frame is subject to rows-then-columns order of filtering. Likewise, a column of pixels of a first block can be filtered after the row of a second block, even though the video frame is subject to columns-then-rows order of filtering.
The deblocking that is performed in accordance with the illustrative embodiment reduces transform blockiness along the horizontal block boundaries and vertical block boundaries. The disclosed techniques are equally applicable to filtering along macroblock boundaries within a video frame and along sub-block boundaries within a macroblock. The techniques of the illustrative embodiment are advantageous over some techniques in the prior art because the deblocker is only required to read in all of the macroblocks in a video frame once, instead of having to read them in twice: once for the filtering across the first dimension (e.g., across rows, etc.) and once for the filtering across the second dimension (e.g., across columns, etc.).
The illustrative embodiment of the present invention comprises: deblocking the topmost row of block (x, y); deblocking the leftmost column of block (x, y) after deblocking the topmost row of block (x, y); and deblocking the bottommost row of block (x+a, y+b) after deblocking the leftmost column of block (x, y); wherein at least one of a and b is a positive integer, and wherein x and y are integers.
The following terms are defined for use in this Specification, including the appended claims:
Receiver 201 receives information from the video decoder, as described below and with respect to
Processor 202 is a general-purpose processor as is well-known in the art that is capable of executing the operating system and user programs in memory 203, and of writing into and removing from memory 203 one or more blocks of video data. The user programs perform the tasks described below and with respect to
Memory 203 is a non-volatile memory that stores:
i. the operating system and user programs for processor 202, and
ii. one or more blocks of video data,
as described with respect to
Transmitter 204 receives deblocked output from processor 202 and transmits it to the post-processing and display systems.
Each macroblock 301 through 320 is a 16×16 array of pixels. It will be clear, however, to those skilled in the art how to apply the present invention to macroblocks that comprise a different size array than 16×16 pixels (e.g., 8×8 pixels, etc.) or to macroblocks of different sizes within a frame (e.g., 4×4-pixel macroblocks co-existing with 16×16-pixel macroblocks, etc.). In each 16×16-pixel macroblock, there are 16 columns of pixels, where the leftmost column consists of pixels A1 through A16 and the rightmost column consists of pixels P1 through P16. Similarly, in each macroblock there are 16 rows of pixels, where the topmost row consists of pixels A1 through P1 and the bottommost row consists of pixels A16 through P16.
As part of the overall deblocking process, one or more rows of pixels near a side (i.e., the top or bottom) within each macroblock are filtered in well-known fashion to produce a smoother visual transition at the horizontal boundaries of the current macroblock. For example, the topmost row of pixels (i.e., row 321) of macroblock 308 is filtered by using adjacent pixels, including the bottommost row of pixels of macroblock 303, which is situated above macroblock 308. Similarly, one or more columns of pixels near a side (i.e., the left-side or right-side) within each macroblock are filtered in well-known fashion to produce a smoother visual transition at the vertical boundaries of the macroblock. For example, the leftmost column of pixels (i.e., column 323) of macroblock 308 is filtered by using adjacent pixels, including the rightmost column of pixels of macroblock 307, which is situated to the left of macroblock 308.
Video frame 300 is deblocked by filtering pixels along horizontal boundaries 331, 332, and 333, as well as by filtering pixels along vertical boundaries 341, 342, 343, and 344. In contrast to some techniques in the prior art, however, frame 300 is not deblocked by filtering all horizontal boundaries first, followed by all vertical boundaries. Instead, frame 300 is deblocked in a manner in which the only requirement in filtering order is that any pixel that belongs to both a filtered row and column of pixels is filtered first as part of the row and then as part of the column. As an example, pixel A1 of macroblock 308 belongs to both row 321 and column 323, in which row 321 is filtered first, followed by column 323. As another example, pixel A16 of macroblock 308 belongs to both row 322 and column 323, in which row 322 is filtered first, followed by column 323.
Subsequently, one or more rows, such as rows 325 through 328, are deblocked. In other words, in accordance with the illustrative embodiment, some rows near a macroblock boundary are deblocked after some columns, as long the row-before-column rule is not violated for a pixel that belongs to both a filtered row and column.
Each macroblock 401 through 420 is a 16×16 array of pixels. It will be clear, however, to those skilled in the art how to apply the present invention to macroblocks that comprise a different size array than 16×16 pixels (e.g., 8×8 pixels, etc.) or to macroblocks of different sizes within a frame (e.g., 4×4-pixel macroblocks co-existing with 16×16-pixel macroblocks, etc.). In each 16×16-pixel macroblock, there are 16 columns of pixels, where the leftmost column consists of pixels A1 through A16 and the rightmost column consists of pixels P1 through P16. Similarly, in each macroblock there are 16 rows of pixels, where the topmost row consists of pixels A1 through P1 and the bottommost row consists of pixels A16 through P16.
As part of the overall deblocking process, one or more columns of pixels near a side (i.e., the left-side or right-side) within each macroblock are filtered in well-known fashion to produce a smoother visual transition at the vertical boundaries of the macroblock. For example, the leftmost column of pixels (i.e., column 421) of macroblock 408 is filtered by using adjacent pixels, including the rightmost column of pixels of macroblock 407, which is situated to the left of macroblock 408. Similarly, one or more rows of pixels near a side (i.e., the top or bottom) within each macroblock are filtered in well-known fashion to produce a smoother visual transition at the horizontal boundaries of the current macroblock. For example, the topmost row of pixels (i.e., row 423) of macroblock 408 is filtered by using adjacent pixels, including the bottommost row of pixels of macroblock 403, which is situated above macroblock 408.
Video frame 400 is deblocked by filtering pixels along vertical boundaries 431, 432, 433, and 434, as well as by filtering pixels along horizontal boundaries 441, 442, and 443. In contrast to some techniques in the prior art, however, frame 400 is not deblocked by filtering all vertical boundaries first, followed by all horizontal boundaries. Instead, frame 400 is deblocked in a manner in which the only requirement in filtering order is that any pixel that belongs to both a filtered column and row of pixels is filtered first as part of the column and then as part of the row. As an example, pixel A1 of macroblock 408 belongs to both column 421 and row 423, in which column 421 is filtered first, followed by row 423. As another example, pixel P1 of macroblock 408 belongs to both column 422 and row 423, in which column 422 is filtered first, followed by row 423.
Subsequently, one or more columns, such as columns 425 through 428, are deblocked. In other words, in accordance with the illustrative embodiment, some columns near a macroblock boundary are deblocked after some rows, as long the column-before-row rule is not violated for a pixel that belongs to both a filtered column and row.
As part of the overall deblocking process, one or more columns of pixels near a side (i.e., the left-side or right-side) within each sub-block are filtered in well-known fashion to produce a smoother visual transition at the vertical boundaries of the sub-block. For example, the leftmost column of pixels (i.e., column 521) of sub-block B3 is filtered by using adjacent pixels, including the rightmost column of pixels of sub-block B2, which is situated to the left of sub-block B3. Similarly, one or more rows of pixels near a side (i.e., the top or bottom) within each sub-block are filtered in well-known fashion to produce a smoother visual transition at the horizontal boundaries of the current sub-block. For example, the topmost row of pixels (i.e., row 523) of sub-block B3 is filtered by using adjacent pixels, including the bottommost row of pixels of sub-block BT3, which is situated above sub-block B3.
Macroblock 408 is deblocked by filtering pixels along vertical boundaries 432, 533, 534, and 535, as well as by filtering pixels along horizontal boundaries 441, 542, 543, and 544. In contrast to some techniques in the prior art, however, macroblock 408 is not deblocked by filtering all vertical boundaries first, followed by all horizontal boundaries. Instead, macroblock 408 is deblocked in a manner in which the only requirement in filtering order is that any pixel that belongs to both a filtered column and row of pixels is filtered first as part of the column and then as part of the row. As an example, pixel I1 of sub-block B3 belongs to both column 521 and row 523, in which column 521 is filtered first, followed by row 523. As another example, pixel L1 of sub-block B3 belongs to both column 522 and row 523, in which column 522 is filtered first, followed by row 523.
Subsequently, one or more columns, such as columns 525, 527, 529, and 531, are deblocked. In other words, in accordance with the illustrative embodiment, some columns in a macroblock are deblocked after some rows, as long the column-before-row rule is not violated for a pixel that belongs to both a filtered column and row.
In some alternative embodiments, as with the row-then-column filtering described above and with respect to
In some alternative embodiments, the filtering along macroblock boundaries is performed in a different row/column ordering than for the filtering along sub-block boundaries. For example, the filtering might be performed along all of the horizontal (or vertical) macroblock boundaries first, followed by filtering along all of the vertical (or horizontal) macroblock boundaries, followed by filtering along all sub-block boundaries that is in accordance with the illustrative embodiment.
At task 601, deblocker 200 receives one or more decoded blocks that comprise block (x, y), wherein x and y are integers.
At task 602, deblocker 200 determines if a pixel that is subject to both column and row filtering is to be filtered row first or column first. If the row is to be filtered first (i.e., row-first mode), task execution proceeds to task 603. If the column is to be filtered first (i.e., column-first mode), task execution proceeds to task 606.
At task 603, deblocker 200 deblocks the topmost row of pixels or bottommost row of pixels, or both, of block (x, y) in well-known fashion. In some embodiments, one or more rows that are adjacent to the filtered outermost row (i.e., topmost or bottommost) are also filtered in well-known fashion.
At task 604, deblocker 200 deblocks the leftmost column of pixels or rightmost column of pixels, or both, of block (x, y) in well-known fashion. In some embodiments, one or more columns that are adjacent to the filtered outermost column (i.e., leftmost or rightmost) are also filtered in well-known fashion. In a row-first mode, the deblockings that are described with respect to task 604 are performed after the deblockings that are described with respect to task 603.
At task 605, deblocker 200 deblocks the topmost row of pixels or bottommost row of pixels, or both, of block (x+a, y+b) in well-known fashion, wherein a and b are integers. In some embodiments, at least one of a and b is a positive integer. In some other embodiments, at least one of a and b is a positive integer, and b is not negative. For example, values for the pair (a, b) can be (1, 0), (−2, 1), (−1, 1), (0, 1), and so forth. In some alternative embodiments, one or more rows that are adjacent to the filtered outermost row (i.e., topmost or bottommost) are also filtered in well-known fashion. In a row-first mode, the deblockings that are described with respect to task 605 are performed after the deblockings that are described with respect to task 604, in accordance with the illustrative embodiment of the present invention. Task execution then ends.
At task 606, deblocker 200 deblocks the leftmost column or rightmost column, or both, of block (x, y) in well-known fashion. In some embodiments, one or more adjacent columns to the filtered outermost column (i.e., leftmost or rightmost) are also filtered in well-known fashion.
At task 607, deblocker 200 deblocks the topmost row of pixels or bottommost row of pixels, or both, of block (x, y) in well-known fashion. In some embodiments, one or more rows that are adjacent to the filtered outermost row (i.e., topmost or bottommost) are also filtered in well-known fashion. In a column-first mode, the deblockings that are described with respect to task 607 are performed after the deblockings that are described with respect to task 606.
At task 608, deblocker 200 deblocks the leftmost column of pixels or rightmost column of pixels, or both, of block (x+a, y+b) in well-known fashion, wherein a and b are integers. In some embodiments, at least one of a and b is a positive integer. In some other embodiments, at least one of a and b is a positive integer, and b is not negative. For example, values for the pair (a, b) can be (1, 0), (−2, 1), (−1, 1), (0, 1), and so forth. In some alternative embodiments, one or more columns that are adjacent to the filtered outermost column (i.e., leftmost or rightmost) are also filtered in well-known fashion. In a column-first mode, the deblockings that are described with respect to task 608 are performed after the deblockings that are described with respect to task 607, in accordance with the illustrative embodiment of the present invention. Task execution then ends.
It is to be understood that the above-described embodiments are merely illustrative of the present invention and that many variations of the above-described embodiments can be devised by those skilled in the art without departing from the scope of the invention. For example, in this Specification, numerous specific details are provided in order to provide a thorough description and understanding of the illustrative embodiments of the present invention. Those skilled in the art will recognize, however, that the invention can be practiced without one or more of those details, or with other methods, materials, components, etc.
Furthermore, in some instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the illustrative embodiments. It is understood that the various embodiments shown in the Figures are illustrative, and are not necessarily drawn to scale. Reference throughout the specification to “one embodiment” or “an embodiment” or “some embodiments” means that a particular feature, structure, material, or characteristic described in connection with the embodiment(s) is included in at least one embodiment of the present invention, but not necessarily all embodiments. Consequently, the appearances of the phrase “in one embodiment,” “in an embodiment,” or “in some embodiments” in various places throughout the Specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments. It is therefore intended that such variations be included within the scope of the following claims and their equivalents.
