Patent Application
20230188712
Disclosed are embodiments related to the processing (e.g., encoding and decoding) of video data.
A video sequence consists of a series of images (a.k.a., pictures) where each image consists of one or more components. Each component can be described as a two-dimensional rectangular array of sample values. It is common that an image in a video sequence consists of three components; one luma component Y where the sample values are luma values and two chroma components Cb and Cr, where the sample values are chroma values. Other examples include Y′ Cb Cr, Yuv and ICTCP. In ICTCP, I is the “intensity luma” component. For the remainder of this document we will refer to any luma component Y′, Y or I as Y or simply luma. It is common that the dimensions of the chroma components are smaller than the luma components by a factor of two in each dimension. For example, the size of the luma component of an HD image would be 1920×1080 and the chroma components would each have the dimension of 960×540. Components are sometimes referred to as color components.
A “block” is a two-dimensional array of samples. In video coding, each component is split into one or more blocks and the coded video bitstream is a series of blocks. It is common in video coding that an image is split into units that cover a specific area of the image. Each unit consist of all blocks from all components that make up that specific area and each block belongs fully to one unit. The macroblock in H.264 and the Coding unit (CU) in HEVC are examples of units.
In HEVC, each image is partitioned into coding tree units (CTU). A CTU consist of an N×N block of luma samples and two M×M corresponding chroma blocks. A CTU in HEVC is like macroblocks in H.264 and earlier standards but in contrast to macroblocks the CTU size is configurable. Most often, however, the CTU size in HEVC is set to 64×64 luma samples. Each CTU can be recursively quadtree split. The root of the quadtree is then associated with the CTU. The quadtree is split until a leaf is reached, which is referred to as the coding unit (CU). A CU in HEVC always consist of a luma block with equal height and width. How each CTU is split is conveyed in the bitstream. The CU is further the root node of two other trees, the prediction tree that has prediction units (PUs) as nodes and the transform tree that has transform units (TUs) as nodes. Some decoding processes in HEVC are done on the CU level, some are done on the PU level and some on TU level. Boundaries between PUs and boundaries between TUs are filtered by a deblocking filter to reduce discontinuities between TUs and PUs. In HEVC there exist two kinds of prediction types for a PU, intra-prediction which only uses prediction from previously decoded samples of the current image for prediction, and inter-prediction which uses prediction form at least one previously decoded image.
In HEVC, deblocking is first applied on vertical boundaries and then on horizontal boundaries. The boundaries are either TU boundaries or PU boundaries. To enable parallel friendly deblocking, the deblocking is performed on an 8×8 sample grid.
A deblocking filter strength parameter (bs) is set for each 4 sample part of the boundary. If the value of bs is larger than 0, then deblocking may be applied. The larger the boundary strength is, the stronger filtering is applied. First it is checked if any of the blocks at a PU boundary between the blocks is an intra predicted block then (bs is set to=2), or if both blocks use inter prediction but and they use different reference frames or have significantly different motion vectors then (bs is set to =1). It is also checked if a TU boundary between the blocks has non-zero transform coefficients in at least one of the blocks (code block flag CBF equal to 1), then (bs is set to =1). This first check sets a boundary strength (bs) which is larger than 0 to indicate that deblocking should be applied for a 4 sample part of the boundary. The larger the boundary strength is the stronger filtering is applied.
To reduce/avoid removing natural structures when deblocking, deblocking edge decisions check that there are not any natural structures on respective sides of the boundary is then applied for luma. In HEVC, gradient calculations are used on respective sides of the boundary using the following inequality: abs(p0−2*p1+p2)+abs(q0−2*q1+q2)<beta, where beta (also denoted “β”) is a parameter based on the quantization parameter for the block and p0, p1, to p2 are samples on one side of the block boundary and q0, q1, to q2 are samples on the other side of the block boundary. The condition is checked at two lines across of the 4 sample part of the boundary, line 0 and 3, and if both conditions are fulfilled, then the luma samples are deblocked for that 4 sample part of the boundary. This is applied for all 4 sample parts of a boundary until all samples of the block boundary have been checked and possibly filtered. Chroma boundaries may always be filtered if one any of the neighbouring blocks are intra coded.
In the current draft of the specification for Versatile Video Coding (VVC) (see reference [1]) (also referred to herein as the “VVC Draft Specification”) a coding tree unit (CTU) is similar to the CTU in HEVC with the difference that the CTU in H.266 has a size of 128×128 luma samples. In VVC, the CTU can be split more flexibly such that a resulting CUs may consist of a rectangular luma block. In VVC, there is no prediction tree or transform tree as in HEVC. However, a CU in VVC can be divided into a multiple of TUs or into a multiple of prediction subblocks.
In the current draft of the specification for VVC, the deblocking is applied on an 4×4 grid for CUs first on vertical boundaries (CU/implicit TU/prediction sub-block boundaries) and then on horizontal boundaries (CU/implicit TU/prediction sub-blocks). Prediction sub-block boundaries inside a CU is filtered on an 8×8 grid. The deblocking is based on HEVC deblocking but also have longer deblocking filters if the size orthogonal to the block boundary is equal to or larger than 32 on at least one side for luma and the other side is larger than 4, modifying at most 7 samples (reading at most 8 samples), if the size orthogonal to the block boundary is less than 32 for one side for luma it modifies at most 3 samples and reading at most 4 samples on that side, and if it is equal to or larger than 8 on both side of a boundary in chroma samples for chroma modifying at most 3 chroma samples and reading at most 4 chroma samples otherwise it modifies at most one sample and reading at most two samples on respective side of the boundary.
The deblocking edge decisions are computed for line 0 and line 3 for a 4 sample segment of the block boundary. Based on the deblocking edge decision either long deblocking filter, strong deblocking filter or weak deblocking filter is applied for filtering lines 0 to 3 of the for sample segment of the block boundary.
The following is an excerpt from the VVC Draft Specification. This excerpt describes a block edge decision process (more specifically, the decision process for luma block edges).
The following is another excerpt from the VVC Draft Specification. This excerpt describes the filtering process for luma block edges:
The following is another excerpt from the VVC Draft Specification. This excerpt describes a decision process for a luma sample:
The following is another excerpt from the VVC Draft Specification. This excerpt describes a filtering process for a luma sample using short filters:
The following is another excerpt from the VVC Draft Specification. This excerpt describes a filtering process for a luma sample using long filters:
Certain challenges presently exist. For example, currently the long deblocking decision is based on checks for line 0 and 3 for each 4 samples boundary segment, which works fine in many cases, but in some corner cases, if there is some structure or details on line 1 on samples p0,1 to p7,1 and q0,1 to q7,1 or on line 2 on samples p0,2 to p7,2 and q0,2 to q7,2, then deblocking can be applied since line 0 and line 3 pass the check which then can cause over filtering of 28 samples for a 4 sample boundary segment, e.g. p0,1 to p6,1, q0,1 to q6,1, p0,2 to p6,2 and q0,2 to q6,2.
This disclosure aims to overcome this problem. For example, the problem can be overcome by checking all lines for true edges before applications of a deblocking filter that can modify more than three samples on at least one side of a block boundary. That is, this disclosure proposes to fix the long deblocking decision such that all lines of respective 4 samples boundary segment are checked to avoid over filtering of lines 1 and 2 due to decision based only on line 0 and line 3. In one embodiment, the proposal ensures that the deblocking filtering is robust and that the fix does not increase worst case complexity for deblocking decisions.
Accordingly, in one aspect there is provided a method for filtering luma block edges. The method includes assigning a value to a first decision variable by performing a block edge decision process using a first set of input samples; assigning a value to a second decision variable by performing the block edge decision process using a second set of input samples; assigning a value to a third decision variable by performing the block edge decision process using a third set of input samples; and assigning a value to a fourth decision variable by performing the block edge decision process using a fourth set of input samples. The method also includes determining that a long filter condition is satisfied. The method also includes, as a result of determining that the long filter condition is satisfied, performing a filtering process for the first set of input samples, the second set of input samples, the third set of input samples and the fourth set of input samples, using a long filter to produce a first set of output samples, a second set of output samples, a third set of output samples and a fourth set of output samples. The long filter condition is satisfied if and only if: the value of the first decision variable is equal to a first value, the value of the second decision variable is equal to the first value, the value of the third decision variable is equal to the first value, and the value of the fourth decision variable is equal to the first value.
In another aspect there is provided a computer program comprising instructions which when executed by processing circuitry of an apparatus causes the apparatus to perform the method. In another aspect there is provided a carrier containing the computer program, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.
In another aspect there is provided an apparatus, where the apparatus is configured to perform the method of any embodiments disclosed herein. In some embodiments, the apparatus includes processing circuitry and a memory containing instructions executable by the processing circuitry, whereby the apparatus is configured to perform the methods disclosed herein.
The embodiments disclosed herein are advantageous in that the embodiments avoid removing natural structure on half of the lines of a four sample boundary segment that currently the deblocking is unware of.
The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.
The embodiments disclosed herein can be used to avoid removing natural structure. This can be applied in encoder 102 and/or decoder 104.
In one embodiment additional checks for line 1 and 2 are added to the long luma deblocking decision to avoid removing structure on those lines when applying long luma deblocking. This can be performed without increasing the worst case for deblocking complexity since it is only applied when the long luma deblocking filter can be used.
In another optional embodiment additional checks for line 1 and 2 are also added for the strong deblocking decision (i.e., When d is less than β and both maxFilterLengthP and maxFilterLengthQ are greater than 2).
Accordingly, this disclosure proposes changes to section 8.8.3.6.1 (Decision process for luma block edges) of the VVC Draft Specification such that the decision process not only checks lines 0 and 3, but also checks lines 1 and 2 so that dSam1 and dSam2 are obtained in addition to dSam0 and dSam3 and then to require dSam0 to dSam3 to all be equal to 1 to set dE to 3, e.g. apply long luma deblocking filter in section 8.8.3.6.7. The below table includes a proposed new section 8.8.3.6.1 to replace the current section 8.8.3.6.1. The below proposed changes with respect to the strong deblocking decision (i.e., g. When d is less than β and both maxFilterLengthP and maxFilterLengthQ are greater than 2) are less preferred than the changes with respect to the long filter and thus are optional.
A1. A method (400, see
A2. The method (400) of embodiment A1, further comprising, prior to determining that the long filter condition is satisfied: assigning (s401) a value to the first decision variable by performing a block edge decision process using the first set of input samples; assigning (s402) a value to the second decision variable by performing the block edge decision process using the second set of input samples; assigning (s403) a value to the third decision variable by performing the block edge decision process using the third set of input samples; and assigning (s404) a value to the fourth decision variable by performing the block edge decision process using the fourth set of input samples.
A3. The method of embodiment A1 or A2, wherein determining that a long filter condition is satisfied consists of determining that the value of a fifth decision variable is equal to 3.
A4. The method of embodiment A3, further comprising setting the value of the fifth decision variable to 3 as a result of determining that: i) the value of the first decision variable is equal to the first value, ii) the value of the second decision variable is equal to the first value, iii) the value of the third decision variable is equal to the first value, and iv) the value of the fourth decision variable is equal to the first value.
A5. The method of any one of embodiments A1-A4, wherein the first set of inputs are from a first line across a block boundary including samples p0,0, p1,0, p2,0, p3,0, p4,0, p5,0, p6,0, p7,0 on one side of the boundary and q0,0, q1,0, q2,0, q3,0, q4,0, q5,0, q6,0, q7,0 on the other side of the boundary, the second set of inputs are from a second line across the block boundary including samples p0,1, p1,1, p2,1, p3,1, p4,1, p5,1, p6,1, p7,1 and q0,1, q1,1, q2,1, q3,1, q4,1, q5,1, q6,1, q7,1 on the other side of the boundary, the third set of inputs are from a third line across the block boundary including samples p0,2, p1,2, p2,2, p3,2, p4,2, p5,2, p6,2, p7,2 and q0,2, q1,2, q2,2, q3,2, q4,2, q5,2, q6,2, q7,2 on the other side of the boundary, and the fourth set of inputs are from a fourth line across the block boundary including samples p0,3, p1,3, p2,3, p3,3, p4,3, p5,3, p6,3, p7,3 and q,3, q1,3, q2,3, q3,3, q4,3, q5,3, q6,3, q7,3 on the other side of the boundary.
A6. The method of any one of embodiments A1-A5, wherein the first set of output samples p′0,0, p′1,0, p′2,0, p′3,0, p′4,0, p′5,0, p′6,0, q′0,0, q′1,0, q′2,0, q′3,0, q′4,0, q′5,0, q′6,0, are derived from the first set of input samples, the second set of output samples p′0,1, p′2,1, p′3,1, p′4,1, p′5,1, p′6,1, q′0,1, q′2,1, q′3,1, q′4,1, q′5,1, q′6,1, are derived from the second set of input samples, the third set of output samples p′0,2, p′1,2, p′2,2, p′3,2, p′4,2, p′5,2, p′6,2, q′0,2, q′1,2, q′2,2, q′3,2, q′4,2, q′5,2, q′6,2, are derived from the third set of input samples, and the fourth set of output samples, p′0,3, p′1,3, p′2,3, p′3,3, p′4,3, p′5,3, p′6,3, q′0,3, q′1,3, q′2,3, q′3,3, q′4,3, q′5,3, q′6,3 are derived from the fourth set of input samples.
A7. The method of any one of embodiments A1-A6, wherein the first set of inputs are from a first line across the block boundary including samples p0,0, p1,0, p2,0, p3,0, p4,0, p5,0 on one side of the boundary and q0,0, q1,0, q2,0, q3,0, q4,0, q5,0 on the other side of the boundary, the second set of inputs are from a second line across the block boundary including samples p0,1, p1,1, p2,1, p3,1, p4,1, p5,1 and q0,1, q1,1, q2,1, q3,1, q4,1, q5,1 on the other side of the boundary, the third set of inputs are from a third line across the block boundary including samples p0,2, p1,2, p2,2, p3,2, p4,2, p5,2 and q0,2, q1,2, q2,2, q3,2, q4,2, q5,2 on the other side of the boundary, and the fourth set of inputs are from a fourth line across the block boundary including samples p0,3, p1,3, p2,3, p3,3, p4,3, p5,3 and q0,3, q1,3, q2,3, q3,3, q4,3, q5,3 on the other side of the boundary, the first set of output samples p′0,0, p′1,0, p′2,0, p′3,0, p′4,0, q′0,0, q′1,0, q′2,0, q′3,0, q′4,0, are derived from the first set of input samples, the second set of output samples p′0,1, p′1,1, p′2,1, p′3,1, p′4,1, q′0,1, q′1,1, q′2,1, q′3,1, q′4,1, are derived from the second set of input samples, the third set of output samples p′0,2, p′1,2, p′2,2, p′3,2, p′4,2, q′0,2, q′1,2, q′2,2, q′3,2, q′4,2, are derived from the third set of input samples, and the fourth set of output samples, p′0,3, p′1,3, p′2,3, p′3,3, p′4,3, q′0,3, q′1,3, q′2,3, q′3,3, q′4,3 are derived from the fourth set of input samples.
A8. The method of any one of embodiments A1-A7, wherein the first set of inputs are from a first line across the block boundary including samples p0,0, p1,0, p2,0, p3,0, p4,0, p5,0, p6,0, p7,0 on one side of the boundary and q0,0, q1,0, q2,0, q3,0 on the other side of the boundary, the second set of inputs are from a second line across the block boundary including samples p0,1, p1,1, p2,1, p3,1, p4,1, p5,1, p6,1, p7,1 and q0,1, q1,1, q2,1, q3,1 on the other side of the boundary, the third set of inputs are from a third line across the block boundary including samples p0,2, p1,2, p2,2, p3,2, p4,2, p5,2, p6,2, p7,2 and q0,2, q1,2, q2,2, q3,2 on the other side of the boundary, and the fourth set of inputs are from a fourth line across the block boundary including samples p0,3, p1,3, p2,3, p3,3, p4,3, p5,3, p6,3, p7,3 and q0,3, q1,3, q2,3, q3,3 on the other side of the boundary, the first set of output samples p′0,0, p′1,0, p′2,0, p′3,0, p′4,0, p′5,0, p′6,0, q′0,0, q′1,0, q′2,0, are derived from the first set of input samples, the second set of output samples p′0,1, p′1,1, p′2,1, p′3,1, p′4,1, p′5,1, p′6,1, q′0,2, q′1,2, q′2,2 are derived from the second set of input samples, the third set of output samples p′0,2, p′1,2, p′2,2, p′3,2, p′4,2, p′5,2, p′6,2, q′0,2, q′1,2, q′2,2, are derived from the third set of input samples, and the fourth set of output samples, p′0,3, p′1,3, p′2,3, p′3,3, p′4,3, p′5,3, p′6,3, q′0,3, q′1,3, q′2,3 are derived from the fourth set of input samples.
A9. The method of any one of embodiments A1-A8, wherein the first set of inputs are from a first line across the block boundary including samples p0,0, p1,0, p2,0, p3,0, p4,0, p5,0 on one side of the boundary and q0,0, q1,0, q2,0, q3,0 on the other side of the boundary, the second set of inputs are from a second line across the block boundary including samples p0,1, p1,1, p2,1, p3,1, p4,1, p5,1 and q0,1, q1,1, q2,1, q3,1 on the other side of the boundary, the third set of inputs are from a third line across the block boundary including samples p0,2, p1,2, p2,2, p3,2, p4,2, p5,2 and q0,2, q1,2, q2,2, q3,2 on the other side of the boundary, and the fourth set of inputs are from a fourth line across the block boundary including samples p0,3, p1,3, p2,3, p3,3, p4,3, p5,3 and q0,3, q1,3, q2,3, q3,3 on the other side of the boundary, the first set of output samples p′0,0, p′1,0, p′2,0, p′3,0, p′4,0, q′0,1, q′1,1, q′2,1, are derived from the first set of input samples, the second set of output samples p′0,1, p′1,1, p′2,1, p′3,1, p′4,1, q′0,2, q′1,2, q′2,2 are derived from the second set of input samples, the third set of output samples p′0,2, p′1,2, p′2,2, p′3,2, p′4,2, q′0,2, q′1,2, q′2,2, are derived from the third set of input samples, and the fourth set of output samples, p′0,3, p′1,3, p′2,3, p′3,3, p′4,3, q′0,3, q′1,3, q′2,3 are derived from the fourth set of input samples.
A10. The method of any one of embodiments A1-A9, wherein the first set of inputs are from a first line across the block boundary including samples p0,0, p1,0, p2,0, p3,0 on one side of the boundary and q0,0, q1,0, q2,0, q3,0 on the other side of the boundary, the second set of inputs are from a second line across the block boundary including samples p0,1, p1,1, p2,1, p3,1 and q0,1, q1,1, q2,1, q3,1 on the other side of the boundary, the third set of inputs are from a third line across the block boundary including samples p0,2, p1,2, p2,2, p3,2 and q0,2, q1,2, q2,2, q3,2 on the other side of the boundary, the fourth set of inputs are from a fourth line across the block boundary including samples p0,3, p1,3, p2,3, p3,3 and q0,3, q1,3, q2,3, q3,3 on the other side of the boundary, the first set of output samples p′0,0, p′1,0, p′2,0, q′0,0, q′1,0, q′2,0 are derived from the first set of input samples, the second set of output samples p′0,1, p′1,1, p′2,1, q′0,1, q′1,1, q′2,1, are derived from the second set of input samples, the third set of output samples p′0,2, p′1,2, p′2,2, q′0,2, q′1,2, q′2,2, are derived from the third set of input samples, and the fourth set of output samples, p′0,3, p′1,3, p′2,3, q′0,3, q′1,3, q′2,3 are derived from the fourth set of input samples.
A11. The method of any of the embodiments A1-A10, wherein the block edge decision process for each input set, k, of samples is based on an edge metrics Abs(p2,k−2*p1,k+p0,k), Abs(q2,k−2*q1,k+q0,k), Abs(p3,k−p0,k), Abs(q3,k−q0,k), Abs(p0,k−q0,k).
A12. The method of A11, wherein the block edge decision process for each input set, k, of samples is based on at least one of the edge metrics Abs(p5, k−2*p4,k+p3,k), Abs(p3,k−p5,k) or Abs(q5,k−2*q4,k+q3,k), Abs(q3,k−q5,k).
A13. The method of any of the embodiments A12, wherein the block edge decision process for each input set, k, of samples is based on at least one of the edge metrics Abs(p4,k−p5,k−p6,k+p7,k), Abs(p3,k−p7,k) or Abs(q4,k−q5,k−q6,k+q7,k) Abs(q3,k−q7,k).
B1. A computer program 543 comprising instructions 544 which when executed by processing circuitry 502 causes the processing circuitry 502 to perform the method of any one of the above embodiments.
B2. A carrier containing the computer program of embodiment B1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium 542.
C1. An apparatus 501, the apparatus being adapted to perform the method of any one of embodiments A1-A13.
D1. An apparatus 501, the apparatus comprising: processing circuitry 502; and a memory 542, said memory containing instructions 544 executable by said processing circuitry, whereby said apparatus is operative to perform the method of any one of the embodiments A1-A13.
Complexity Analysis
Long deblocking filter can be used when at least one side has a width for vertical boundaries or height for horizontal boundaries of 32 samples or more and the other side has width for vertical boundaries or height for horizontal boundaries of 8 samples or more.
The current long filter decision for that case has for each 4 samples boundary segment to make calculations used for both long and strong deblocking filter but also calculations specific for long filter decision and then also calculations specific for strong filter decision in case long filter not is selected. In total about 2 segments*(28+46+14)/(32*8)=2*88/(32*8)=176/(32*8)=0.69 operations per sample.
The additional complexity for line 1 and 2 are 2 segments*(28+46−2)/(32*8)=2*72=144/(32*8)=0.56 operations per sample.
The total number of operations after the suggested fix is 0.69+0.56=1.25 operations per sample.
The worst case for decision calculations is not modified since that happens when all blocks are 8×8 where the number of operations per sample for decisions are about 2 segments*(28+14)/(8*8)=2*42/64=1.28 operations per sample.
Calculations in general for both long and strong deblocking filter (10 abs+4 shifts+14 adds=28 op):
Additional calculations for long filter of one side (6 abs+6 shifts+27 adds+7 cmp=46 op):
Additional calculations for strong filter (7 adds+7 cmp=14op):
Results
The objective performance for the proposal is shown below in comparison to VTM-8.0 using CTC. The objective results show a small gain. Encoding time is measured on simulations run on a cluster with machines with same capability but are not reliable. Decoding time is measured by running test and anchor on same machine without yuv output. Encoding and decoding time is similar as for the anchor.
It is proposed to fix the long luma deblocking decision such that not only lines 0 and 3 are checked but also lines 1 and 2 are checked before application of long luma deblocking filtering. It is proposed to include the fix to the specification and software to reduce the risk of over filtering of luma samples of lines 1 and 2 which can be achieved without increasing worst case complexity for deblocking decisions.
While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.
Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2021/058381 | 3/30/2021 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63004130 | Apr 2020 | US |
