Coding and decoding of video coding modes

Description

TECHNICAL FIELD

This disclosure is related to video and image coding and decoding technologies.

BACKGROUND

Digital video accounts for the largest bandwidth use on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, it is expected that the bandwidth demand for digital video usage will continue to grow.

SUMMARY

In one example aspect, a method of processing video is disclosed. The method includes performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion of the current block includes determining whether a use of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique to the current block is enabled or disabled, and wherein the determining the use of the BIO technique or the DMVR technique is based on a cost criterion associated with the current block.

In another example aspect, a method of processing video is disclosed. The method includes performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion of the current block includes determining whether a use of a decoder-side motion vector refinement (DMVR) technique to the current block is enabled or disabled, and wherein the DMVR technique includes refining motion information of the current block based on a cost criterion other than a mean removed sum of absolute differences (MRSAD) cost criterion.

In another example aspect, a method of processing video is disclosed. The method includes performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion of the current block includes determining whether a use of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique to the current block is enabled or disabled, and wherein the determining the use of the BIO technique or the DMVR technique is based on computing that a mean value difference of a pair of reference blocks associated with the current block exceeds a threshold value.

In another example aspect, a method of processing video is disclosed. The method includes modifying a first reference block to generate a first modified reference block, and a second reference block to generate a second modified reference block, wherein both the first reference block and the second reference block are associated with a current block of visual media data; determining differences between the first modified reference block and the second modified reference block, the differences including one or more of: a sum of absolute transformed differences (SATD), a mean removed sum of absolute transformed differences (MRSATD), a sum of squares error (SSE), a mean removed sum of squares error (MRSSE), a mean value difference, or gradient values; and performing a conversion between the current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion includes a use of the differences between the first modified reference block and the second modified reference block generated from respectively modifying the first reference block and the second reference block.

In another example aspect, a method of processing video is disclosed. The method includes determining a temporal gradient or a modified temporal gradient using reference pictures associated with a current block of visual media data, the temporal gradient or the modified temporal gradient indicative of differences between the reference pictures; and performing a conversion between the current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion includes a use of a bi-directional optical flow (BIO) technique based in part on the temporal gradient or the modified temporal gradient.

In another example aspect, a method of processing video is disclosed. The method includes determining a first temporal gradient using reference pictures associated with a first video block or a sub-block thereof; determining a second temporal gradient using reference pictures associated with a second video block or a sub-block thereof; performing a modification of the first temporal gradient and a modification of the second temporal gradient to generate a modified first temporal gradient and a modified second temporal gradient, wherein the modification of the first temporal gradient associated with the first video block is different from the modification of the second temporal gradient associated with the second video block; and performing a conversion of the first video block and the second video block to their corresponding coded representation.

In another example aspect, a method of processing video is disclosed. The method includes modifying one or both of a first inter reference block and a second inter reference block associated with a current block; determining, based on using the one or both modified first inter reference block and/or the modified second inter reference block, a spatial gradient associated with the current block in accordance with applying a bi-directional optical (BIO) flow technique; and performing a conversion between the current block and a corresponding coded representation, wherein the conversion includes a use of the spatial gradient associated with the current block.

In another example aspect, a method of processing video is disclosed. The method includes performing a determination, by a processor, that a flag which can be signaled at multiple levels indicates, at least in part, that one or both of a decoder-side motion vector refinement (DMVR) technique or a bi-directional optical flow (BIO) technique is to be enabled for a current block; and performing a conversion between the current block and a corresponding coded representation, wherein the coded representation includes the flag indicating whether the one or both of the DMVR technique and/or the BIO technique is enabled.

In another example aspect, a method of processing video is disclosed. The method includes performing a determination, by a processor that a decoder-side motion vector refinement (DMVR) technique is to be enabled for a current block, wherein the determination is based exclusively on a height of the current block; and performing a conversion between the current block and a corresponding coded representation.

In another example aspect, a method of processing video is disclosed. The method includes performing a conversion between a current block of visual media data and a corresponding coded representation of visual media data, wherein the conversion includes a use of rules associated with one or both of a decoder-side motion vector refinement (DMVR) technique or a bi-directional optical flow (BIO) technique on the current block, wherein the rules associated with the DMVR technique are consistent with application to the BIO technique; and wherein determining whether the use of the one or both of the BIO technique or the DMVR technique on the current block is enabled or disabled is based on applying the rules.

In another example aspect, the above-described methods may be implemented by a video decoder apparatus that comprises a processor.

In another example aspect, the above-described methods may be implemented by a video encoder apparatus that comprises a processor.

In yet another example aspect, these methods may be embodied in the form of processor-executable instructions and stored on a computer-readable program medium.

These, and other, aspects are further described in the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of bilateral matching.

FIG. 2 shows an example of template matching.

FIG. 3 shows an example of unilateral motion estimation (ME) in Frame-Rate Up Conversion (FRUC).

FIG. 4 shows an example of optical flow trajectory.

FIGS. 5A and 5B show examples of bi-directional optical flow (BIO) without block extension.

FIG. 6 shows an example of bilateral matching with 6 points search.

FIG. 7 shows examples of an adaptive integer search pattern and a half sample search pattern.

FIG. 8 is a block diagram of an example of a video processing apparatus.

FIG. 9 shows a block diagram of an example implementation of a video encoder.

FIG. 10 is a flowchart for an example of a video processing method.

FIG. 11 is a flowchart for an example of a video processing method.

FIG. 12 is a flowchart for an example of a video processing method.

FIG. 13 is a flowchart for an example of a video processing method.

FIG. 14 is a flowchart for an example of a video processing method.

FIG. 15 is a flowchart for an example of a video processing method.

FIG. 16 is a flowchart for an example of a video processing method.

FIG. 17 is a block diagram of an example video processing system in which disclosed techniques may be implemented.

FIG. 18 is a flowchart for an example of a video processing method.

FIG. 19 is a flowchart for an example of a video processing method.

FIG. 20 is a flowchart for an example of a video processing method.

FIG. 21 is a flowchart for an example of a video processing method.

FIG. 22 is a flowchart for an example of a video processing method.

FIG. 23 is a flowchart for an example of a video processing method.

FIG. 24 is a flowchart for an example of a video processing method.

FIG. 25 is a flowchart for an example of a video processing method.

FIG. 26 is a flowchart for an example of a video processing method.

FIG. 27 is a flowchart for an example of a video processing method.

DETAILED DESCRIPTION

To improve compression ratio of video, researchers are continually looking for new techniques by which to encode video. The present disclosure provides various techniques that can be used by a decoder of video bitstreams to improve the quality of decompressed or decoded digital video. Furthermore, a video encoder may also implement these techniques during the process of encoding in order to reconstruct decoded frames used for further encoding.

Section headings are used in the present disclosure for improving readability and do not limit the scope of techniques and embodiments described in each section only to that section. Furthermore, while certain terms from various existing video codec standards are used, the disclosed technologies are not limited only to these video standards or their successors and are applicable to other video codec standards. Furthermore, in some cases, techniques are disclosed using corresponding coding steps, and it will be understood that, at a decoder, the corresponding decoding steps in reverse order will be performed. In addition, coding may also be used to perform transcoding in which a video is represented from one coded representation (e.g., one bitrate) to another coded representation (e.g., a different bitrate).

1. SUMMARY

The present disclosure is related to video coding technologies. Specifically, it is related to motion compensation in video coding. It may be applied to the existing video coding standard like High Efficiency Video Coding (HEVC), or the standard Versatile Video Coding (VVC) to be finalized. It may be also applicable to future video coding standards or video codec.

2. BACKGROUND

Video coding standards have evolved primarily through the development of the well-known International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) and International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) standards. The ITU-T produced H.261 and H.263, ISO/IEC produced Moving Picture Experts Group (MPEG)-1 and MPEG-4 Visual, and the two organizations jointly produced the H.262/MPEG-2 Video and H.264/MPEG-4 Advanced Video Coding (AVC) and H.265/HEVC standards. Since H.262, the video coding standards are based on the hybrid video coding structure wherein temporal prediction plus transform coding are utilized. To explore the future video coding technologies beyond HEVC, Joint Video Exploration Team (JVET) was founded by Video Coding Experts Group (VCEG) and MPEG jointly in 2015. Since then, many new methods have been adopted by JVET and put into the reference software named Joint Exploration Model (JEM). In April 2018, the Joint Video Expert Team (JVET) between VCEG (Q6/16) and ISO/IEC Joint Technical Committee (JTC1) SC29/WG11 (MPEG) was created to work on the VVC standard targeting at 50% bitrate reduction compared to HEVC.

The latest version of VVC draft, i.e., Versatile Video Coding (Draft 2) could be found at: http://phenix.it-sudparis.eu/jvet/doc_end_user/documents/11_Ljubljana/wg11/JVET-K1001-v7.zip. The latest reference software of VVC, named VTM, could be found at: https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/tags/VTM-2.1.

FIG. 9 is a block diagram of an example implementation of a video encoder. FIG. 9 shows that the encoder implementation has a feedback path built in in which the video encoder also performs video decoding functionality (reconstructing compressed representation of video data for use in encoding of next video data).

2.1 Pattern Matched Motion Vector Derivation

Pattern matched motion vector derivation (PMMVD) mode is a special merge mode based on Frame-Rate Up Conversion (FRUC) techniques. With this mode, motion information of a block is not signalled but derived at decoder side.

A FRUC flag is signalled for a coding unit (CU) when its merge flag is true. When the FRUC flag is false, a merge index is signalled and the regular merge mode is used. When the FRUC flag is true, an additional FRUC mode flag is signalled to indicate which method (bilateral matching or template matching) is to be used to derive motion information for the block.

At encoder side, the decision on whether using FRUC merge mode for a CU is based on rate distortion (RD) cost selection as done for normal merge candidate. That is the two matching modes (bilateral matching and template matching) are both checked for a CU by using RD cost selection. The one leading to the minimal cost is further compared to other CU modes. If a FRUC matching mode is the most efficient one, FRUC flag is set to true for the CU and the related matching mode is used.

Motion derivation process in FRUC merge mode has two steps. A CU-level motion search is first performed, then followed by a Sub-CU level motion refinement. At CU level, an initial motion vector is derived for the whole CU based on bilateral matching or template matching. First, a list of motion vector (MV) candidates is generated and the candidate which leads to the minimum matching cost is selected as the starting point for further CU level refinement. Then a local search based on bilateral matching or template matching around the starting point is performed and the MV results in the minimum matching cost is taken as the MV for the whole CU. Subsequently, the motion information is further refined at sub-CU level with the derived CU motion vectors as the starting points.

For example, the following derivation process is performed for a W×H CU motion information derivation. At the first stage, MV for the whole W×H CU is derived. At the second stage, the CU is further split into M×M sub-CUs. The value of M is calculated as in (16), D is a predefined splitting depth which is set to 3 by default in the JEM. Then the MV for each sub-CU is derived as:

$\begin{matrix} M = \max {4, \min {\frac{M}{2^{D}}, \frac{N}{2^{D}}}} & (1) \end{matrix}$

As shown in FIG. 1, the bilateral matching is used to derive motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. Under the assumption of continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks shall be proportional to the temporal distances, i.e., TD0 and TD1, between the current picture and the two reference pictures. As a special case, when the current picture is temporally between the two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, the bilateral matching becomes mirror based bi-directional MV.

As shown in FIG. 2, template matching is used to derive motion information of the current CU by finding the closest match between a template (top and/or left neighbouring blocks of the current CU) in the current picture and a block (same size to the template) in a reference picture. Except the aforementioned FRUC merge mode, the template matching is also applied to advanced motion vector prediction (AMVP) mode. In the JEM, as done in HEVC, AMVP has two candidates. With template matching method, a new candidate is derived. If the newly derived candidate by template matching is different to the first existing AMVP candidate, it is inserted at the very beginning of the AMVP candidate list and then the list size is set to two (meaning remove the second existing AMVP candidate). When applied to AMVP mode, only CU level search is applied.

CU Level MV Candidate Set

The MV candidate set at CU level can include:

- Original AMVP candidates if the current CU is in AMVP mode,
- all merge candidates,
- several MVs in the interpolated MV field, which is introduced in section 2.1.1.3, and
- top and left neighbouring motion vectors

When using bilateral matching, each valid MV of a merge candidate is used as an input to generate a MV pair with the assumption of bilateral matching. For example, one valid MV of a merge candidate is (MVa, refa) at reference list A. Then the reference picture refb of its paired bilateral MV is found in the other reference list B so that refa and refb are temporally at different sides of the current picture. If such a refb is not available in reference list B, refb is determined as a reference which is different from refa and its temporal distance to the current picture is the minimal one in list B. After refb is determined, MVb is derived by scaling MVa based on the temporal distance between the current picture and refa, refb.

Four MVs from the interpolated MV field are also added to the CU level candidate list. More specifically, the interpolated MVs at the position (0, 0), (W/2, 0), (0, H/2) and (W/2, H/2) of the current CU are added.

When FRUC is applied in AMVP mode, the original AMVP candidates are also added to CU level MV candidate set.

At the CU level, up to 15 MVs for AMVP CUs and up to 13 MVs for merge CUs are added to the candidate list.

Sub-CU Level MV Candidate Set

The MV candidate set at sub-CU level can include:

- an MV determined from a CU-level search,
- top, left, top-left and top-right neighbouring MVs,
- scaled versions of collocated MVs from reference pictures,
- up to 4 advanced temporal motion vector prediction (ATMVP) candidates, and
- up to 4 STMVP candidates.

The scaled MVs from reference pictures are derived as follows. All the reference pictures in both lists are traversed. The MVs at a collocated position of the sub-CU in a reference picture are scaled to the reference of the starting CU-level MV.

ATMVP and spatial temporal motion vector prediction (STMVP) candidates are limited to the four first ones.

At the sub-CU level, up to 17 MVs are added to the candidate list.

Generation of Interpolated MV Field

Before coding a frame, interpolated motion field is generated for the whole picture based on unilateral motion estimation (ME). Then the motion field may be used later as CU level or sub-CU level MV candidates.

First, the motion field of each reference pictures in both reference lists is traversed at 4×4 block level. For each 4×4 block, if the motion associated to the block passing through a 4×4 block in the current picture (as shown in FIG. 3) and the block has not been assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distance TD0 and TD1 (the same way as that of MV scaling of temporal motion vector prediction (TMVP) in HEVC) and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to a 4×4 block, the block's motion is marked as unavailable in the interpolated motion field.

Interpolation and Matching Cost

When a motion vector points to a fractional sample position, motion compensated interpolation can be performed. To reduce complexity, bi-linear interpolation instead of regular 8-tap HEVC interpolation is used for both bilateral matching and template matching.

The calculation of matching cost is a bit different at different steps. When selecting the candidate from the candidate set at the CU level, the matching cost is the absolute sum difference (SAD) of bilateral matching or template matching. After the starting MV is determined, the matching cost C of bilateral matching at sub-CU level search is calculated as follows:

C=SAD+w·(|MV_x−MV_x^s|+|MV_y−MV_y^s|) (2)

where w is a weighting factor which is empirically set to 4, MV and MV^sindicate the current MV and the starting MV, respectively. SAD is still used as the matching cost of template matching at sub-CU level search.

In FRUC mode, MV is derived by using luma samples only. The derived motion will be used for both luma and chroma for motion compensation (MC) inter prediction. After MV is decided, final MC is performed using 8-taps interpolation filter for luma and 4-taps interpolation filter for chroma.

MV Refinement

MV refinement is a pattern based MV search with the criterion of bilateral matching cost or template matching cost. In the JEM, two search patterns are supported—an unrestricted center-biased diamond search (UCBDS) and an adaptive cross search for MV refinement at the CU level and sub-CU level, respectively. For both CU and sub-CU level MV refinement, the MV is directly searched at quarter luma sample MV accuracy, and this is followed by one-eighth luma sample MV refinement. The search range of MV refinement for the CU and sub-CU step are set equal to 8 luma samples.

Selection of Prediction Direction in Template Matching FRUC Merge Mode

In the bilateral matching merge mode, bi-prediction is always applied since the motion information of a CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. There is no such limitation for the template matching merge mode. In the template matching merge mode, the encoder can choose among uni-prediction from list0, uni-prediction from list1 or bi-prediction for a CU. The selection is based on a template matching cost as follows:

If costBi <= factor * min (cost0, cost1)

bi-prediction is used;

Otherwise, if cost0 <= cost1

uni-prediction from list0 is used;

Otherwise,

uni-prediction from list1 is used;

where cost0 is the SAD of list0 template matching, cost1 is the SAD of list1 template matching and costBi is the SAD of bi-prediction template matching. The value of factor is equal to 1.25, which means that the selection process is biased toward bi-prediction.

The inter prediction direction selection is only applied to the CU-level template matching process.

Hybrid Intra and Inter Prediction

In JVET-L0100, multi-hypothesis prediction is proposed, wherein hybrid intra and inter prediction is one way to generate multiple hypotheses.

When the multi-hypothesis prediction is applied to improve intra mode, multi-hypothesis prediction combines one intra prediction and one merge indexed prediction. In a merge CU, one flag is signaled for merge mode to select an intra mode from an intra candidate list when the flag is true. For luma component, the intra candidate list is derived from 4 intra prediction modes including direct current (DC), planar, horizontal, and vertical modes, and the size of the intra candidate list can be 3 or 4 depending on the block shape. When the CU width is larger than the double of CU height, horizontal mode is exclusive of the intra mode list and when the CU height is larger than the double of CU width, vertical mode is removed from the intra mode list. One intra prediction mode selected by the intra mode index and one merge indexed prediction selected by the merge index are combined using weighted average. For chroma component, direct mode (DM) is always applied without extra signaling. The weights for combining predictions are described as follow. When DC or planar mode is selected, or the coding block (CB) width or height is smaller than 4, equal weights are applied. For those CBs with CB width and height larger than or equal to 4, when horizontal/vertical mode is selected, one CB is first vertically/horizontally split into four equal-area regions. Each weight set, denoted as (w_intra_i, w_inter_i), where i is from 1 to 4 and (w_intra₁, w_inter₁)=(6, 2), (w_intra₂, w_inter₂)=(5, 3), (w_intra₃, w_inter₃)=(3, 5), and (w_intra₄, w_inter₄)=(2, 6), will be applied to a corresponding region. (w_intra₁, w_inter₁) is for the region closest to the reference samples and (w_intra₄, w_inter₄) is for the region farthest away from the reference samples. Then, the combined prediction can be calculated by summing up the two weighted predictions and right-shifting 3 bits. Moreover, the intra prediction mode for the intra hypothesis of predictors can be saved for reference of the following neighboring CUs.

Bi-Directional Optical Flow

In BIO, motion compensation is first performed to generate the first predictions (in each prediction direction) of the current block. The first predictions are used to derive the spatial gradient, the temporal gradient and the optical flow of each subblock/pixel within the block, which are then used to generate the second prediction, i.e., the final prediction of the subblock/pixel. The details are described as follows.

Bi-directional Optical flow (BIO) is sample-wise motion refinement which is performed on top of block-wise motion compensation for bi-prediction. The sample-level motion refinement doesn't use signalling.

Let I^(k)be the luma value from reference k (k=0, 1) after block motion compensation, and ∂I^(k)/∂x, ∂I^(k)/∂y are horizontal and vertical components of the I^(k)gradient, respectively. Assuming the optical flow is valid, the motion vector field (v_x, v_y) is given by the equation:

∂I^(k)∂t+v_x∂I^(k)/∂x+v_y∂I^(k)/∂y=0. (3)

Combining this optical flow equation with Hermite interpolation for the motion trajectory of each sample results in a unique third-order polynomial that matches both the function values I^(k)and derivatives ∂I^(k)/∂x, ∂I^(k)/∂y at the ends. The value of this polynomial at t=0 is the BIO prediction:

pred_BIO=½·(I⁽⁰⁾+I⁽¹⁾+v_x/2·(τ₁∂I⁽⁰⁾/∂x)+v_y/2·(τ₁∂I⁽¹⁾/∂y−τ₀∂I⁽⁰⁾/∂y)). (4)

Here, τ_oand τ₁denote the distances to the reference frames as shown on FIG. 4. Distances τ_oand τ₁are calculated based on picture order count (POC) for Ref0 and Ref1: τ₀=POC(current)−POC(Ref0), τ₁=POC(Ref1)−POC(current). If both predictions come from the same time direction (either both from the past or both from the future) then the signs are different (i.e., τ_o·τ₁<o). In this case, BIO is applied only if the prediction is not from the same time moment (i.e., τ_o≠τ₁), both referenced regions have non-zero motion (MVx_o, MVy_o, MVx₁, MVy₁≠o) and the block motion vectors are proportional to the time distance (MVx_o/MVx₁=MVy_o/MVy₁=−τ_o/τ₁).

The motion vector field (v_x, v_y) is determined by minimizing the difference Δ between values in points A and B (intersection of motion trajectory and reference frame planes on FIG. 9). Model uses only first linear term of a local Taylor expansion for Δ as:

Δ=(I⁽⁰⁾−I⁽¹⁾₀+v_x(τ₁∂I⁽¹⁾∂x+τ₀∂I⁽⁰⁾/∂x)v_y(τ₁∂I⁽¹⁾/∂y+τ₀∂I⁽⁰⁾∂y)) (5)

All values in Equation 5 depend on the sample location (i′, j′), which was omitted from the notation so far. Assuming the motion is consistent in the local surrounding area, the value of Δ can be minimized inside the (2M+1)×(2M+1) square window Ω centered on the currently predicted point (i, j), where M is equal to 2:

$\begin{matrix} (v_{x}, v_{y}) = \underset{v_{x}, v_{y}}{\arg \min} \sum_{[i^{'}, j] \in Ω} Δ^{2} [i^{'}, j^{'}] & (6) \end{matrix}$

For this optimization problem, the JEM uses a simplified approach making first a minimization in the vertical direction and then in the horizontal direction. This results in

$\begin{matrix} v_{x} = (s_{1} + r) > m ? clip 3 (- thBIO, thBIO, - \frac{s_{3}}{(s_{1} + r)}) : 0 & (7) \end{matrix}$

$\begin{matrix} v_{y} = (s_{5} + r) > m ? clip 3 (- thBIO, thBIO, - \frac{s_{6} - v_{x} s_{2} / 2}{(s_{5} + r)}) : 0 & (8) \end{matrix}$

$where,$

$\begin{matrix} s_{1} = \sum_{[i^{'}, j] \in Ω} {(τ_{1} \partial I^{(1)} / \partial x + τ_{0} \partial I^{(0)} / \partial x)}^{2}; s_{3} = \sum_{[i^{'}, j] \in Ω} (I^{(1)} - I^{(0)}) (τ_{1} \partial I^{(1)} / \partial x + τ_{0} \partial I^{(0)} / \partial x); & (9) \end{matrix}$

$s_{2} = \sum_{[i^{'}, j] \in Ω} (τ_{1} \partial I^{(1)} / \partial x + τ_{0} \partial I^{(0)} / \partial x) (τ_{1} \partial I^{(1)} / \partial y + τ_{0} \partial I^{(0)} / \partial y);$

$s_{5} = \sum_{[i^{'}, j] \in Ω} {(τ_{1} \partial I^{(1)} / \partial y + τ_{0} \partial I^{(0)} / \partial y)}^{2}; s_{6} = \sum_{[i^{'}, j] \in Ω} (I^{(1)} - I^{(0)}) (τ_{1} \partial I^{(1)} / \partial y + τ_{0} \partial I^{(0)} / \partial y)$

In order to avoid division by zero or a very small value, regularization parameters r and m are introduced in Equations 7 and 8 where:

r=500·4^3d-8 (10)
m=700·4^d-8 (11)

Here d is bit depth of the video samples.

In order to keep the memory access for BIO the same as for regular bi-predictive motion compensation, all prediction and gradients values, I^(k), ∂I^(k)/∂x, ∂I^(k)/∂y, are calculated only for positions inside the current block. In Equation 9, (2M+1)×(2M×1) square window Ω centered in currently predicted point on a boundary of predicted block can access positions outside of the block as shown in FIG. 5A. In the JEM, values of I^(k), ∂I^(k)/∂x, ∂I^(k)/∂y outside of the block are set to be equal to the nearest available value inside the block. For example, this can be implemented as padding, as shown in FIG. 5B.

With BIO, it's possible that the motion field can be refined for each sample. To reduce the computational complexity, a block-based design of BIO is used in the JEM. The motion refinement is calculated based on 4×4 block. In the block-based BIO, the values of s_nin Equation 9 of all samples in a 4×4 block are aggregated, and then the aggregated values of s_nin are used to derived BIO motion vectors offset for the 4×4 block. More specifically, the following formula is used for block-based BIO derivation:

$\begin{matrix} s_{1, b_{k}} = \sum_{(x, y) \in b_{k}} \sum_{[i^{'}, j] \in Ω (x, y)} {(τ_{1} \partial I^{(1)} / \partial x + τ_{0} \partial I^{(0)} / \partial x)}^{2}; s_{3, b_{k}} = \sum_{(x, y) \in b_{k}} \sum_{[i^{'}, j] \in Ω} (I^{(1)} - I^{(0)}) (τ_{1} \partial I^{(1)} / \partial x + τ_{0} \partial I^{(0)} / \partial x); & (12) \end{matrix}$

$s_{2, b_{k}} = \sum_{(x, y) \in b_{k}} \sum_{[i^{'}, j] \in Ω} (τ_{1} \partial I^{(1)} / \partial x + τ_{0} \partial I^{(0)} / \partial x) (τ_{1} \partial I^{(1)} / \partial y + τ_{0} \partial I^{(0)} / \partial y);$

$s_{5, b_{k}} = \sum_{(x, y) \in b_{k}} \sum_{[i^{'}, j] \in Ω} {(τ_{1} \partial I^{(1)} / \partial y + τ_{0} \partial I^{(0)} / \partial y)}^{2}; s_{6, b_{k}} = \sum_{(x, y) \in b_{k}} \sum_{[i^{'}, j] \in Ω} (I^{(1)} - I^{(0)}) (τ_{1} \partial I^{(1)} / \partial y + τ_{0} \partial I^{(0)} / \partial y)$

where b_kdenotes the set of samples belonging to the k-th 4×4 block of the predicted block. s_nin Equations 7 and 8 are replaced by ((s_n,bk)>>4) to derive the associated motion vector offsets.

In some cases, MV regiment of BIO might be unreliable due to noise or irregular motion. Therefore, in BIO, the magnitude of MV regiment is clipped to a threshold value thBIO. The threshold value is determined based on whether the reference pictures of the current picture are all from one direction. If all the reference pictures of the current picture are from one direction, the value of the threshold is set to 12×2^14-d; otherwise, it is set to 12×2^13-d.

Gradients for BIO are calculated at the same time with motion compensation interpolation using operations consistent with HEVC motion compensation process (two dimensional (2D) separable finite impulse response (FIR)). The input for this 2D separable FIR is the same reference frame sample as for motion compensation process and fractional position (fracX, fracY) according to the fractional part of block motion vector. In case of horizontal gradient al lax signal first interpolated vertically using BIOfilterS corresponding to the fractional position fracY with de-scaling shift d−8, then gradient filter BIOfilterG is applied in horizontal direction corresponding to the fractional position fracX with de-scaling shift by 18−d. In case of vertical gradient ∂I/∂y day first gradient filter is applied vertically using BIOfilterG corresponding to the fractional position fracY with de-scaling shift d−8, then signal displacement is performed using BIOfilterS in horizontal direction corresponding to the fractional position fracX with de-scaling shift by 18−d. The length of interpolation filter for gradients calculation BIOfilterG and signal displacement BIOfilterF is shorter (6-tap) in order to maintain reasonable complexity. Table 1 shows the filters used for gradients calculation for different fractional positions of block motion vector in BIO. Table 2 shows the interpolation filters used for prediction signal generation in BIO.

TABLE 1

Filters for gradients calculation in BIO

Fractional
Interpolation filter for

pel position
gradient(BIOfilterG)

0
{ 8, −39, −3, 46, −17, 5}

1/16
{ 8, −32, −13, 50, −18, 5}

⅛
{ 7, −27, −20, 54, −19, 5}

3/16
{ 6, −21, −29, 57, −18, 5}

¼
{ 4, −17, −36, 60, −15, 4}

5/16
{ 3, −9, −44, 61, −15, 4}

⅜
{ 1, −4, −48, 61, −13, 3}

7/16
{ 0, 1, −54, 60, −9, 2}

½
{ −1, 4, −57, 57, −4, 1}

TABLE 2

Interpolation filters for prediction signal generation in BIO

Fractional
Interpolation filter for prediction

pel position
signal(BIOfilterS)

0
{ 0, 0, 64, 0, 0, 0}

1/16
{ 1, −3, 64, 4, −2, 0}

⅛
{ 1, −6, 62, 9, −3, 1}

3/16
{ 2, −8, 60, 14, −5, 1}

¼
{ 2, −9, 57, 19, −7, 2}

5/16
3, −10, 53, 24, −8, 2}

⅜
{ 3, −11, 50, 29, −9, 2}

7/16
{ 3, −11, 44, 35, −10, 3}

½
{ 3, −10, 35, 44, −11, 3}

In the JEM, BIO is applied to all bi-predicted blocks when the two predictions are from different reference pictures. When local intensity compensation (LIC) is enabled for a CU, BIO is disabled.

In the JEM, overlapped block motion compensation (OBMC) is applied for a block after normal MC process. To reduce the computational complexity, BIO is not applied during the OBMC process. This means that BIO is only applied in the MC process for a block when using its own MV and is not applied in the MC process when the MV of a neighboring block is used during the OBMC process.

A two-stage early termination method is used to conditionally disable the BIO operations depending on the similarity between the two prediction signals. The early termination is first applied at the CU-level and then at the sub-CU-level. Specifically, the proposed method first calculates the SAD between the L0 and L1 prediction signals at the CU level. Given that the BIO is only applied to luma, only the luma samples can be considered for the SAD calculation. If the CU-level SAD is no larger than a predefined threshold, the BIO process is completely disabled for the whole CU. The CU-level threshold is set to 2^(BDepth-9)per sample. If the BIO process is not disabled at the CU level, and if the current CU includes multiple sub-CUs, the SAD of each sub-CU inside the CU will be calculated. Then, the decision on whether to enable or disable the BIO process is made at the sub-CU-level based on a predefined sub-CU-level SAD threshold, which is set to 3*2^(BDepth-10)per sample.

2.4 Specification for BDOF in VVC

Specification of BDOF (in JVET-N1001-v2) is as follows:

8.5.7.4 Bidirectional optical flow prediction process

Inputs to this process are:

- two variables nCbW and nCbH specifying the width and the height of the current coding block,

- two (nCbW + 2)x(nCbH + 2) luma prediction sample arrays predSamplesL0 and predSamplesL1,

- the prediction list utilization flags predFlagL0 and predFlagL1,

- the reference indices refIdxL0 and refIdxL1,

- the bidirectional optical flow utilization flags bdofUtilizationFlag[ xIdx ][ yIdx ] with

xIdx = 0..( nCbW>> 2) − 1, yIdx = 0..( nCbH >> 2 ) − 1.

Output of this process is the (nCbW)x(nCbH) array pbSamples of luma prediction sample values.

Variables bitDepth, shift1, shift2, shift3, shift4, offset4, and mvRefineThres are derived as follows:

- The variable bitDepth is set equal to BitDepthy.

- The variable shift1 is set to equal to Max( 2, 14 − bitDepth ).

- The variable shift2 is set to equal to Max( 8, bitDepth − 4 ).

- The variable shift3 is set to equal to Max( 5, bitDepth − 7 ).

- The variable shift4 is set equal to Max( 3, 15 − bitDepth ) and the variable offset4 is set equal to

1 << ( shift4 − 1 ).

- The variable mvRefineThres is set equal to Max( 2, 1 << ( 13 − bitDepth ) ).

For xIdx = 0..( nCbW >> 2 ) − 1 and yIdx = 0..( nCbH >> 2 ) − 1, the following applies:

- The variable xSb is set equal to ( xIdx << 2) + 1 and ySb is set equal to ( yIdx << 2) + 1.

- If bdofUtilizationFlag[ xSbIdx ][ yIdx ] is equal to FALSE, for x = xSb − 1..xSb + 2, y = ySb − 1..ySb + 2, the

prediction sample values of the current subblock are derived as follows:

pbSamples[ x ][ y ] = Clip3( 0, ( 2^bitDepth) − 1, ( predSamplesL0[ x + 1 ][ y + 1 ] + offset2 +
(8-852)

predSamplesL1[ x + 1][ y + 1] ) >> shift2 )

- Otherwise (bdofUtilizationFlag[ xSbIdx ][ yIdx ] is equal to TRUE), the prediction sample values of the current

subblock are derived as follows:

- For x =xSb − 1..xSb + 4, y = ySb − 1..ySb + 4, the following ordered steps apply:

1. The locations ( h_x, v_y) for each of the corresponding sample locations ( x, y ) inside the prediction

sample arrays are derived as follows:

h_x= Clip3( 1, nCbW, x)
(8-853)

v_y= Clip3( 1, nCbH, y )
(8-854)

2. The variables gradientHL0[ x ][ y ], gradient VL0[ x ][ y ], gradientHL1 [ x ][ y ]
and

gradientVL1[ x ][ y ] are derived as follows:

gradientHL0[ x ][ y ] = (predSamplesL0[ h_x+ 1 ][v_y] − predSampleL0[ h_x− 1 ][ v_y] ) >> shift1
(8-855)

gradientVL0[ x ][ y ] = (predSampleL0[ h_x][ v_y+ 1 ] − predSampleL0[ h_x][v_y− 1 ]) >> shift1
(8-856)

gradientHL1[ x ][ y ] = (predSamplesL1[ h_x+ 1 ][v_y] − predSampleL1[ h_x− 1 ][ v_y] ) >> shift1
(8-857)

gradientVL1[ x ][ y ] = (predSampleL1[ h_x][ v_y+ 1 ] − predSampleL1[ h_x][v_y− 1 ] ) >> shift1
(8-858)

3. The variables temp[ x ][ y ], tempH[ x ][ y ] and tempV[ x ][ y ] are derived as follows:

diff[ x ][ y ] = (predSamplesL0[ h_x][ v_y] >> shift2 ) − ( predSamplesL1 [ h_x][ v_y] >> shift2 )
(8-859)

tempH[ x ][ y ] = (gradientHL0[ x ][ y ] + gradientHL1[ x ][ y ] ) >> shift3
(8-860)

tempV[ x ][ y ] = (gradientVL0[ x ][ y ] + gradient VL1[ x ][ y ] ) >> shift3
(8-861)

- The variables sGx2, sGy2, sGxGy, sGxdI and sGydI are derived as follows:

sGx2 = Σ_iΣ_j( tempH[ xSb + i ][ ySb + j ] * tempH[ xSb + i ][ ySb + j ] ) with i, j = −1..4
(8-862)

sGy2= Σ_iΣ_j(tempV[ xSb + i ][ ySb + j ] * tempV[ xSb +i ][ ySb + j ] ) with i, j = −1..4
(8-863)

sGxGy= Σ_iΣ_j(tempH[ xSb +i ][ ySb + j ] * tempV[ xSb + i ][ ySb + j ] ) with i, j −1..4
(8-864)

sGxdI = Σ_iΣ_j( −tempH[ xSb + i ][ ySb + j ] * diff[ xSb + i ][ ySb + j ] ) with i, j = −1..4
(8-865)

sGydI = Σ_iΣ_j( −tempV[ xSb + i ][ ySb + j ] * diff[ xSb +i ][ ySb + j ] ) with i, j = −1..4
(8-866)

- The horizontal and vertical motion offset of the current subblock are derived as:

v_x= sGx2 > 0 ? Clip3( −mvRefineThres, mvRefineThres,
(8-867)

−(sGxdI << 3) >> Floor( Log2(sGx2 ) ) ) : 0

v_y= sGy2 >0 ? Clip3(−mvRefineThres, mvRefineThres, ( (sGydI << 3) −
(8-868)

( ( v_x* sGxGy_m) << 12 + v_x* sGxGy_s) >> 1) >> Floor( Log2(sGx2 ) ) ) : 0

- For x =xSb − 1..xSb + 2, y = ySb − 1..ySb + 2, the prediction sample values of the current sub-block are

derived as follows:

bdofOffset = Round( ( v_x* ( gradientHL1[ x + 1 ][ y + 1 ] − gradientHL0[ x + 1 ][ y + 1 ] ) ) >> 1 )
(8-869)

+ Round( ( v_y* (gradientVL1[ x + 1][ y + 1 ] − gradientVL0[ x + 1 ][ y + 1 ] ) ) >> 1 )

[Ed. (JC): Round( ) operation is defined for float input. The Round( ) operation seems redundant here

since the input is an integer value. To be confirmed by the proponent]

pbSamples[ x ][ y ] = Clip3( 0, ( 2^bitDepth) − 1, ( predSamplesL0[ x + 1 ][ y + 1 ] + offset4 +
(8-870)

predSamplesL1[ x + 1 ][ y + 1 ] + bdofOffset ) >> shift4 )

The spatial gradient is calculated as follows:

gradientHL0[ x ][ y ] = (predSamplesL0[ h_x+ 1 ][v_y] − predSampleL0[ h_x− 1 ][ v_y] ) >> shift1

(8-855)

On the other hand, temporal gradient is calculated as follows:

diff[ x ][ y ] = (predSamplesL0[ h_x][ v_y] >> shift2 ) − ( predSamplesL1[ h_x][ v_y] >> shift2 )
(8-859)

Thus, the calculation of spatial gradient and temporal gradient is not aligned.

2.5 Decoder-Side Motion Vector Refinement

In bi-prediction operation, for the prediction of one block region, two prediction blocks, formed using a motion vector (MV) of list0 and a MV of list1, respectively, are combined to form a single prediction signal. In JVET-K0217, the decoder-side motion vector refinement (DMVR) method, the two motion vectors of the bi-prediction are further refined by a bilateral matching process.

In the proposed method DMVR is applied only in Merge and Skip modes, if the following condition is true:

(POC−POC0)*(POC−POC1)<0,

where POC is the picture order count of current to be encoded picture, POC0 and POC1 are picture order counts of the references for the current picture.

The signaled merge candidate pair is used as input to DMVR process and are denoted initial motion vectors (MV0, MV1). The search points that are searched by DMVR obey the motion vector difference mirroring condition. In other words any point that is checked by DMVR, denoted by candidate motion vector pair (MV0′, MV1′) obey the following two equations:

MV0′=MV0+MV_diff
MV1′=MV1−MV_diff

where MV_diffrepresents the points in the search space in one of the reference pictures.

After the construction of the search space the uni-lateral predictions are constructed using regular 8-tap discrete cosine transform based interpolation filter (DCTIF) interpolation filter. Bilateral matching cost function is calculated by using mean removed sum of absolute differences (MRSAD) between the two predictions (FIG. 6) and the search point resulting in the minimum cost is selected as the refined MV pair. For the MRSAD calculation 16 bit precision of samples is used (which is the output of the interpolation filtering), and no clipping and no rounding operations are applied before MRSAD calculation. The reason for not applying rounding and clipping is to reduce internal buffer requirement.

FIG. 6 shows an example of bilateral matching with 6 points search.

In the proposed method the integer precision search points are chosen by the Adaptive pattern method. The cost, corresponding to the central points (pointed by the initial motion vectors) is calculated firstly. The other 4 costs (in sign shape) is calculated by the two predictions, located at the opposite sides of each other by the central point. Last 6^thpoint at the angle is chosen by the gradient of the previous calculated costs (FIG. 7).

FIG. 7 shows examples of an adaptive integer search pattern and a half sample search pattern.

The output of the DMVR process is the refined motion vector pair corresponding to the minimal cost.

If after one iteration the minimum cost is achieved at the central point of the search space, i.e. the motion vectors are not changed, and the refinement process is terminated. Otherwise, the best cost further is regarded as center, and the process continues, while the minimal cost does not correspond to the central point and the search range is not exceeded.

Half sample precision search is applied only if application of half-pel search does not exceed the search range. In this case only 4 MRSAD calculations are performed, corresponding to plus shape points around the central one, which is chosen as the best during the integer precision search. At the end the refined motion vector pair is output that correspond to the minimal cost point.

Some simplifications and improvements are further proposed in JVET-L0163.

Reference Sampling Padding

Reference sample padding is applied in order to extend the reference sample block that is pointed by the initial motion vector. If the size of the coding block are given by “w” and “h”, then it is assumed that a block of size w+7 and h+7 is retrieved from the reference picture buffer. The retrieved buffer is then extended by 2 samples in each direction by repetitive sample padding using the nearest sample. Afterwards the extended reference sample block is used to generate the final prediction once the refined motion vector is obtained (which can deviate from the initial motion vector 2 samples in each direction).

It is noted that this modification eliminates the external memory access requirement of DMVR completely without any coding loss.

Bilinear Interpolation instead of 8-tap DCTIF

According to the proposal bilinear interpolation is applied during the DMVR search process, which means that the predictions used in MRSAD computation are generated using bilinear interpolation. Once the final refined motion vectors are obtained regular 8-tap DCTIF interpolation filter is applied to generate final predictions.

Disabling of DMVR for Small Blocks

DMVR is disabled for blocks 4×4, 4×8 and 8×4.

Early Termination based on MV Difference Between Merge Candidates

An additional condition is imposed on DMVR to confine the MV refinement process. With it, DMVR is conditionally disabled when the below condition is satisfied.

The MV difference between the selected merge candidate and any of the previous ones in the same merge list is less than a pre-defined threshold (that is, ¼, ½- and 1-pixel-wide intervals for CUs with less than 64 pixels, less than 256 pixels and at least 256 pixels, respectively).

Early Termination based on SAD cost at the Center Search Coordinate

The sum of absolute difference (SAD) between the two prediction signals (L0 and L1 prediction) using the initial motion vectors of the current CU is calculated. If the SAD is no larger than a predefined threshold, i.e., 2^(BDepth-9)per sample, the DMVR is skipped; otherwise, the DMVR is still applied to refine the two motion vectors of the current block.

DMVR Application Condition

The DMVR application condition is (POC−POC1)×(POC−POC2)<0 as it is implemented in BMS2.1 is replaced by the new condition (POC−POC1)==(POC2−POC). This means that DMVR is applied only if reference pictures are in opposite time directions and are equidistant to current picture.

MRSAD Computation using Every Second Row

The MRSAD cost is computed only for odd numbered rows of a block, the even numbered samples rows are not considered. Accordingly, the number of operations for the MRSAD calculation is halved.

2.6 Related Method

In the patent application identified by Application No. PCT/CN2018/098691 (which is incorporated by reference herein), entitled “Motion Refinement for Visual Media Coding,” filed Aug. 4, 2018, a MV update method and a two-step inter prediction method are proposed. The derived MV between reference block 0 and reference block 1 in BIO are scaled and added to the original motion vector of list 0 and list 1. Meanwhile, the updated MV is used to perform motion compensation and a second inter prediction is generated as the final prediction. The temporal gradient is modified by removing the mean difference between reference block 0 and reference block 1.

2.7 DMVR in VVC Draft 4

The usage of DMVR in JVET-M1001_v7 (VVC working draft 4, version 7) is defined as follows:

- When all of the following conditions are true, dmvrFlag is set equal to 1:

- sps_dmvr_enabled_flag is equal to 1

- Current block is not coded with triangular prediction mode, AMVR affine mode, sub-

block mode (including merge affine mode, and ATMVP mode)

- merge_flag[ xCb ][ yCb ] is equal to 1

- both predFlagL0[ 0 ][ 0 ] and predFlagL1[ 0 ][ 0 ] are equal to 1

- mmvd_flag[ xCb ][ yCb ] is equal to 0

- DiffPicOrderCnt( currPic, RefPicList[ 0 ][ refIdxL0 ]) is equal to

DiffPicOrderCnt( RefPicList[ 1 ][ refIdxL1 ], currPic )

- cbHeight is greater than or equal to 8

- cbHeight*cbWidth is greater than or equal to 64

3. EXAMPLES OF PROBLEMS SOLVED BY EMBODIMENTS

In BIO, difference between two reference blocks or sub-blocks are calculated in the early termination stage, meanwhile, the temporal gradient is also calculated. Because the temporal gradient is actually the difference (or right shifted difference) between two reference pixels, calculating both the difference and the temporal gradient is not meaningful.

In DMVR, the MRSAD calculation is used to decide the refine motion vector of one block.

In BIO, the SAD calculation is used to decide whether BIO should be enabled/disabled for one block or one sub-block using all samples of one block/one sub-block which increases the computation complexity.

The calculation method is different for spatial gradient and temporal gradient.

4. EXAMPLES OF EMBODIMENTS

Denote SATD as sum of absolute transformed differences, MRSATD as mean removed sum of absolute transformed differences, and SSE as sum of squares error, and MRSSE as mean removed sum of squares error.

The detailed techniques below should be considered as examples to explain general concepts. These techniques should not be interpreted in a narrow way. Furthermore, these inventions can be combined in any manner.

In the following discussion, SatShift(x, n) is defined as

$SatShift (x, n) = {\begin{matrix} (x + offsset 0) >> n & if x \geq 0 \\ - ((- x + offset 1) >> n) & if x < 0 \end{matrix}$

Shift(x, n) is defined as Shift(x, n)=(x+offset0)>>n.

In one example, offset0 and/or offset1 are set to (1<<n)>>1 or (1<<(n−1)). In another example, offset0 and/or offset1 are set to 0.

In another example, offset0=offset1=((1<<n)>>1)−1 or ((1<<(n−1)))−1.

In gradient calculation of BDOF, difference between two neighboring (either spatial neighboring or temporal neighboring) or/and non-adjacent samples may be calculated, and right-shift may be performed during the gradient calculation. Suppose the two neighboring samples are neig0 and neig1, and the right shift value is shift1, and the gradient to be calculated is grad. Note that shift1 may be different for spatial gradient and temporal gradient.

- 1. It is proposed to align the method used in calculating spatial gradient and temporal gradient.
  - a. In one example, gradient is calculated according to the shifted sample differences.
    - i. Alternatively, gradient is calculated according to the modified sample (e.g., via shifting) differences.
  - b. In one example, in gradient calculation, subtraction may be performed before right shift. E.g., grad=(neig0−neig1)>>shift1.
  - c. In one example, in gradient calculation, subtraction may be performed after right shift. E.g., grad=(neig0>>shift1)−(neig1>>shift1).
  - d. In one example, in gradient calculation, subtraction may be performed before right shift and an offset may be added before right shift. E.g., grad=(neig0−neig1+offset)>>shift1. The offset may be equal to 1<<(shift1−1) or 1<<shift1>>1.
  - e. In one example, in gradient calculation, subtraction may be performed after right shift and an offset may be added before right shift. E.g., grad=((neig0+offset)>>shift1)−((neig1+offset)>>shift1). The offset may be equal to 1<<(shift−1) or 1<<shift>>1.
  - f. In one example, the gradient may be calculated as SatShift(neig0−neig1, shift1).
    - i. Alternatively, the gradient may be calculated as SatShift(neig0, shift1)−SatShift(neig1, shift1).
- 2. It is proposed to use other criteria to decide the enabling/disabling of BIO or/and DMVR in the early termination stage, such as SATD or MRSATD or SSE or MRSSE or mean value difference or gradient values.
  - a. In one example, the block level and sub-block level enabling/disabling decisions may choose different rules, e.g., one with SAD and the other with SATD.
  - b. In one example, for a block/sub-block, if the gradient values (horizontal and/or vertical) or the averaged gradient values or the range of gradient values satisfy a condition, (e.g., larger than a threshold or outside a given range), BIO and/or DMVR may be disabled.
  - c. It is proposed that the criteria used to decide the enabling/disabling BIO/DMVR may be signaled from the encoder to the decoder in video parameter set (VPS)/sequence parameter set (SPS)/picture parameter set (PPS)/slice header/tile group header.
- 3. It is proposed to use other criteria to decide the refined motion vector of one block in DMVR process, such as SATD or MRSATD or SSE or MRSSE to replace MRSAD.
  - a. In one example, the refined motion vector of one sub-block in DMVR process, such as SATD or MRSATD or SSE or MRSSE to replace MRSAD.
  - b. In one example, if SATD (or MRSATD) is applied, the whole block is split into M×N sub-blocks and SATD (or MRSATD) is calculated for each sub-block. The SATDs (or MRSATDs) for all or some of the sub-blocks are summed up to get the SATD (or MRSATD) value for the whole block.
- 4. BIO or/and DMVR may be disabled when mean value difference of two reference blocks of one block is larger than a threshold (T1).
  - a. BIO may be disabled when mean value difference of two reference sub-blocks of one sub-block is larger than a threshold (T2).
  - b. The thresholds T1 and/or T2 may be pre-defined.
  - c. The thresholds T1 and/or T2 may be dependent on the block dimension.
- 5. It is proposed that in the early termination stage of BIO, before calculating the difference (e.g., SAD/SATD/SSE, etc.) between the two reference blocks/sub-blocks, the reference blocks or/and sub-blocks may be first modified.
  - a. In one example, mean of the reference block or/and sub-block may be calculated and then subtracted by the reference block or/and sub-block.
  - b. In one example, methods disclosed in App. No. PCT/CN2018/096384, (which is incorporated by reference herein), entitled “Motion Prediction Based on Updated Motion Vectors,” filed on Jul. 20, 2018, may be used to calculate the mean value of the reference block or/and sub-block, i.e., mean value is calculated for some representative positions.
- 6. It is proposed that in the early termination stage of BIO or/and DMVR, the difference (e.g., SAD/SATD/SSE/MRSAD/MRSATD/MRSSE, etc.) between the two reference blocks or/and sub-blocks may be calculated only for some representative positions.
  - a. In one example, only difference of even rows is calculated for the block or/and sub-block.
  - b. In one example, only difference of four corner samples of one block/sub-block is calculated for the block or/and sub-block.
  - c. In one example, the methods disclosed in U.S. Provisional Application No. 62/693,412, (which is incorporated by reference herein) entitled “Decoder Side Motion Vector Derivation in Video Coding,” filed Jul. 2, 2018, may be used to select the representative positions.
  - d. In one example, the difference (e.g., SAD/SATD/SSE/MRSAD/MRSATD/MRSSE, etc.) between the two reference blocks may be calculated only for some representative sub-blocks.
  - e. In one example, the difference (e.g., SAD/SATD/SSE/MRSAD/MRSATD/MRSSE, etc.) calculated for representative positions or sub-blocks are summed up to get the difference for the whole block/sub-block.
- 7. It is proposed that temporal gradient (temporal gradient at position (x,y) is defined as G(x,y)=P0(x,y)−P1(x,y), where P0(x,y) and P1(x,y) represent the prediction at (x,y) from two different reference pictures) or modified temporal gradient is used as the difference (instead of SAD) in the early termination stage of BIO, and the threshold used in early termination may be adjusted accordingly.
  - a. In one example, absolute sum of the temporal gradients is calculated and used as the difference of the two reference blocks or/and sub-blocks.
  - b. In one example, absolute sum of the temporal gradients is calculated only on some representative positions for the block or/and sub-block.
  - c. In one example, the methods disclosed in U.S. Provisional Application No. 62/693,412, (which is incorporated by reference herein) entitled “Decoder Side Motion Vector Derivation in Video Coding,” filed Jul. 2, 2018, may be used to select the representative positions.
- 8. It is proposed that the temporal gradient modification process may be performed adaptively for different blocks/sub-blocks.
  - a. In one example, the temporal gradient is modified only when the absolute mean difference (or SAD/SATD/SSE, etc.) between the two reference blocks is greater than a threshold T, for example, T=4.
  - b. In one example, the temporal gradient is modified only when the absolute mean difference (or SAD/SATD/SSE, etc.) between the two reference blocks is less than a threshold T, for example, T=20.
  - c. In one example, the temporal gradient is modified only when the absolute mean difference (or SAD/SATD/SSE, etc.) between the two reference blocks is in the range of [T1, T2], for example, T1=4, T2=20.
  - d. In one example, if the absolute mean difference (or SAD/SATD/SSE etc.) between the two reference blocks is greater than a threshold T (for example, T=40), BIO is disabled.
  - e. In one example, these thresholds may be predefined implicitly.
  - f. In one example, these thresholds may be signaled in SPS/PPS/picture/slice/tile level.
  - g. In one example, these thresholds may be different for different CU, largest coding unit (LCU), slice, tile or picture.
    - i. In one example, these thresholds may be designed based on decoded/encoded pixel values.
    - ii. In one example, these thresholds may be designed differently for different reference pictures.
  - h. In one example, the temporal gradient is modified only when (absolute) mean of the two (or anyone of the two) reference blocks is greater than a threshold T, for example, T=40.
  - i. In one example, the temporal gradient is modified only when (absolute) mean of the two (or anyone of the two) reference blocks is smaller than a threshold T, for example, T=100.
  - j. In one example, the temporal gradient is modified only when (absolute) mean of the two (or anyone of the two) reference blocks are in the range of [T1, T2], for example, T1=40, T2=100.
  - k. In one example, the temporal gradient is modified only when (absolute) mean of the two (or anyone of the two) reference blocks is greater/less than the absolute mean difference (or SAD/SATD etc.) multiplied by T, in one example, T=4.5.
  - l. In one example, the temporal gradient is modified only when (absolute) mean of the two (or anyone of the two) reference blocks is in the range of the absolute mean difference (or SAD/SATD etc.) multiplied by [T1, T2], in one example, T1=4.5, T2=7.
- 9. It is proposed that in hybrid intra and inter prediction mode, the two inter reference blocks may be modified when calculating the spatial gradients in BIO, or they may be modified before performing the entire BIO procedure.
  - a. In one example, the intra prediction block and the inter prediction block in each prediction direction are weighted averaged (using same weighting method as in hybrid inter and inter prediction) to generate two new prediction blocks, denoted as wAvgBlkL0 and wAvgBlkL1, which are used to derive the spatial gradients in BIO.
  - b. In one example, wAvgBlkL0 and wAvgBlkL1 are used to generate the prediction block of the current block, denoted as predBlk. Then, wAvgBlkL0, wAvgBlkL1 and predBlk are further used for the BIO procedure, and the refined prediction block generated in BIO is used as the final prediction block.
- 10. It is proposed that a DMVR or/and BIO flag may be signaled at block level to indicate whether DMVR or/and BIO is enabled for the block.
  - a. In one example, such flag may be signaled only for AMVP mode, and in merge mode, such flag may be inherited from spatial or/and temporal neighboring blocks.
  - b. In one example, whether BIO or/and DMVR is enabled or not may be decided jointly by the signaled flag and the on-the-fly decision (for example, the decision based on SAD in the early termination stage). The signaled flag may indicate whether the on-the-fly decision is correct or not.
  - c. Such flag is not signaled for uni-predicted blocks.
  - d. Such flag may be not signaled for bi-predicted blocks whose two reference pictures are both preceding pictures or following pictures in display order.
  - e. Such flag may be not signaled for bi-predicted blocks if POC_diff(curPic, ref0) is not equal to POC_diff(ref1, curPic), wherein POC_diff ( ) calculates the POC difference between two pictures, and ref0 and ref1 are the reference pictures of current picture.
  - f. Such a flag is not signaled for intra coded blocks. Alternatively, furthermore, such a flag is not signaled for blocks coded with the hybrid intra and inter prediction mode.
    - Alternatively, such a flag is not signaled for current picture referencing block, i.e. the reference picture is the current picture.
  - g. Whether to signal the flag may depend on the block dimension. For example, if the block size is smaller than a threshold, such a flag is not signaled. Alternatively, if the block width and/or height is equal to or larger than a threshold, such a flag is not signaled.
  - h. Whether to signal the flag may depend on the motion vector precision. For example, if the motion vector is in integer precision, such a flag is not signaled.
  - i. If such a flag is not signaled, it may be derived to be true or false implicitly.
  - j. A flag may be signaled at slice header/tile header/PPS/SPS/VPS to indicate whether this method is enabled or not.
  - k. Such signaling method may depend on the temporal layer of the picture, for example, it may be disabled for picture with high temporal layer.
  - l. Such signaling method may depend on the quantization parameter (QP) of the picture, for example, it may be disabled for picture with high QP.
- 11. Instead of checking both block height and block size, it is proposed to decide whether to enable or disable DMVR according to the block height only.
  - a. In one example, DMVR may be enabled when the block height is greater than T1 (e.g., T1=4).
  - b. In one example, DMVR may be enabled when the block height is equal to or greater than T1 (e.g., T1=8).
- 12. The above methods which are applied to DMVR/BIO may be only applicable to other decoder-side motion vector derivation (DMVD) methods, such as prediction refinement based on optical flow for the affine mode.
  - a. In one example, the condition check for usage determination of DMVR and BIO may be aligned, such as whether block height satisfies same threshold.
    - i. In one example, DMVR and BIO may be enabled when the block height is equal to or greater than T1 (e.g., T1=8).
    - ii. In one example, DMVR and BIO may be enabled when the block height is greater than T1 (e.g., T1=4).

5. EMBODIMENT
5.1 Embodiment #1

The usage of DMVR in WET-M1001_v7 (VVC working draft 4, version 7) is modified as follows:

- When all of the following conditions are true, dmvrFlag is set equal to 1:

- sps_dmvr_enabled_flag is equal to 1

- Current block is not coded with triangular prediction mode, AMVR affine mode,

sub-block mode (including merge affine mode, and ATMVP mode)

- merge_flag[ xCb ][ yCb ] is equal to 1

- both predFlagL0[ 0 ][ 0 ] and predFlagL1[ 0 ][ 0 ] are equal to 1

- mmvd_flag[ xCb ][ yCb ] is equal to 0

- DiffPicOrderCnt( currPic, RefPicList[ 0 ][ refIdxL0 ]) is equal to

DiffPicOrderCnt( RefPicList[ 1 ][ refIdxL1 ], currPic )

- cbHeight is greater than or equal to 8

custom character

That means, “cbHeight*cb Width is greater than or equal to 64” is deleted.

5.2 Embodiment #2

The newly added parts are highlighted in bold face italics, and the deleted part are highlighted in strikethrough.

i. One Example

tempH[ x ][ y ] = (gradientHL0[ x ][ y ] + gradientHL1 [ x ][ y ] ) >> shift3
(8-860)

tempV[ x ][ y ] = (gradientVL0[ x ][ y ] + gradientVL1[ x ][ y ] ) >> shift3
(8-861)

- The variables sGx2, sGy2, sGxGy, sGxdI and sGydI are derived as follows:

sGx2 = Σ_iΣ_j( tempH[ xSb +i ][ ySb + j ] * tempH[ xSb + i ][ ySb + j ] ) with i, j = −1..4
(8-862)

sGy2 = Σ_iΣ_j(tempV[ xSb +i ][ ySb + j ] * tempV[ xSb +i ][ ySb + j ] ) with i, j = −1..4
(8-863)

sGxGy = Σ_iΣ_j(tempH[ xSb + i ][ ySb + j ] * tempV[ xSb +i ][ ySb + j ] ) with i, j −1..4
(8-864)

sGxdI = Σ_iΣ_j( −tempH[ xSb + i ][ ySb + j ] * diff[ xSb +i ][ ySb + j ] ) with i, j = −1..4
(8-865)

sGydI = Σ_iΣ_j( −tempV[ xSb + i ][ ySb + j ] * diff[ xSb + i ][ ySb + j ] ) with i, j = −1..4
(8-866)

- The horizontal and vertical motion offset of the current subblock are derived as:

v_x= sGx2 > 0 ? Clip3( −mvRefineThres, mvRefineThres,
(8-867)

−( sGxdI << 3 ) >> Floor( Log2(sGx2 ) ) ) : 0

v_y= sGy2 > 0 ? Clip3( −mvRefineThres, mvRefineThres, ( ( sGydI << 3 ) −
(8-868)

( ( v_x* sGxGym ) << 12+ v_x* sGxGys ) >> 1) >> Floor( Log2(sGx2))) : 0

- For x =xSb − 1..xSb + 2, y = ySb − 1..ySb + 2, the prediction sample values of the current sub-block are

derived as follows:

bdofOffset = Round( ( v_x*(gradientHL1[x + 1][y + 1] − gradientHL0[x + 1][y + 1 ] ) ) >> 1)
(8-869)

+ Round( ( v_y* (gradientVL1[ x + 1][ y + 1] − gradientVL0[x + 1][y + 1 ] ) ) >> 1)

[Ed. (JC): Round( ) operation is defined for float input. The Round( ) operation seems redundant here

since the input is an integer value. To be confirmed by the proponent]

pbSamples[ x ][ y ] = Clip3( 0, ( 2^bitDepth) − 1, ( predSamplesL0[ x + 1 ][ y + 1 ] + offset4 +
(8-870)

predSamplesL1[ x + 1 ][ y + 1 ] + bdofOffset ) >> shift4 )

ii. One Example

9. The variables temp[ x ][ y ], tempH[ x ][ y ] and tempV[ x ][ y ] are derived as follows:

diff[ x ][ y ] = (predSamplesL0[ h_x][ v_y] >> shift2 ) − ( predSamplesL1[ h_x][ v_y] >> shift2 )
(8-859)

tempH[ x ][ y ] = (gradientHL0[ x ][ y ] + gradientHL1[ x ][ y ] ) >> shift3
(8-860)

tempV[ x ][ y ] = (gradientVL0[ x ][ y ] + gradientVL1[ x ][ y ] ) >> shift3
(8-861)

- The variables sGx2, sGy2, sGxGy, sGxdI and sGydI are derived as follows:

sGx2 = Σ_iΣ_j( tempH[ xSb + i ][ ySb + j ] * tempH[ xSb + i ][ ySb + j ] ) with i, j = −1..4
(8-862)

sGy2 = Σ_iΣ_j(tempV[ xSb + i ][ ySb + j ] * tempV[ xSb + i ][ ySb + j ] ) with i, j = −1..4
(8-863)

sGxGy = Σ_iΣ_j(tempH[ xSb + i ][ ySb + j ] * tempV[ xSb + i ][ ySb + j ] ) with i, j −1..4
(8-864)

sGxdI = Σ_iΣ_j( −tempH[ xSb + i ][ ySb + j ] * diff[ xSb + i ][ ySb + j ] ) with i, j = −1..4
(8-865)

sGydI = Σ_iΣ_j( −tempV[ xSb + i ][ ySb + j ] * diff[ xSb + i ][ ySb + j ] ) with i, j = −1..4
(8-866)

- The horizontal and vertical motion offset of the current subblock are derived as:

v_x= sGx2 > 0 ? Clip3( −mvRefineThres, mvRefineThres,
(8-867)

−( sGxdI << 3 ) >> Floor( Log2(sGx2 ) ) ) : 0

v_y= sGy2 > 0 ? Clip3(−mvRefineThres, mvRefineThres, ( ( sGydI << 3) −
(8-868)

( ( v_x* sGxGym ) << 12 + v_x* sGxGys) >> 1) >> Floor( Log2(sGx2))) : 0

- For x =xSb − 1..xSb + 2, y = ySb − 1..ySb + 2, the prediction sample values of the current sub-block are

derived as follows:

bdofOffset = Round( ( v_x* ( gradientHL1[ x + 1][ y + 1] − gradientHL0[x + 1][y + 1 ] ) ) >> 1 )

(8-869)

+ Round( ( v_y* (gradientVL1[ x + 1][ y + 1] - gradientVL0[x + 1][y + 1 ] ) ) >> 1 )

[Ed. (JC): Round( ) operation is defined for float input. The Round( ) operation seems redundant here

since the input is an integer value. To be confirmed by the proponent]

pbSamples[ x ][ y ] = Clip3( 0, ( 2^bitDepth) − 1, ( predSamplesL0[ x + 1 ][ y + 1 ] + offset4 +
(8-870)

predSamplesL1[ x + 1 ][ y + 1 ] + bdofOffset ) >> shift4 )

iii. One Example

For xIdx = 0 .. ( nCbW>> 2) - 1 and yIdx = 0 .. ( nCbH >> 2) − 1, the following applies:

- The variable xSb is set equal to ( xIdx << 2) + 1 and ySb is set equal to ( yIdx << 2) + 1.

- If bdofUtilizationFlag[ xSbIdx ][ yIdx ] is equal to FALSE, for x = xSb − 1..xSb + 2, y = ySb − 1..ySb + 2, the

prediction sample values of the current subblock are derived as follows:

pbSamples[ x ][ y ] = Clip3( 0, ( 2^bitDepth) − 1, ( predSamplesL0[ x + 1 ][ y + 1 ] + offset2 +
(8-852)

predSamplesL1[ x + 1 ][ y + 1]) >> shift2)

- Otherwise (bdofUtilizationFlag[ xSbIdx ][ yIdx ] is equal to TRUE), the prediction sample values of the current

subblock are derived as follows:

- For x =xSb − 1..xSb + 4, y = ySb − 1..ySb + 4, the following ordered steps apply:

10. The locations ( h_x, v_y) for each of the corresponding sample locations ( x, y ) inside the prediction

sample arrays are derived as follows:

h_x= Clip3( 1, nCbW, x )
(8-853)

v_y= Clip3( 1, nCbH, y )
(8-854)

11. The variables gradientHL0[ x ][ y ], gradient VL0[ x ][ y ], gradientHL1[ x ][ y ] and

gradientVL1[ x ][ y ] are derived as follows:

gradientHL0[ x ][ y ] = (predSamplesL0[ h_x+ 1 ][v_y] − predSampleL0[ h_x− 1 ][ v_y] + offset5 ) >>

shift1 (8-855)

gradientVL0[ x ][ y ] = (predSampleL0[ h_x][ v_y+ 1 ] − predSampleL0[ h_x][v_y− 1 ] + offset5) >> s

hift1 (8-856)

gradientHL1[ x ][ y ] = (predSamplesL1[ h_x+ 1 ][v_y] − predSampleL1[ h_x− 1 ][ v_y] + offset5) >> s

hift1 (8-857)

gradientVL1[ x ][ y ] = (predSampleL1[ h_x][ v_y+ 1 ] − predSampleL1[ h_x][v_y− 1 ] + offset5) >> s

hift1 (8-858)

12. The variables temp[ x ][ y ], tempH[ x ][ y ] and tempV[ x ][ y ] are derived as follows:

custom character

tempH[ x ][ y ] = (gradientHL0[ x ][ y ] + gradientHL1 [ x ][ y ] ) >> shift3
(8-860)

tempV[ x ][ y ] = (gradientVL0[ x ][ y ] + gradient VL1[ x ][ y ] ) >> shift3
(8-861)

- The variables sGx2, sGy2, sGxGy, sGxdI and sGydI are derived as follows:

sGx2 = Σ_iΣ_j( tempH[ xSb + i ][ ySb + j ] * tempH[ xSb + i ][ ySb + j ] ) with i, j = −1..4
(8-862)

sGy2 = Σ_iΣ_j(tempV[ xSb + i ][ ySb + j ] * tempV[ xSb + i ][ ySb + j ] ) with i, j = −1..4
(8-863)

sGxGy = Σ_iΣ_j(tempH[ xSb + i ][ ySb + j ] * tempV[ xSb + i ][ ySb + j ] ) with i, j −1..4
(8-864)

sGxdI = Σ_iΣ_j( −tempH[ xSb + i ][ ySb + j ] * diff[ xSb + i ][ ySb + j ] ) with i, j = −1..4
(8-865)

sGydI = Σ_iΣ_j( −tempV[ xSb + i ][ ySb + j ] * diff[ xSb + i ][ ySb + j ] ) with i, j = −1..4
(8-866)

- The horizontal and vertical motion offset of the current subblock are derived as:

v_x= sGx2 > 0 ? Clip3( −mvRefineThres, mvRefineThres,
(8-867)

−( sGxdI << 3 ) >> Floor( Log2(sGx2 ) ) ) : 0

v_y= sGy2 > 0 ? Clip3( −mvRefineThres, mvRefineThres, ( ( sGydI << 3) −
(8-868)

( ( v_x* sGxGym ) << 12 + v_x* sGxGys )>> 1) >> Floor( Log2(sGx2 ) ) ) : 0

- For x =xSb − 1..xSb + 2, y = ySb − 1..ySb + 2, the prediction sample values of the current sub-block are

derived as follows:

bdofOffset = Round( ( v_x* ( gradientHL1[ x + 1][ y + 1] − gradientHL0[x + 1][y + 1 ] ) ) >> 1 )

(8-869)

+ Round( ( v_y* (gradientVL1[ x + 1 ][ y + 1] − gradientVL0[ x + 1 ][ y + 1 ] ) ) >>1 )

[Ed. (JC): Round( ) operation is defined for float input. The Round( ) operation seems redundant here

since the input is an integer value. To be confirmed by the proponent]

pbSamples[ x ][ y ] = Clip3( 0, ( 2^bitDepth) − 1, ( predSamplesL0[ x + 1 ][ y + 1 ] + offset4 +
(8-870)

predSamplesL1[ x + 1 ][ y + 1 ] + bdofOffset ) >> shift4 )

FIG. 8 is a block diagram of a video processing apparatus 800. The apparatus 800 may be used to implement one or more of the methods described herein. The apparatus 800 may be embodied in a smartphone, tablet, computer, Internet of Things (IoT) receiver, and so on. The apparatus 800 may include one or more processors 802, one or more memories 804 and video processing hardware 806. The processor(s) 802 may be configured to implement one or more methods described in the present disclosure. The memory (memories) 804 may be used for storing data and code used for implementing the methods and techniques described herein. The video processing hardware 806 may be used to implement, in hardware circuitry, some techniques described in the present disclosure. The video processing hardware 806 may be partially or completely includes within the processor(s) 802 in the form of dedicated hardware, or graphical processor unit (GPU) or specialized signal processing blocks.

FIG. 10 is a flowchart for a method 1000 of processing a video. The method 1000 includes performing a determination (1005) of characteristics of a first video block, the characteristics including differences between reference blocks associated with the first video block, the differences including one or more of a sum of absolute transformed differences (SATD), a mean removed sum of absolute transformed differences (MRSATD), a sum of squares error (SSE), a mean removed sum of squares error (MRSSE), a mean value difference, or gradient values, determining (1010) an operational state of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique based on the characteristics of the first video block, the operational state being one of enabled or disabled, and performing (1015) further processing of the first video block consistent with the operational state of one or both of the BIO technique or the DMVR technique.

FIG. 11 is a flowchart for a method 1100 of processing a video. The method 1100 includes modifying (1105) a first reference block to generate a first modified reference block, and a second reference block to generate a second modified reference block, the first reference block and the second reference block associated with a first video block, performing (1110) differences between the first modified reference block and the second modified reference block, the differences including one or more of a sum of absolute transformed differences (SATD), a mean removed sum of absolute transformed differences (MRSATD), a sum of squares error (SSE), a mean removed sum of squares error (MRSSE), a mean value difference, or gradient values, and performing (1115) further processing of the first video block based on the differences between the first modified reference block and the second modified reference block.

FIG. 12 is a flowchart for a method 1200 of processing a video. The method 1200 includes determining (1205) differences between a portion of a first reference block and a portion of a second reference block that are associated with a first video block, the differences including one or more of a sum of absolute transformed differences (SATD), a mean removed sum of absolute transformed differences (MRSATD), a sum of squares error (SSE), a mean removed sum of squares error (MRSSE), a mean value difference, or gradient values, and performing (1210) further processing of the first video block based on the differences.

FIG. 13 is a flowchart for a method 1300 of processing a video. The method 1300 includes determining (1305) a temporal gradient or a modified temporal gradient using reference pictures associated with a first video block, the temporal gradient or the modified temporal gradient indicative of differences between the reference pictures, and performing (1310) further processing of the first video block using a bi-directional optical flow (BIO) coding tool in accordance with the differences.

FIG. 14 is a flowchart for a method 1400 of processing a video. The method 1400 includes determining (1405) a temporal gradient using reference pictures associated with a first video block, modifying (1410) the temporal gradient to generate a modified temporal gradient, and performing (1415) further processing of the first video block using the modified temporal gradient.

FIG. 15 is a flowchart for a method 1500 of processing a video. The method 1500 includes modifying (1505) one or both of a first inter reference block and a second inter reference block associated with a first video block, determining (1510) a spatial gradient in accordance with a bi-directional optical flow coding tool (BIO) using one or both of the modified first inter reference block or the modified second inter reference block, and performing (1515) further processing of the first video block based on the spatial gradient.

FIG. 16 is a flowchart for a method 1600 of processing a video. The method 1600 includes performing (1605) a determination that a flag which can be signaled at multiple levels indicates that one or both of a decoder-side motion vector refinement (DMVR) or a bi-directional optical flow (BIO) is to be enabled for a first video block, and performing (1610) further processing of the first video block, the processing including applying one or both of DMVR or BIO consistent with the flag.

With reference to methods 1000, 1100, 1200, 1300, 1400, 1500, and 1600, some examples of determining use of bi-directional optical flow (BIO) or decoder-side motion vector refinement (DMVR) are described in Section 4 of the present disclosure. For example, as described in Section 4, differences between reference blocks can be determined and the differences can be used to enable or disable BIO or DMVR.

With reference to methods 1000, 1100, 1200, 1300, 1400, 1500, and 1600, a video block may be encoded in the video bitstream in which bit efficiency may be achieved by using a bitstream generation rule related to motion information prediction.

The methods can include wherein the operational state of the BIO technique or the DMVR technique is different between a block-level and a sub-block level.

The methods can include determining that one or more of the gradient values, an average of the gradient values, or a range of the gradient values are within a threshold range, wherein determining the operational state is based on the determination the gradient values, the average of the gradient values, or the range of the gradient values are within the threshold range.

The methods can include wherein determining the operational state is further based on information signaled from an encoder to a decoder in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a tile group header, a picture header, or a slice header.

The methods can include determining a refined motion vector of the first video block based on the SATD, MRSATD, SSE, or MRSSE, and wherein performing further processing is based on the refined motion vector.

The methods can include wherein determining the refined motion vector is based on SATD or MRSATD, the method further comprising: determining SATD or MRSATD for each sub-block of the first video block; and generating SATD or MRSATD for the first video block based on a summation of the SATD or MRSATD for each sub-block, wherein further processing of the first video block is based on the generated SATD or MRSATD.

The methods can include determining that a mean value difference of two reference blocks of the first video block is larger than a threshold value, and wherein one or both of BIO or DMVR is in a disabled operational state based on the mean value difference of the two reference blocks.

The methods can include determining that a mean value difference of two reference sub-blocks of a sub-block of the first video block is larger than a threshold value, and wherein one or both of BIO or DMVR is in a disabled operational state based on the mean value difference of the two reference sub-blocks.

The methods can include wherein the threshold value is pre-defined.

The methods can include determining dimensions of the first video block, and wherein the threshold value is based on the dimensions of the first video block.

The methods can include wherein modifying the first reference block and the second reference block includes subtracting a mean of the first reference block from the first reference block.

The methods can include wherein the portions of the first reference block and the second reference block include even rows.

The methods can include wherein the portions of the first reference block and the second reference block include corner samples.

The methods can include wherein the portions of the first reference block and the second reference block include representative sub-blocks.

The methods can include wherein differences between the representative sub-blocks are summed to generate a difference for the first reference block or the second reference block.

The methods can include wherein the differences are related to an absolute sum of the temporal gradient.

The methods can include wherein modifying the temporal gradient is based on an absolute mean difference between the reference blocks being greater than a threshold value.

The methods can include wherein the threshold value is 4.

The methods can include wherein modifying the temporal gradient is based on an absolute mean difference between the reference blocks being less than a threshold value.

The methods can include wherein the threshold value is 20.

The methods can include wherein modifying the temporal gradient is based on an absolute mean difference between the reference blocks being within a threshold range.

The methods can include wherein BIO is in a disabled operational state based on the absolute mean difference being greater than a threshold value.

The methods can include wherein the threshold value or the threshold range is indicated in VPS, SPS, PPS, a picture, a slice, or a tile level.

The methods can include wherein the threshold value or the threshold range is different for different coding units (CUs), largest coding units (LCUs), slices, tiles, or pictures.

The methods can include wherein the threshold value or the threshold range is based on a decoded or encoded pixel value.

The methods can include wherein the threshold value or the threshold range is based on a reference picture.

The methods can include wherein determining the spatial gradient includes determining a weighted average of an intra prediction block and an inter prediction block in each prediction direction.

The methods can include wherein the flag is provided in advanced motion vector prediction (AMVP) mode, and in merge mode the flag is inherited from one or both of spatial neighboring blocks or temporal neighboring blocks.

The methods can include wherein the flag is not signaled for uni-predicted blocks.

The methods can include wherein the flag is not signaled for bi-predicted blocks with reference pictures that are preceding pictures or following pictures in display order.

The methods can include wherein the flag is not signaled for bi-predicted blocks.

The methods can include wherein the flag is not signaled for intra coded blocks.

The methods can include wherein the flag is not signaled for blocks coded with hybrid intra and inter prediction mode.

The methods can include wherein the flag is signaled based on a dimension of the first video block.

The methods can include wherein the flag is signaled in a VPS, a SPS, or a PPS.

The methods can include wherein the flag is based on a temporal layer of a picture associated with the first video block.

The methods can include wherein the flag is based on a quantization parameter (QP) of a picture associated with the first video block.

FIG. 17 is a block diagram showing an example video processing system 1700 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of the system 1700. The system 1700 may include input 1702 for receiving video content. The video content may be received in a raw or uncompressed format, e.g., 8 or 10 bit multi-component pixel values, or may be in a compressed or encoded format. The input 1702 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interface include wired interfaces such as Ethernet, passive optical network (PON), etc. and wireless interfaces such as wireless fidelity (Wi-Fi) or cellular interfaces.

The system 1700 may include a coding component 1704 that may implement the various coding or encoding methods described in the present disclosure. The coding component 1704 may reduce the average bitrate of video from the input 1702 to the output of the coding component 1704 to produce a coded representation of the video. The coding techniques are therefore sometimes called video compression or video transcoding techniques. The output of the coding component 1704 may be either stored, or transmitted via a communication connected, as represented by the component 1706. The stored or communicated bitstream (or coded) representation of the video received at the input 1702 may be used by the component 1708 for generating pixel values or displayable video that is sent to a display interface 1710. The process of generating user-viewable video from the bitstream representation is sometimes called video decompression. Furthermore, while certain video processing operations are referred to as “coding” operations or tools, it will be appreciated that the coding tools or operations are used at an encoder and corresponding decoding tools or operations that reverse the results of the coding will be performed by a decoder.

Examples of a peripheral bus interface or a display interface may include universal serial bus (USB) or high definition multimedia interface (HDMI) or Displayport, and so on. Examples of storage interfaces include serial advanced technology attachment (SATA), peripheral component interconnect (PCI), integrated drive electronics (IDE) interface, and the like. The techniques described in the present disclosure may be embodied in various electronic devices such as mobile phones, laptops, smartphones or other devices that are capable of performing digital data processing and/or video display.

It will be appreciated that the disclosed techniques may be embodied in video encoders or decoders to improve compression efficiency when the coding units being compressed have shaped that are significantly different than the traditional square shaped blocks or rectangular blocks that are half-square shaped. For example, new coding tools that use long or tall coding units such as 4×32 or 32×4 sized units may benefit from the disclosed techniques.

In some implementations, a method of video processing may be performed as follows:

- using, during a conversion between a video block and a bitstream representation of the video block, a filtering method for calculating a spatial gradient and a temporal gradient, and
- performing the conversion using the filtering.

Here, the conversion includes generating the bitstream representation from pixel values of the video block or generating the pixels values from the bitstream representation.

In some embodiments, the spatial and temporal gradients are calculated using shifted sample differences.

In some embodiments, the spatial and temporal gradients are calculated using modified samples.

Additional details of this method are provided in item 1 discussed in Section 4.

FIG. 18 is a flowchart for an example of a video processing method. Steps of this method are discussed in example 2 of Section 4 of this disclosure. The method includes (at step 1805) performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion of the current block includes determining whether a use of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique to the current block is enabled or disabled, and wherein the determining the use of the BIO technique or the DMVR technique is based on a cost criterion associated with the current block.

FIG. 19 is a flowchart for an example of a video processing method. Steps of this method are discussed in example 3 of Section 4 of this disclosure. The method includes (at step 1905) performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion of the current block includes determining whether a use of a decoder-side motion vector refinement (DMVR) technique to the current block is enabled or disabled, and wherein the DMVR technique includes refining motion information of the current block based on a cost criterion other than a mean removed sum of absolute differences (MRSAD) cost criterion.

FIG. 20 is a flowchart for an example of a video processing method. Steps of this method are discussed in example 4 of Section 4 of this disclosure. The method includes (at step 2005) performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion of the current block includes determining whether a use of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique to the current block is enabled or disabled, and wherein the determining the use of the BIO technique or the DMVR technique is based on computing that a mean value difference of a pair of reference blocks associated with the current block exceeds a threshold value.

FIG. 21 is a flowchart for an example of a video processing method. Steps of this method are discussed in example 6 of Section 4 of this disclosure. The method includes (at step 2105) modifying a first reference block to generate a first modified reference block, and a second reference block to generate a second modified reference block, wherein both the first reference block and the second reference block are associated with a current block of visual media data. The method further includes (at step 2110) determining differences between the first modified reference block and the second modified reference block, the differences including one or more of: a sum of absolute transformed differences (SATD), a mean removed sum of absolute transformed differences (MRSATD), a sum of squares error (SSE), a mean removed sum of squares error (MRSSE), a mean value difference, or gradient values. The method includes (at step 2115) performing a conversion between the current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion includes a use of the differences between the first modified reference block and the second modified reference block generated from respectively modifying the first reference block and the second reference block.

FIG. 22 is a flowchart for an example of a video processing method. Steps of this method are discussed in example 7 of Section 4 of this disclosure. The method includes (at step 2205) determining a temporal gradient or a modified temporal gradient using reference pictures associated with a current block of visual media data, the temporal gradient or the modified temporal gradient indicative of differences between the reference pictures. The method includes (at step 2210) performing a conversion between the current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion includes a use of a bi-directional optical flow (BIO) technique based in part on the temporal gradient or the modified temporal gradient.

FIG. 23 is a flowchart for an example of a video processing method. Steps of this method are discussed in example 8 of Section 4 of this disclosure. The method includes (at step 2305) determining a first temporal gradient using reference pictures associated with a first video block or a sub-block thereof. The method includes (at step 2310) determining a second temporal gradient using reference pictures associated with a second video block or a sub-block thereof. The method includes (at step 2315) performing a modification of the first temporal gradient and a modification of the second temporal gradient to generate a modified first temporal gradient and a modified second temporal gradient, wherein the modification of the first temporal gradient associated with the first video block is different from the modification of the second temporal gradient associated with the second video block. The method includes (at step 2320) performing a conversion of the first video block and the second video block to their corresponding coded representation.

FIG. 24 is a flowchart for an example of a video processing method. Steps of this method are discussed in example 9 of Section 4 of this disclosure. The method includes (at step 2405) modifying one or both of a first inter reference block and a second inter reference block associated with a current block. The method includes (at step 2410) determining, based on using the one or both modified first inter reference block and/or the modified second inter reference block, a spatial gradient associated with the current block in accordance with applying a bi-directional optical (BIO) flow technique. The method includes (at step 2415) performing a conversion between the current block and a corresponding coded representation, wherein the conversion includes a use of the spatial gradient associated with the current block.

FIG. 25 is a flowchart for an example of a video processing method. Steps of this method are discussed in example 10 of Section 4 of this disclosure. The method includes (at step 2505) performing a determination, by a processor, that a flag which can be signaled at multiple levels indicates, at least in part, that one or both of a decoder-side motion vector refinement (DMVR) technique or a bi-directional optical flow (BIO) technique is to be enabled for a current block. The method includes (at step 2510) performing a conversion between the current block and a corresponding coded representation, wherein the coded representation includes the flag indicating whether the one or both of the DMVR technique and/or the BIO technique is enabled.

FIG. 26 is a flowchart for an example of a video processing method. Steps of this method are discussed in example 11 of Section 4 of this disclosure. The method includes (at step 2605) performing a determination, by a processor that a decoder-side motion vector refinement (DMVR) technique is to be enabled for a current block, wherein the determination is based exclusively on a height of the current block. The method includes (at step 2610) performing a conversion between the current block and a corresponding coded representation.

FIG. 27 is a flowchart for an example of a video processing method. Steps of this method are discussed in example 12 of Section 4 of this disclosure. The method includes (at step 2705) performing a conversion between a current block of visual media data and a corresponding coded representation of visual media data, wherein the conversion includes a use of rules associated with one or both of a decoder-side motion vector refinement (DMVR) technique or a bi-directional optical flow (BIO) technique on the current block, wherein the rules associated with the DMVR technique are consistent with application to the BIO technique, and wherein determining whether the use of the one or both of the BIO technique or the DMVR technique on the current block is enabled or disabled is based on applying the rules.

Some embodiments of the present technology are discussed in clause-based format.

1. A method of visual media processing, comprising:

- performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data,
- wherein the conversion of the current block includes determining whether a use of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique to the current block is enabled or disabled, and
- wherein the determining the use of the BIO technique or the DMVR technique is based on a cost criterion associated with the current block.

2. The method of clause 1, wherein the cost criterion is based on one or more of: a sum of absolute transformed differences (SATD), a mean removed sum of absolute transformed differences (MRSATD), a sum of squares error (SSE), a mean removed sum of squares error (MRSSE), a mean value difference, or gradient values.

3. The method of any one or more of clauses 1-2, wherein the cost criterion is associated with a sub-block of the current block.

4. The method of clause 3, wherein a sub-block-level cost criterion is different from a block-level cost criterion.

5. The method of any one or more of clauses 1-4, further comprising:

- upon determining that one or more of the gradient values, an average of the gradient values, or a range of the gradient values is outside a threshold range, disabling application of the BIO technique and/or the DMVR technique.

6. The method of clause 1, wherein the cost criterion associated with the current block is signaled in the coded representation.

7. The method of clause 6, wherein the cost criterion is signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a tile group header, a picture header, or a slice header.

8. A method of visual media processing, comprising:

- performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data,
- wherein the conversion of the current block includes determining whether a use of a decoder-side motion vector refinement (DMVR) technique to the current block is enabled or disabled, and
- wherein the DMVR technique includes refining motion information of the current block based on a cost criterion other than a mean removed sum of absolute differences (MRSAD) cost criterion.

9. The method of clause 8, wherein the cost criterion associated with the current block is based on one or more of: a sum of absolute transformed differences (SATD), a mean removed sum of absolute transformed differences (MRSATD), a sum of squares error (SSE), or a mean removed sum of squares error (MRSSE).

10. The method of any one or more of clauses 8-9, wherein the cost criterion is associated with a sub-block of the current block.

11. The method of clause 10, further comprising:

- splitting the current block into multiple sub-blocks of size M×N, wherein the cost criterion is based on the motion information associated with each of the multiple sub-blocks; and
- generating costs corresponding to each of the multiple sub-blocks.

12. The method of clause 11, further comprising:

- summing at least a subset of the costs corresponding to each of the multiple sub-blocks to generate a resulting cost associated with the current block.

13. A method of visual media processing, comprising:

- performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data,
- wherein the conversion of the current block includes determining whether a use of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique to the current block is enabled or disabled, and
- wherein the determining the use of the BIO technique or the DMVR technique is based on computing that a mean value difference of a pair of reference blocks associated with the current block exceeds a threshold value.

14. The method of clause 13, wherein the threshold value is a first threshold value, further comprising:

- upon determining that a mean value difference of a pair of reference sub-blocks associated with a sub-block of the current block exceeds a second threshold value, disabling application of the BIO technique and/or the DMVR technique.

15. The method of clause 14, wherein the first threshold value and/or the second threshold value are predefined numbers.

16. The method of clause 14, wherein the first threshold value and/or the second threshold value are based on dimensions of the current block.

17. A method of visual media processing, comprising:

- modifying a first reference block to generate a first modified reference block, and a second reference block to generate a second modified reference block, wherein both the first reference block and the second reference block are associated with a current block of visual media data;
- determining differences between the first modified reference block and the second modified reference block, the differences including one or more of: a sum of absolute transformed differences (SATD), a mean removed sum of absolute transformed differences (MRSATD), a sum of squares error (SSE), a mean removed sum of squares error (MRSSE), a mean value difference, or gradient values; and
- performing a conversion between the current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion includes a use of the differences between the first modified reference block and the second modified reference block generated from respectively modifying the first reference block and the second reference block.

18. The method of clause 17, wherein the modifying the first reference block and the second reference block includes:

- computing a first arithmetic mean based on sample values included in the first reference block and a second arithmetic mean based on sample values included in the second reference block;
- subtracting the first arithmetic mean from samples included in the first reference block and the second arithmetic mean from samples included in the second reference block.

19. The method of clause 18, wherein the first arithmetic mean and the second arithmetic mean are based on a subset of samples respectively included in the first reference block and the second reference block.

20. The method of any one or more of clauses 17-19, wherein the first reference block and/or the second reference block are sub-blocks associated with the current block.

21. A method of visual media processing, comprising:

- determining a temporal gradient or a modified temporal gradient using reference pictures associated with a current block of visual media data, the temporal gradient or the modified temporal gradient indicative of differences between the reference pictures; and
- performing a conversion between the current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion includes a use of a bi-directional optical flow (BIO) technique based in part on the temporal gradient or the modified temporal gradient.

22. The method of clause 21, further comprising:

- prematurely terminating the BIO technique, in response to determining that the temporal gradient or the modified temporal gradient is less than or equal to a threshold.

23. The method of clause 22, further comprising:

- adjusting the threshold based on the number of samples used for calculating an absolute sum of the temporal gradient or the modified gradient.

24. The method of any one or more of clauses 21-23, wherein the differences are related to an absolute sum of the temporal gradient.'

25. The method of any one or more of clauses 21-24, wherein the differences between the reference pictures correspond to differences between a first portion of a first reference picture and a second portion of a second reference picture.

26. The method of any one or more of clauses 21-25, wherein the reference pictures are associated with a sub-block of the current block.

27. A method of visual media processing, comprising:

- determining a first temporal gradient using reference pictures associated with a first video block or a sub-block thereof;
- determining a second temporal gradient using reference pictures associated with a second video block or a sub-block thereof;
- performing a modification of the first temporal gradient and a modification of the second temporal gradient to generate a modified first temporal gradient and a modified second temporal gradient, wherein the modification of the first temporal gradient associated with the first video block is different from the modification of the second temporal gradient associated with the second video block; and
- performing a conversion of the first video block and the second video block to their corresponding coded representation.

28. The method of clause 27, wherein the modification of the first temporal gradient and/or the modification of the second temporal gradient is conditionally based on an absolute mean difference between the reference pictures associated with the first video block and/or the second video block being greater than a threshold value.

29. The method of clause 28, wherein the threshold value is 4.

30. The method of clause 27, wherein the modification of the first temporal gradient and/or the modification of the second temporal gradient is conditionally based on an absolute mean difference between the reference pictures associated with the first video block and/or the second video block being less than a threshold value.

31. The method of clause 30, wherein the threshold value is 20.

32. The method of clause 27, wherein the modification of the first temporal gradient and/or the modification of the second temporal gradient is conditionally based on an absolute mean difference between the reference pictures associated with the first video block and/or the second video block being within a threshold range.

33. The method of any of clauses 27-32, further comprising:

- disabling a use of a bi-directional optical flow (BIO) technique on the first video block and/or the second block based on an absolute mean difference between the reference pictures associated with the first video block and/or the second video block being greater than a threshold value.

34. The method of any one or more of clauses 27-33, wherein the threshold value or the threshold range is indicated in VPS, SPS, PPS, a picture, a slice, or a tile level associated with the first video block and/or the second video block.

35. The method of any one or more of clauses 27-33, wherein the threshold value or the threshold range are implicitly predefined parameters.

36. The method of any one or more of clauses 27-33, wherein the threshold value or the threshold range is different for different coding units (CUs), largest coding units (LCUs), slices, tiles, or pictures associated with the first video block and/or the second video block.

37. The method of any one or more of clauses 27-33, wherein the threshold value or the threshold range is based on a decoded or an encoded pixel value associated with the first video block and/or the second video block.

38. The method of any one or more of clauses 27-33, wherein the threshold value or the threshold range for a first set of reference pictures is different from the threshold value or the threshold range for a second set of reference pictures.

39. The method of clause 27, wherein the modification of the first temporal gradient and/or the modification of the second temporal gradient is conditionally based on an absolute mean of the reference pictures associated with the first video block and/or the second video block being greater than a threshold value.

40. The method of clause 39, wherein the threshold value is 40.

41. The method of clause 27, wherein the modification of the first temporal gradient and/or the modification of the second temporal gradient is conditionally based on an absolute mean of the reference pictures associated with the first video block and/or the second video block being smaller than a threshold value.

42. The method of clause 41, wherein the threshold value is 100.

43. The method of clause 27, wherein the modification of the first temporal gradient and/or the modification of the second temporal gradient is conditionally based on an absolute mean of the reference pictures associated with the first video block and/or the second video block being within a threshold range.

44. The method of clause 27, wherein the modification of the first temporal gradient and/or the modification of the second temporal gradient is conditionally based on an absolute mean of the reference pictures associated with the first video block and/or the second video block being greater than an absolute mean difference of the reference pictures associated with the first video block and/or the second video block times a multiplication factor.

45. The method of clause 27, wherein the modification of the first temporal gradient and/or the modification of the second temporal gradient is conditionally based on an absolute mean of the reference pictures associated with the first video block and/or the second video block being less than an absolute mean difference of the reference pictures associated with the first video block and/or the second video block times a multiplication factor.

46. The method of any one or more of clauses 44-45, wherein the multiplication factor is 4.5.

47. A method of visual media processing, comprising:

- modifying one or both of a first inter reference block and a second inter reference block associated with a current block;
- determining, based on using the one or both modified first inter reference block and/or the modified second inter reference block, a spatial gradient associated with the current block in accordance with applying a bi-directional optical (BIO) flow technique; and
- performing a conversion between the current block and a corresponding coded representation, wherein the conversion includes a use of the spatial gradient associated with the current block.

48. The method of clause 47, wherein determining the spatial gradient includes:

- generating two prediction blocks based on a weighted averaging of an intra prediction block and an inter prediction block associated with the current block; and
- using the two prediction blocks for determining the spatial gradient associated with the current block.

49. The method of clause 48, further comprising:

- generating, using the BIO technique, a refined prediction block from the two prediction blocks; and
- using the refined prediction block for predicting sub-blocks and/or samples of the current block.

50. A method of visual media processing, comprising:

- performing a determination, by a processor, that a flag which can be signaled at multiple levels indicates, at least in part, that one or both of a decoder-side motion vector refinement (DMVR) technique or a bi-directional optical flow (BIO) technique is to be enabled for a current block; and
- performing a conversion between the current block and a corresponding coded representation, wherein the coded representation includes the flag indicating whether the one or both of the DMVR technique and/or the BIO technique is enabled.

51. The method of clause 50, wherein the flag is signaled in the coded representation in response to detecting that an advanced motion vector prediction (AMVP) technique is enabled for the current block.

52. The method of clause 50, wherein the flag is derived from one or both of spatial neighboring blocks or temporal neighboring blocks associated with the current block in response to detecting that a merge mode is enabled for the current block.

53. The method of clause 52, wherein, the flag is inherited from a selected merging candidate if the selected merging candidate is a spatial merging candidate.

54. The method of clause 52, wherein, the flag is inherited from a selected merging candidate if the selected merging candidate is a temporal merging candidate.

55. The method of clause 50, wherein, a cost criterion associated with the current block is used to determine whether the one or both of the DMVR technique and/or the BIO technique is enabled, and the flag signaled in the coded representation is used to indicate whether such determination is correct or not.

56. The method of clause 55, wherein the cost criterion associated with the current block is a sum of absolute difference (SAD) between two reference blocks of the current block, and wherein the determination that the one or both of the DMVR technique and/or the BIO technique is enabled applies when the cost criterion is greater than a threshold.

57. The method of clause 50, further comprising:

- upon determining that the current block is a uni-predicted block, skipping signaling of the flag in the coded representation.

58. The method of clause 50, further comprising:

- upon determining that the current block is a bi-predicted block associated with a pair of reference pictures both of which are either preceding or succeeding in a display order, skipping signaling of the flag in the coded representation.

59. The method of clause 50, further comprising:

- upon determining that the current block is a bi-predicted block associated with a pair of reference pictures with different picture order count (POC) distances from a current picture associated with the current block, skipping signaling of the flag in the coded representation.

60. The method of clause 50, further comprising:

- upon determining that the current block is an intra coded block, skipping signaling of the flag in the coded representation.

61. The method of clause 50, further comprising:

- upon determining that the current block is a hybrid intra and inter predicted block, skipping signaling of the flag in the coded representation.

62. The method of clause 50, further comprising:

- upon determining that the current block is associated with at least one block of a picture same as a reference block, skipping signaling of the flag in the coded representation.

63. The method of clause 50, further comprising:

- upon determining that a dimension of the current block is smaller than a threshold value, skipping signaling of the flag in the coded representation.

64. The method of clause 50, further comprising:

- upon determining that a dimension of the current block is greater than or equal to a threshold value, skipping signaling of the flag in the coded representation.

65. The method of clause 50, further comprising:

- upon determining that a precision of motion information associated with the current block is an integer precision, skipping signaling of the flag in the coded representation.

66. The method of clause 50, further comprising:

- upon determining that a temporal layer associated with the picture containing the current block is beyond a threshold value, skipping signaling of the flag in the coded representation.

67. The method of clause 50, further comprising:

- upon determining that a quantization parameter associated with the current block is beyond a threshold value, skipping signaling of the flag in the coded representation.

68. The method of any one or more of clauses 50-67, further comprising:

- in response to determining that signaling of the flag in the coded representation is skipped, deriving a value of the flag as a Boolean true or false.

69. The method of any one or more of clauses 50-67, further comprising:

- upon determining that the flag is a Boolean true, enabling the one or both of the DMVR technique or the BIO technique.

70. The method of any one or more of clauses 50-67, further comprising:

- upon determining that the flag is a Boolean false, disabling the one or both of the DMVR technique or the BIO technique.

71. The method of any one or more of clauses 50-67, further comprising:

- upon determining that the flag is a Boolean true, the determination of the enabling or disabling one or both of the DMVR technique or the BIO technique based on at least one cost criterion is determined as correct.

72. The method of any one or more of clauses 50-67, further comprising:

- upon determining that the flag is a Boolean false, the determination of the enabling or disabling one or both of the DMVR technique or the BIO technique based on at least one cost criterion is determined as incorrect.

73. The method of any one or more of clause 50-67, wherein the flag is signaled in a slice header, a picture header, a tile header, a Video Parameter Set (VPS), a Sequence Parameter Set (SPS), or a Picture Parameter Set (PPS).

74. The method of clause 50, wherein a first flag is signaled to indicate whether the DMVR technique is enabled or not, and a second flag is signaled to indicate whether the BIO technique is enabled or not.

75. The method of any one or more of clauses 64-74, further comprising:

- upon determining that the flag for the DMVR technique is a Boolean true, disabling the DMVR technique for a slice, a tile, a video, a sequence or a picture.

76. The method of any one or more of clauses 64-74, further comprising:

- upon determining that the flag for the DMVR technique is a Boolean false, enabling the DMVR technique for a slice, a tile, a video, a sequence or a picture.

77. The method of any one or more of clauses 64-74, further comprising:

- upon determining that the flag for the BIO technique is a Boolean true, disabling the BIO technique for a slice, a tile, a video, a sequence or a picture.

78. The method of any one or more of clauses 64-74, further comprising:

- upon determining that the flag for the BIO technique is a Boolean false, enabling the BIO technique for a slice, a tile, a video, a sequence or a picture.

79. A method of visual media processing, comprising:

- performing a determination, by a processor that a decoder-side motion vector refinement (DMVR) technique is to be enabled for a current block, wherein the determination is based exclusively on a height of the current block; and
- performing a conversion between the current block and a corresponding coded representation.

80. The method of clause 79, further comprising:

- in response to determining that the DMVR technique is enabled, verifying that the height of the current block is greater than or exceeds a threshold parameter.

81. The method of clause 80, wherein the threshold parameter equals 4.

82. The method of clause 80, wherein the threshold parameter equals 8.

83. A method of visual media processing, comprising:

- performing a conversion between a current block of visual media data and a corresponding coded representation of visual media data, wherein the conversion includes a use of rules associated with one or both of a decoder-side motion vector refinement (DMVR) technique or a bi-directional optical flow (BIO) technique on the current block, wherein the rules associated with the DMVR technique are consistent with application to the BIO technique; and
- wherein determining whether the use of the one or both of the BIO technique or the DMVR technique on the current block is enabled or disabled is based on applying the rules.

84. The method of clause 83, wherein a rule to determine whether the DMVR technique is enabled is same as a rule to determine whether the BIO technique is enabled.

85. The method of clause 84, wherein the rule to determine whether the BIO technique and/or the DMVR technique is enabled specifies verifying that a height of the current block is greater than or equal to a threshold value.

86. The method of clause 84, wherein the rule to determine whether the BIO technique and/or the DMVR technique is enabled specifies verifying that both of a width and a height of the current block are greater than or equal to a threshold value.

87. The method of any one or more of clauses 85 or 86, wherein the threshold value is 4 or 8.

88. The method of clause 84, wherein the rule to determine whether the BIO technique and/or the DMVR technique is enabled specifies verifying that a size of the current block is greater than or equal to a threshold value.

89. The method of clause 86, wherein the threshold value is 64 or 128.

90. The method of clause 84, wherein the rule to determine whether the BIO technique and/or the DMVR technique is enabled specifies verifying that the current block is not coded in a Bi-prediction with CU-level Weight (BCW) mode, wherein unequal weights are used for two reference blocks from two reference lists.

91. The method of clause 84, wherein the rule to determine whether the BIO technique and/or the DMVR technique is enabled specifies verifying that the current block is a bi-predicted block associated with a pair of reference pictures with a same picture order count (POC) distance from a current picture associated with the current block.

92. The method of clause 91, wherein the pair of reference pictures include a preceding picture and a succeeding picture of the current picture associated with the current block, in display order.

93. A video decoding apparatus comprising a processor configured to implement a method recited in one or more of clauses 1 to 92.

94. A video encoding apparatus comprising a processor configured to implement a method recited in one or more of clauses 1 to 92.

95. A computer program product having computer code stored thereon, the code, when executed by a processor, causes the processor to implement a method recited in any of clauses 1 to 92.

96. A method, apparatus or system described in the present disclosure.

The disclosed and other solutions, examples, embodiments, modules and the functional operations described in this disclosure can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this disclosure and their structural equivalents, or in combinations of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more them. The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this disclosure can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random-access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and compact disc, read-only memory (CD ROM) and digital versatile disc read-only memory (DVD-ROM) disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

While the present disclosure contains many specifics, these should not be construed as limitations on the scope of any subject matter or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular techniques. Certain features that are described in the present disclosure in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in the present disclosure should not be understood as requiring such separation in all embodiments.

Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in the present disclosure.

Claims

1. A method of processing video data, comprising: determining, for a current block of a video, an initial prediction sample based on motion compensation;refining the initial prediction sample, based on an optical flow refinement technology, with a prediction sample offset to acquire a final prediction sample; andperforming a conversion between the current block and a bitstream of the video based on the final prediction sample,wherein the prediction sample offset is determined based on at least one spatial gradient of the initial prediction sample, wherein the spatial gradient is calculated based on at least a difference between two first prediction samples from a same reference picture list,wherein before calculating the difference between the two first prediction samples, values of the two first prediction samples are right shifted with a first value,wherein the prediction sample offset is determined further based on at least one temporal gradient, wherein the temporal gradient is calculated based on at least a difference between two second prediction samples from different reference picture lists,wherein a shifting rule of the difference between the two second prediction samples is same as that of the difference between the two first prediction samples, and the shifting rule indicates an order of a right-shifting operation and a subtraction operation,wherein for a sample location (x,y) in the current block, the two first prediction samples have locations (hx+1, vy) and (hx−1, vy) corresponding to the same reference picture list or locations (hx, vy+1) and (hx, vy−1) corresponding to the same reference picture list, andwherein the same reference picture list is a same reference picture list 0 or a same reference picture list 1, wherein hx=Clip3(1, nCbW, x) and vy=Clip3(1, nCbH, y), wherein nCbW and nCbH are a width and a height of the current block, and wherein Clip3 is a clipping function which is defined as:
2. The method of claim 1, wherein before calculating the difference between the two second prediction samples, values of the two second prediction samples are right shifted with a second value.
3. The method of claim 2, wherein for the sample location (x,y) in the current block, the two second prediction samples have locations (hx, vy) corresponding to a reference picture list 0 and a reference picture list 1.
4. The method of claim 2, wherein the first value is different from the second value.
5. The method of claim 1, wherein whether the optical flow refinement technology is enabled is based on a condition related with a size of the current block.
6. The method of claim 5, wherein whether a decoder-side motion vector refinement technique is enabled for the current block is based on a same condition, wherein the decoder-side motion vector refinement technique is used to derive a refined motion information of the current block based on a cost between at least one prediction sample acquired based on at least one reference sample of reference picture list 0 and at least one prediction sample acquired based on at least one reference sample of reference picture list 1.
7. The method of claim 6, wherein the optical flow refinement technology and the decoder-side motion vector refinement technique are enabled at least based on a height of the current block is equal to or greater than threshold (T1).
8. The method of claim 7, wherein T1=8.
9. The method of claim 1, wherein performing the conversion includes decoding the current block from the bitstream.
10. The method of claim 1, wherein performing the conversion includes encoding the current block into the bitstream.
11. An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to: determine, for a current block of a video, an initial prediction sample based on motion compensation;refine the initial prediction sample, based on an optical flow refinement technology, with a prediction sample offset to acquire a final prediction sample; andperform a conversion between the current block and a bitstream of the video based on the final prediction sample,wherein the prediction sample offset is determined based on at least one spatial gradient of the initial prediction sample, wherein the spatial gradient is calculated based on at least a difference between two first prediction samples from a same reference picture list,wherein before calculating the difference between the two first prediction samples, values of the two first prediction samples are right shifted with a first value,wherein the prediction sample offset is determined further based on at least one temporal gradient, wherein the temporal gradient is calculated based on at least a difference between two second prediction samples from different reference picture lists,wherein a shifting rule of the difference between the two second prediction samples is same as that of the difference between the two first prediction samples, and the shifting rule indicates an order of a right-shifting operation and a subtraction operation,wherein for a sample location (x,y) in the current block, the two first prediction samples have locations (hx+1, vy) and (hx−1, vy) corresponding to the same reference picture list or locations (hx, vy+1) and (hx, vy−1) corresponding to the same reference picture list, andwherein the same reference picture list is a same reference picture list 0 or a same reference picture list 1, wherein hx=Clip3(1, nCbW, x) and vy=Clip3(1, nCbH, y), wherein nCbW and nCbH are a width and a height of the current block, and wherein Clip3 is a clipping function which is defined as:
12. The apparatus of claim 11, wherein before calculating the difference between the two second prediction samples, values of the two second prediction samples are right shifted with a second value.
13. The apparatus of claim 12, wherein for the sample location (x,y) in the current block, the two second prediction samples have locations (hx, vy) corresponding to a reference list picture 0 and a reference picture list 1.
14. The apparatus of claim 12, wherein the first value is different from the second value.
15. The apparatus of claim 11, wherein whether the optical flow refinement technology is enabled is based on a condition related with a size of the current block, and wherein whether a decoder-side motion vector refinement technique is enabled for the current block is based on a same condition, wherein the decoder-side motion vector refinement technique is used to derive a refined motion information of the current block based on a cost between at least one prediction sample acquired based on at least one reference sample of reference picture list 0 and at least one prediction sample acquired based on at least one reference sample of reference picture list 1.
16. A non-transitory computer-readable storage medium storing instructions that cause a processor to: determine, for a current block of a video, an initial prediction sample based on motion compensation;refine the initial prediction sample, based on an optical flow refinement technology, with a prediction sample offset to acquire a final prediction sample; andperform a conversion between the current block and a bitstream of the video based on the final prediction sample,wherein the prediction sample offset is determined based on at least one spatial gradient of the initial prediction sample, wherein the spatial gradient is calculated based on at least a difference between two first prediction samples from a same reference picture list,wherein before calculating the difference between the two first prediction samples, values of the two first prediction samples are right shifted with a first value,wherein the prediction sample offset is determined further based on at least one temporal gradient, wherein the temporal gradient is calculated based on at least a difference between two second prediction samples from different reference picture lists,wherein a shifting rule of the difference between the two second prediction samples is same as that of the difference between the two first prediction samples, and the shifting rule indicates an order of a right-shifting operation and a subtraction operation,wherein for a sample location (x,y) in the current block, the two first prediction samples have locations (hx+1, vy) and (hx−1, vy) corresponding to the same reference picture list or locations (hx, vy+1) and (hx, vy−1) corresponding to the same reference picture list, andwherein the same reference picture list is a same reference picture list 0 or a same reference picture list 1, wherein hx=Clip3(1, nCbW, x) and vy=Clip3(1, nCbH, y), wherein nCbW and nCbH are a width and a height of the current block, and wherein Clip3 is a clipping function which is defined as:
17. The non-transitory computer-readable storage medium of claim 16, wherein before calculating the difference between the two second prediction samples, values of the two second prediction samples are right shifted with a second value, and wherein for the sample location (x,y) in the current block, the two second prediction samples have locations (hx, vy) corresponding to a reference picture list 0 and a reference picture list 1, and wherein the first value is different from the second value.
18. The non-transitory computer-readable storage medium of claim 16, wherein whether the optical flow refinement technology is enabled is based on a condition related with a size of the current block, and wherein whether a decoder-side motion vector refinement technique is enabled for the current block is based on a same condition, wherein the decoder-side motion vector refinement technique is used to derive a refined motion information of the current block based on a cost between at least one prediction sample acquired based on at least one reference sample of reference picture list 0 and at least one prediction sample acquired based on at least one reference sample of reference picture list 1.
19. A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises: determining, for a current block of the video, an initial prediction sample based on motion compensation;refining the initial prediction sample, based on an optical flow refinement technology, with a prediction sample offset to acquire a final prediction sample; andgenerating the bitstream based on the final prediction sample,wherein the prediction sample offset is determined based on at least one spatial gradient of the initial prediction sample, wherein the spatial gradient is calculated based on at least a difference between two first prediction samples from a same reference picture list,wherein before calculating the difference between the two first prediction samples, values of the two first prediction samples are right shifted with a first value,wherein the prediction sample offset is determined further based on at least one temporal gradient, wherein the temporal gradient is calculated based on at least a difference between two second prediction samples from different reference picture lists,wherein a shifting rule of the difference between the two second prediction samples is same as that of the difference between the two first prediction samples, and the shifting rule indicates an order of a right-shifting operation and a subtraction operation,wherein for a sample location (x,y) in the current block, the two first prediction samples have locations (hx+1, vy) and (hx−1, vy) corresponding to the same reference picture list or locations (hx, vy+1) and (hx, vy−1) corresponding to the same reference picture list, andwherein the same reference picture list is a same reference picture list 0 or a same reference picture list 1, wherein hx=Clip3(1, nCbW, x) and vy=Clip3(1, nCbH, y), wherein nCbW and nCbH are a width and a height of the current block, and wherein Clip3 is a clipping function which is defined as:
20. The non-transitory computer-readable recording medium of claim 19, wherein before calculating the difference between the two second prediction samples, values of the two second prediction samples are right shifted with a second value, and wherein for the sample location (x,y) in the current block, the two second prediction samples have locations (hx, vy) corresponding to a reference picture list 0 and a reference picture list 1, and wherein the first value is different from the second value.

Priority Claims (3)

Number	Date	Country	Kind
PCT/CN2018/116371	Nov 2018	WO	international
PCT/CN2019/081155	Apr 2019	WO	international
PCT/CN2019/085796	May 2019	WO	international

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No. 17/317,522, filed on May 11, 2021, which is a continuation of International No. PCT/CN2019/119763, filed on Nov. 20, 2019, which claims the priority to and benefits of International Patent Application No. PCT/CN2018/116371, filed on Nov. 20, 2018, International Patent Application No. PCT/CN2019/081155, filed on Apr. 2, 2019, and International Patent Application No. PCT/CN2019/085796, filed on May 7, 2019. For all purposes under the U.S. law, the entire disclosures of the aforementioned applications are incorporated by reference as part of the disclosure of this application.

US Referenced Citations (298)

Number	Name	Date	Kind
2018132	Kingsbury	Oct 1935	A
6005627	Odaka	Dec 1999	A
6480615	Sun	Nov 2002	B1
6829303	Pearlstein	Dec 2004	B1
7627037	Li	Dec 2009	B2
9215470	Karczewicz	Dec 2015	B2
9247246	Lu	Jan 2016	B2
9294777	Wang	Mar 2016	B2
9374578	Mukherjee	Jun 2016	B1
9445103	Xu	Sep 2016	B2
9509995	Xu	Nov 2016	B2
9521425	Chen	Dec 2016	B2
9549200	Zhou	Jan 2017	B1
9554150	Zhang	Jan 2017	B2
9596448	Thirumalai	Mar 2017	B2
9609343	Chen	Mar 2017	B1
9628795	Zhang	Apr 2017	B2
9641852	Lu	May 2017	B2
9654792	Chiu	May 2017	B2
9667996	Chen	May 2017	B2
9756336	Zhang	Sep 2017	B2
9762927	Chen	Sep 2017	B2
9906813	Zhang	Feb 2018	B2
9955186	Chon	Apr 2018	B2
10009615	Gisquet	Jun 2018	B2
10165252	An	Dec 2018	B2
10230980	Liu	Mar 2019	B2
10244253	Chen	Mar 2019	B2
10257539	An	Apr 2019	B2
10268901	Garud	Apr 2019	B2
10271048	Zhang	Apr 2019	B2
10334281	Zhang	Jun 2019	B2
10341677	Sung	Jul 2019	B2
10349050	Shima	Jul 2019	B2
10390044	Karczewicz	Aug 2019	B2
10477237	Liu	Nov 2019	B2
10523964	Chuang	Dec 2019	B2
10587859	An	Mar 2020	B2
10609423	Chuang	Mar 2020	B2
10645382	Zhang	May 2020	B2
10687069	Li	Jun 2020	B2
10701366	Chen	Jun 2020	B2
10757420	Zhang	Aug 2020	B2
10764592	Zhang	Sep 2020	B2
10778997	Zhang	Sep 2020	B2
10779002	Chen	Sep 2020	B2
10785494	Chien	Sep 2020	B2
10805630	Li	Oct 2020	B2
10805650	Wang	Oct 2020	B2
10812806	Zhang	Oct 2020	B2
10855992	Ye	Dec 2020	B2
10887597	Liu	Jan 2021	B2
10893267	Jang	Jan 2021	B2
10897617	Huang	Jan 2021	B2
10904565	Chuang	Jan 2021	B2
10939130	Xiu	Mar 2021	B2
10986360	Thirumalai	Apr 2021	B2
11057642	Zhang	Jul 2021	B2
11070842	Choi	Jul 2021	B2
11166037	Chiang	Nov 2021	B2
11206419	Lee	Dec 2021	B2
11259044	Jeong	Feb 2022	B2
11277624	Zhang	Mar 2022	B2
11284088	Zhang	Mar 2022	B2
11509923	Zhang	Nov 2022	B1
11509927	Liu	Nov 2022	B2
11509929	Liu	Nov 2022	B2
11516480	Zhang	Nov 2022	B2
11533477	Liu	Dec 2022	B2
11546632	Koo	Jan 2023	B2
11553201	Liu	Jan 2023	B2
11558634	Liu	Jan 2023	B2
11570461	Jeong	Jan 2023	B2
11582460	Wang	Feb 2023	B2
11632566	Liu	Apr 2023	B2
11641467	Liu	May 2023	B2
11706443	Liu	Jul 2023	B2
11838539	Liu	Dec 2023	B2
11843725	Zhang	Dec 2023	B2
11889108	Liu et al.	Jan 2024	B2
11930165	Zhang et al.	Mar 2024	B2
11956449	Zhang	Apr 2024	B2
11956465	Liu et al.	Apr 2024	B2
12041267	Liu et al.	Jul 2024	B2
20040213348	Kim	Oct 2004	A1
20050007492	Renner	Jan 2005	A1
20050201468	Tsai	Sep 2005	A1
20060008000	Ye	Jan 2006	A1
20070009044	Tourapis	Jan 2007	A1
20070047648	Tourapis	Mar 2007	A1
20070160153	Sullivan	Jul 2007	A1
20070188607	Jia	Aug 2007	A1
20080063075	Kondo	Mar 2008	A1
20080086050	Misic	Apr 2008	A1
20090304087	Shibahara	Dec 2009	A1
20110043706	Van Beek	Feb 2011	A1
20110090969	Sung	Apr 2011	A1
20110176611	Huang	Jul 2011	A1
20120057632	Sato	Mar 2012	A1
20120069906	Sato	Mar 2012	A1
20120128071	Celetto	May 2012	A1
20120163711	Nagone	Jun 2012	A1
20120230405	An	Sep 2012	A1
20120257678	Zhou	Oct 2012	A1
20130010864	Teng	Jan 2013	A1
20130051467	Zhou	Feb 2013	A1
20130089145	Guo	Apr 2013	A1
20130136179	Lim	May 2013	A1
20130156096	Yang	Jun 2013	A1
20130202037	Wang	Aug 2013	A1
20130272415	Zhou	Oct 2013	A1
20130279596	Gisquet	Oct 2013	A1
20130287097	Song	Oct 2013	A1
20140002594	Chan	Jan 2014	A1
20140003512	Sato	Jan 2014	A1
20140071235	Zhang	Mar 2014	A1
20140072041	Seregin	Mar 2014	A1
20140177706	Fernandes	Jun 2014	A1
20140226721	Joshi	Aug 2014	A1
20140286408	Zhang	Sep 2014	A1
20140294078	Seregin	Oct 2014	A1
20150030073	Chen	Jan 2015	A1
20150043634	Lin	Feb 2015	A1
20150063440	Pang	Mar 2015	A1
20150181216	Zhang	Jun 2015	A1
20150195527	Zhou	Jul 2015	A1
20150201200	Cheong	Jul 2015	A1
20150229926	Puri	Aug 2015	A1
20150229955	Seregin	Aug 2015	A1
20150264396	Zhang	Sep 2015	A1
20150264406	Kim	Sep 2015	A1
20150271524	Zhang	Sep 2015	A1
20150365649	Chen	Dec 2015	A1
20150373334	Rapaka	Dec 2015	A1
20150373358	Pang	Dec 2015	A1
20150382009	Chen	Dec 2015	A1
20160057420	Pang	Feb 2016	A1
20160100189	Pang	Apr 2016	A1
20160105670	Pang	Apr 2016	A1
20160219278	Chen	Jul 2016	A1
20160219302	Liu	Jul 2016	A1
20160227214	Rapaka	Aug 2016	A1
20160249056	Tsukuba	Aug 2016	A1
20160286229	Li	Sep 2016	A1
20160286232	Li	Sep 2016	A1
20160330439	Yu	Nov 2016	A1
20160337661	Pang	Nov 2016	A1
20160345011	Naing	Nov 2016	A1
20160360205	Chang	Dec 2016	A1
20160366416	Liu	Dec 2016	A1
20170034526	Rapaka	Feb 2017	A1
20170085917	Hannuksela	Mar 2017	A1
20170094285	Said	Mar 2017	A1
20170094305	Li	Mar 2017	A1
20170238020	Karczewicz	Aug 2017	A1
20170280159	Xu	Sep 2017	A1
20170302966	Xu	Oct 2017	A1
20170332095	Zou	Nov 2017	A1
20170332099	Lee	Nov 2017	A1
20170339405	Wang	Nov 2017	A1
20170339425	Jeong	Nov 2017	A1
20170347096	Hong	Nov 2017	A1
20180014028	Liu	Jan 2018	A1
20180041762	Ikai	Feb 2018	A1
20180048909	Liu	Feb 2018	A1
20180070102	Zhang	Mar 2018	A1
20180070105	Jin	Mar 2018	A1
20180098063	Chen	Apr 2018	A1
20180098097	Huang	Apr 2018	A1
20180109806	Zhou	Apr 2018	A1
20180176563	Zhao	Jun 2018	A1
20180176582	Zhao	Jun 2018	A1
20180176587	Panusopone	Jun 2018	A1
20180176596	Jeong	Jun 2018	A1
20180184117	Chen	Jun 2018	A1
20180192071	Chuang	Jul 2018	A1
20180192072	Chen	Jul 2018	A1
20180199057	Chuang	Jul 2018	A1
20180241998	Chen	Aug 2018	A1
20180242024	Chen	Aug 2018	A1
20180249156	Heo	Aug 2018	A1
20180249172	Chen	Aug 2018	A1
20180262773	Chuang	Sep 2018	A1
20180270498	Nakagami	Sep 2018	A1
20180278942	Zhang	Sep 2018	A1
20180278949	Karczewicz	Sep 2018	A1
20180278950	Chen	Sep 2018	A1
20180288410	Park	Oct 2018	A1
20180295385	Alshin	Oct 2018	A1
20180309983	Heo	Oct 2018	A1
20180310017	Chen	Oct 2018	A1
20180324417	Karczewicz	Nov 2018	A1
20180352223	Chen	Dec 2018	A1
20180352226	An	Dec 2018	A1
20180376148	Zhang	Dec 2018	A1
20180376149	Zhang	Dec 2018	A1
20180376166	Chuang	Dec 2018	A1
20190045183	Chen	Feb 2019	A1
20190045184	Zhang	Feb 2019	A1
20190045214	Ikai	Feb 2019	A1
20190045215	Chen	Feb 2019	A1
20190191180	An	Jun 2019	A1
20190222848	Chen	Jul 2019	A1
20190222865	Zhang	Jul 2019	A1
20190238883	Chen	Aug 2019	A1
20190306502	Gadde	Oct 2019	A1
20190313115	Chao	Oct 2019	A1
20190320197	Chen	Oct 2019	A1
20190320199	Chen	Oct 2019	A1
20190335170	Lee	Oct 2019	A1
20190387234	Wang	Dec 2019	A1
20200021833	Xu	Jan 2020	A1
20200029087	Lim	Jan 2020	A1
20200029091	Chien	Jan 2020	A1
20200045336	Xiu	Feb 2020	A1
20200051288	Lim	Feb 2020	A1
20200053386	Abe	Feb 2020	A1
20200068218	Chen	Feb 2020	A1
20200077086	Lee	Mar 2020	A1
20200092545	Xu	Mar 2020	A1
20200128258	Chen	Apr 2020	A1
20200137416	Esenlik	Apr 2020	A1
20200137422	Misra	Apr 2020	A1
20200177878	Choi	Jun 2020	A1
20200213590	Kim	Jul 2020	A1
20200221110	Chien	Jul 2020	A1
20200221122	Ye	Jul 2020	A1
20200252605	Xu	Aug 2020	A1
20200260070	Yoo	Aug 2020	A1
20200260096	Ikai et al.	Aug 2020	A1
20200277878	Avis	Sep 2020	A1
20200296414	Park	Sep 2020	A1
20200304805	Li	Sep 2020	A1
20200314432	Wang	Oct 2020	A1
20200336738	Xiu	Oct 2020	A1
20200344475	Zhu	Oct 2020	A1
20200359024	Misra	Nov 2020	A1
20200366902	Jeong	Nov 2020	A1
20200374543	Liu	Nov 2020	A1
20200382795	Zhang	Dec 2020	A1
20200382807	Liu	Dec 2020	A1
20200396453	Zhang	Dec 2020	A1
20200413069	Lim	Dec 2020	A1
20200413082	Li	Dec 2020	A1
20210006790	Zhang	Jan 2021	A1
20210006803	Zhang	Jan 2021	A1
20210029356	Zhang	Jan 2021	A1
20210029362	Liu	Jan 2021	A1
20210029366	Zhang	Jan 2021	A1
20210029368	Zhang	Jan 2021	A1
20210029370	Li	Jan 2021	A1
20210029372	Zhang	Jan 2021	A1
20210037238	Park	Feb 2021	A1
20210037256	Zhang	Feb 2021	A1
20210051339	Liu	Feb 2021	A1
20210051348	Zhang	Feb 2021	A1
20210051349	Zhang	Feb 2021	A1
20210058618	Zhang	Feb 2021	A1
20210058637	Zhang	Feb 2021	A1
20210058647	Zhang	Feb 2021	A1
20210076050	Zhang	Mar 2021	A1
20210076063	Liu	Mar 2021	A1
20210092378	Zhang	Mar 2021	A1
20210092431	Zhang	Mar 2021	A1
20210092435	Liu et al.	Mar 2021	A1
20210105463	Zhang	Apr 2021	A1
20210105485	Zhang	Apr 2021	A1
20210112248	Zhang	Apr 2021	A1
20210120243	Zhang	Apr 2021	A1
20210144366	Zhang	May 2021	A1
20210144388	Zhang	May 2021	A1
20210144392	Zhang	May 2021	A1
20210144400	Liu	May 2021	A1
20210160527	Chuang	May 2021	A1
20210168357	Toma	Jun 2021	A1
20210211716	Zhang	Jul 2021	A1
20210227245	Liu	Jul 2021	A1
20210227246	Liu	Jul 2021	A1
20210227250	Liu	Jul 2021	A1
20210235083	Liu	Jul 2021	A1
20210266530	Liu	Aug 2021	A1
20210266585	Liu	Aug 2021	A1
20210274205	Park	Sep 2021	A1
20210274213	Xiu	Sep 2021	A1
20210281865	Liu et al.	Sep 2021	A1
20210297688	Xu	Sep 2021	A1
20210314586	Li	Oct 2021	A1
20210329257	Sethuraman	Oct 2021	A1
20210344952	Xiu	Nov 2021	A1
20210368172	Lim	Nov 2021	A1
20210377553	Galpin	Dec 2021	A1
20210385481	Liu	Dec 2021	A1
20210392371	Lee	Dec 2021	A1
20220014761	Zhang	Jan 2022	A1
20220078431	Chujoh	Mar 2022	A1
20220086481	Liu	Mar 2022	A1
20220368916	Zhang	Nov 2022	A1
20230239492	Lee	Jul 2023	A1

Foreign Referenced Citations (147)

Number	Date	Country
1665300	Sep 2005	CN
101267562	Sep 2008	CN
101711481	May 2010	CN
101877785	Nov 2010	CN
101911706	Dec 2010	CN
102037732	Apr 2011	CN
102811346	Dec 2012	CN
102934444	Feb 2013	CN
103155563	Jun 2013	CN
103202016	Jul 2013	CN
103370937	Oct 2013	CN
103561263	Feb 2014	CN
103650507	Mar 2014	CN
103765897	Apr 2014	CN
103931184	Jul 2014	CN
104079944	Oct 2014	CN
104094605	Oct 2014	CN
104170381	Nov 2014	CN
104702957	Jun 2015	CN
104737537	Jun 2015	CN
105075263	Nov 2015	CN
105103556	Nov 2015	CN
105122803	Dec 2015	CN
105163116	Dec 2015	CN
105493505	Apr 2016	CN
105578198	May 2016	CN
105637872	Jun 2016	CN
105723454	Jun 2016	CN
105847804	Aug 2016	CN
105850133	Aug 2016	CN
105959698	Sep 2016	CN
106797476	May 2017	CN
106973297	Jul 2017	CN
107005713	Aug 2017	CN
107079162	Aug 2017	CN
107113424	Aug 2017	CN
107113425	Aug 2017	CN
107360419	Nov 2017	CN
107431820	Dec 2017	CN
107646195	Jan 2018	CN
107852490	Mar 2018	CN
107852499	Mar 2018	CN
107896330	Apr 2018	CN
107925775	Apr 2018	CN
107995489	May 2018	CN
108028929	May 2018	CN
108028931	May 2018	CN
108141603	Jun 2018	CN
108141604	Jun 2018	CN
108293113	Jul 2018	CN
108293131	Jul 2018	CN
108352074	Jul 2018	CN
108353166	Jul 2018	CN
108353184	Jul 2018	CN
108370441	Aug 2018	CN
108541375	Sep 2018	CN
108702515	Oct 2018	CN
108781282	Nov 2018	CN
108781294	Nov 2018	CN
109191514	Jan 2019	CN
110267045	Sep 2019	CN
111010569	Apr 2020	CN
111010581	Apr 2020	CN
113170097	Apr 2024	CN
113170171	Apr 2024	CN
111083489	May 2024	CN
111083484	Jun 2024	CN
111436228	Jun 2024	CN
111436229	Jun 2024	CN
111436226	Aug 2024	CN
111083491	Sep 2024	CN
2800368	Nov 2014	EP
3264768	Jan 2018	EP
3264769	Jan 2018	EP
3301918	Apr 2018	EP
3301920	Apr 2018	EP
3367681	Aug 2018	EP
3376764	Sep 2018	EP
3383045	Oct 2018	EP
3657794	May 2020	EP
3849184	Jul 2021	EP
549417	Aug 2024	IN
2006187025	Jul 2006	JP
2007036889	Feb 2007	JP
2012191298	Oct 2012	JP
2013240046	Nov 2013	JP
2015510357	Apr 2015	JP
2017139776	Aug 2017	JP
2018023121	Feb 2018	JP
2022507281	Jan 2022	JP
2022521554	Apr 2022	JP
19980030414	Jul 1998	KR
100203281	Jun 1999	KR
20180107762	Oct 2018	KR
20180119084	Nov 2018	KR
201740734	Nov 2017	TW
201742465	Dec 2017	TW
201830968	Aug 2018	TW
2005022919	Mar 2005	WO
2008048489	Apr 2008	WO
2011021913	Feb 2011	WO
2013111596	Aug 2013	WO
2013188457	Dec 2013	WO
2014082680	Jun 2014	WO
2014165555	Oct 2014	WO
2015023689	Feb 2015	WO
2015062002	May 2015	WO
2015137723	Sep 2015	WO
2015180014	Dec 2015	WO
2015192353	Dec 2015	WO
2016072775	May 2016	WO
2016078511	May 2016	WO
2016123749	Aug 2016	WO
2016141609	Sep 2016	WO
2016160609	Oct 2016	WO
2017036399	Mar 2017	WO
2017082670	May 2017	WO
2017133661	Aug 2017	WO
2017138393	Aug 2017	WO
2017138417	Aug 2017	WO
2017156669	Sep 2017	WO
2017197146	Nov 2017	WO
2017209328	Dec 2017	WO
2018002024	Jan 2018	WO
2018028559	Feb 2018	WO
2018033661	Feb 2018	WO
2018048265	Mar 2018	WO
2018062892	Apr 2018	WO
2018067823	Apr 2018	WO
2018070152	Apr 2018	WO
2018092869	May 2018	WO
2018113658	Jun 2018	WO
2018116802	Jun 2018	WO
2018119233	Jun 2018	WO
2018121506	Jul 2018	WO
2018128417	Jul 2018	WO
2018129172	Jul 2018	WO
2018156628	Aug 2018	WO
2018166357	Sep 2018	WO
2018169989	Sep 2018	WO
2018171796	Sep 2018	WO
2018210315	Nov 2018	WO
2020103852	May 2020	WO
2020167097	Aug 2020	WO
2020186119	Sep 2020	WO
2020190896	Sep 2020	WO
2021058033	Apr 2021	WO

Non-Patent Literature Citations (221)

Entry
Yun et al. “Study on the Development of Video Coding Standard VVC” Content Production & Broadcasting, Academy of Broadcasting Science, Sep. 2018, 45(9): 26-31.
Xiao Zhenjian, “Research on Low Complexity Intra and Inter Compression Algorithm Based on HEVC”, Dissertation for the Master Degree in Engineering, Harbin Institute of Technology, Jun. 2017.
Vandendorpe et al. “Statistical Properties of Coded Interlaced and Progressive Image Sequences,” IEEE Transactions on Image Processing, Jun. 1999, 8(6):749-761.
Sugio et al. “Parsing Robustness for Merge/AMVP,” Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 6th Meeting: Torino, IT, Jul. 14-22, 2011, document JCTVC-F470, WG11 No. m20900, 2011.
Xiu et al. “Description of Core Experiment 9 (CE9): Decoder Side Motion Vector Derivation,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 12th Meeting: Macao, CN, Oct. 3-12, 2018, document JVET-L01029, 2018.
Xu et al. “CE10-Related: Inter Prediction Sample Filtering,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN Oct. 3-12, 2018, document JVET-L0375, 2018.
Document: JVET-L0104, Chen, Y., et al., “AHG5: Reducing WC worst-case memory bandwidth by restricting bidirectional 4×4 inter CUs/Sub-blocks,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 12th Meeting: Macao, CN, Oct. 3-12, 2018, 3 pages.
Document: JVET-L0100-v3, Chiang, M., et al., “CE10.1.1: Multi-hypothesis prediction for improving AMVP mode, skip or merge mode, and intra mode,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 12th Meeting: Macao, CN, Oct. 3-12, 2018, 14 pages.
Notice of Allowance from U.S. Appl. No. 17/483,570 dated Aug. 7, 2023.
Non-Final Office Action from U.S. Appl. No. 18/071,324 dated Aug. 14, 2023.
Notice of Allowance from U.S. Appl. No. 17/534,968 dated Oct. 12, 2023.
Extended European Search Report from European Application No. 19883887.2 dated Aug. 20, 2021.
Partial Supplementary European Search Report from European Application No. 19885858.1 dated Oct. 28, 2021.
Extended European Search Report from European Application No. 19885858.1 dated Feb. 16, 2022.
Extended European Search Report from European Application No. 20766860.9 dated Feb. 16, 2022.
Extended European Search Report from European Application No. 20766773.4 dated Feb. 25, 2022.
Extended European Search Report from European Application No. 19883617.3 dated Apr. 11, 2022.
International Search Report and Written Opinion from International Patent Application No. PCT/CN2019/117508 dated Feb. 1, 2020 (9 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2019/117512 dated Jan. 31, 2020 (9 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2019/117519 dated Feb. 18, 2020 (12 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2019/117523 dated Feb. 18, 2020 (10 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2019/117528 dated Jan. 31, 2020 (9 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2019/117580 dated Jan. 23, 2020 (10 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2019/118779 dated Feb. 7, 2020 (9 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2019/118788 dated Jan. 23, 2020 (8 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2020/078107 dated Jun. 4, 2020 (10 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2020/078108 dated May 29, 2020 (12 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2020/080824 dated Jun. 30, 2020 (10 pages).
Non-Final Office Action from U.S. Appl. No. 17/534,968 dated Apr. 26, 2023.
Non-Final Office Action from U.S. Appl. No. 17/154,680 dated Mar. 16, 2021.
Non-Final Office Action from U.S. Appl. No. 17/154,795 dated Apr. 21, 2021.
Non-Final Office Action from U.S. Appl. No. 17/154,736 dated Apr. 27, 2021.
Notice of Allowance from U.S. Appl. No. 17/154,736 dated Aug. 3, 2021.
Non-Final Office Action from U.S. Appl. No. 17/356,321 dated Aug. 13, 2021.
Notice of Allowance from U.S. Appl. No. 17/356,275 dated Sep. 10, 2021.
Final Office Action from U.S. Appl. No. 17/154,639 dated Sep. 22, 2021.
Notice of Allowance from U.S. Appl. No. 17/154,639 dated Dec. 1, 2021.
Final Office Action from U.S. Appl. No. 17/154,795 dated Jan. 25, 2022.
Non-Final Office Action from U.S. Appl. No. 17/356,321 dated Jun. 7, 2022.
Final Office Action from U.S. Appl. No. 17/356,321 dated Oct. 5, 2022.
Final Office Action from U.S. Appl. No. 17/483,570 dated May 15, 2023.
Non-Final Office Action from U.S. Appl. No. 17/230,004 dated Jun. 14, 2022.
Non-Final Office Action from U.S. Appl. No. 17/225,470 dated Oct. 6, 2022.
Extended European Search Report from European Patent Application No. 20782973.0 dated Mar. 7, 2022.
Extended European Search Report from European Patent Application No. 19887639.3 dated Mar. 15, 2022.
Non-Final Office Action from U.S. Appl. No. 17/244,633 dated Apr. 29, 2022.
Non-Final Office Action from U.S. Appl. No. 17/244,633 dated Jan. 6, 2022.
Notice of Allowance from U.S. Appl. No. 17/405,179 dated Jan. 12, 2022.
Non-Final Office Action from U.S. Appl. No. 17/225,504 dated Jan. 19, 2022.
Document: JVET-J0025-v2, Chen, H., et al., “Description of SDR, HDR and 360° video coding technology proposal by Huawei, GoPro, HiSilicon, and Samsung,” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 10th Meeting: San Diego, US, Apr. 10-20, 2018, 132 pages.
Alshina et al. “Bi-Directional Optical Flow,” Joint Collaborative Team on Video Coding (JCTVC) of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC 29/WG 11 3rd Meeting, Guangzhou, CN Oct. 7-15, 2010, document JCTVC-C204, 2010.
Bross et al. “CE3: Multiple Reference Line Intra Prediction (Test1.1.1, 1.1.2, 1.1.3 and 1.1.4),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0283, 2018.
Bross et al. “Versatile Video Coding (Draft 3),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L1001, 2018.
Cha et al. “Improved Combined Inter-Intra Prediction Using Spatial-Variant Weighted Coefficient,” IEEE, School of Electric and Computer Engineering, Hong Kong University of Science and Technology, 2011.
Chen et al. “A Pre-Filtering Approach to Exploit Decoupled Prediction and Transform Block Structures in Video Coding,” IEEE, Department of Electrical and Computer Engineering, Santa Barbara, CA, 2014.
Chen et al. “Generalized Bi-Prediction for Inter Coding,” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 3rd Meeting, Geneva, CH, May 26-Jun. 1, 2016, document JVET-C0047, 2016.
Chen et al. “CE4: Affine Merge Enhancement (Test 2.10),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018, document JVET-K0186, 2018.
Chen et al. “AHG5: Reducing VVC Worst-Case Memory Bandwidth by Restricting Bi-Directional4×4InterCUs/Sub-blocks,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0104, 2018.
Chen et al. “CE4: Cross-Model Inheritance for Affine Candidate Derivation (Test 4.1.1),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0363, 2018.
Chen et al. “CE4: Common Base for Affine Merge Mode (Test 4.2.1),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0366, 2018.
Chen et al. “CE4: Affine Merge Enhancement with Simplification (Test 4.2.2),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0368, 2018.
Chen et al. “CE4-Related: Reducing Worst Case Memory Bandwidth in Inter Prediction,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0371, 2018.
Chen et al. “CE2.5.1: Simplification of SBTMVP,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 13th Meeting, Marrakech, MA, Jan. 9-18, 2019, document JVET-M0165, 2019.
Chen et al. “Algorithm Description for Versatile Video Coding and Test Model 4 (VTM 4)” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 13th Meeting: Marrakech, MA, Jan. 9-18, 2019, document JVET-M1002, 2019.
Chiang et al. CE10.1: Combined and Multi-Hypothesis Prediction, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 11th Meeting: Ljubljana, SI, Jul. 10-18, 2018, document JVET-K0257, 2018.
Chen et al. “CE2-related: Worst-case Memory Bandwidth Reduction for VVC,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 13th Meeting: Marrakech, MA, Jan. 9-18, 2019, document JVET-M0400, 2019.
Deng et al. “CE4-1.14 Related: Block Size Limitation of Enabling TPM and GEO,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 16th Meeting, Geneva, CH, Oct. 1-11, 2019, document JVET-P0663, 2019.
Gao et al. “CE4-Related: Sub-block MV Clipping in Affine Prediction,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0317, 2018.
Gao et al. “CE4-Related: Sub-block MV Clipping in Planar Motion Vector Prediction,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0319, 2018.
He et al. “CE4-Related: Encoder Speed-Up and Bug Fix for Generalized Bi-Prediction in BMS-2.1,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0296, 2018.
Hsu et al. “Description of Core Experiment 10: Combined and Multi-Hypotheisis Prediction,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN Oct. 3-12, 2018. document JVET-L1030, 2018.
Jin et al. “Combined Inter-Intra Prediction for High Definition Video Coding,” Picture Coding Symposium, Nov. 2007.
Kakino et al. “6.1 The Role of Deblocking Filters: Deblocking Filter to Remove One Block Distortion,” H.264 /AVC Text Book Three Editions, 2013.
Lee et al. “CE4: Simplified Affine MVP List Construction (Test 4.1.4),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macau, CN, Oct. 8-12, 2018, document JVET-L0141, 2018.
Li et al. “AHG5: Reduction of Worst Case Memory Bandwidth,” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting: Macao, CN, Oct. 3-12, 2018, document JVET-L0122, 2018.
Liao et al. CE10: Triangular Prediction Unit Mode (CE10.3.1 and CE10.3.2),M Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018 document JVET-K0144, 2018.
Liao et al. “CE10.3.1.b: Triangular Prediction Unit Mode,” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0124, 2018.
Lin et al. “CE4.2.3: Affine Merge Mode,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN Oct. 3-12, 2018, document JVET-L0088, 2018.
Liu et al. “CE2-Related: Disabling Bi-Prediction or Inter-Prediction for Small Blocks,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 14th Meeting, Geneva, CH, Mar. 19-27, 2019, document JVET-N0266, 2018.
Murakami et al. “High Efficiency Video Coding,” HEVC / H.265, 2013. pp. 85-88 8, 1 2 5-1 3 High-efficiency image symbolization technology, p. 85-88; 109-119, 125-136 (Murakami et al. “High Efficiency Video Coding,” HEVC / H.265, 2013.).
Pham Van et al. “CE4-Related: Affine Restrictions for the Worst-Case Bandwidth Reduction,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0396, 2018.
Pham Van et al. “Non-CE3: Removal of Chroma 2×N BlocksinCIIPMode,” JointVideoExperts Team(JVET)ofITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 16th Meeting, Geneva, CH Oct. 1-11, 2019, document JVET-P0596, 2019.
Racape et al. “CE3-Related: Wide-Angle Intra Prediction for Non-Square Blocks,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018, document JVET-K0500, 2018.
Su et al. “CE4-Related: Generalized Bi-Prediction Improvements,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0197, 2018.
Su et al. “CE4-Related: Generalized Bi-Prediction Improvements Combined from JVET-L0197 and JVET-L0296,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0646, 2018.
Sullivan et al. “Overview of the High Efficiency Video Coding (HEVC) Standard,” IEEE Transactions on Circuits and Systems for Video Technology, Dec. 2012, 22(12):1649-1668.
Winken et al.CE10: Multi-Hypothesis Inter Prediction (Tests 1.2.a-1.2.c). Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 12th Meeting: Macao, CN, Oct. 3-12, 2018, document JVET-L0148, 2018.
Yang et al. “CE4-Related: Control Point MV Offset for Affine Merge Mode,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0389, 2018.
Zhang et al.“CE4-Related: History-based Motion Vector Prediction,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/ SC 29/WG 11 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018, document JVET-K0104, 2018.
Zhang et al.“CE4.5.2: Motion Compensated Boundary Pixel Padding,” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018, document JVET-K0363, 2018.
Zhou et al. “AHG7: A Combined Study on JCTVC-I0216 and JCTVC-I0107,” Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG 11 9th Meeting, Geneva, Switzerland, Apr. 27-May 7, 2012, document JCTVC I0425, 2012.
English translation of WO2020167097A1, Aug. 20, 2020.
H.265/HEVC, https://www.itu.int/rec/T-REC-H.265.(only website), Jul. 12, 2023.
VTM-2.0.1;http:vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/tags/VTM-2.0.1, Jul. 12, 2023.
Sethuraman, Sriram. “CE9: Results of DMVR Related Tests CE9.2.1 and CE9.2.2,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 13th Meeting, Marrakech, MA, Jan. 9-18, 2019, document JVET-M0147, 2019.
Xiu et al. “CE9.1.3: Complexity Reduction on Decoder-Side Motion Vector Refinement (DMVR)” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC, document JVET-K0342, 2018.
Document: JVET-L0188-v3, Chen, F., et al., “CE9: Unidirectional Template based DMVR and its Combination with Simplified Bidirectional DMVR (Test 9.2.10 and Test 9.2.11),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 12th Meeting: Macao, CN, Oct. 3-12, 2018, 6 pages.
Zhang et al.“Fast Coding Unit Depth Selection Algorithm for Inter-frame Prediction of HEVC”, Computer Engineering, Oct. 2018, 44(10):258-263.
Document: JVET-L1002-v1, Chen, J., “Algorithm description for Versatile Video Coding and Test Model 3 (VTM 3),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 12th Meeting: Macao, CN, Oct. 3-12, 2018, 48 pages.
Partial Supplementary European Search Report from European Patent Application No. 19887639.3 dated Oct. 27, 2021.
International Search Report and Written Opinion from International Patent Application No. PCT/CN2019/119634 dated Feb. 26, 2020 (11 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/IB2019/058994 dated Jan. 2, 2020 (16 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/IB2019/058995 dated Jan. 17, 2020 (16 pages).
Intemational Search Report and Written Opinion from International Patent Application No. PCT/IB2019/058996 dated Jan. 2, 2020 (15 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/IB2019/058997 dated Jan. 16, 2020 (18 pages).
Non-Final Office Action from U.S. Appl. No. 17/154,485 dated Mar. 23, 2021.
Final Office Action from U.S. Appl. No. 17/154,485 dated Jul. 27, 2021.
Non-Final Office Action from U.S. Appl. No. 17/225,470 dated Nov. 26, 2021.
International Search Report and Written Opinion from International Patent Application No. PCT/CN2019/119742 dated Feb. 19, 2020 (12 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2019/119756 dated Feb. 7, 2020 (10 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2019/119763 dated Feb. 26, 2020 (12 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2020/082937 dated Jun. 30, 2020 (10 pages).
International Search Report and Written Opinion from International Patent Application No. PCT/CN2020/088927 dated Aug. 12, 2020 (9 pages).
Non-Final Office Action from U.S. Appl. No. 17/317,522 dated Mar. 1, 2022.
Notice of Allowance from U.S. Appl. No. 17/990,065 dated Nov. 13, 2023.
Non-Final Office Action from U.S. Appl. No. 17/317,522 dated Sep. 6, 2022.
Final Office Action from U.S. Appl. No. 17/317,522 dated Apr. 12, 2023.
Notice of Allowance from U.S. Appl. No. 18/071,324 dated Dec. 6, 2023.
Non-Final Office Action from U.S. Appl. No. 18/301,819 dated Jan. 17, 2024.
Non-Final Office Action from U.S. Appl. No. 17/317,522 dated Sep. 26, 2023.
Document: JVET-L0192, Gisquest, C., et al., “CE11: Higher precision modification for VVC deblocking filter (Test 2.1),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 12th Meeting: Macao, CN, Oct. 3-12, 2018, 7 pages.
Document: JVET-M0454-v1, Dias, A., et al., Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and 1S0/IEC JTC 1/SC 29/WG 11 13th Meeting: Marrakech, MA, Jan. 9-18, 2019, 6 pages.
Chinese Office Action from Chinese Application No. 2023/105110 dated Apr. 23, 2024, 9 pages. With English Translation.
Chinese Patent Document from Chinese Patent Application No. CN113170097B dated Apr. 9, 2024, 96 pages. With English Translation.
Final Office Action from U.S. Appl. No. 17/317,522 dated Mar. 7, 2024, 32 pages.
Chinese Office Action from Chinese Application No. 201980005122.0 dated Mar. 11, 2024, 25 pages. With English Translation.
Bross et al. “Versatile Video Coding (Draft 2),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018, document JVET-K1001, 2018.
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/tags/VTM-2.1, Jul. 26, 2021.
Chiang et al. “CE10.1.1: Multi-Hypothesis Prediction for Improving AMVP Mode, Skip or Merge Mode, and Intra Mode,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0100, 2018.
ITU-T H.265 “High efficiency video coding” Series H: Audiovisual and Multimedia Systems Infrastructure of audiovisual services—Coding of moving video, Telecommunication Standardization Sector of ITU, Feb. 2018.
Chen et al. “Algorithm Description of Joint Exploration Test Model 7 (JEM 7),” Joint Video Exploration Team (JVET) of ITU-T SG WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 7th Meeting, Torino, IT, Jul. 13-21, 2017, document JVET-G1001, 2011.
JEM-7.0: https://jvet.hhi.fraunhofer.de/svn/svn_HMJEMSoftware/tags/ HM-16.6-JEM-7.0, Jul. 26, 2021.
Akula et al. “Description of SOR, HOR and 360 degrees Video Coding Technology Proposal Considering Mobile Application Scenario by Samsung, Huawei, GoPro, and HiSilicon,” buJoint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and 1SO/IEC JTC 1/SC 29/WG 11, 10th Meeting, San Diego, US, Apr. 10-20, 2018, document JVET-J0024, 2018.
Esenlik et al. “CE9: DMVR with Bilateral Matching (Test2.9),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018, document JVET-K0217, 2018.
Esenlik et al. “CE9: Report on the Results of Tests CE9.2. 15 and CE9.2.16,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0163, 2018.
Bross et al. “Versatile Video Coding (Draft 4),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 13th Meeting, Marrakech, MA, Jan. 9-18, 2019, document JVET-M1001, 2019.
Bross et al. “Versatile Video Coding (Draft 5),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 14th Meeting, Geneva, CH, Mar. 19-27, 2019, documentJVET-N1001, 2019.
Xiu et al. “CE9.5.3: Bi-Directional Optical Flow (BIO) Simplification,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and 1SO/IEC JTC 1/SC 29/WG 11, 11th, Meeting, Ljubljana, SI, Jul. 10-18, 2018, document JVET-K0344, 2018.
Lai et al. “CE9-Related: BIO Simplification,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC /SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0099, 2018.
Luo et al. “CE9.2.7: Complexity Reduction on Decoder-Side Motion Vector Refinement (DMVR),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and 1SO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0196, 2018.
Liu et al. “CE9-Related: Simplification of Decoder Side Motion Vector Derivation,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018, document JVET-K0105, 2018.
Alshin et al. “AHG6: On BIO Memory Bandwidth,” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 4th Meeting, Chengdu, CN, Oct. 15-21, 2016, document JVET-D0042, 2016.
Alshin et al. “Bi-Directional Optical Flow for Improving Motion Compensation,” Dec. 8-10, 2010, 28th Picture Coding Symposium, PCS2010, Nagoya, Japan, pp. 422-425.
Blaser et al. “Geometry-based Partitioning for Predictive Video Coding with Transform Adaptation,” 2018, IEEE.
Chen et al. “CE9.5.2: BIO with Simplified Gradient Calculation, Adaptive BIO Granularity, and Applying BIO to Chroma Components,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018, document JVET-K0255, 2018.
Chuang et al. “EE2-Related: A Simplified Gradient Filter for Bi-Directional Optical Flow (BIO),” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 7th Meeting, Torino, IT, Jul. 13-21, 2017, document G0083, 2017.
Esenlik et al. “Description of Core Experiment 9 (CE9): Decoder Side Motion Vector Derivation,” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 10th Meeting, San Diego, US, Apr. 10-20, 2018, document JVET-J1029, 2018.
Hsu et al. “Description of Core Experiment 10: Combined and Multi-Hypothesis Prediciton,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 10th Meeting, San Diego, US, Apr. 10-20, 2018, document JVET-J1030, 2018.
ITU-T H.265 “High efficiency video coding” Series H: Audiovisual and Multimedia Systems Infrastructure of audiovisual services—Coding of moving video, TelecommunicationStandardization Sector of ITU, Available at address: https://www.itu.int/rec/T-REC-H.265 (Nov. 2019).
“Information Technology—High Efficiency Coding and Media Delivery in Heterogeneous Environments—Part 2: High Efficiency Video Coding” Apr. 20, 2018, ISO/DIS 23008, 4th Edition.
Jeong et al. “CE4 Ultimate Motion Vector Expression (Test 4.5.4),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0054, 2018.
Kamp et al. “Decoder Side Motion Vector Derivation for Inter Frame Video Coding,” 2008, IEEE, RWTH Aachen University, Germany.
Kamp et al. “Fast Decoder Side Motion Vector Derivation for Inter Frame Video Coding,” 2009, RWTH Aachen University, Germany.
Klomp et al. “Decoder-Side Block Motion Estimation for H.264 / MPEG-4 AVC Based Video Coding,” 2009, IEEE, Hannover, Germany, pp. 1641-1644.
Li et al. “CE4-Related: Affine Merge Mode with Prediction Offsets,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0320, 2018.
Liu et al. “CE9-Related: Motion Vector Refinement in Bi-Directional Optical Flow,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0333, 2018.
Murakami et al., “Advanced B Skip Mode with Decoder-side Motion Estimation,” Hitachi, 2012, 37th VCEG Meeting at Yokohama, VCEG-AK12.
Rosewarne et al. “High Efficiency Video Coding (HEVC) Test Model 16 (HM 16) Improved Encoder Description Update 7,” Joint Collaborative Team on Video Coding (JCT-VC) ITU-T SG 16 WP3 and ISO/IEC JTC1/SC29/WG11, 25th Meeting, Chengdu, CN, Oct. 14-21, 2019, document JCTVC-Y1002, 2016.
Su et al. “CE4.4.1: Generalized Bi-Prediction for Intercoding,” Joint Video Exploration Team of ISO/IEC JTC 1/SC 29/WG 11 and ITU-T SG 16, Ljubljana, Jul. 10-18, 2018, document No. JVET-K0248, 2018.
Ueda et al. “TE1.a: Implementation Report of Refinement Motion Compensation Using DMVD on TMuC,” Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, 3rd Meeting, Guangzhou, CN Oct. 7-15, 2010, document JCTVC-C138, 2010.
Winken et al. “CE10: Multi-Hypothesis Inter Prediction (Tests 1.5-1.8),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018, document JVET-K0269, 2018.
Xiu et al. “CE9-Related: Complexity Reduction and Bit-Width Control for Bi-Directional Optical Flow (BIO),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0256, 2018.
Zhang et al. “CE4-Related: Interweaved Prediction for Affine Motion Compensation,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018, document JVET-K0102, 2018.
Bross et al. “Versatile Video Coding (Draft 7),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 16th Meeting: Geneva, CH, Oct. 1-11, 2019, document JVET-P2001, 2019.
Esenlik et al. “BoG Report on PROF/BDOF Harmonization Contributions (CE4 and CE9 related),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 15th Meeting: Gothenburg, SE, Jul. 3-12, 2019, document JVET-01133, 2019.
Liu et al. “Non-CE9: Unified Gradient Calculations in BDOF,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 an ISO/IEC JTC 1/SC 29/WG 11 15th Meeting: Gothenburg, SE, Jul. 3-12, 2019, document JVET-O0570, 2019.
Park et al. “Non-CE9 : Mismatch Between Text Specification and Reference Software on BDOF and DMVR,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, Mar. 19-27, 2019, document JVET-N0444, 2019.
Sethuraman et al. “Non-CE9: Methods for BDOF Complexity Reduction,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 13th Meeting: Marrakech, MA, Jan. 9-18, 2019, document JVET-M0517, 2019.
Toma et al. “Description of SDR Video Coding Technology Proposal by Panasonic,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 10th Meeting, San Diego, US, Apr. 10-20, 2018, document JVET-J0020, 2018.
Xiu et al. “CE10-Related: Simplification on Combined Inter and Intra Prediction (CIIP),” Joint Video Experts Team (JVET)of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, Mar. 19-27, 2019, document JVET-N0327, 2019.
Chujoh et al. “CE9-related: An Early Termination of DMVR,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 12th Meeting: Macao, CN, Oct. 3-12, 2018. document JVET-L0367, 2018.
Esenlik et al. “Simplified DMVR for Inclusion in VVC,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 an ISO/IEC JTC 1/SC 29/WG 1 12th Meeting: Macao, CN, Oct. 3-12, 2018, document JVET-L0670, 2018.
Luo et al. “CE2-Related: Prediction Refinement with Optical Flow for Affine Mode,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting, Geneva, CH, Mar. 19-27, 2019, document JVET-N0236, 2019.
Albrecht et al. “Description of SDR, HDR, and 360 Degree Video Coding Technology Proposal by Fraunhofer HHI,” Joint Video Experts Team (JVET0 of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 10th Meeting, San Diego, US, Apr. 10-20, 2018, document JVET-J0014, 2018.
Alshin et al. “EE3: Bi-Directional Optical Flow w/o Block Extension,” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 5th Meeting, Geneva, CH, Jan. 12-20, 2020, document JVET-E0028, 2017.
Japanese Office Action from Japanese Application No. 2023-105110 dated Apr. 23, 2024, 9 pages. With English Translation.
Non-Final Office Action from U.S. Appl. No. 1/317,522 dated Jul. 1, 2024, 23 pages.
Chinese Office Action from Chinese Patent Application No. 201911007809.6 dated Jul. 2, 2024, 13 pages.
Chinese Office Action from Chinese Patent Application No. 201980005114.6 dated Jul. 19, 2024, 7 pages.
Chinese Office Action from Chinese Patent Application No. 201980005122.0 dated May 31, 2024, 7 pages.
Document: JVET-N1001-v6, Bross, B., et al., “Versatile Video Coding (Draft 5),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, Mar. 19-27, 2019, 384 pages.
Document: JVET-M0313-r1, Liao, R., et al., “CE4: Motion compensation constraints for complexity reduction (test 4.5.1 and test 4.5.2),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 13th Meeting: Marrakech, MA, Jan. 9-18, 2019, 8 pages.
Korean Notice of Allowance from Korean Patent Application No. 10/2021-7027315 dated Sep. 25, 2024, 10 pages.
Andersson, K., et al., “Combined Intra Inter Prediction Coding Mode,” Document: VCEG-AD11 URL: http://wftp3.itu.int/av-arch/video-site/0610_Han/VCEG-AD11.zip, Oct. 18, 2006, 4 pages.
Bross B., et al., “Versatile Video Coding (Draft 2),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 11th Meeting: Ljubljana, SI, Jul. 10-18, 2018, Document: JVET-K1001-v7, 280 Pages.
Bross B., et al., “Versatile Video Coding (Draft 2),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC29/WG 11, 11th Meeting: Ljubljana, SI, Jul. 10-18, 2018, Document: JVET-K1001-v1, 43 Pages.
Bross B., et al., “Versatile Video Coding (Draft 3),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting: Macao, CN, Oct. 3-12, 2018, Document: JVET-L1001-v5, 193 Pages.
Bross B., et al., “Versatile Video Coding (Draft 5),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 14th Meeting: Geneva, CH, Mar. 19-27, 2019, Document: JVET-N1001-v5, 374 Pages.
Chen et al. “CE9-Related: Simplified DMVR with Reduced Internal Memory,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L0098, 2018.
Chiang M-S., et al., “CE10.1.1: Multi-Hypothesis Prediction for Improving AMVP Mode, Skip or Merge Mode, and Intra Mode,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting: Macao, CN, Oct. 3-12, 2018, Document: JVET-L0100-v1, pp. 1-13, (13 Pages) (cited in CN201980074019.1 mailed May 22, 2023).
Final Office Action from U.S. Appl. No. 17/317,522 dated Nov. 27, 2024, 23 pages.
Final Office Action from U.S. Appl. No. 18/531,153 dated Nov. 29, 2024, 25 pages.
First Office Action for Chinese Application No. 201980076196.3, mailed Aug. 8, 2022, 17 Pages.
Gao M., et al., “CE4-Related: Sub-Block MV Clipping in Affine Prediction,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting: Macao, CN, Oct. 3, 2018-Oct. 12, 2018, Document: JVET-L0317-r1, 3 Pages.
Hellman T., et al., “AHG7: Reducing HEVC Worst-Case Memory Bandwidth by Restricting Bidirectional 4×8 and 8×4 Prediction Units,” Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 9th Meeting: Geneva, CH, Apr. 27, 2012-May 7, 2012, Document: JCTVC-10216v2, 9 Pages.
Hsu C-W., et al., “Description of Core Experiment 10: Combined and Multi-Hypothesis Prediction,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, Document: JVET-L1030-v1, 10 Pages.
Japanese Notice of Reasons for Rejection from Japanese Patent Application No. 2024-039982 dated Dec. 10, 2024, 30 pages.
Japanese Notice of Reasons for Rejection from Japanese Patent Application No. 23023-132610 dated Nov. 26, 2024, 19 pages.
Kakino S., et al., “H.264/AVC Textbook,” Revision Three Editions, Impress Standard textboox; 2010, pp. 144-148, 12 Pages, with English Translation.
Karczewicz M., et al., “CE8-Related: Quantized Residual BDPCM,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 14th Meeting: Geneva, CH, Mar. 19-27, 2019, Document: JVET-N0413, 336 Pages.
Kondo K., et al., “AHG7: Modification of Merge Candidate Derivation to Reduce MC Memory Bandwidth,” Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 9th Meeting: Geneva, CH, Apr. 27, 2012-May 7, 2012, Document: JCTVC-10107-r1, 9 Pages.
Lai C-Y., et al., “CE9-Related: BIO Simplification,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, Document: JVET-L0099, 7 Pages.
Lai C-Y., et al., “CE9-Related: BIO Simplification,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting: Macao, CN, Oct. 3-12, 2018, Document: JVET-L0099-v1, 5 Pages.
Luo J(D)., et al., “CE2-Related: Prediction Refinement with Optical Flow for Affine Mode,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 14th Meeting: Geneva, CH, Mar. 19, 2019-Mar. 27, 2019, Document: JVET-N0236, 7 Pages.
Luo J.D., et al., “CE2-related: Prediction Refinement with Optical Flow for Affine Mode,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 14th Meeting: Geneva, CH, Mar. 19-27, 2019, Document: JVET-N0236-r1, 7 Pages.
Luo J.D, et al., “CF2-Related: Prediction Refinement with Optical Flow for Affine Mode,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 14th Meeting: Geneva, CH, Mar. 19-27, 2019, Document: JVET-N0236, 25 Pages.
Murakami, A., et al., “High-efficiency image symbolization technology; HEVC/H.265; High Efficiency Video Coding,” Nose Software Information Center, May 26, 2022, 40 pages. with English Translation.
Murakami et aL “High Efficiency Video Coding,” HEVC / H.265, First Edition, Feb. 25, 2013. High-efficiency Image symbolization technology, Ohmsha Co., Ltd., p. 109-119.
Non-Final Office Action for U.S. Appl. No. 17/483,570, mailed Nov. 25, 2022, 52 Pages.
Non-Final Office Action from U.S. Appl. No. 18/531,153 dated Jul. 3, 2024, 100 pages.
Notice of Allowance for U.S. Appl. No. 17/230,004, mailed Dec. 16, 2022, 19 Pages.
Notice of Reasons for Refusal for Japanese Application No. 2021-525770, mailed May 10, 2022, 9 Pages.
Notice of Reasons for Refusal for Japanese Application No. 2021-557132, mailed Sep. 13, 2022, 10 Pages.
Notice of Reasons for Refusal from Japanese Patent Application No. 2021-549770 dated Mar. 22, 2023, 15 Pages.
Partial Supplementary European Search Report for European Application No. 19883617.3, mailed Oct. 28, 2021, 11 pages.
Patent Certificate of Chinese Patent Application No. 201911008574.2 dated May 14, 2024.
Sethuraman S., “Non-CE9: Methods for BDOF Complexity Reduction,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 13th Meeting: Marrakech, MA, Jan. 9-18, 2019, Document: JVET-M0517-v2, 4 Pages.
Su Y-C., et al., “CE4-Related: Generalized Bi-Prediction Improvements Combined from JVET-L0197 and JVET-L0296,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting: Macao, CN, Oct. 3-12, 2018, Document: JVET-L0646-v1, 39 Pages.
VTM software, Retrieved from the internet: https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM.git, Feb. 7, 2023, 3 pages.
Xiu X., et al., “Description of Core Experiment 9 (CE9): Decoder Side Motion Vector Derivation,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting: Macao, CN, Oct. 3, 2018-Oct. 12, 2018, Document: JVET-L1029-v2, 12 Pages.
Xiu X., et al., “Description of Core Experiment 9 (CE9): Decoder Side Motion Vector Derivation,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting: Macao, CN, Oct. 3-12, 2018, Document: JVET-L01029, 11 Pages.

Related Publications (1)

	Number	Date	Country
	20240137554 A1	Apr 2024	US

Continuations (2)

	Number	Date	Country
Parent	17317522	May 2021	US
Child	18531153		US
Parent	PCT/CN2019/119763	Nov 2019	WO
Child	17317522		US

Coding and decoding of video coding modes

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

Field of Search

CPC

International Classifications

Term Extension

Abstract