Patent Application
20240146954
The present application concerns picture and/or video coding and in particular codecs supporting block partitioning and skip mode.
Many picture and/or video codecs treat the pictures in units of blocks. For example, predictive codecs use a block granularity in order to achieve a good compromise between very precisely set prediction parameters set at a high spatial resolution with, however, spending too much side information for the prediction parameters on the one hand and too coarsely set prediction parameters, causing the amount of bits necessitated to encode the prediction residual to increase due to the lower spatial resolution of the prediction parameters, on the other hand. In effect, the optimum setting for the prediction parameters lies somewhere between both extremes.
Several attempts have been made in order to obtain the optimum solution for the above-outlined problem. For example, instead of using a regular subdivision of a picture into blocks regularly arranged in rows and columns, multi-tree partitioning subdivision seeks to increase the freedom of subdividing a picture into blocks at a reasonable demand for subdivision information. Nevertheless, even multi-tree subdivision necessitates the signalization of a remarkable amount of data and the freedom in subdividing a picture is quite restricted even in case of using such multi-tree subdivisioning.
In order to enable a better tradeoff between the amount of side information necessitated in order to signalize the picture subdivision on the one hand and the freedom in subdividing the picture on the other hand, merging of blocks may be used in order to increase the number of possible picture subdivisionings at a reasonable amount of additional data necessitated in order to signalize the merging information. For blocks being merged, the coding parameters need to be transmitted within the bitstream in full merely once, similarly as if the resulting merged group of blocks was a directly subdivided portion of the picture.
In order to additionally increase the efficiency in encoding the picture content, skip mode has been introduced into some block-based picture codecs, the skip mode enabling the encoder to refrain from transmitting the residual data of a certain block to the decoder. That is, the skip mode is a possibility to suppress residual data transmission for certain blocks. The ability to suppress the transmission of residual data for certain blocks results in a broader granularity interval for encoding the coding/prediction parameters within which an optimum tradeoff between coding quality on the one hand and total bit rate spent on the other hand may be expected: naturally, increasing the spatial resolution of the encoding of the coding/prediction parameters results in an increase of the side information rate while decreasing, however, the residuum thereby lowering the rate necessitated to encode the residual data. However, due to the availability of the skip mode, it may be favorable to obtain an abrupt coding rate saving by merely moderately further increasing the granularity at which the coding/prediction parameters are transmitted so that the residuum is so small that a separate transmission of the residuum may be left away.
However, there is still a need for achieving better coding efficiency, due to remaining redundancies newly caused by the combination of block merging and skip mode usage.
An embodiment may have an apparatus configured to decode a bitstream into which a picture is coded, the sample array of the picture being partitioned into sample sets each of which is associated with respective coding parameters, the apparatus being configured to be, for a current sample set, commonly responsive to a flag within the bitstream as to a first decision as to whether the coding parameters associated with the current sample set are to be set according to a merge candidate or to be retrieved from the bitstream, and a second decision as to whether the current sample set of the picture is to be reconstructed merely based on a prediction signal depending on the coding parameters associated with the current sample set, without any residual data, or to be reconstructed by refining the prediction signal depending on the coding parameters associated with the current sample set by means of residual data within the bitstream, such that the apparatus if the flag within the bitstream signals that the coding parameters associated with the current sample set are to be set according to a merge candidate, obtains the coding parameters associated with the current sample set by setting same according to a merge candidate, and reconstructs the current sample set of the picture merely based on a prediction signal depending on the coding parameters without any residual data.
According to another embodiment, an apparatus for decoding a bitstream having a picture encoded therein may have: a subdivider configured to subdivide the picture into sample sets of samples; a merger configured to merge the sample sets into groups of one or more sample sets each; a decoder configured to decode the picture using coding parameters varying across the picture in units of the groups of sample sets, wherein the decoder is configured to decode the picture by predicting the picture, decoding a prediction residual for predetermined sample sets and combining the prediction residual and a prediction resulting from predicting the picture, for the predetermined sample sets; an extractor configured to extract the prediction residual and the coding parameters from the bitstream, along with one or more syntax elements for each of at least a subset of the sample sets, signaling as to whether the respective sample set is to be merged into one of the groups along with another sample set or not, wherein the merger is configured to perform the merging responsive to the one or more syntax elements, wherein one of the possible states of the one or more syntax elements signalizes that the respective sample set is to be merged into one of the groups along with another sample set and has no prediction residual encoded and inserted into the bitstream.
Another embodiment may have an apparatus for encoding a picture into a bitstream, the sample array of the picture being partitioned into sample sets each of which is associated with respective coding parameters, the apparatus being configured to encode, for a current sample set, a flag into the bitstream commonly signaling as to whether the coding parameters associated with the current sample set are to be set according to a merge candidate or to be retrieved from the bitstream, and as to whether the current sample set of the picture is to be reconstructed merely based on a prediction signal depending on the coding parameters associated with the current sample set, without any residual data, or to be reconstructed by refining the prediction signal depending on the coding parameters associated with the current sample set by means of residual data within the bitstream, such that the apparatus if the flag within the bitstream signals that the coding parameters associated with the current sample set are to be set according to a merge candidate, the coding parameters associated with the current sample set are to be set according to a merge candidate, and the current sample set of the picture is to be reconstructed merely based on a prediction signal depending on the coding parameters without any residual data.
According to another embodiment, an apparatus for encoding a picture may have: a subdivider configured to subdivide the picture into sample sets of samples; a merger configured to merge the sample sets into groups of one or more sample sets each; an encoder configured to encode the picture using coding parameters varying across the picture in units of the groups of sample sets, wherein the encoder is configured to encode the picture by predicting the picture and encoding a prediction residual for predetermined sample sets; a stream generator configured to insert the prediction residual and the coding parameters into a bitstream, along with one or more syntax elements for each of at least a subset of the sample sets, signaling as to whether the respective sample set is merged into one of the groups along with another sample set or not, wherein one of the possible states of the one or more syntax elements signalizes that the respective sample set is to be merged into one of the groups along with another sample set and has no prediction residual encoded and inserted into the bitstream.
Another embodiment may have a method for decoding a bitstream into which a picture is coded, the sample array of the picture being partitioned into sample sets each of which is associated with respective coding parameters, the method having, for a current sample set, commonly reacting to a flag within the bitstream as to a first decision as to whether the coding parameters associated with the current sample set are to be set according to a merge candidate or to be retrieved from the bitstream, and a second decision as to whether the current sample set of the picture is to be reconstructed merely based on a prediction signal depending on the coding parameters associated with the current sample set, without any residual data, or to be reconstructed by refining the prediction signal depending on the coding parameters associated with the current sample set by means of residual data within the bitstream, such that if the flag within the bitstream signals that the coding parameters associated with the current sample set are to be set according to a merge candidate, the coding parameters associated with the current sample set are obtained by setting same according to a merge candidate, and the current sample set of the picture are reconstructed merely based on a prediction signal depending on the coding parameters without any residual data.
According to another embodiment, a method for decoding a bitstream having a picture encoded therein may have the steps of: subdividing the picture into sample sets of samples; merging the sample sets into groups of one or more sample sets each; decoding the picture using coding parameters varying across the picture in units of the groups of sample sets, wherein the decoder is configured to decode the picture by predicting the picture, decoding a prediction residual for predetermined sample sets and combining the prediction residual and a prediction resulting from predicting the picture, for the predetermined sample sets; extracting the prediction residual and the coding parameters from the bitstream, along with one or more syntax elements for each of at least a subset of the sample sets, signaling as to whether the respective sample set is to be merged into one of the groups along with another sample set or not, wherein the merger is configured to perform the merging responsive to the one or more syntax elements, wherein one of the possible states of the one or more syntax elements signalizes that the respective sample set is to be merged into one of the groups along with another sample set and has no prediction residual encoded and inserted into the bitstream.
Still another embodiment may have a method for encoding a picture into a bitstream, the sample array of the picture being partitioned into sample sets each of which is associated with respective coding parameters, the method having encoding, for a current sample set, a flag into the bitstream commonly signaling as to whether the coding parameters associated with the current sample set are to be set according to a merge candidate or to be retrieved from the bitstream, and as to whether the current sample set of the picture is to be reconstructed merely based on a prediction signal depending on the coding parameters associated with the current sample set, without any residual data, or to be reconstructed by refining the prediction signal depending on the coding parameters associated with the current sample set by means of residual data within the bitstream, such that if the flag within the bitstream signals that the coding parameters associated with the current sample set are to be set according to a merge candidate, the coding parameters associated with the current sample set are to be set according to a merge candidate, and the current sample set of the picture is to be reconstructed merely based on a prediction signal depending on the coding parameters without any residual data.
According to another embodiment, a method for encoding a picture may have the steps of: subdividing the picture into sample sets of samples; merging the sample sets into groups of one or more sample sets each; encoding the picture using coding parameters varying across the picture in units of the groups of sample sets, wherein the encoder is configured to encode the picture by predicting the picture and encoding a prediction residual for predetermined sample sets; inserting the prediction residual and the coding parameters into a bitstream, along with one or more syntax elements for each of at least a subset of the sample sets, signaling as to whether the respective sample set is merged into one of the groups along with another sample set or not, wherein one of the possible states of the one or more syntax elements signalizes that the respective sample set is to be merged into one of the groups along with another sample set and has no prediction residual encoded and inserted into the bitstream.
Another embodiment may have a computer program having a program code for performing, when running on a computer, an inventive method.
The idea underlying the present invention is that a further coding efficiency increase may be achieved if a common signalization is used within the bitstream with regard to both activation of merging and activation of the skip mode. That is, one of the possible states of one or more syntax elements within the bitstream may signalize for a current sample set of a picture that the respective sample set is to be merged and has no prediction residual encoded and inserted into the bitstream. Alternatively speaking, a common flag may commonly signalize whether the coding parameters associated with a current sample set are to be set according to a merge candidate or to be retrieved from the bitstream, and whether the current sample set of the picture is to be reconstructed merely based on a prediction signal depending on the coding parameters associated with the current sample set, without any residual data, or to be reconstructed by refining the prediction signal depending on the coding parameters associated with the current sample set by means of residual data within the bitstream.
The inventors of the present invention found out that this introduction of a common signalization of the activation of the merging on the one hand and the activation of the skip mode on the other hand saves bit rate as additional overhead in order to signalize the activation of the merging and/or skip mode separately from each other may be reduced or may merely have to be spent in case of the merging and the skip mode not being activated concurrently.
Embodiments of the present invention will be detailed subsequently referring to the appended drawings, in which:
With respect to the following description, it is noted that whenever the same reference sign is used in connection with different figures, the explanations with regard to the respective element presented with respect to one of these figures shall equally apply to the other figures, provided that such transferring of explanations from one figure to the other does not conflict with the remaining description of this other figure.
The picture 20 is, although not explicitly shown in
For example, the picture 20 may be subdivided into a regular arrangement of blocks 40 so that the blocks 40 are arranged in rows and columns as exemplarily shown in
In any case, the encoder 10 is configured to encode, for current sample set 40, a flag into the bitstream 30 commonly signaling whether the coding parameters associated with the current sample set 40 are to be set according to a merge candidate or to be retrieved from the bitstream 30, and whether the current sample set of the picture 20 is to be reconstructed merely based on a prediction signal depending on the coding parameters associated with the current sample set, without any residual data, or to be reconstructed by refining the prediction signal depending on the coding parameters associated with the current sample set 40 by means of a residual data within the bitstream 30. For example, the encoder 10 is configured to encode, for current sample set 40, a flag into the bitstream 30 commonly signaling, if assuming a first state, that the coding parameters associated with the current sample set 40 are to be set according to a merge candidate rather than to be retrieved from the bitstream 30, and that the current sample set of the picture 20 is to be reconstructed merely based on a prediction signal depending on the coding parameters associated with the current sample set, without any residual data, and if assuming any other state that the coding parameters associated with the current sample set 40 are to be retrieved from the bitstream 30, or that the current sample set of the picture 20 is to be reconstructed by refining the prediction signal depending on the coding parameters associated with the current sample set 40 by means of a residual data within the bitstream 30. This means the following. The encoder 10 supports merging of blocks 40. The merging is facultative. That is, not every block 40 is subject to merging. For some blocks 40 it is, in some, for example, rate-distortion optimization sense favorable to merge the current block 40 with a merge candidate, but for others the opposite is true. In order to decide whether a certain block 40 should be made the subject of merging, the encoder 10 determines a set or list of merge candidates and checks, for each of these merge candidates, whether merging the current block 40 with that merge candidate forms the most advantageous coding option in, for example, rate-distortion optimization sense. The encoder 10 is configured to determine the set or list of merge candidates for a current block 40 based on previously encoded portions of bitstream 30. For example, encoder 10 derives at least a portion of the set or list of merge candidates by adopting the coding parameters associated with locally and/or temporally neighboring blocks 40 which have been previously encoded in accordance with the encoding order applied by encoder 10. Temporal neighborhood denotes, for example, blocks of previously encoded pictures of a video to which picture 20 belongs, with the temporally neighboring blocks thereof being spatially located so as to spatially overlap the current block 40 of the current picture 20. Accordingly, for this portion of the set or list of merge candidates, there is a one to one association between each merge candidate and the spatially and/or temporally neighboring blocks. Each merge candidate has coding parameters associated therewith. If the current block 40 is merged with any of the merge candidates, encoder 10 sets the coding parameters of the current block 40 in accordance with the merge candidate. For example, encoder 10 may set the coding parameters of the current block 40 to be equal to the respective merge candidate, i.e. encoder 10 may copy the coding parameters of the current block 40 from the respective merge candidate. Thus, for this just-outlined portion of the set or list of merge candidates, the coding parameters of a merge candidate are directly adopted from a spatially and/or temporally neighboring block, or the coding parameters of the respective merge candidate is obtained from the coding parameters of such a spatially and/or temporally neighboring block by adopting same, i.e. setting the merge candidate equal thereto, while, however, taking domain changes into account by, for example, scaling the adopted coding parameters in accordance with the domain change. For example, at least a part of the coding parameters being subject to merging could encompass motion parameters. Motion parameters may, however, refer to different reference picture indices. To be more precise, motion parameters to be adopted may refer to a certain time interval between the current picture and the reference picture, and in merging the current block with the respective merge candidate having respective motion parameters, encoder 10 may be configured to scale the motion parameters of the respective merge candidate in order to adapt its time interval to the time interval selected for the current block.
In any case, the merge candidates described so far have in common that all of them have coding parameters associated therewith, and there is a one to one association between these merge candidates and neighboring blocks. Accordingly, merging blocks 40 with any of the just-outlined merge candidates may be thought of as a merging of these blocks into groups of one or more blocks 40 so that the coding parameters do not vary across the picture 20 within these groups of blocks 40, except for the scaling adaptions or the like. Effectively, the merging with any of the just-outlined merge candidates reduces the granularity at which the coding parameters vary over the picture 20. Beyond that, the merging with any of the just-outlined merge candidates results in an additional freedom in subdividing picture 20 into blocks 40 and groups of blocks 40, respectively. Thus, in this regard the merging of blocks 40 into such groups of blocks may be thought of causing the encoder 10 to encode the picture 20 using coding parameters which vary across the picture 20 in units of these groups of blocks 40.
Besides the just-mentioned merge candidates, encoder 10 may also add merge candidates to the set/list of merge candidates, which are a result of a combination of two or more neighboring blocks' coding parameters, such as an arithmetic mean, a geometric mean thereof or a median of the coding parameters of neighboring blocks and the like.
Thus, effectively, encoder 10 reduces the granularity at which coding parameters are explicitly transmitted within bitstream 30 compared to the granularity defined by the subdivision of picture 20 into blocks 40. Some of these blocks 40 form groups of blocks using one and the same coding parameters by use of the merging option outlined above. Some blocks are coupled to each other via merging, but use different coding parameters correlated among each other via respective scaling adaptations and/or combinational functions. Some blocks 40 are not subject to merging, and accordingly encoder 10 encodes the coding parameters into bitstream 30 directly.
The encoder 10 uses the coding parameters of blocks 40 thus defined in order to determine a prediction signal for picture 20. Encoder 10 performs this determination of the prediction signal block-wise in that the prediction signal depends on the coding parameters associated with the respective block 40.
Another decision performed by encoder 10 is whether the residuum, i.e. the difference between the prediction signal and the original picture content at the respective local area of the current block 40, is to be transmitted within bitstream 30 or not. That is, encoder 10 decides for blocks 40 whether skip mode shall be applied to the respective block or not. If skip mode is applied, the encoder 10 encodes picture 20 within the current portion 40 merely in the form of the prediction signal derived from, or depending on, the coding parameters associated with the respective block 40, and in case of the skip mode being deselected, encoder 10 encodes the picture 20 into bitstream 30 within block 40 using both, the prediction signal as well as the residual data.
In order to save bit rate for signaling the decision with regard to the merging on the one hand and the skip mode on the other hand, encoder 10 commonly signals both decisions using one flag for a block 40. To be more precise, the common signalization may be realized such that the activation of both the merging and the skip mode is commonly indicated by the flag of the respective block 40 within bitstream 30 assuming a first possible flag state, whereas the other flag state of the flag merely indicates to the decoder that either one of the merging or skip mode is not activated. For example, encoder 10 may decide for a certain block 40 to activate merging, but deactivate the skip mode. In that case, encoder 10 uses the other flag state in order to signal within bitstream 30 the deactivation of at least one of the merging and the skip mode, while subsequently signaling within bitstream 30 the activation of merging by use of another flag, for example. Accordingly, encoder 10 has to transmit this further flag merely in case a block 40 for which the merging and the skip mode is not activated concurrently. In embodiments further described below, the first flag is called mrg_cbf or skip_flag while the subsidiary merge indicator flag is called mrg or merge_flag. It has been found out by the inventors of the present application that this co-use of one signalization state in order to commonly signal the activation of merging and skip mode reduces the overall bit rate of bitstream 30.
As to the signalization state just-mentioned, it should be noted that such a signalization state may be determined by the state of one bit of bitstream 30. However, encoder 10 may be configured to entropy encode bitstream 30, and accordingly the correspondence between the signalization state of the flag and the bitstream 30 may be more complicated. In that case, the state could correspond to one bit of bitstream 30 in the entropy-decoded domain. Even further, the signalization state may correspond to one of the two states of the flag for which code words are assigned in accordance with the variable length coding scheme. In case of arithmetic coding, the signalization state commonly signaling the activation of merging and skip mode, may correspond to one of the symbols of the symbol alphabet underlying the arithmetic encoding scheme.
As outlined above, the encoder 10 signals the concurrent activation of the merging and the skip mode using a flag within bitstream 30. As will be outlined in more detail below, this flag may be transmitted within a syntax element which has more than two possible states. This syntax element may, for example, signal other coding options as well. Details are described in more detail below. In that case, however, one of the possible states of the one or more syntax elements signalizes the concurrent activation. That is, whenever the just-mentioned syntax element of a current block 40 assumes this predetermined possible state, the encoder 10 signalizes thereby the activation of both the merging and the skip mode. The decoder thus needs no further signalization regarding the activation of merging and the activation of skip mode, respectively.
With regard to the description outlined above, it should be noted that the partitioning of picture 20 into blocks 40 may not represent the finest resolution at which coding parameters are determined for picture 20. Rather, encoder 10 may accompany each block 40 with further partitioning information in order to signal within the bitstream 30 one of supported partitioning patterns for partitioning the current block 40 into sub-blocks 50 and 60, respectively, i.e. sample subsets. In that case, the concurrent merging/skip decision is performed by encoder 10 in units of blocks 40, whereas coding parameters along with, for example, subsidiary merge decision and/or skip mode decision separated from each other, are defined for picture 20 in units of the sub-partitioning of blocks 40, i.e. in units of sub-blocks 50 and 60 in the exemplarily shown block 40 of
In accordance with some embodiments of the present application, the encoder 10 avoids bit efficiency penalties resulting from the co-use of the block partitioning of blocks 40 on the one hand and the merging of sub-blocks 50 and 60 on the other hand. To be more precise, the encoder 10 may decide as to whether it is in some, for example, rate-distortion optimization sense better to further partition block 40, and as to which of supported partitioning patterns should be used for a current block 40 in order to adapt the granularity at which certain coding parameters are set or defined within the current block 40 of picture 20. As will be outlined in more detail below, the coding parameters may, for example, represent prediction parameters such as inter prediction parameters. Such inter prediction parameters may, for example, comprise a reference picture index, a motion vector and the like. The supported partitioning patterns may, for example, comprise a non-partitioning mode, i.e., an option according to which the current block 40 is not further partitioned, a horizontally partitioning mode, i.e., an option according to which the current block 40 is subdivided along a horizontally extending line into an upper or top portion and a bottom or lower portion and a vertically partitioning mode, i.e., an option according to which the current block 40 is vertically subdivided along a vertically extending line into a left portion and a right portion. Beyond this, the supported partitioning patterns may also comprise an option according to which the current block 40 is further regularly subdivided into four further blocks each assuming one quarter of current block 40. Further, the partitioning may pertain all blocks 40 of the picture 20 or merely a proper subset thereof such as those having a certain coding mode associated therewith, such as the inter prediction mode. Similarly, it is noted that merging may, per se, merely be available for certain blocks, such as those coded in the inter prediction mode. In accordance with an embodiment further outlined below, the aforementioned commonly interpreted state also signals concurrently that the respective block is of the inter prediction mode rather than the intra prediction mode. Accordingly, one state of the aforementioned flag for block 40 may signal that this block is an inter prediction coded block which is not further partitioned and for which both the merging and the skip mode are activated. However, as a subsidiary decision in case of the flag assuming the other state, each partition or sample subset 50 and 60 may individually be accompanied by a further flag within bitstream 30 in order to signal whether merging shall be applied to the respective partition 50 and 60 or not. Further, different subsets of the supported partitioning modes may be available for blocks 40, depending, for example, on the block size, the subdivision level of the block 40 in case of the same being a multi-tree subdivision leaf block, in combination or individually.
That is, the subdivision of picture 20 into blocks so as to obtain, inter alia, block 40 may be fixed or signaled within the bitstream. Similarly, the partitioning pattern to be used for further partitioning current block 40 may be signaled within the bitstream 30 in the form of partitioning information. Accordingly, the partitioning information may, thus, be thought of as being a kind of extension of the subdivision of picture 20 into blocks 40. On the other hand, an additional relevance of the original granularity of subdivision of picture 20 into blocks 40 may still remain. For example, the encoder 10 may be configured to signalize within the bitstream 30 the coding mode to be used for the respective portion or block 40 of picture 20 at the granularity defined by block 40 while the encoder 10 may be configured to vary the coding parameters of the respective coding mode within the respective block 40 at an increased (finer) granularity defined by the respective partitioning pattern chosen for the respective block 40. For example, the coding mode signaled at the granularity of blocks 40 may distinguish between intra prediction mode, inter prediction mode and the like, such as temporal inter prediction mode, inter-view prediction mode etc. The sort of coding parameters associated with the one or more sub-blocks (partitions) resulting from the partitioning of the respective block 40, then depends on the coding mode assigned to the respective block 40. For example, for an intra-coded block 40, the coding parameters may comprise a spatial direction along which picture content of previously decoded portions of picture 20 are used to fill the respective block 40. In case of an inter-coded block 40, the coding parameters may comprise, inter alia, a motion vector for motion-compensated prediction.
To be more precise, encoder 10 may be configured to use block merging in an effective way along with block partitioning. As far as the block merging is concerned, encoder 10 may determine for each partition 50 and 60, a respective set of coding parameter candidates. The encoder may be configured to determine the sets of coding parameter candidates for each of the partitions 50 and 60 based on coding parameters associated with previously decoded blocks. In particular, at least some of the coding parameter candidates within the sets of coding parameter candidates may be equal to, i.e. may be adopted from, the coding parameters of previously decoded partitions. Additionally or alternatively, at least some of the coding parameter candidates may be derived from coding parameter candidates associated with more than one previously coded partition, by way of a suitable combination such as a median, mean or the like. However, since the encoder 10 is configured to perform the determination of the reduced set of coding parameter candidates and, if more than one such coding parameter candidate remains after removal, the choice among the remaining non-removed coding parameter candidates, for each of the non-first partitions 60 in order to set coding parameters associated with the respective partition depending on the one non-removed or chosen coding parameter candidate, the encoder 10 is configured to perform the removal such that coding parameter candidates which would lead, effectively, to a re-uniting of partitions 50 and 60, are removed. That is, syntax constellations are effectively avoided according to which an effective partitioning situation is coded more complex than in case of directly signaling this partitioning merely by use of the partitioning information alone.
Moreover, as the sets of coding parameter candidates gets smaller, the amount of side information necessitated to encode the merging information into the bitstream 30 may decrease due to the lower number of elements in these candidate sets. In particular, as the decoder is able to determine and subsequently reduce the sets of coding parameter candidates in the same way as the encoder of
In accordance with the just-outlined possibility of reducing the merge candidate list, the encoder 10 may be configured to determine the merge candidates to be removed by way of a comparison of their coding parameters with the coding parameters of the partition, the merging with which would yield another supported partitioning pattern. This way of treating the coding parameter candidates would, effectively, remove at least one coding parameter candidate in the illustrative case of
Before describing an embodiment of a decoder fitting to the just-outlined embodiment of
After having described an encoder according to an embodiment of the present invention with regard to
That is, the decoder's functionality largely coincides with that of the encoder described with respect to
Analogously to
Thus, comparing
As already outlined above, the extractor 88 is configured to interpret one of the possible states of a flag or syntax element for a current block as a signal that concurrently signals the activation of the merging and the skip mode. Concurrently, extractor 88 may interpret the state to also signal a predetermined one among the supported partitioning patterns for the current block 40. For example, the predetermined partitioning pattern may be the non-partitioning mode according to which block 40 remains unpartitioned and thus forms a partition itself. Accordingly, extractor 88 expects bitstream 30 to comprise partitioning information signaling the partitioning of block 40 merely in case of the respective flag or syntax element not assuming the concurrently signaling state. As will be outlined in more detail below, the partitioning information may be conveyed within bitstream 30 via a syntax element which, concurrently, controls the coding mode of the current block 40, i.e. divide up blocks 40 into ones being inter coded and ones being intra coded. In that case, the commonly signaling state of the first flag/syntax element may also be interpreted as a signalization of the inter prediction coding mode. For each of the partitions resulting from the signaled partitioning information, extractor 88 may extract another merging flag from bitstream in case of the first flag/syntax element for block 40 not assuming the commonly signaling state concurrently signaling activation of the merging and the skip mode. In that case, the skip mode may inevitably be interpreted by extractor 88 to be switched off, and although merging may be activated by bitstream 30 individually for the partitions, the residual signal is extracted from bitstream 30 for this current block 40.
Thus, the decoder 80 of
For example, the decoder 80 may be configured to, if a number of the non-removed coding parameter candidates is non-zero, set coding parameters associated with the respective partition 60 depending on one of the non-removed parameter candidates. For example, the decoder 80 sets the coding parameters of partition 60 so as to be equal to one of the non-removed coding parameter candidate, with or without additional refinement and/or with or without scaling in accordance with a temporal distance to which the coding parameters refer, respectively. For example, the coding parameter candidate to merge with out of the non-removed candidates, may have another reference picture index associated therewith than a reference picture index explicitly signaled within the bitstream 30 for partition 60. In that case, the coding parameters of the coding parameter candidates may define motion vectors, each related to a respective reference picture index, and the decoder 80 may be configured to scale the motion vector of the finally chosen non-removed coding parameter candidate in accordance with the ratio between both reference picture indices. Thus, in accordance with the just-mentioned alternative, the coding parameters being subject to merging, would encompass the motion parameters, whereas reference picture indices would be separate therefrom. However, as indicated above, in accordance with alternative embodiments, the reference picture indices could also be a part of the coding parameters being subject to merging.
It equally applies for the encoder of
As already discussed above, the coding parameters may be prediction parameters and the decoder 80 may be configured to use the prediction parameters of the partitions 50 and 60 in order to derive a prediction signal for the respective partition. Naturally, the encoder 10 performs the derivation of the prediction signal in the same way, too. The encoder 10, however, additionally sets the prediction parameters along with all the other syntax elements within bitstream 30 in order to achieve some optimization in a suitable optimization sense.
Further, as already described above, the encoder may be configured to insert an index to a (non-removed) coding parameter candidate merely in case the number of (non-removed) coding parameter candidate for a respective partition is greater than one. Accordingly, the decoder 80 may be configured to, depending on the number of (non-removed) coding parameter candidates for, for example, partition 60, merely expect the bitstream 30 to comprise a syntax element specifying which of the (non-removed) coding parameter candidate is employed for merging, if the number of (non-removed) coding parameter candidates is greater than one. However, the case of the candidate set getting smaller in number than two, could be generally excluded from occurring by extending, as described above, the list/set of candidates using combined coding parameters, i.e. parameters having been derived by combination of the coding parameters of more than one—or more than two—previously coded partitions, with restricting the performance of the candidate set reduction to those candidates having been obtained by adopting, or derivation from, the coding parameters of exactly one previously coded partition. The opposite is possible as well, i.e. generally removing all coding parameter candidates having the same value as those of the partition resulting in another supported partitioning pattern.
Regarding the determination, the decoder 80 acts as encoder 10 does. That is, decoder 80 may be configured to determine the set of merge candidates for the partition or the partitions of a block 40 based on coding parameters associated with previously decoded partitions. That is, a coding order may not only be defined among the partitions 50 and 60 of a respective block 40, but also among blocks 40 of picture 20 itself. All the partitions having been coded prior to partition 60 may, thus, serve the basis for the determination of the set of merge candidates for any of the subsequent partitions, such as partition 60 in case of
Coinciding with the description of
Further, in case of the encoder introducing subdivision information into the bitstream in order to subdivide picture 20 into the blocks 40, the decoder 80 may be configured to recover the subdivision of picture 20 into such coding blocks 40 according to the subdivision information in the bitstream 30.
With regard to
Before describing further possible details of further embodiments below, a possible internal structure of encoder and decoder of
The elements of
A possible implementation of the internal structure of decoder 80 of
Briefly describing the functionality of the decoder of
As becomes clear from comparing
In the above description of
As already noted above, each coding block 40 may be parallelly subdivided into transform blocks so that transform blocks may represent a different subdivision of the respective coding block 40. To each of these transform blocks, which are not shown in
In the following, further embodiments of the present invention are described. While the above embodiments concentrated on the relation between the block merging on the one hand and the block partitioning on the other hand, the following description also includes aspects of the present application relating to other coding principles known in present codecs, such as SKIP/DIRECT modes. Nevertheless, the subsequent description shall not be regarded as merely describing separate embodiments, i.e., embodiments separated from those described above. Rather, the description below also reveals possible implementation details for the embodiments described above. Accordingly, the description below uses reference signs of the figures already described above, so that a respective possible implementation described below, shall define possible variations of embodiments described above, too. Most of these variations may be individually transferred to the above embodiments.
In other words, embodiments of the present application describe methods for reducing the side information rate in image and video coding applications by combined signaling of merging and the absence of residual data for sets of samples. In other words, the side information rate in image and video coding applications is reduced by combining syntax elements indicating the usage of merging schemes and syntax elements indicating the absence of residual data.
Further, before describing these variations and further details, an overview over picture and video codecs is presented.
In image and video coding applications, the sample arrays associated with a picture are usually partitioned into particular sets of samples (or sample sets), which may represent rectangular or quadratic blocks or any other collection of samples including arbitrarily shaped regions, triangles, or any other shapes. The subdivision of the samples arrays may be fixed by the syntax or the subdivision is (at least partly) signaled inside the bitstream. To keep the side information rate for signaling the subdivision information small, the syntax usually allows only a limited number of choices resulting in simple partitioning such as the subdivision of blocks into smaller blocks. An often used partitioning scheme is the partitioning of square block into four smaller square blocks, or into two rectangular blocks of the same size, or into two rectangular blocks of different sizes, where the actually employed partitioning is signaled inside the bitstream. The sample sets are associated with particular coding parameters, which may specify prediction information or residual coding modes, etc. In video coding applications, a partitioning is often done for the purpose of motion representation. All samples of a block (inside a partitioning pattern) are associated with the same set of motion parameters, which may include parameters specifying the type of prediction (e.g., list 0, list 1, or bi-prediction; and/or translational or affine prediction or a prediction with a different motion model), parameters specifying the employed reference pictures, parameters specifying the motion with respect to the reference pictures (e.g., displacement vectors, affine motion parameter vectors, or motion parameter vectors for any other motion model), which are usually transmitted as a difference to a predictor, parameters specifying the accuracy of motion parameters (e.g., half-sample or quarter-sample accuracy), parameters specifying the weighting of the reference sample signal (e.g., for the purpose of illumination compensation), or parameters specifying the interpolation filter that is employed for deriving the motion compensated prediction signal of the current block. It is assumed that for each sample set, individual coding parameters (e.g., for specifying the prediction and/or residual coding) are transmitted. In order to achieve an improved coding efficiency, this invention presents a method and particular embodiments for merging two or more sample sets into so-called groups of sample sets. All sample sets of such a group share the same coding parameters, which can be transmitted together with one of the sample sets in the group. By doing so, the coding parameters do not need to be transmitted for each sample set of the group of sample sets individually, but instead the coding parameters are transmitted only once for the whole group of sample sets.
As a result the side information rate for transmitting the coding parameters is reduced and the overall coding efficiency is improved. As an alternative approach, an additional refinement for one or more of the coding parameters can be transmitted for one or more of the sample sets of a group of sample sets. The refinement can be either applied to all sample sets of a group or only to the sample set for which it is transmitted.
Some embodiments of the present invention combine the merging process with a partitioning of a block into various sub-blocks 50, 60 (as mentioned above). Usually, image or video coding systems support various partitioning patterns for a block 40. As an example, a square block can be either not be partitioned or it can be partitioned into four square blocks of the same size, or into two rectangular blocks of the same size (where the square block can be vertically or horizontally divided), or into rectangular blocks of different sizes (horizontally or vertically). The described exemplary partition patterns are illustrated in
Some embodiments of the present invention reduce the side information rate and thus increase the coding efficiency for a combination of the concept of merging with the concept of providing different partitioning patterns for a block. If we look at the example partitioning patterns in
As an example, if the current partitioning pattern specifies the subdivision into two rectangular blocks, as shown in
Another aspect that needs to be considered is that the merging concept is in some sense similar to the SKIP or DIRECT modes that are found in video coding designs. In SKIP/DIRECT modes, basically no motion parameters are transmitted for a current block, but are inferred from a spatial and/or temporal neighborhood. In a particular efficient concept of the SKIP/DIRECT modes, a list of motion parameter candidates (reference frame indices, displacement vectors, etc.) is created from a spatial and/or temporal neighborhood and an index into this list is transmitted that specifies which of the candidate parameters is chosen. For bi-predicted blocks (or multi-hypothesis frames), a separate candidate can be signaled for each reference list. Possible candidates may include the block to the top of the current block, the block to the left of the current block, the block to the top-left of the current block, the block to the top-right of the current block, the median predictor of various of these candidates, the co-located block in one or more previous reference frames (or any other already coded block, or a combination obtained from already coded blocks).
Combining SKIP/DIRECT with the merge concept means that a block can be coded using either a SKIP/DIRECT or a merging mode. Although the SKIP/DIRECT and merging concepts are similar there are differences between the two concepts which are explained in more detail in section 1. The main difference between SKIP and DIRECT is that the SKIP mode further signals that no residual signal is transmitted. When the merging concept is used, usually a flag is transmitted that signals whether a block contains non-zero transform coefficient levels.
In order to achieve an improved coding efficiency, the embodiments described above and below combine the signaling whether a sample set uses the coding parameters of another sample set and the signaling whether no residual signal is transmitted for the block. The combined flag indicates that a sample set uses coding parameters of another sample set and that no residual data is transmitted. For this case only one flag, instead of two, needs to be transmitted.
As mentioned above, some embodiments of the present invention also provide an encoder with a greater freedom for creating a bitstream, since the merging approach significantly increases the number possibilities for selecting a partitioning for the sample arrays of a picture without introducing redundancy in the bitstream. Since the encoder can choose between more options, e.g., for minimizing a particular rate-distortion measure, the coding efficiency can be improved. As an example, some of the additional patterns that can be represented by a combination of sub-partitioning and merging (e.g., the patterns in
In the following, some possible detailed implementation for the embodiments outlined above are described, such as for the encoders in
In state-of-the-art image and video coding standards, the possibilities for subdividing a picture (or a plane group) into blocks that are provided by the syntax are very limited. Usually, it can only be specified whether and (potentially how) a block of a predefined size can be subdivided into smaller blocks. As an example, the largest block size in H.264 is 16×16. The 16×16 blocks are also referred to as macroblocks and each picture is partitioned into macroblocks in a first step. For each 16×16 macroblock, it can be signaled whether it is coded as 16×16 block, or as two 16×8 blocks, or as two 8×16 blocks, or as four 8×8 blocks. If a 16×16 block is subdivided into four 8×8 block, each of these 8×8 blocks can be either coded as one 8×8 block, or as two 8×4 blocks, or as two 4×8 blocks, or as four 4×4 blocks. The small set of possibilities for specifying the partitioning into blocks in state-of-the-art image and video coding standards has the advantage that the side information rate for signaling the subdivision information can be kept small, but it has the disadvantage that the bit rate necessitated for transmitting the prediction parameters for the blocks can become significant as explained in the following. The side information rate for signaling the prediction information does usually represent a significant amount of the overall bit rate for a block. And the coding efficiency could be increased when this side information is reduced, which, for instance, could be achieved by using larger block sizes. It is also possible to increase the set of supported partitioning patterns in comparison to H.264. For example, the partitioning patterns depicted in
If the bit rate that is saved by reducing the number of coded prediction parameters is larger than the bit rate that is additionally spend for coding the merging information, the described merging does result in increased coding efficiency. It should further be mentioned that the described syntax extension (for the merging) provides the encoder with additional freedom in selecting the partitioning of a picture or plane group into blocks, without introducing redundancy. The encoder is not restricted to do the subdivision first and then to check whether some of the resulting blocks have the same set of prediction parameters. As one simple alternative, the encoder could first determine the subdivision as in state-of-the-art coding techniques. And then it could check for each block, whether a merging with one of its neighbor blocks (or the associated already determined group of blocks) reduces a rate-distortion cost measure. At this, the prediction parameters associated with the new group of blocks can be re-estimated (e.g., by performing a new motion search) or the prediction parameters that have already been determined for the current block and the neighboring block or group of blocks could be evaluated for the new group of blocks. An encoder can also directly check (a subset of) the patterns that are provided by a combination of splitting and merging; i.e., the motion estimation and mode decision can be done with the resulting shapes as already mentioned above. The merging information can be signaled on a block basis. Effectively, the merging could also be interpreted as inference of the prediction parameters for a current block, where the inferred prediction parameters are set equal to the prediction parameters of one of the neighboring blocks.
For other modes than SKIP, additional flags, like the CBF, are needed to signal that no residual signal is transmitted. There are two variants of SKIP/DIRECT modes in the state-of-the-art video coding standard in H.264, which are selected on a picture level: the temporal direct mode and the spatial direct mode. Both direct modes are only applicable to B pictures. In temporal direct mode, the reference index for reference picture list 0 is set equal to 0 and the reference index for reference picture list 1 as well as the motion vectors for both reference lists are derived based on the motion data of the co-located macroblock in the first reference picture in reference picture list 1. The temporal DIRECT mode uses the motion vector from the temporal collocated block and scales the motion vector according to the temporal distance between the current and the collocated block. In spatial direct mode, the reference indexes and motion vectors for both reference picture lists are basically inferred based on the motion data in a spatial neighborhood. The reference indexes are chosen as the minimum of the corresponding reference indexes in the spatial neighborhood and each motion vector component is set equal to the median of the corresponding motion vector components in the spatial neighborhood. The SKIP mode can only be used for coding 16×16 macroblocks in H.264 (in P and B pictures) and the DIRECT mode can be used for coding 16×16 macroblocks or 8×8 sub-macroblocks. In contrast to the DIRECT mode, if merging is applied to a current block, all prediction parameters may be copied from the block the current block is merged with. Merging can also be applied to arbitrary block sizes resulting in the above-mentioned more flexible partitioning patterns where all samples of one pattern are predicting using the same prediction parameters.
The basic idea of the embodiments outlined above and below is to reduce the bit rate that is necessitated for transmitting the CBF flags by combining the merge and CBF flags. If a sample set uses merging and no residual data is transmitted, one flag is transmitted signaling both.
In order to reduce the side information rate in image and video coding applications, particular sets of samples (which may represent rectangular or quadratic blocks or arbitrarily shaped regions or any other collection of samples) are usually associated with a particular set of coding parameters. For each of these sample sets, the coding parameters are included in the bitstream. The coding parameters may represent prediction parameters, which specify how the corresponding set of samples is predicted using already coded samples. The partitioning of the sample arrays of a picture into sample sets may be fixed by the syntax or may be signaled by corresponding subdivision information inside the bitstream. Multiple partitioning patterns for a block may be allowed. The coding parameters for the sample sets are transmitted in a predefined order, which is given by the syntax. It can be signaled for a current set of samples that it is merged (e.g., for the purpose of prediction) with one or more other sample sets into a group of sample sets. The possible set of values for the corresponding merging information may be adapted to the employed partitioning pattern, in a way that particular partitioning patterns cannot be represented by a combination of other partitioning patterns and corresponding merging data. The coding parameters for a group of sample sets need to be transmitted only once. Additional to the prediction parameters, residual parameters (such as transform and quantization side information and transform coefficient levels) may be transmitted. If the current sample set is merged, side information describing the merging process are transmitted. This side information will be further referred to as merging information. The embodiments described above and below describe a concept by which the signaling of the merging information is combined with the signaling of the coded block flag (specifying whether residual data is present for a block).
In a particular embodiment, the merging information contains a combined, so called mrg_cbf, flag that is equal to 1 if the current sample set is merged and no residual data is transmitted. In this case no further coding parameters and residual parameters are transmitted. If the combined mrg_cbf flag is equal to 0, another flag indicating whether merging is applied or not is coded. Furthermore flags indicating that no residual parameters are transmitted are coded. In CABAC and context adaptive VLC, the contexts for probability derivation (and VLC table switching) for syntax elements related to merging information can be chosen as a function of already transmitted syntax elements and/or decoded parameters (such as the combined mrg_cbf flag).
In an embodiment, the merging information containing a combined mrg_cbf flag is coded before coding parameters (e.g. prediction information and subdivision information).
In an embodiment, the merging information containing a combined mrg_cbf flag is coded after a subset of the coding parameters (e.g. prediction information and subdivision information). For every sample set, resulting from the subdivision information, merging information may be coded.
In embodiments described further below with respect to
The following embodiments are described for sets of samples that represent rectangular and quadratic blocks, but it can be extended to arbitrarily shaped regions or other collections of samples in a straightforward way. The embodiments describe combinations of syntax elements related to the merging scheme and syntax elements indicating the absence of residual data. Residual data may include residual side information as well as transform coefficient levels. For all the embodiments, the absence of residual data is specified by means of a coded block flag (CBF) but it can be expressed by other means or flags as well. A CBF equal to 0 relates to the case in which no residual data is transmitted.
In the following, the subsidiary merging activating flag is called mrg whereas later on, with respect to
A possible combination of the merging flag and the CBF flag by using one syntax element is described in this section. The description of this possible combination outlined below, may be transferred to any of the above described shown in
In an embodiment, up to three syntax elements are transmitted for specifying the merging information and the CBF.
The first syntax element, which is called mrg_cbf in the following, specifies whether the current set of samples is merged with another sample set and whether all corresponding CBFs are equal to 0. The mrg_cbf syntax element may only be coded if a derived set of candidate sample sets is not empty (after the potential removal of the candidates that would produce a partitioning that could be signaled by different partitioning pattern without merging). However, it may be guaranteed by default that the list of merge candidates never vanishes, that there is at least one or even at least two merge candidates available. In an embodiment of the invention, if a derived set of candidate sample sets is not empty, the mrg_cbf syntax element is coded as follows.
The values 0 and 1 for the mrg_cbf syntax elements can also be switched.
The second syntax element, further called mrg specifies whether the current set of samples is merged with another sample set. If the mrg_cbf syntax element is equal to 1 the mrg syntax element is not coded and inferred to be equal to 1 instead. If the mrg_cbf syntax element is not present (because the derived set of candidate samples is empty), the mrg syntax element is also not present, but inferred to be equal to 0. However, it may be guaranteed by default that the list of merge candidates never vanishes, that there is at least one or even at least two merge candidates available
The third syntax element, further called mrg_idx, which is only coded if the mrg syntax element is equal to 1 (or inferred to be equal to 1), specifies which of the sets of candidate sample sets is employed for merging. In an embodiment, the mrg_idx syntax element is only coded if a derived set of candidate sample sets contains more than one candidate sample set. In a further embodiment, the mrg_idx syntax element is only coded if at least two sample sets of a derived set of candidate sample sets are associated with different coding parameters.
It should be mentioned that the merge candidate list can even be fixed in order to decouple the parsing and the reconstruction to improve the parsing throughput and to be more robust in regard to information loss. To be more precise, the decoupling can be assured by using a fixed assignment of list entries and codewords. This would not necessitate to fix the length of the list. However, concurrently fixing the length of the list by adding additional candidates allows for compensating the coding efficiency loss of the fixed (longer) codewords. Thus, as described before, the merge index syntax element may only be transmitted if the list of candidates contains more than one candidate. This would necessitate, however, to derive the list prior to parsing merge index, preventing to carry out these two processes in parallel. To allow for an increased parsing throughput and to make the parsing process more robust with regard to transmission errors, it is possible to remove this dependency by using a fixed codeword for each index value and a fixed number of candidates. If this number may not be reached by a candidate selection, it is possible to derive ancillary candidates to complete the list. These additional candidates may include so-called combined candidates, which are built from motion parameters of possibly different candidates already in the list, and zero motion vectors.
In an embodiment, the merging information for a set of samples is coded after a subset of the prediction parameters (or, more generally, the particular coding parameters that are associated with the sample sets) has been transmitted. The subset of prediction parameters may consist of one or more reference picture indices or one or more components of a motion parameter vector or a reference picture index and one or more components of a motion parameter vector, etc.
In an embodiment the mrg_cbf syntax element of the merging information is coded only for a reduced set of partitioning modes. A possible set of partitioning modes is presented in
In another embodiment the mrg_cbf syntax element of the merging information is coded only for one block of a partitioning where this partitioning is one of the possible partitioning modes shown in
In a further embodiment of the invention, the merging information for a set of samples is coded before the prediction parameters (or, more generally, the particular coding parameters that are associated with the sample sets). The merging information, containing the mrg_cbf, the mrg and the mrg_idx syntax element, is coded in a manner as described in the first embodiment above. The prediction or coding parameters and the residual parameters are only transmitted if the merging information signals that the current set of samples is not merged with another set of samples and that the CBF, for at least one of the components, is equal to 1. In an embodiment, if the mrg_cbf syntax element specifies that the current block is merged and the CBFs for all components are equal to 0, there will be no more signalization needed after the merging information, for this current block.
In another embodiment of the invention, the syntax elements mrg_cbf, mrg, and mrg_idx are combined and coded as one or two syntax elements. In one embodiment, mrg_cbf and mrg are combined into one syntax element, which specifies any of the following cases: (a) the block is merged and it doesn't contain residual data, (b) the block is merged and contains residual data (or may contain residual data), (c) the block is not merged. In another embodiment, the syntax elements mrg and mrg_idx are combined into one syntax element. If N is being the number of merge candidates, the combined syntax element specifies one of the following cases: the block is not merged, the block is merged with candidate 1, the block is merged with candidate 2, . . . , the block is merged with candidate N. In a further embodiment of the invention, the syntax elements mrg_cfb, mrg, and mrg_idx are combined into one syntax element, which specifies one of the following cases (with N being the number of candidates): the block is not merged, the block is merged with candidate 1 and doesn't contain residual data, the block is merged with candidate 2 and doesn't contain residual data, . . . , the block is merged with candidate N and doesn't contain residual data, the block is merged with candidate 1 and contains (or may contain) residual data, the block is merged with candidate 2 and contains (or may contain) residual data, . . . , the block is merged with candidate N and contains (or may contain) residual data. The combined syntax elements may be transmitted with a variable length code or may be transmitted by arithmetic coding or may be transmitted by binary arithmetic coding using any particular binarization scheme.
The SKIP/DIRECT modes may be supported for all or only particular block sizes and/or block shapes. In an extension of the SKIP/DIRECT modes as specified in the state-of-the-art video coding standard H.264, a set of candidate blocks is used for the SKIP/DIRECT modes. The difference between SKIP and DIRECT is whether residual parameters are sent or not. The parameters (e.g., for prediction) of SKIP and DIRECT can be inferred from any of the corresponding candidates. A candidate index is coded, which signals which candidate is used to infer the coding parameters. If multiple predictions are combined to form the final prediction signal for the current block (as in bi-predictive blocks used in H.264 B-frames) every prediction can refer to a different candidate. Thus for every prediction a candidate index can be coded.
In an embodiment of the invention, the candidate list for SKIP/DIRECT may contain different candidate blocks than the candidate list for merge modes. An example is illustrated in
The following notation is used for describing following embodiments:
In an embodiment, both SKIP/DIRECT modes and block merging modes are supported. SKIP/DIRECT modes use the original set of candidates, set_mvp_ori. The merging information related to the block merging mode may contain the combined mrg_cbf syntax element.
In another embodiment, both SKIP/DIRECT mode and block merging modes are supported, but the SKIP/DIRECT modes use a modified set of candidates, set_mvp_comb. This modified set of candidates can be a particular subset of the original set set_mvp_ori. In an embodiment, the modified set of candidates is composed of Corner blocks and a Collocated block. In another embodiment, the modified set of candidates is only composed of the Collocated block. Further subsets are possible.
In another embodiment, the merging information containing the mrg_cbf syntax element is coded before the SKIP mode related parameters.
In another embodiment, the SKIP mode related parameters are coded before the merging information containing the mrg_cbf syntax element.
According to another embodiment, DIRECT mode may not be activated (is not even present) and block merging has an extended set of candidates with SKIP mode replaced by mrg_cbf.
In an embodiment, the candidate list for block merging may contain different candidate blocks. An example is illustrated in
It should be mentioned that position of candidates for block merging can be the same than the list of MVP in inter-prediction in order to save memory access.
Further, the list can be “fixed” in the manner outlined above in order to decouple the parsing and the reconstruction to improve the parsing throughput & to be more robust in regard to information loss.
In an embodiment, if the mrg_cfb syntax element is equal to 0 (it signals that the block is either not merged or that it contains nonzero residual data), a flag is transmitted that signals whether or not all components (e.g., a luma and two chroma components) of the residual data are zero. This flag is not transmitted if mrg_cfb is equal to 1. In a particular configuration, this flag is not transmitted if mrg_cfb is equal to 0 and the syntax element mrg specifies that the block is merged.
In another embodiment, if the mrg_cfb syntax element is equal to 0 (it signals that the block is either not merged or that it contains nonzero residual data), a separate syntax element for each component is transmitted that signals whether or not the residual data for the component are zero.
Different context models could be used for mrg_cbf
Further, an apparatus for decoding a bitstream having a picture encoded therein has been described, comprising a subdivider configured to subdivide the picture into sample sets;
One of the possible states of the one or more syntax elements signalizes that the respective sample set is to be merged into one of the disjoint sets along with another sample set and has no prediction residual encoded and inserted into the bitstream.
The extractor may also be configured to extract subdivision information from the bitstream, and the subdivider is configured to subdivide the picture into sample sets responsive to the subdivision information.
The extractor and the merger may, for example, be configured to sequentially step through the sample sets according to a sample set scan order, and to, for a current sample set, extract a first binary syntax element (mrg_cbf) from the bitstream;
if the first binary syntax element assumes a first binary state, merge the current sample set into one of the disjoint sets by inferring the coding parameters for the current sample set to be equal to the coding parameters associated with this disjoint set, skip the extraction of the prediction residual for the current sample set and step to the next sample set in sample set scan order;
if the first binary syntax element assumes a second binary state, extract a second syntax element (mrg, mrg_idx) from the bitstream; and depending on the second syntax element, merge the current sample set into one of the disjoint sets by inferring the coding parameters for the current sample set to be equal to the coding parameters associated with this disjoint set or perform the extraction of the coding parameters for the current sample set, with extracting at least one further syntax element concerning the prediction residual for the current sample set.
The one or more syntax elements for each of at least a subset of the sample sets may also signal with which of a set of predetermined candidate sample sets neighboring the respective sample set, the respective sample set is to be merged, if the respective sample set is to be merged into any one of the disjoint sets along with another sample set.
The extractor may be configured to, if the one or more syntax elements does not signal that the respective sample set is to be merged into any of the disjoint sets along with another sample set,
extract from the bitstream one or more further syntax elements (SKIP/DIRECT mode) signaling as to whether, and from which of a further set of predetermined candidate sample sets neighboring the respective sample set, at least a part of the coding parameters for the respective sample set is to be predicted.
In that case, the set of predetermined candidate sample sets and the further set of predetermined candidate sample sets may be disjoint or intersect each other with respect to a minority of the predetermined candidate sample sets of the set of predetermined candidate sample sets and the further set of predetermined candidate sample sets, respectively.
The extractor may also be configured to extract subdivision information from the bitstream, and the subdivider is configured to hierarchically subdivide the picture into sample sets responsive to the subdivision information, and the extractor is configured to sequentially step through child sample sets of a parent sample set, comprised by the sample sets into which the picture is sub-divided, and, for a current child sample set, extract a first binary syntax element (mrg_cbf) from the bitstream; if the first binary syntax element assumes a first binary state, merge the current child sample set into one of the disjoint sets by inferring the coding parameters for the current child sample set to be equal to the coding parameters associated with this disjoint set, skip the extraction of the prediction residual for the current child sample set and step to the next child sample sets; if the first binary syntax element assumes a second binary state, extract a second syntax element (mrg, mrg_idx) from the bitstream; and depending on the second syntax element, merge the current child sample set into one of the disjoint sets by inferring the coding parameters for the current child sample set to be equal to the coding parameters associated with this disjoint set or perform the extraction of the coding parameters for the current child sample set, with extracting at least one further syntax element concerning the prediction residual for the current child sample set, and then stepping to the next child sample sets, with, for the next child sample sets, skipping extraction of the first binary syntax element if the first binary syntax element of the current child sample set assumes the first binary state and starting with extracting the second syntax element instead, and extracting the first binary syntax element if the first binary syntax element of the current child sample set assumes the second binary state.
Lets assume, for example, a parent sample set (CU) is split into two child sample sets (PU). If, then, for the first PU, the first binary syntax element (merge_cbf) has the first binary state 1) the first PU uses merge and the first and the second PU (the whole CU) do not have residual data in the bitstream, and 2) for the second PU second binary syntax element (merge_flag, merge_idx) is signaled. If, however, the first binary syntax element for the first PU has the second binary state, then 1) for the first PU, the second binary syntax element (merge_flag, merge_idx) is signaled, and residual data is in the bitstream as well, while 2) for the second PU, the first binary syntax element (merge_cbf) is signaled. Thus, it could be that the merge_cbf is also signaled on a PU level, i.e. for successive child sample sets, if merge_cbf is in a secondary binary state for all previous child sample sets. If merge_cbf is in a first binary state for a successive child sample set, all child sample sets following this child sample set do not have residual data in the bitstream. For example, for a CU split into, for example, 4 PUs, it is possible that merge_cbf is in a first binary state for the second PU meaning that the third and fourth PU in coding order do not have residual data in the bitstream but the first PU has or can have.
The first and second binary syntax elements may be coded using context-adaptive variable length coding or context-adaptive (binary) arithmetic coding and the contexts for coding the syntax elements are derived based on the values for these syntax elements in already coded blocks.
As described in other embodiments, the syntax element merge_idx may only be transmitted if the list of candidates contains more than one candidate. This necessitates to derive the list prior to parsing merge index, preventing to carry out these two processes in parallel. To allow for an increased parsing throughput and to make the parsing process more robust with regard to transmission errors, it is possible to remove this dependency by using a fixed codeword for each index value and a fixed number of candidates. If this number may not be reached by a candidate selection, it is possible to derive ancillary candidates to complete the list. These additional candidates may include so-called combined candidates, which are built from motion parameters of possibly different candidates already in the list, and zero motion vectors
In another embodiment, the syntax for signaling which of the blocks of the candidate set is simultaneously adapted at encoder and decoder. If for example, 3 choices of blocks for merging are given, those three choices are only present in the syntax and are considered for entropy coding. The probabilities for all other choices are considered to be 0 and the entropy codec is adjusted simultaneously at encoder and decoder.
The prediction parameters that are inferred as a consequence of the merging process may represent the complete set of the prediction parameters that are associated with a block or they may represent of subset of these prediction parameters (e.g., the prediction parameters for one hypothesis of a block for which multi-hypotheses prediction is used).
In an embodiment, the syntax elements related to the merging information are entropy coded using context modeling.
One way of transferring the above-outlined embodiments to a specific syntax is explained in the following with respect to the following figures. In particular,
In the currently described embodiment, the above-mentioned option is used according to which merging is merely usable for pictures for which the inter prediction mode is available. That is, intra-coded slices/pictures do not use merging anyway. This is visible from
Accordingly, if skip_flag is set, the function prediction_unit is called at 408 with denoting the current coding unit as being a prediction unit. This is, however, not the only possibility for switching on the merging option. Rather, if the skip_flag related to the whole coding unit is not set at 406, the prediction type of the coding unit of the non-intra-picture slice is signaled at 410 by syntax element pred_type with, depending thereon, calling function prediction_unit for any partition of the current coding unit at, for example, 412 in case of the current coding unit being not further partitioned. In
As is visible from
As already indicated above, skip_flag=1 concurrently signals that no residual data is to be transmitted. This is derivable from the fact that the transmission of the residual data at 426 in
Up to now, the embodiment of
See, for example,
Table 1 specifies the significance of the possible states of the syntax elements cu_split_pred_part_mode in case of the current coding unit have a size which is not the smallest one in the quad-tree subdivisioning of the current tree-root block. The possible states are listed in at the outermost left-hand column of table 1. As table 1 refers to the case where the current coding unit does not have the smallest size, there is a state of cu_split_pred_part_mode, namely the state 0, which signals that the current coding unit is not an actual coding unit but has to be subdivided into further four units which are then traversed in depth-first traversal order as outlined by calling function coding tree at 432 again. That is, cu_split_pred_part_mode=0 signals that the current quad-tree subdivision unit of the current tree-root block is to be subdivided into four further smaller units again, i.e. split_coding_unit_flag=1. However, if cu_split_pred_part_mode assumes any other possible state, then split_coding_unit_flag=0 and the current unit forms a leaf block of the current tree-root block, i.e. a coding unit. In that case, one of the remaining possible states of cu_split_pred_part_mode represents the above-described commonly signaling state which concurrently signals that the current coding unit is subject to merging and has the skip mode activated, indicated by skip_flag equaling 1 in the third column of table 1, while concurrently signaling that no further partition of the current coding unit takes place, i.e. PART_2N×2N is chosen as the partitioning mode. cu_split_pred_part_mode also has a possible state which signals the activation of merging with the skip mode being deactivated. This is the possible state 2, corresponding to skip_flag=0 while merge_flag=1 with the no partitioning mode being active, i.e. PART_2N×2N. That is, in that case the merge_flag is signaled before hand rather than within the prediction_unit syntax. In the remaining possible states of cu_split_pred_part_mode, the inter prediction mode with other partitioning modes is signaled, with these partitioning modes partitioning the current coding unit into more than one partition.
Table 2 shows the significance or semantics of the possible states of cu_split_pred_part_mode in case of the current coding unit having the smallest size possible in accordance with the quad-tree subdivision of the current tree-root block. In that case, all possible states of cu_split_pred_part_mode corresponds to no further subdivision according to split_coding_unit_flag=0. However, the possible state 0 signals that skip_flag=1, i.e. concurrently signals that merging is activated and the skip mode is active. Moreover, same signals that no partitioning takes place, i.e. partitioning mode PART_2N×2N. Possible state 1 corresponds to possible state 2 of table 1 and the same applies to possible state 2 of table 2, which corresponds to possible state 3 of table 1.
Although the above description of the embodiment of
skip_flag[x0][y0] equal to 1 specifies that for the current coding unit (see 40 in the figures), when decoding a P or B slice, no more syntax elements except the motion vector predictor indices (merge_idx) are parsed after skip_flag[x0][y0]. skip_flag[x0][y0] equal to 0 specifies that the coding unit is not skipped. The array indices x0, y0 specify the location (x0, y0) of the top-left luma sample of the considered coding unit relative to the top-left luma sample of the picture (see 20 in the figures).
When skip_flag[x0][y0] is not present, it shall be inferred to be equal to 0.
As described above if skip_flag[x0][y0] is equal to 1,
cu_split_pred_part_mode[x0][y0] specifies split_coding_unit_flag and when the coding unit is not split the skip_flag[x0][y0], the merge_flag[x0][y0], PredMode and PartMode of a coding unit. The array indices x0 and y0 specify the location (x0, y0) of the top-left luma sample of the coding unit relative to the top-left luma sample of the picture.
merge_flag[x0][y0] specifies whether the inter prediction parameters for the current prediction_unit (see 50 and 60 in the figures, i.e. the partition within coding unit 40) are inferred from a neighboring inter-predicted partition. The array indices x0, y0 specify the location (x0, y0) of the top-left luma sample of the considered prediction block relative to the top-left luma sample of the picture.
merge_idx[x0][y0] specifies the merging candidate index of the merging candidate list where x0, y0 specify the location (x0, y0) of the top-left luma sample of the considered prediction block relative to the top-left luma sample of the picture.
Although not specifically indicated in the above description of
In order to perform the above-mentioned optional removal of those candidates which would lead to another partitioning mode which would also have been available, the following functions could be used:
In particular, the candidate N, i.e. the coding/prediction parameters stemming from the prediction_unit/partition covering pixel N=(B0, B1, B2, A0, A1), i.e. position (xN, yN), is removed from the candidate list if any of the following conditions is true (please see
In this regard, please note that position or location (xP, yP) denotes the uppermost pixel of the current partition/prediction unit. That is, in accordance with the first item, all coding parameter candidates are checked which have been derived by directly adopting the respective coding parameters of neighboring prediction units, namely prediction unit N. The other additional coding parameter candidates may, however, be checked in the same manner as to whether same are equal to the coding parameters of the respective prediction unit emerging with which would result in obtaining another partitioning pattern also supported by the syntax. In accordance with the embodiments just described, the equality of the coding parameters encompasses a check of the equality of the motion vector, i.e. mvLX, the reference index, i.e. reflxLX, and the prediction flag predFlagLX indicating that the parameters, i.e. motion vector and reference index, associated with reference list X, with X being 0 or 1, are used in inter prediction.
Please note that the just-described possibility for removal of coding parameter candidates of neighboring prediction units/partitions would also be applicable in case of supporting asymmetric partitioning modes shown in the right-hand half of
Regarding the embodiment
Further, it is noted that three contexts could be used for the skip_flag, merge_flag and the merge_idx, respectively.
Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.
Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blue-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.
Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.
Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.
Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.
In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary.
A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.
A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.
A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.
A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.
In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are performed by any hardware apparatus.
While this invention has been described in terms of several advantageous embodiments, there are alterations, permutations, and equivalents which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations, and equivalents as fall within the true spirit and scope of the present invention.
The present application is a continuation of U.S. patent application Ser. No. 16/251,785 filed Jan. 18, 2019, which is a continuation of U.S. patent application Ser. No. 15/890,168, filed Feb. 6, 2018, now U.S. Pat. No. 10,382,776, which is a continuation of U.S. patent application Ser. No. 13/875,779 filed May 2, 2013, now U.S. Pat. No. 9,924,193, which is a continuation of International Application PCT/EP2011/069408, filed Nov. 4, 2011, which claims priority to U.S. Provisional Application 61/410,246, filed Nov. 4, 2010, each of which is incorporated herein by reference in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| 61410246 | Nov 2010 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17027168 | Sep 2020 | US |
| Child | 18363057 | US | |
| Parent | 16251785 | Jan 2019 | US |
| Child | 17027168 | US | |
| Parent | 15890168 | Feb 2018 | US |
| Child | 16251785 | US | |
| Parent | 13875779 | May 2013 | US |
| Child | 15890168 | US | |
| Parent | PCT/EP2011/069408 | Nov 2011 | US |
| Child | 13875779 | US |
