Patent Application
20240013446
Video coding (video encoding and decoding) is used in a wide range of digital video applications, for example broadcast digital TV, video transmission over internet and mobile networks, real-time conversational applications such as video chat, video conferencing, DVD and Blu-ray discs, video content acquisition and editing systems, and camcorders of security applications.
The amount of video data needed to depict even a relatively short video can be substantial, which may result in difficulties when the data is to be streamed or otherwise communicated across a communications network with limited bandwidth capacity. Thus, video data is generally compressed before being communicated across modern day telecommunications networks. The size of a video could also be an issue when the video is stored on a storage device because memory resources may be limited. Video compression devices often use software and/or hardware at the source to code the video data prior to transmission or storage, thereby decreasing the quantity of data needed to represent digital video images. The compressed data is then received at the destination by a video decompression device that decodes the video data. With limited network resources and ever increasing demands of higher video quality, improved compression and decompression techniques that improve compression ratio with little to no sacrifice in picture quality are desirable.
Neural networks and deep-learning techniques making use of neural networks have now been used for some time, also in the technical field of encoding and decoding of videos, images and the like.
In such cases, the bitstream usually represents or is data that can reasonably be represented by a two-dimensional matrix of values. For example, this holds for bitstreams that represent or are images, video sequences or the like data. Apart from 2D data, the neural network and the framework referred to in the present disclosure may be applied to further source signals such as audio signals, which are typically represented as a ID signal, or other signals.
For example, neural networks comprising a plurality of downsampling layers may apply a downsampling (convolution, in the case of the downsampling layer being a convolution layer) to an input to be encoded, like a picture. By applying this downsampling to the input picture, its size is reduced and this can be repeated until a final size is obtained. Such neural networks can be used for both, image recognition with deep-learning neural networks and encoding of pictures. Correspondingly, such networks can be used to decode an encoded picture. Other source signals such as signals with less or more than two dimensions may be also processed by similar networks.
It may be desirable to provide a neural network framework which may be efficiently applied to various different signals possibly differing in size.
Embodiments of the disclosure presented here may allow for reducing a size of a bitstream obtained from an encoder that encodes a picture input where this bitstream carries information of the encoded picture while, at the same time, ensuring that the original picture can be reconstructed with as few losses of information as possible.
Some embodiments presented herein provide a method of encoding a picture using a neural network according to independent claim 1 as well as a method for decoding a bitstream using a neural network according to claim 39 as well as an encoder for encoding a bitstream according to claims 77 to 79 as well as a decoder for decoding a bitstream according to claims 80 to 82 and a computer-readable storage medium comprising computer-executable instructions according to claim 83.
The present disclosure provides a method for encoding a picture using a neural network, NN, wherein the NN comprises at least two sub-networks, wherein at least one sub-network of the at least two sub-networks comprises at least two downsampling layers, wherein the at least one sub-network applies a downsampling to an input representing a matrix having a size S1 in at least one dimension, the method comprising:
In the context of the present disclosure, the picture may be understood as a still picture or a moving picture in the sense of a video or a video sequence or portions thereof. Specifically, a picture may refer to a part of a total or bigger picture or a total or bigger or longer video sequence. In this regard, the invention is not limited. Additionally, a picture may also be referred to as an image or a frame in the context of the present disclosure. A picture may in any case be considered to be representable by a two- or more dimensional array of values. These values are typically referred to as “samples”. This two- or more dimensional array may specifically have the form of a matrix that can then be processed by a neural network including downsampling layers of the neural network in the manner as specified above and as will be specified further below.
In the present disclosure, a sub-network or specifically a sub-network of an encoder may be considered a part of the neural network where this part comprises a subset of the layers of the neural network. In this regard, the sub-networks of the neural network are not restricted to only comprising downsampling layers or to all comprise the same number of downsampling layers. Specifically, one sub-network may comprise two downsampling layers whereas another sub-network may only comprise one downsampling layer and another layer that does not apply a downsampling to the input but transforms it in another way. A further sub-network may comprise even more than two downsampling layers, for example 3, 5 or 10 downsampling layers.
In the context of the present disclosure, the combined downsampling ratio of a sub-network may be an integer value that corresponds to or represents a product of the downsampling ratios of all downsampling layers in a sub-network. It may be obtained by calculating the product of all downsampling ratios of a given sub-network or it may be a preset value that is available (for example to an encoder) in addition to the downsampling ratios of the downsampling layers of a sub-network. The combined downsampling ratio of a sub-network may be a predetermined number that represents the ratio between the sizes of the input of the sub-network and the output of the sub-network.
The bitstream according to the present disclosure may be or may comprise the encoded picture. Additionally, the bitstream output by the neural network may comprise additional information, also referred to as side information herein below. This additional information may refer to information that is necessary for decoding the bitstream to reconstruct the image that is encoded by the bitstream. For example, this information may comprise the combined downsampling ratio as already mentioned above or the downsampling ratios of the respective downsampling layers of the sub-networks.
The bitstream may generally be considered to be reduced in size or to comprise a representation of the original picture that is reduced in size compared to the original picture in at least one dimension. This may, for example, mean that a two-dimensional representation (for example in the form of a matrix) of the encoded picture is only half of the size of the original picture in, for example, the height or the width. The bitstream may be considered as the representation of the input image in a binary format (comprising “0”s and “1”s). The goal of the video compression is to reduce the size of the bitstream as much as possible while keeping the quality of the reconstructed picture that can be obtained based on or from the bitstream at an acceptable level.
In the context presented herein, the term “size” may refer to, for example, a number of samples in one or more dimensions (the width or the height of the picture) and/or to the number of pixels that represent the picture. Additionally, the size may represent a resolution of the picture. The resolution is usually specified in terms of number of samples per picture or picture area where this picture area might be one-dimensional or two-dimensional.
In general terms, the output of the neural network (or in general an output of one or more of the layers of the network) may have a third dimension. This third dimension may have a bigger size than the corresponding dimension of the input picture. The third dimension can represent the number of feature maps that may also be referred to as channels. In a specific example, the size of the third dimension might be three at the input (the original picture input of the neural network, e.g. with 3 color components) and 192 at the output (i.e. the feature maps before binarization (encoding into the bitstream)). The size of feature maps is typically increased by the encoder in order to classify the input efficiently.
The downsampling applied by a downsampling layer of the neural network can be achieved in any known or technically reasonable way. Specifically, this may comprise a downsampling by applying a convolution to an input of the respective downsampling layer of the neural network. The downsampling can be performed in one dimension only or it can also be performed on two dimensions of the input picture or input in general when represented in the form of a matrix. This pertains to both, the downsampling applied by a sub-network in total and the downsampling applied by each downsampling layer of a respective sub-network. For example, while a sub-network might apply a downsampling to an input in two dimensions, a first downsampling layer of this sub-network might only apply a downsampling in one dimension whereas another downsampling layer of the sub-network applies a downsampling to the input in another dimension or in two dimensions.
In general, the disclosure presented herein is not limited to particular ways of downsampling. One or more of the layers of the neural network discussed below may apply downsampling in a way that is different from convolutions for example by deleting (removing) every second, third or the like row and/or column of the input picture or input feature map (when seen in the representation of the two-dimensional matrix).
Embodiments presented herein are to be understood as referring to a rescaling that is applied immediately before the processing of an input by a sub-network comprising downsampling layers but not within the sub-network. This means, as regards the rescaling, the sub-network is, although comprising a plurality of layers and potentially a large number of downsampling layers, considered as one entity that applies a downsampling with a combined downsampling ratio to an input and the rescaling of the input is applied so that a size S of the rescaled input matches an integer multiple of the combined downsampling ratio of the sub-network by which this input is to be processed.
As will be explained further below, the rescaling that is applied to an input of a sub-network is not necessarily applied before each sub-network. Specifically, some embodiments may comprise that a determination is made, before applying any rescaling to an input of a sub-network, whether this input or the size of the input already matches an integer multiple of the combined downsampling ratio of the respective sub-network. If this is the case, the rescaling may not be applied to the input or an “identical rescaling” may be applied whereby the size S of the input is not changed. The term “rescaling” herein is used in the same meaning as “resizing”.
By applying the rescaling to the input on a per-sub-network basis, it can be accounted for each sub-network potentially providing an “intermediate output” or intermediate bitstream that is output by a sub-network. Considering, for example, a case where the output provided when encoding a picture does not only comprise a single bitstream but is made up of a plurality of bitstreams that are obtained when having processed an input picture by only a first number of sub-networks of a neural network and a second bitstream by having processed the original input by all sub-networks of the neural network. In this case, the rescaling applied on a per sub-network basis can result in a reduced size of at least one of the bitstreams, consequently resulting in a reduced size of the combined bitstream, thereby allowing for keeping the quality of the encoded picture high (when decoded again) while keeping the size of the bitstream comparably low.
It is noted that the combined downsampling ratio may be determined according to all downsampling ratios of the downsampling layers of the at least one sub-network in isolation without regard to other downsampling layers of other sub-networks. Specifically, when obtaining the size , rk,m>1 for the sub-network k. Here, the term rk,m indicates a downsampling ratio of a downsampling layer m of the sub-network k. The sub-network may comprise a total number of M (M being a natural number larger than 0) downsampling layers. When the index m in rk,m is used to enumerate the downsampling layers of the sub-network k in the order they process an input, then m may begin with 1 and may take values up to M. Also other ways of enumerating the downsampling layers and their respective downsampling ratios rk,m may be used. M may take values beginning with 0 or −1. Generally, a downsampling layer m of the sub-network k may have an associated downsampling ratio rk,m so as to provide information to which sub-network k and which downsampling layer k within the sub-network k this downsampling ratio belongs. It is noted that the index k may only be provided in order to enumerate the sub-networks. It may be of integer value beginning with 0. It may also comprise integer values larger than or equal to −1 or may start at any reasonable starting point, for example also k=−10. Regarding the value of the index k and also the value of the index m, the invention is not limited though natural numbers larger than or equal to 0 or larger than or equal to −1 are preferred.
It is noted that the product mentioned above for obtaining the combined downsampling ratio may be explicitly calculated or may, for example, be obtained by using the downsampling ratios of the downsampling layers and a look-up table where the look-up table might, for example, comprise entries that represent a combined downsampling ratio and the respective combined downsampling ratio of a sub-network may be obtained by using the downsampling ratios of the of the sub-network as indices to the table. Likewise, the index k may act as an index to the lookup table. Alternatively, the combined downsampling ratio may be a preset or pre-calculated value that is stored for and/or associated with each sub-network.
In one embodiment, the NN comprises a number of K∈ sub-networks k, k≤K, k∈
, that each comprise at least two downsampling layers, wherein the method further comprises:
The index k may begin with 0 and may thus be larger than or equal to 0. Also other starting values may be chosen, for example k being larger than or equal to −1 or k may begin with 1, i.e. k being larger than or equal to 1. Regarding the selection of the index k, the invention is not limited and any way of differentiating between the respective sub-networks is encompassed by the present disclosure.
With this embodiment, the rescaling before each sub-network is only applied as necessary and only as necessary for the respective sub-network, which may result in a further reduction of the size of the bitstream.
In a further embodiment, at least two of the sub-networks each provide a sub-bitstream as output. A sub-bitstream may be regarded as a complete bitstream on its own. Nevertheless, the output of the neural network, which is referred to as a “bitstream” as well, may be made up from or may comprise at least some of the sub-bitstreams that are obtained by the respective sub-networks. In line with this embodiment, at least two of all sub-networks provide a respective sub-bitstream as output. This may have advantages in combination with the rescaling being applied on a per-sub-network basis.
In one embodiment, before applying the rescaling to the input with the size Sk, a determination is made whether Sk is an integer multiple of the combined downsampling ratio Rk of the sub-network k and, if it is determined that Sk is not an integer multiple of the combined downsampling ratio Rk of the sub-network k, the rescaling is applied to the input so that the size Sk is changed in the at least one dimension so that .
This means that
In one embodiment it is provided that, if the size Sk of the input is an integer multiple of the combined downsampling ratio Rk of the sub-network k, no rescaling to a size
In one embodiment, the determination whether Sk is an integer multiple of the combined downsampling ratio Rk comprises comparing the size Sk to an allowed input size of the sub-network k.
The allowed input size may, for example, be obtained from a look-up table or may be calculated by obtaining a series of potential integer multiples of the combined downsampling ratio.
In a more specific embodiment, the allowed input size of the sub-network k is calculated based on at least one of the combined downsampling ratio Rk and the size Sk. This calculation allows for obtaining the appropriate allowed input size for the respective sub-network specifically depending on the actual size of the input that is to be processed by the sub-network, making it applicable also to varying input sizes.
In a further embodiment, the comparing comprises calculating a difference between Sk and the allowed input size of the sub-network k.
Calculating the difference may be done by subtracting the size Sk of the input to the sub-network k from the allowed input size to this sub-network. In this context, the allowed input size may be considered to be identical to
In one embodiment, the allowed input size is determined according to
By using these operations, it is possible to determine those sizes that are closest to the actual size Sk of the input to the sub-network k depending on the combined downsampling ratio Rk. Specifically, it can thus be determined what the closest larger integer multiple of the combined downsampling ratio is (using the ceil function) and what the closest smaller integer multiple of the combined downsampling ratio is (using the floor function). Thereby, if at all rescaling is to be applied, it is done in a way that requires the least amount of modifications to the original size of the input, resulting in as little as possible additional information being added to the input or being removed from the input.
In one embodiment,
determined and, if
the rescaling is applied to the input with the size Sk. In an alternative or additional embodiment,
is determined and, if
the rescaling is applied to the input with the size Sk.
If these values were (both) equal to 0, this would mean that the input size Sk to the sub-network k is already an integer multiple of the respective combined downsampling ratio Rk of this sub-network, making a rescaling to a different size
In a further embodiment, the size
More specifically, the size
In specific cases, the determination of the size
the size S is determined using
With these embodiments, the size
In a further embodiment, the rescaling applied to an input of a sub-network k is independent from the combined downsampling ratios Rl, l≠k of other sub-networks of the NN and/or the rescaling applied to an input of a sub-network k is independent from downsampling ratios rl,m, l≠k of downsampling layers of other sub-networks of the NN. By considering each sub-network k in isolation from the other sub-networks or their downsampling layers and applying a corresponding rescaling only depending on the values of the sub-network k itself to an input Sk to this sub-network, the advantage in the reduction in size of bitstreams may be increased further.
It can further be provided that the input to a sub-network k has a size Sk in the at least one dimension that has a value that is between a closest smaller integer multiple of the combined downsampling ratio Rk of the sub-network k and a closest larger integer multiple of the combined downsampling ratio Rk of the sub-network k and wherein, depending on a condition, the size Sk of the input is changed during the rescaling to either match the closest smaller integer multiple of the combined downsampling ratio Rk or to match the closest larger integer multiple of the combined downsampling ratio Rk. The condition can, for example, depend on characteristics of the sub-network or an intention to, for example, only add information to an original input size when applying the rescaling (i.e. always increasing the size of the input if a rescaling is necessary) or to make as few as possible changes to the input by either removing information (for example by cropping) or adding information (for example by padding).
With this, only modifications to the input with the size Sk that are helpful in ensuring that the rescaling results in a rescaled input that can be processed by the sub-network are applied.
In one embodiment, the input to a sub-network k has a size Sk in the at least one dimension that has a value that is not an integer multiple of the combined downsampling ratio Rk of the sub-network k, wherein the size Sk of the input is changed during the rescaling to either match the closest smaller integer multiple of the combined downsampling ratio Rk or to match the closest larger integer multiple of the combined downsampling ratio Rk.
In a further embodiment, the input to a sub-network k has a size Sk in the at least one dimension, wherein lRk≤Sk≤Rk(l+1), l∈ and Rk is the combined downsampling ratio of the sub-network k and wherein the size Sk is either rescaled to
This constitutes a reasonable formulation for obtaining the closest larger and closest smaller integer multiples of the combined downsampling ratio to Rk of the sub-network k for an input having a size Sk and makes a flexible adaption of the rescaling also to varying input sizes possible.
In a further embodiment it may be provided that, if the size Sk of the input is closer to the closest smaller integer multiple of the combined downsampling ratio Rk of the sub-network k than to the closest larger integer multiple of the combined downsampling ratio Rk, the size Sk of the input is reduced to a size
Specifically, it may be provided that reducing the size Sk of the input to the size
and removing the samples denoted with S up to
This holds if M is an integer multiple of 2 and in other cases may, for example, comprise removing
samples from the left border and removing
samples from the right border if M is no integer multiple of 2. Removing samples from both borders may be preferred in order to not change information of the original input in a biased way by removing samples from a single border while removing samples from a single border. However, cropping by removing samples from only one border may be computationally more efficient in some cases.
In one embodiment, if the size Sk of the input to the sub-network k is closer to the closest larger integer multiple of the combined downsampling ratio Rk of the sub-network k than to the closest smaller integer multiple of the combined downsampling ratio Rk, the size Sk of the input is increased to a size
Specifically, it can be provided that increasing the size Sk of the input to the size
In a more specific embodiment, the padding information obtained from the input with the size Sk is applied as redundant padding information to increase the size Sk of the input to the size
It can also be provided that the padding information is or comprises at least one value of the input with the size Sk that is closest to a region in the input to which the redundant padding information is to be added. Specifically, if for example a number of M samples is to be added to a border of an input, these M samples and their respective values may be taken from the M samples that are closest to this border of the input. Thereby, it may be avoided that unintentional relations to other portions of the input are artificially created.
In a further embodiment, the size Sk of the input to the sub-network k is increased to a size
In one embodiment, the condition referred to above makes use of Min(|Sk−lRk|, |Sk−Rk(l+1)|) and the condition may comprise that, if Min throws |Sk−lRk|, then the size Sk of the input is reduced to
In a more specific embodiment, l is determined using at least one of the size Sk of the input to the sub-network k and the combined downsampling ratio Rk of the sub-network k. Because the number l for calculating the closest smaller and closest larger integer multiple may not be preset in view of varying input sizes Sk, it may be obtained in some way. By using the combined downsampling ratio Rk and the input size Sk, l can be obtained in a way that depends on the actual input size, making it possible to obtain l in a flexible way.
Specifically, l may be determined by
and/or l+1 may be determined by
This allows for a computationally efficient calculation l or l+1, respectively. l and l+1 can be calculated in two steps using both, floor and ceil. Alternatively, it is also possible to only calculate l using floor and then obtaining l+1 from this calculation. Alternatively, it can also be envisaged to calculate l+1 using the ceil function and then obtaining l from this value.
It may further be provided that at least one of the downsampling layers of at least one sub-network applies a downsampling to the input in the two dimensions and the downsampling ratio in the first dimension is equal to the downsampling ratio in the second dimension.
It can still further be provided that the downsampling ratios of all downsampling layers of a sub-network are equal. Specifically, the downsampling ratios could all be equal to 2.
In one embodiment, all sub-networks comprise the same number of downsampling layers. The number of downsampling layers of a sub-network k may be denoted with Mk, where Mk is a natural number. Mk may then have the same value M for all sub-networks k.
It can further be provided that the downsampling ratios of all downsampling layers of all sub-networks are equal.
The downsampling ratios of the respective downsampling layers m may be denoted with rm, where m corresponds to the actual number of the downsampling layer specifically in the direction of the processing of the input through the sub-network. In this context, it may also be envisaged to denote the downsampling ratios with rk,m where k is a natural number and m is a natural number and k indicates the sub-network k to which the downsampling layer m with the downsampling ratio rk,m belongs.
It can further be provided that at least two sub-networks of the NN have different numbers of downsampling layers.
At least one downsampling ratio rk,m of at least one downsampling layer m of a sub-network k may further be different from at least one downsampling ratio rl,n of at least one downsampling layer n of a sub-network l. Specifically, the sub-networks k and l are different sub-networks. The downsampling layer m and the downsampling layer n may further be at different positions within the sub-networks k and l when seen in processing order of the input through the sub-networks.
It can be provided that, if it is determined that the size Sk of an input to a sub-network k is no integer multiple of the combined downsampling ratio Rk, the rescaling comprises applying an interpolation filter. In this context, interpolation may be used to increase the size by calculating, using for example two neighboring sample values of the input with the size Sk, an intermediate sample value and adding it in between the neighboring samples as a new sample, thereby adding a sample to the input and increasing the size Sk by l. This can be done as often as necessary in order to increase the input size Sk to the size
The interpolation can be mathematically more complex than in the above example and can comprise, for example, not only the immediate neighbors but may be obtained by considering the values of at least four adjacent samples. Interpolation may also be done in a multi-dimensional manner to obtain, for example, an intermediate sample value from four sample values in a two-dimensional matrix that comprise four samples in two neighboring columns and rows. Thereby, an efficient increase or decrease of the size Sk of the input may be obtained that makes use of the originally available information, resulting preferably in as few information losses as possible.
The present disclosure further provides a method for decoding a bitstream representing a picture using a neural network, NN, wherein the NN comprises at least two sub-networks, wherein at least one sub-network of the at least two sub-networks comprises at least two upsampling layers, wherein the at least one sub-network applies an upsampling to an input representing a matrix having a size T1 in at least one dimension, the method comprising:
In this context, the upsampling may be considered an inverse to the downsampling applied according to the preceding embodiments. Thereby, when having processed the bitstream with the neural network, a reconstructed or decoded picture can be obtained as output of the neural network. The combined upsampling ratio U1 of the first sub-network and the combined upsampling ratio U2 of the second sub-network may be obtained in different ways or may, for example, be pre-calculated or the like.
The upsampling applied by an upsampling layer of the neural network can be achieved in any known or technically reasonable way. Specifically, this may comprise an upsampling by applying a de-convolution to an input of the respective upsampling layer of the neural network. The upsampling can be performed in one dimension only or it can also be performed on two dimensions of the input when represented in the form of a matrix. This pertains to both, the upsampling applied by a sub-network in total and the upsampling applied by each upsampling layer of a respective sub-network. For example, while a sub-network might apply an upsampling to an input in two dimensions, a first upsampling layer of this sub-network might only apply an upsampling in one dimension whereas another upsampling layer of the sub-network applies an upsampling to the input in another dimension or in two dimensions.
In general, the disclosure presented herein is not limited to particular ways of upsampling. One or more of the layers of the neural network discussed below may apply an upsampling in a way that is different from de-convolutions for example by adding intermediate rows or columns, like between every two or four rows and/or columns of the input (when seen in the representation of the two-dimensional matrix).
Embodiments presented herein are to be understood as referring to a rescaling that is applied immediately after the processing of an input by a sub-network comprising upsampling layers but not within the sub-network. This means, as regards the rescaling, the sub-network is, although comprising a plurality of layers and potentially a large number of upsampling layers, considered as one entity that applies an upsampling with a combined upsampling ratio to an input and the rescaling of the output of the sub-network is applied so that a size T of the rescaled output matches a size T that may, for example, be a target input size for the subsequent sub-network.
It is noted that the combined upsampling ratio may be determined according to all upsampling ratios of the upsampling layers of the at least one sub-network in isolation without regard to other upsampling layers of other sub-networks. More specifically, the combined upsampling ratio Uk of a sub-network k, k being a natural number and denoting the position of the sub-network in the processing order of the input, may be obtained by calculating the product of the upsampling ratios u of all upsampling layers of the sub-network k. This may be represented as Uk=Πm uk,m, uk,m, k, m∈, uk,m>1 for the sub-network k. Here, the term uk,m indicates a upsampling ratio of an upsampling layer m of the sub-network k. The sub-network may comprise a total number of M (M being a natural number larger than 0) upsampling layers. When the index m in uk,m is used to enumerate the upsampling layers of the sub-network k in the order they process an input, then m may begin with 1 and may take values up to M. Also other ways of enumerating the upsampling layers and their respective upsampling ratios uk,m may be used. M may take values beginning with 0 or −1. Generally, an upsampling layer m of the sub-network k may have an associated upsampling ratio uk,m so as to provide information to which sub-network k and which upsampling layer m within the sub-network k this upsampling ratio belongs. It is noted that the index k may only be provided in order to enumerate the sub-networks. It may be of integer value beginning with 0. It may also comprise integer values larger than or equal to −1 or may start at any reasonable starting point, for example also k=−10. Regarding the value of the index k and also the value of the index m, the invention is not limited though natural numbers larger than or equal to 0 or larger than or equal to −1 are preferred.
It is noted that the product mentioned above for obtaining the combined upsampling ratio may be explicitly calculated or may, for example, be obtained by using the upsampling ratios of the upsampling layers and a look-up table where the look-up table might, for example, comprise entries that represent a combined upsampling ratio and the respective combined upsampling ratio of a sub-network may be obtained by using the upsampling ratios of the of the sub-network as indices to the table. Likewise, the index k may act as an index to the lookup table. Alternatively, the combined upsampling ratio may be a preset or pre-calculated value that is stored for and/or associated with each sub-network.
With this method, it is possible to obtain a decoded picture even from a bitstream encoding a picture that has reduced size and was, for example, obtained using one or more of the above embodiments.
In one embodiment the method further comprises receiving, by at least two sub-networks, a sub-bitstream. The sub-bitstreams received by each of these at least two sub-networks may be different. As was already indicated above, during encoding, a first sub-bitstream may be obtained by processing the originally input picture by only a subset of the available sub-networks and providing an output after this partial processing of the input picture. The second sub-bitstream or the second sub-bitstream is then, for example, obtained after having processed the input picture by the whole neural network and thus by all downsampling layers. For decoding the picture, the process can be in inverse order so that the sub-bitstream that was processed only by a subset of the sub-networks of the encoder is likewise only processed by the last few sub-networks before obtaining the decoded picture.
In a further embodiment, at least one upsampling layer of at least one sub-network comprises a transposed convolution or convolution layer. The transposed convolution may be implemented as the inverse of the convolution that has, for example, been applied in a corresponding encoder encoding the picture.
In a further embodiment, the information comprises at least one of a target size of the decoded picture comprising at least one of a height H of the decoded picture and a width W of the decoded picture, the combined upsampling ratio U1, the combined upsampling ratio U2, at least one upsampling ratio u1m of an upsampling layer of the first sub-network, at least one upsampling ratio u2m of an upsampling layer of the second sub-network, a target output size of the second sub-network, the size
. Using one or more of these pieces of information can result in a reliable reconstruction of the image.
It can be provided that the information is obtained from at least one of: the bitstream, a second bitstream, information available at a decoder. While some of the information can advantageously be included in the bitstream like, for example, the height and width of the original picture, some other information, like for example the upsampling ratios, can be already available at the decoder that performs the decoding method according to one of the above embodiments. This is because this information may usually not be known to the encoder but may be available at the decoder, making it more efficient to obtain this information from the decoder itself and not having to include it, as further information, in the bitstream provided to the decoder. An additional bitstream can also be used in order to, for example, separate the additional information on the one side from the information pertaining or constituting the encoded picture on the other side, making it computationally easier to distinguish between such information. The other benefit of including an additional bitstream may be to speed up the processing by means of parallel processing. If for example a sub-network only requires one part of the bitstream, and the second sub-network only requires the second part of the bitstream (each part being disjoint), it is advantageous to divide the single bitstream into two bitstreams. This way it is possible to start the processing of the first sub-network independently from the second subnetwork, increasing the parallel processing capability. In one embodiment, the rescaling for changing the size is determined based on a formula depending on
and U2, wherein
is a target output size of the output of the second sub-network and U2 is the combined upsampling ratio of the second sub-network. The target output size
can be preset or can be calculated inversely for example based on an intended size of the decoded picture. With this embodiment, the size
can be determined based on information pertaining to the sub-networks and/or the output to be obtained.
It may further be provided the rescaling for changing the size is determined based on a formula depending on U2 and N, where N is the total number of sub-networks proceeding the first sub-network in processing order of the bitstream through the NN.
With this, the size is depending on the number of proceeding sub-networks and further depends on the actual output size to be obtained for the decoded picture.
Specifically, the formula may be given by
wherein Toutput is the target size of the output of the NN. It can also be provided that the formula is given by
wherein Toutput is the target size of the output of the NN and U is a combined upsampling ratio. In this case, an indication for indicating the size Toutput may be included in the bitstream. By including the size Toutput in the bitstream, also varying output sizes to be obtained by the decoding can be signaled to the decoding.
In a further embodiment, the formula is given by
or wherein the formula is given by
An indication may be provided in and obtained from the bitstream that indicates which of the multiple predefined formulas are selected. With this, it is possible to signal to the decoding what processing has been applied during encoding, thereby making a reconstruction of the picture possible in a reliable way.
The method may further comprise, before the rescaling the output with the size . Thereby, unnecessary rescaling or calculations potentially involved in determining which rescaling is to be applied can be avoided.
In this regard it can be provided that, if it is determined that the size , no rescaling changing the size
. This “identical” rescaling does not applying a change to the size
In one embodiment, the method further comprises determining whether T2 is larger than or whether
. By determining whether
, further consequences on the rescaling to be applied can be made. Specifically, if it is determined that
, the rescaling may comprise applying a cropping to the output with the size
. Thereby, a computationally efficient reduction of the size of
is provided. Alternatively, if it is determined that
, the rescaling may comprise applying a padding to the output with the size
.
Specifically the cropping operation corresponds to discarding samples from the edges of the output such that the size . In the decoder, usually the cropping operation is applied at the end of a sub-network. The reason is that in an encoder it is usually preferable to apply padding for resizing before a sub-network, since padding ensures that no information is lost and that the information is not altered (just the size of the input comprising the information is increased). Since, in a decoder, the operations applied by an encoder are reverted, the cropping is applied after a sub-network.
Specifically, the padding may comprise padding the output with the size
In a further embodiment, the padding information obtained from the output with the size . Applying redundant padding does not add additional information but adds already present information to the output which may result in fewer distortions in the reconstructed image.
More specifically, the padding may comprise reflection padding or repetition padding.
It can further be provided that the padding information is or comprises at least one value of the output with the size
In a further embodiment, if it is determined that , the rescaling comprises applying an interpolation filter. Applying interpolation can be useful to increase the quality of the reconstructed picture.
In a further embodiment, the information is provided in the bitstream or a further bitstream and comprises a combined downsampling ratio Rk of at least one sub-network k that comprises at least one downsampling layer m of an encoder that encoded the bitstream, wherein the sub-network k corresponds, in the order of processing the input, to the sub-network of the decoder. Thereby, it can be ensured that the processing performed during the decoding is reversed by the encoding.
It may further be provided that at least one upsampling layer of at least one sub-network of the NN applies an upsampling in the two dimensions and the upsampling ratio in the first dimension is equal to the upsampling ratio in the second dimension.
Furthermore, the upsampling ratios of all upsampling layers of a sub-network may be equal. This can be implemented computationally efficient.
In one embodiment, all sub-networks comprise the same number of upsampling layers. The number of upsampling layers greater than or equal to 2.
It can also be provided that the upsampling ratios of all upsampling layers of all sub-networks are equal.
Alternatively, at least two sub-networks of the NN may have different numbers of upsampling layers. Optionally, at least one upsampling ratio uk,m of at least one upsampling layer m of a sub-network k may different from at least one upsampling ratio ul,n of at least one upsampling layer n of a sub-network l. The indices k, l, m, n may be integer values greater than 0 and may indicate the position of the sub-network or the upsampling layer in the processing order of the input to the NN, respectively.
It can be provided that the sub-networks k and l are different sub-networks. Furthermore, the upsampling layer m and the upsampling layer n may be at different positions within the sub-networks k and l when seen in processing order of the input through the sub-networks.
It can further be provided that the combined upsampling ratios of at least two different sub-networks are equal or the combined upsampling ratios of all sub-networks may be are pairwise disjoint.
In view of the above embodiments, it may be provided that the NN comprises, in the processing order of the bitstream through the NN, a further unit that applies a transformation to the input that does not change the size of the input in the at least one dimension, wherein the method comprises applying the rescaling after the processing of the input by the further unit and before processing the input by the following sub-network of the NN, if the rescaling results in an increase of the size of the input in the at least one dimension, and/or wherein the method comprises applying the rescaling before the processing of the input by the further unit, if the rescaling comprises a decrease of the size of the input in the at least one dimension. By applying the rescaling before or after the respective further unit, the rescaling can be implemented in a computationally efficient way avoiding, for example, a rescaling to an input that would, by the further unit, be changed anyway, potentially making, for example, an interpolation less reliable.
Specifically, the further unit may be or may comprise a batch normalizer and/or a rectified linear unit, ReLU. Such units are part of some neural networks nowadays and can increase the quality of the processing of an input through the neural network.
The bitstream may comprise sub-bitstreams corresponding to distinct color channels of the picture and wherein the NN comprises sub-neural networks, sNN, that are each adapted to apply a method according to any of the above embodiments to the sub-bitstream provided as input to the sNN. The applications of such sub-neural networks can be provided in a way that each of the sub-neural networks performs a rescaling and processing of an input in line with any of the above embodiments without the sub-networks influencing each other. They may thus be independent from each other and process their respective input independent from each other which can also comprise that a different rescaling is applied by one of the sub-neural networks compared to the rescaling that is applied during the processing of an input to another sub-neural network. Furthermore, the sub-neural networks are not necessarily identical with respect to their construction regarding the sub-networks or the respective layers or the structure of the layers within the sub-networks.
Regarding the encoding, it can be provided that, if the rescaling comprises increasing the size Sm to the size
and, if the rescaling comprises reducing the size Sm to the size
Regarding the decoding, it may further be provided that, if the rescaling comprises increasing the size , the size
is given by
and, if the rescaling comprises reducing the size , the size
is given by
The present disclosure further provides an encoder for encoding a picture, wherein the encoder comprises a receiver for receiving a picture and one or more processors configured to implement a neural network, NN, the NN comprising, in a processing order of a picture through the NN, at least two sub-networks, wherein each sub-network comprises at least two layers, wherein the at least two layers of at least one sub-network of the at least two sub-networks comprise at least one downsampling layer that is adapted to apply a downsampling to an input, and a transmitter for outputting a bitstream, wherein the encoder is adapted to perform a method according to any of the above embodiments.
Further, an encoder for encoding a bitstream is provided, wherein the encoder comprises one or more processors for implementing a neural network, NN, wherein the one or more processors are adapted to perform a method according to any of the above embodiments.
Furthermore, an encoder for encoding a picture is provided, the encoder comprising one or more processors configured to implement a neural network, NN, the NN comprising, in a processing order of a picture through the NN, at least two sub-networks, wherein at least one sub-network of the at least two sub-networks comprises at least two downsampling layers and wherein the at least one sub-network is adapted to apply a downsampling to an input representing a matrix having a size S1 in at least one dimension, wherein the encoder and/or the one or more processors are adapted to encode a picture by:
Further embodiments of the encoder are configured to implement the features of the encoding methods explained above.
These embodiments allow for implementing the advantages of the encoding method explained in the above embodiments in encoders.
Moreover, a decoder for decoding a bitstream representing a picture is provided, wherein the decoder comprises a receiver for receiving a bitstream and one or more processors configured to implement a neural network, NN, the NN comprising, in a processing order of a bitstream through the NN, at least two sub-networks, wherein each sub-network comprises at least two layers, wherein the at least two layers of each of the at least two sub-network comprise at least one upsampling layer, wherein each upsampling layer is adapted to apply upsampling to an input, and a transmitter for outputting a decoded picture, wherein the decoder is adapted to perform any of the methods of the above embodiments.
A decoder for decoding a bitstream representing a picture is also provided in the present disclosure, wherein the decoder comprises one or more processors for implementing a neural network, NN, wherein the one or more processors are adapted to perform a method according to any of the above embodiments.
Furthermore, a decoder for decoding a bitstream representing a picture is provided, wherein the decoder comprises a receiver for receiving a bitstream and one or more processors configured to implement a neural network, NN, the NN comprising, in a processing order of a bitstream through the NN, at least two sub-networks, wherein at least one sub-network of the at least two sub-networks comprises at least two upsampling layers, wherein the at least one sub-network is adapted to apply an upsampling to an input representing a matrix having a size T1 in at least one dimension, wherein the encoder and/or the one or more processors are configured to decode a bitstream by:
Further embodiments of the decoder are configured to implement the features of the decoding methods explained above.
These embodiments advantageously implement the above embodiments for decoding a bitstream in a decoder.
Furthermore, a computer-readable (non-transitory) storage medium is provided that comprises computer executable instructions that, when executed on a computing system, cause the computing system to execute a method according to any of the above embodiments.
In the following, some embodiments are described with reference to the FIGS. The
In the following description, reference is made to the accompanying FIGS., which form part of the disclosure, and which show, by way of illustration, specific aspects of the present disclosure or specific aspects in which embodiments of the present disclosure may be used. It is understood that the embodiments may be used in other aspects and comprise structural or logical changes not depicted in the FIGS. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.
For instance, it is understood that a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa. For example, if one or a plurality of specific method steps are described, a corresponding device may include one or a plurality of units, e.g. functional units, to perform the described one or plurality of method steps (e.g. one unit performing the one or plurality of steps, or a plurality of units each performing one or more of the plurality of steps), even if such one or more units are not explicitly described or illustrated in the FIGS. On the other hand, for example, if a specific apparatus is described based on one or a plurality of units, e.g. functional units, a corresponding method may include one step to perform the functionality of the one or plurality of units (e.g. one step performing the functionality of the one or plurality of units, or a plurality of steps each performing the functionality of one or more of the plurality of units), even if such one or plurality of steps are not explicitly described or illustrated in the FIGS. Further, it is understood that the features of the various exemplary embodiments and/or aspects described herein may be combined with each other, unless specifically noted otherwise.
Video coding typically refers to the processing of a sequence of pictures, which form the video or video sequence. Instead of the term “picture” the term “frame” or “image” may be used as synonyms in the field of video coding. Video coding (or coding in general) comprises two parts video encoding and video decoding. Video encoding is performed at the source side, typically comprising processing (e.g. by compression) the original video pictures to reduce the amount of data required for representing the video pictures (for more efficient storage and/or transmission). Video decoding is performed at the destination side and typically comprises the inverse processing compared to the encoder to reconstruct the video pictures. Embodiments referring to “coding” of video pictures (or pictures in general) shall be understood to relate to “encoding” or “decoding” of video pictures or respective video sequences. The combination of the encoding part and the decoding part is also referred to as CODEC (Coding and Decoding).
In case of lossless video coding, the original video pictures can be reconstructed, i.e. the reconstructed video pictures have the same quality as the original video pictures (assuming no transmission loss or other data loss during storage or transmission). In case of lossy video coding, further compression, e.g. by quantization, is performed, to reduce the amount of data representing the video pictures, which cannot be completely reconstructed at the decoder, i.e. the quality of the reconstructed video pictures is lower or worse compared to the quality of the original video pictures.
Several video coding standards belong to the group of “lossy hybrid video codecs” (i.e. combine spatial and temporal prediction in the sample domain and 2D transform coding for applying quantization in the transform domain). Each picture of a video sequence is typically partitioned into a set of non-overlapping blocks and the coding is typically performed on a block level. In other words, at the encoder the video is typically processed, i.e. encoded, on a block (video block) level, e.g. by using spatial (intra picture) prediction and/or temporal (inter picture) prediction to generate a prediction block, subtracting the prediction block from the current block (block currently processed/to be processed) to obtain a residual block, transforming the residual block and quantizing the residual block in the transform domain to reduce the amount of data to be transmitted (compression), whereas at the decoder the inverse processing compared to the encoder is applied to the encoded or compressed block to reconstruct the current block for representation. Furthermore, the encoder duplicates the decoder processing loop such that both will generate identical predictions (e.g. intra- and inter predictions) and/or re-constructions for processing, i.e. coding, the subsequent blocks. Recently, some parts or the entire encoding and decoding chain has been implemented by using a neural network or, in general, any machine learning or deep learning framework.
In the following embodiments of a video coding system 10, a video encoder 20 and a video decoder 30 are described based on
As shown in
The source device 12 comprises an encoder 20, and may additionally, i.e. optionally, comprise a picture source 16, a pre-processor (or pre-processing unit) 18, e.g. a picture pre-processor 18, and a communication interface or communication unit 22. Some embodiments of the present disclosure (e.g. relating to an initial rescaling or rescaling between two proceeding layers) may be implemented by the encoder 20. Some embodiments (e.g. relating to an initial rescaling) may be implemented by the picture pre-processor 18.
The picture source 16 may comprise or be any kind of picture capturing device, for example a camera for capturing a real-world picture, and/or any kind of a picture generating device, for example a computer-graphics processor for generating a computer animated picture, or any kind of other device for obtaining and/or providing a real-world picture, a computer generated picture (e.g. a screen content, a virtual reality (VR) picture) and/or any combination thereof (e.g. an augmented reality (AR) picture). The picture source may be any kind of memory or storage storing any of the aforementioned pictures.
In distinction to the pre-processor 18 and the processing performed by the pre-processing unit 18, the picture or picture data 17 may also be referred to as raw picture or raw picture data 17.
Pre-processor 18 is configured to receive the (raw) picture data 17 and to perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. Pre-processing performed by the pre-processor 18 may, e.g., comprise trimming, color format conversion (e.g. from RGB to YCbCr), color correction, or de-noising. It can be understood that the pre-processing unit 18 may be optional component.
The video encoder 20 is configured to receive the pre-processed picture data 19 and provide encoded picture data 21.
Communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and to transmit the encoded picture data 21 (or any further processed version thereof) over communication channel 13 to another device, e.g. the destination device 14 or any other device, for storage or direct reconstruction.
The destination device 14 comprises a decoder 30 (e.g. a video decoder 30), and may additionally, i.e. optionally, comprise a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32) and a display device 34.
The communication interface 28 of the destination device 14 is configured receive the encoded picture data 21 (or any further processed version thereof), e.g. directly from the source device 12 or from any other source, e.g. a storage device, e.g. an encoded picture data storage device, and provide the encoded picture data 21 to the decoder 30.
The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoded picture data 21 or encoded data 13 via a direct communication link between the source device 12 and the destination device 14, e.g. a direct wired or wireless connection, or via any kind of network, e.g. a wired or wireless network or any combination thereof, or any kind of private and public network, or any kind of combination thereof.
The communication interface 22 may be, e.g., configured to package the encoded picture data 21 into an appropriate format, e.g. packets, and/or process the encoded picture data using any kind of transmission encoding or processing for transmission over a communication link or communication network.
The communication interface 28, forming the counterpart of the communication interface 22, may be, e.g., configured to receive the transmitted data and process the transmission data using any kind of corresponding transmission decoding or processing and/or de-packaging to obtain the encoded picture data 21.
Both, communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces as indicated by the arrow for the communication channel 13 in
The decoder 30 is configured to receive the encoded picture data 21 and provide decoded picture data 31 or a decoded picture 31 (further details will be described below, e.g., based on
The post-processor 32 of destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), e.g. the decoded picture 31, to obtain post-processed picture data 33, e.g. a post-processed picture 33. The post-processing performed by the post-processing unit 32 may comprise, e.g. color format conversion (e.g. from YCbCr to RGB), color correction, trimming, or re-sampling, or any other processing, e.g. for preparing the decoded picture data 31 for display, e.g. by display device 34.
Some embodiments of the disclosure may be implemented by the decoder 30 or by the post-processor 32.
The display device 34 of the destination device 14 is configured to receive the post-processed picture data 33 for displaying the picture, e.g. to a user or viewer. The display device 34 may be or comprise any kind of display for representing the reconstructed picture, e.g. an integrated or external display or monitor. The displays may, e.g. comprise liquid crystal displays (LCD), organic light emitting diodes (OLED) displays, plasma displays, projectors, micro LED displays, liquid crystal on silicon (LCoS), digital light processor (DLP) or any kind of other display.
Although
As will be apparent for the skilled person based on the description, the existence and (exact) split of functionalities of the different units or functionalities within the source device 12 and/or destination device 14 as shown in
The encoder 20 (e.g. a video encoder 20) or the decoder 30 (e.g. a video decoder 30) or both encoder 20 and decoder 30 may be implemented via processing circuitry as shown in
Source device 12 and destination device 14 may comprise any of a wide range of devices, including any kind of handheld or stationary devices, e.g. notebook or laptop computers, mobile phones, smart phones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video gaming consoles, video streaming devices (such as content services servers or content delivery servers), broadcast receiver device, broadcast transmitter device, or the like and may use no or any kind of operating system. In some cases, the source device 12 and the destination device 14 may be equipped for wireless communication. Thus, the source device 12 and the destination device 14 may be wireless communication devices.
In some cases, video coding system 10 illustrated in
For convenience of description, some embodiments are described herein, for example, by reference to High-Efficiency Video Coding (HEVC) or to the reference software of Versatile Video coding (VVC), the next generation video coding standard developed by the Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Motion Picture Experts Group (MPEG). One of ordinary skill in the art will understand that embodiments of the invention are not limited to HEVC or VVC.
The video coding device 400 comprises ingress ports 410 (or input ports 410) and receiver units (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 to process the data; transmitter units (Tx) 440 and egress ports 450 (or output ports 450) for transmitting the data; and a memory 460 for storing the data. The video coding device 400 may also comprise optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress ports 410, the receiver units 420, the transmitter units 440, and the egress ports 450 for egress or ingress of optical or electrical signals.
The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGAs, ASICs, and DSPs. The processor 430 is in communication with the ingress ports 410, receiver units 420, transmitter units 440, egress ports 450, and memory 460. The processor 430 comprises a coding module 470. The coding module 470 implements the disclosed embodiments described above. For instance, the coding module 470 implements, processes, prepares, or provides the various coding operations. The inclusion of the coding module 470 therefore provides a substantial improvement to the functionality of the video coding device 400 and effects a transformation of the video coding device 400 to a different state. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.
The memory 460 may comprise one or more disks, tape drives, and solid-state drives and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory 460 may be, for example, volatile and/or non-volatile and may be a read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and/or static random-access memory (SRAM).
A processor 502 in the apparatus 500 can be a central processing unit. Alternatively, the processor 502 can be any other type of device, or multiple devices, capable of manipulating or processing information now-existing or hereafter developed. Although the disclosed implementations can be practiced with a single processor as shown, e.g., the processor 502, advantages in speed and efficiency can be achieved using more than one processor.
A memory 504 in the apparatus 500 can be a read only memory (ROM) device or a random access memory (RAM) device in an implementation. Any other suitable type of storage device can be used as the memory 504. The memory 504 can include code and data 506 that is accessed by the processor 502 using a bus 512. The memory 504 can further include an operating system 508 and application programs 510, the application programs 510 including at least one program that permits the processor 502 to perform the methods described here. For example, the application programs 510 can include applications 1 through N, which further include a video coding application that performs the methods described here.
The apparatus 500 can also include one or more output devices, such as a display 518. The display 518 may be, in one example, a touch sensitive display that combines a display with a touch sensitive element that is operable to sense touch inputs. The display 518 can be coupled to the processor 502 via the bus 512.
Although depicted here as a single bus, the bus 512 of the apparatus 500 can be composed of multiple buses. Further, the secondary storage 514 can be directly coupled to the other components of the apparatus 500 or can be accessed via a network and can comprise a single integrated unit such as a memory card or multiple units such as multiple memory cards. The apparatus 500 can thus be implemented in a wide variety of configurations.
In the following, more specific, non-limiting, and exemplary embodiments of the invention are described. Before that, some explanations will be provided aiding in the understanding of the disclosure:
Artificial neural networks (ANN) or connectionist systems are computing systems vaguely inspired by the biological neural networks that constitute animal brains. An ANN is based on a collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain. Each connection, like the synapses in a biological brain, can transmit a signal to other neurons. An artificial neuron that receives a signal then processes it and can signal neurons connected to it. In ANN implementations, the “signal” at a connection is a real number, and the output of each neuron can be computed by some non-linear function of the sum of its inputs. The connections are called edges. Neurons and edges typically have a weight that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Neurons may have a threshold such that a signal is sent only if the aggregate signal crosses that threshold. Typically, neurons are aggregated into layers. Different layers may perform different transformations on their inputs. Signals travel from the first layer (the input layer), to the last layer (the output layer), possibly after traversing the layers multiple times.
The original goal of the ANN approach was to solve problems in the same way that a human brain would. Over time, attention moved to performing specific tasks, leading to deviations from biology. ANNs have been used on a variety of tasks, including computer vision.
The name “convolutional neural network” (CNN) indicates that the network employs a mathematical operation called convolution. Convolution is a specialized kind of linear operation. Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers. A convolutional neural network consists of an input and an output layer, as well as multiple hidden layers. Input layer is the layer to which the input is provided for processing. For example, the neural network of
When programming a CNN for processing pictures or images, the input is a tensor with shape (number of images)×(image width)×(image height)×(image depth). Then, after passing through a convolutional layer, the image becomes abstracted to a feature map, with shape (number of images)×(feature map width)×(feature map height)×(feature map channels). A convolutional layer within a neural network should have the following attributes. Convolutional kernels defined by a width and height (hyper-parameters). The number of input channels and output channels (hyper-parameter). The depth of the convolution filter (the input channels) should be equal to the number channels (depth) of the input feature map.
In the past, traditional multilayer perceptron (MLP) models have been used for image recognition. However, due to the full connectivity between nodes, they suffered from high dimensionality, and did not scale well with higher resolution images. A 1000×1000-pixel image with RGB color channels has 3 million weights, which is too high to feasibly process efficiently at scale with full connectivity. Also, such network architecture does not take into account the spatial structure of data, treating input pixels which are far apart in the same way as pixels that are close together. This ignores locality of reference in image data, both computationally and semantically. Thus, full connectivity of neurons is wasteful for purposes such as image recognition that are dominated by spatially local input patterns. CNN models mitigate the challenges posed by the MLP architecture by exploiting the strong spatially local correlation present in natural images. The convolutional layer is the core building block of a CNN. The layer's parameters consist of a set of learnable filters (the above-mentioned kernels), which have a small receptive field, but extend through the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the entries of the filter and the input and producing a 2-dimensional activation map of that filter. As a result, the network learns filters that activate when it detects some specific type of feature at some spatial position in the input.
Stacking the activation maps for all filters along the depth dimension forms the full output volume of the convolution layer. Every entry in the output volume can thus also be interpreted as an output of a neuron that looks at a small region in the input and shares parameters with neurons in the same activation map. A feature map, or activation map, is the output activations for a given filter. Feature map and activation has same meaning. In some papers it is called an activation map because it is a mapping that corresponds to the activation of different parts of the image, and also a feature map because it is also a mapping of where a certain kind of feature is found in the image. A high activation means that a certain feature was found.
Another important concept of CNNs is pooling, which is a form of non-linear downsampling. There are several non-linear functions to implement pooling among which max pooling is the most common. It partitions the input image into a set of non-overlapping rectangles and, for each such sub-region, outputs the maximum. Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. The pooling layer serves to progressively reduce the spatial size of the representation, to reduce the number of parameters, memory footprint and amount of computation in the network, and hence to also control overfitting. It is common to periodically insert a pooling layer between successive convolutional layers in a CNN architecture. The pooling operation provides another form of translation invariance.
The above-mentioned ReLU is the abbreviation of rectified linear unit, which applies the non-saturating activation function. It effectively removes negative values from an activation map by setting them to zero. It increases the nonlinear properties of the decision function and of the overall network without affecting the receptive fields of the convolution layer. Other functions are also used to increase nonlinearity, for example the saturating hyperbolic tangent and the sigmoid function. ReLU is often preferred to other functions because it trains the neural network several times faster without a significant penalty to generalization accuracy.
After several convolutional and max pooling layers, the high-level reasoning in the neural network is done via fully connected layers. Neurons in a fully connected layer have connections to all activations in the previous layer, as seen in regular (non-convolutional) artificial neural networks. Their activations can thus be computed as an affine transformation, with matrix multiplication followed by a bias offset (vector addition of a learned or fixed bias term).
An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner. The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for dimensionality reduction, by training the network to ignore signal “noise”. Along with the reduction side, a reconstructing side is learnt, where the autoencoder tries to generate from the reduced encoding a representation as close as possible to its original input, hence its name.
Picture size: refers to the width or height or the width-height pair of a picture. Width and height of an image is usually measured in number of luma samples.
Downsampling: Downsampling is a process, where the sampling rate (sampling interval) of the discrete input signal is reduced. For example if the input signal is an image which has a size of height h and width w (or H and W as referred to below likewise), and the output of the downsampling is a height h2 and a width w2, at least one of the following holds true:
In one example implementation, downsampling can be implemented as keeping only each m-th sample, discarding the rest of the input signal (which, in the context of the invention, basically is a picture).
Upsampling: Upsampling is a process, where the sampling rate (sampling interval) of the discrete input signal is increased. For example if the input image has a size of h and w (or H and W as referred to below likewise), and the output of the downsampling is h2 and w2, at least one of the following holds true:
Resampling: downsampling and upsampling processes are both examples of resampling. Resampling is a process where the sampling rate (sampling interval) of the input signal is changed.
Interpolation filtering: During the upsampling or downsampling processes, filtering can be applied to improve the accuracy of the resampled signal and to reduce the aliasing affect. An interpolation filter usually includes a weighted combination of sample values at sample positions around the resampling position. It can be implemented as:
f(xr,yr)=Σs(x,y)C(k)
Where f( ) is the resampled signal, (xr, yr) are the resampling coordinates, C(k) are interpolation filter coefficients and s(x,y) are or is the input signal. The summation operation is performed for (x,y) that are in the vicinity of (xr, yr).
Cropping: Trimming off the outside edges of a digital image. Cropping can be used to make an image smaller (in number of samples) and/or to change the aspect ratio (length to width) of the image.
Padding: padding refers to increasing the size of the input image (or image) by generating new samples at the borders of the image. This can be done, for example, by either using sample values that are predefined or by using sample values of the positions in the input image.
Resizing: Resizing is a general term where the size of the input image is changed. It might be done using one of the methods of padding or cropping. It can be done by a resizing operation using interpolation. In the following, resizing may also be referred to as rescaling.
Integer division: Integer division is division in which the fractional part (remainder) is discarded.
Convolution: convolution is given by the following general equation. Below f( ) can be defined as the input signal and g( ) can be defined as the filter.
Downsampling layer: A processing layer, such as a layer of a neural network that results in a reduction of at least one of the dimensions of the input. In general, the input might have 3 or more dimensions, where the dimensions might comprise number of channels, width and height. However, the present disclosure is not limited to such signals. Rather, signals which may have one or two dimensions (such as audio signal or an audio signal with a plurality of channels) may be processed. The downsampling layer usually refers to reduction of the width and/or height dimensions. It can be implemented with convolution, averaging, max-pooling etc. operations. Also other ways of downsampling are possible and the invention is not limited in this regard.
Upsampling layer: A processing layer, such as a layer of a neural network that results in an increase of one of the dimensions of the input. In general, the input might have 3 or more dimensions, where the dimensions might comprise number of channels, width and height. The upsampling layer usually refers to increase in the width and/or height dimensions. It can be implemented with de-convolution, replication etc. operations. Also, other ways of upsampling are possible and the invention is not limited in this regard.
Some deep learning based image and video compression algorithms follow the Variational Auto-Encoder framework (VAE), e.g. G-VAE: A Continuously Variable Rate Deep Image Compression Framework, (Ze Cui, Jing Wang, Bo Bai, Tiansheng Guo, Yihui Feng), available at: https://arxiv.org/abs/2003.02012.
The VAE framework could be counted as a nonlinear transforming coding model.
The transforming process can be mainly divided into four parts:
The latent space can be understood as a representation of compressed data in which similar data points are closer together in the latent space. Latent space is useful for learning data features and for finding simpler representations of data for analysis.
The quantized latent representation T, ŷ and the side information {circumflex over (z)} of the hyperprior 3 are included into a bitstream 2 (are binarized) using arithmetic coding (AE).
Furthermore, a decoder 604 is provided that transforms the quantized latent representation to the reconstructed image {circumflex over (x)}, {circumflex over (x)}=g(ŷ). The signal {circumflex over (x)} is the estimation of the input image x. It is desirable that x is as close to {circumflex over (x)} as possible, in other words the reconstruction quality is as high as possible. However, the higher the similarity between {circumflex over (x)} and x, the higher the amount of side information necessary to be transmitted. The side information includes bitstream1 and bitstream2 shown in
In
The arithmetic decoding (AD) 606 is the process of reverting the binarization process, where binary digits are converted back to sample values. The arithmetic decoding is provided by the arithmetic decoding module 606.
It is noted that the present disclosure is not limited to this particular framework. Moreover the present disclosure is not restricted to image or video compression, and can be applied to object detection, image generation, and recognition systems as well.
In
The purpose of the second subnetwork is to obtain statistical properties (e.g. mean value, variance and correlations between samples of bitstream 1) of the samples of “bitstream1”, such that the compressing of bitstream 1 by the first subnetwork is more efficient. The second subnetwork generates a second bitstream “bitstream2”, which comprises the said information (e.g. mean value, variance and correlations between samples of bitstream1).
The second network includes an encoding part which comprises transforming 603 of the quantized latent representation ŷ into side information z, quantizing the side information z into quantized side information {circumflex over (z)}, and encoding (e.g. binarizing) 609 the quantized side information {circumflex over (z)} into bitstream2. In this example, the binarization is performed by an arithmetic encoding (AE). A decoding part of the second network includes arithmetic decoding (AD) 610, which transforms the input bitstream2 into decoded quantized side information {circumflex over (z)}′. The {circumflex over (z)} ′ might be identical to {circumflex over (z)}, since the arithmetic encoding end decoding operations are lossless compression methods. The decoded quantized side information {circumflex over (z)} ′ is then transformed 607 into decoded side information ŷ′. ŷ′ represents the statistical properties of ŷ (e.g. mean value of samples of ŷ, or the variance of sample values or like). The decoded latent representation ŷ′ is then provided to the above-mentioned Arithmetic Encoder 605 and Arithmetic Decoder 606 to control the probability model of ŷ.
The
The
Similarly in
As indicated above, the VAE can be split into different logical units that perform different actions. This is exemplified in
Specifically, as is seen in
The encoding can make use of a convolution, as will be explained in further detail below with respect to
The output of the arithmetic encoding module is the bitstream1. The bitstream1 and bitstream2 are the output of the encoding of the signal, which are then provided (transmitted) to the decoding process.
Although the unit 901 is called “encoder”, it is also possible to call the complete subnetwork described in
The remaining parts in the figure (quantization unit, hyper encoder, hyper decoder, arithmetic encoder/decoder) are all parts that either improve the efficiency of the encoding process or are responsible for converting the compressed output y into a series of bits (bitstream). Quantization may be provided to further compress the output of the NN encoder 901 by a lossy compression. The AE 905 in combination with the hyper encoder 903 and hyper decoder 907 used to configure the AE 905 may perform the binarization which may further compress the quantized signal by a lossless compression. Therefore, it is also possible to call the whole subnetwork in
A majority of Deep Learning (DL) based image/video compression systems reduce dimensionality of the signal before converting the signal into binary digits (bits). In the VAE framework for example, the encoder, which is a non-linear transform, maps the input image x into y, where y has a smaller width and height than x. Since the y has a smaller width and height, hence a smaller size, the (size of the) dimension of the signal is reduced, and, hence, it is easier to compress the signal y. It is noted that in general, the encoder does not necessarily need to reduce the size in both (or in general all) dimensions. Rather, some exemplary implementations may provide an encoder which reduces size only in one (or in general a subset of) dimension.
The general principle of compression is exemplified in
The reduction in the size of the input signal is exemplified in the
One can see from the
One of the methods for reduction of the signal size is downsampling. Downsampling is a process where the sampling rate of the input signal is reduced. For example if the input image has a size of h and w, and the output of the downsampling is h2 and w2, at least one of the following holds true:
The reduction in the signal size usually happens step by step along the chain of processing layers, not all at once. For example if the input image x has dimensions (or size of dimensions) of h and w (indicating the height and the width), and the latent space y has dimensions h/16 and w/16, the reduction of size might happen at 4 layers during the encoding, wherein each layer reduces the size of the signal by a factor of 2 in each dimension.
Some deep learning based video/image compression methods employ multiple downsampling layers. As an example the VAE framework,
In
According to some embodiments, the layers 801 to 804 of the encoder may be considered one sub-network of the encoder and the layers 830, 805 and 806 may be considered to be a second sub-network of the encoder. Similarly, the layers 812, 811 and 820 may be (in processing order through the decoder) considered a first sub-network of the decoder while the layers 810, 809, 808, 807 may be considered a second-subnetwork of the decoder. Sub-networks may be considered to be accumulations of layers of the neural network, specifically of downsampling layers of the encoder and upsampling layers of the decoder. The accumulation could be arbitrary. However, it may be provided that a sub-network is an accumulation of layers of the neural network that process an input and provide, after processing the input, an output bitstream or that receive a bitstream as input that is potentially not received by all other sub-networks of the neural network. In this context, the sub-networks of the encoder are the ones that output the first bitstream (first sub-network) and the second bitstream (second sub-network). The sub-networks of the decoder are those that receive the second bitstream (first sub-network) and the first bitstream (second sub-network).
This relation of the sub-networks is not mandatory. For example, layers 801 and 802 may be one sub-network of the encoder and layers 803 and 804 may be considered another sub-network of the encoder. Various other configurations are possible.
When seen in the processing order of bitstream2 through the decoder, the upsampling layers are run through in reverse order, i.e. from upsampling layer 812 to upsampling layer 807. Each upsampling layer is shown here to provide an upsampling with an upsampling ratio of 2, which is indicated by the ↑. It is, of course, not necessarily the case that all upsampling layers have the same upsampling ratio and also other upsampling ratios like 3, 4, 8 or the like may be used. The layers 807 to 812 are implemented as convolutional layers (conv). Specifically, as they may be intended to provide an operation on the input that is reverse to that of the encoder, the upsampling layers may apply a deconvolution operation to the input received so that its size is increased by a factor corresponding to the upsampling ratio. However, the present disclosure is not generally limited to deconvolution and the upsampling may be performed in any other manner such as by bilinear interpolation between two neighboring samples, or by nearest neighbor sample copying, or the like.
Extending this to the above explained accumulation of layers to a sub-network, it may be considered that a sub-network has a combined downsampling ratio (or combined upsampling ratio) associated with it, where the combined downsampling ratio and/or the combined upsampling ratio may be obtained from the downsampling ratios and/or upsampling ratios of the downsampling layers or upsampling layers in the respective sub-network.
At the encoder, for example, the combined downsampling ratio of a sub-network may be obtained from calculating the product of the downsampling ratios of all downsampling layers of the sub-network. Correspondingly, at the decoder, the combined upsampling ratio of a sub-network of the decoder may be obtained from calculating the product of the upsampling ratios of all upsampling layers. Other alternatives, like for example obtaining the combined upsampling ratio from a table using the upsampling ratios of the upsampling layers of the respective sub-network, as already mentioned above, can also be applied to obtain the combined upsampling ratio.
In the first subnetwork, some convolutional layers (801 to 803) are followed by generalized divisive normalization (GDN) at the encoder side and by the inverse GDN (IGDN) at the decoder side. In the second subnetwork, the activation function applied is ReLu. It is noted that the present disclosure is not limited to such implementation and in general, other activation functions may be used instead of GDN or ReLu.
The image and video compression systems in general cannot process arbitrary input image sizes. The reason is that some of the processing units (such as transform unit, or motion compensation unit) in a compression system operate on a smallest unit, and if the input image size is not integer multiple of the smallest processing unit, it is not possible to process the image.
As an example, HEVC specifies four transform units (TUs) sizes of 4×4, 8×8, 16×16, and 32×32 to code the prediction residual. Since the smallest transform unit size is 4×4, it is not possible to process an input image that has a size of 3×3 using an HEVC encoder and decoder. Similarly if the image size is not a multiple of 4 in one dimension, it is also not possible to process the image, since it is not possible to partition the image into sizes that are processable by the valid transform units (4×4, 8×8, 16×16, and 32×32). Therefore, it is a requirement of the HEVC standard that the input image must be a multiple of a minimum coding unit size, which is 8×8. Otherwise the input image is not compressible by HEVC. Similar requirements have been posed by other codecs, too. In order to make use of existing hardware or software, or in order to maintain some interoperability or even portions of the existing codecs, it may be desirable to maintain such limitation. However, the present disclosure is not limited to any particular transform block size.
Some DNN (deep neural network) or NN (neural network) based image and video compression systems utilize multiple downsampling layers. In
The term “deep” in deep neural networks usually refers to the number of processing layers that are applied sequentially to the input. When the number of the layers is high, the neural network is called a deep neural network, though there is no clear description or guidance on which networks should be called a deep network. Therefore for the purposes of this application there is no major difference between a DNN and an NN. DNN may refer to a NN with more than one layer.
During downsampling, for example in the case of convolutions being applied to the input, fractional (final) sizes for the encoded picture can be obtained in some cases. Such fractional sizes cannot be reasonably processed by a subsequent layer of the neural network or by a decoder.
Stated differently, some downsampling operations (like convolutions) may expect (e.g. by design) that the size of the input to a specific layer of the neural network fulfills specific conditions so that the operations performed within a layer of the neural network performing the downsampling or following the downsampling are still well defined mathematical operations. For example, for a downsampling layer having a downsampling ratio r>1, r∈ that reduces the size of the input in at least one dimension by the ratio r, a reasonable output is obtained if the input has a size in this dimension that is an integer multiple of the downsampling ratio r. The downsampling by r means that the number of input samples in one dimension (e.g. width) or more dimensions (e.g. width and height) is divided by two to obtain number of output samples.
To provide a numeric example, a downsampling ratio of a layer may be 4. A first input has a size 512 in the dimension to which the downsampling is applied. 512 is an integer multiple of 4 because 128×4=512. Processing of the input can thus be performed by the downsampling layer resulting in a reasonable output. A second input may have a size of 513 in the dimension to which the downsampling is applied. 513 is not an integer multiple of 4 and this input can thus not be processed reasonably by the downsampling layer or a subsequent downsampling layer if they are, e.g. by design, expecting certain (e.g. 512) input size. In view of this, in order to ensure that an input can be processed by each layer of the neural network in a reasonable way (in compliance with a predefined layer input size) even if the size of the input is not always the same, a rescaling may be applied before processing the input by the neural network. This rescaling comprises changing or adapting the actual size of the input to the neural network (e.g. to the input layer of the neural network), so that it is fulfilling the above condition with respect to all of the downsampling layers of the neural network. This rescaling is done by increasing or decreasing a size of the input in the dimension to which the downsampling is applied so that the size S=KΠiri, where ri are the downsampling ratios of the downsampling layers and K is an integer greater than zero. In other words, the input size S of the input picture (signal) in the downsampling direction is adapted to be an integer multiple of a product of all downsampling ratios applied to the input picture (signal) in the network processing chain in the downsampling direction (dimension).
Thereby, the size of the input to the neural network has a size that ensures that each layer can process its respective input, e.g. in compliance with a layer's predefined input size configuration.
By providing such rescaling, however, there are limits to the reduction in the size of a picture that is to be encoded and, correspondingly, the size of the encoded picture that can be provided to a decoder for, for example, reconstructing the encoded information also has a lower limit. Furthermore, with the approaches provided so far, a significant amount of entropy may be added to the bitstream (when increasing its size by the rescaling) or a significant amount of information loss can occur (if reducing the size of the bitstream by the rescaling). Both can have negative influence on the quality of the bitstream after the decoding.
It is, therefore, difficult to obtain high quality of encoded/decoded bitstreams and the data they represent while, at the same time, providing encoded bitstreams with reduced size.
Since the size of the output of a layer in a network cannot be fractional (there needs to be an integer number of rows and columns of samples), there is a restriction in the input image size. In
In order to solve this problem, it would be possible to use the method of padding the input image with zeros to make it a multiple of 64 samples in each direction. According to this solution the input image size can be extended in width and height by the following amount:
where “Int” is an integer conversion. The integer conversion may calculate the quotient of a first value a and a second value b and may then provide an output that ignores all fractional digits, thus only being an integer number. The newly generated sample values can be set equal to 0.
The other possibility of solving the issue described above is to crop the input image, i.e. discard rows and columns of samples from ends of the input image, to make the input image size a multiple of 64 samples. The minimum amount of rows and samples that needs to be cropped out can be calculated as follows:
where wdiff and wdiff correspond to an amount of sample rows and columns respectively, that need to be discarded from sides of the image.
Using the above, the new size of the input image in horizontal (hnew) and vertical (wnew) dimensions is as follows:
In the case of padding:
In the case of cropping:
This is also shown in the
The total number of downsampling operations and strides defines conditions on the input channel size, i.e. the size of the input to the neural network.
Here, if input channel size is an integer multiple of 64=2×2×2×2×2×2, then the channel size remains integer after all proceeding downsampling operations. By applying corresponding upsampling operations in the decoder during the upsampling, and by applying the same rescaling at the end of the processing of the input through the upsampling layers, the output size is again identical to the input size at the encoder.
Thereby, a reliable reconstruction of the original input is obtained.
In
This mode of changing the size of the input as explained above may still have some drawbacks:
In
when the rescaling is applied before processing the input with the neural network and when the rescaling is applied in a way that allows processing the input without further rescaling between layers of the neural network, respectively. A and B are scalar parameters that describe the compression ratio. The higher the compression ratio, the smaller the numbers A and B. The total size of the bitstream is therefore given as
Since the goal of the compression is to reduce the size of the bitstream while keeping the quality of the reconstructed image high, it is apparent that the hnew and wnew should be as small as possible to reduce the bitrate.
Therefore, the problem of “padding with zero” is the increase in the bitrate due to an increase in the input size. In other words, the size of the input image is increased by adding redundant data to the input image, which means that more side information must be transmitted from the encoder to the decoder for reconstruction of the input signal. As a result, the size of the bitstream is increased.
As an example, using the encoder/decoder pair in
The problem with the second approach (cropping of the input image) is the loss of information. Since the goal of compression and decompression is the transmission of the input signal while keeping the fidelity high, it is against the purpose to discard part of the signal. Therefore, cropping is not advantageous unless it is known that there are some parts of the input signal that are unwanted, which is usually not the case.
According to one embodiment, the size adjustment of the input image is performed in front of every sub-network of the DNN based picture or video compression system as explained above with relation to
Additionally, a resizing operation can be applied at the end, e.g. at the output of an upsampling layer, if a corresponding downsampling layer has applied resizing at the (its) input. The corresponding layer of a downsampling layer can be found by counting the number of upsampling layers starting from the reconstructed image and counting the number of downsampling layers starting from the input image. This is exemplified by
The resizing operation applied at the input of a downsampling layer (or corresponding sub-network comprising one or more downsampling layers) and the resizing operation applied at the output of an upsampling layer (or corresponding sub-network comprising one or more upsampling layers) are complementary, such that the size of the data at the output of both is kept the same.
As a result, the increase in the size of the bitstreams is minimized. An exemplary embodiment can be explained with reference to
In
In this embodiment, before an input to a sub-network, for example the sub-network 2, is provided to the sub-network, but after it has been processed by the previous sub-network (in this case, the sub-network 1), the input is resized by applying a resizing operation so that the input to the sub-network 2 has a size that is an integer multiple of R2. R2 represents the combined downsampling ratio of the sub-network 2 and may be a preset value and may thus be already available at the encoder. In this embodiment, this resizing operation is performed before each sub-network so that the above condition is fulfilled for the specific sub-network and its respective combined downsampling ratio. In other words, the size S of the input is adapted to or set as an integer multiple of the combined downsampling ratio of the following (following the downsampling in the sequence of processing) sub-network.
In
An embodiment is demonstrated in
Some embodiments can be applied to
In some embodiments, two options for rescaling the input may exist and one of them may be chosen depending, for example, on the circumstance or a condition as will be explained further below. These embodiments are described with reference to
The first option 1501 may comprise padding the input, for example with zeros or redundant information from the input itself in order to increase the size of the input to a size that matches an integer multiple of the combined downsampling ratio. At the decoder side, in order to rescale, cropping may be used in this option in order to reduce the size of the input to a size that matches, for example, a target input size of the proceeding sub-network.
This option can be implemented computationally efficient, but it is only possible to increase the size at the encoder side.
The second option 1502 may utilize interpolation at the encoder and interpolation at the decoder for rescaling/resizing the input. This means, interpolation may be used to increase the size of an input to an intended size, like an integer multiple of the combined downsampling ratio of a proceeding sub-network of the encoder, or a target input size of a proceeding sub-network of the decoder, or interpolation may be used to decrease the size of the input to an intended size, like an integer multiple of the combine downsampling ratio of a proceeding sub-network comprising at least one downsampling layer, or a target input size of a proceeding sub-network comprising at least one upsampling layer. Thereby, it is possible to apply resizing at the encoder by either increasing or decreasing the size of the input. Further, in this option 1502, different interpolation filters may be used, thereby providing spectral characteristics control.
The different options 1501 and 1502 can be signaled, for example in the bitstream as side information. The differentiation between the first option (option 1) 1501 and the second option (option 2) 1502 can be signaled with an indication, such as a syntax element methodIdx, which may take one of two values. For example a first value (e.g. 0) is for indicating padding/cropping, and a second value (e.g. 1) is for indicating interpolation being used for the resizing. For example, a decoder may receive a bitstream encoding a picture and comprising, potentially, side information including an element methodIdx. Upon parsing this bitstream, the side information can be obtained and the value of methodIdx derived. Based on the value of methodIdx, the decoder can then proceed with a corresponding resizing or rescaling method, using padding/cropping if methodIdx has a first value or using interpolation of methodIdx has a second value.
This is shown in
It is noted that, even though the embodiment of
A further indication or flag may be provided as shown in
Like for the indication methodIdx, also SCIdx may be obtained by a decoder by parsing a bitstream that potentially also decodes the picture to be reconstructed. Upon obtaining the value for SCIdx, downsizing or upsizing may be chosen.
In addition or alternatively to the above described indications, as shown in
In some exemplary implementations, the RFIdx may be indicated conditionally for the second option 1502, which may comprise that RFIdx is signaled if methodIdx=1 and not signaled if methodIdx=0. The RFIdx may have a size of more than one bit and may signal, for example, depending on its value, which interpolation filter is used in the interpolation for realizing the resizing. Alternatively or additionally, RFIdx may specify the filter coefficients from the plurality of interpolation filters. This may be, for instance, Bilinear, Bicubic, Lanczos3, Lanczos5, Lanczos8 among others.
As indicated above, at least one of methodIdx, SCIdx and RFIdx or all of them or at least two of them may be provided in a bitstream which may be the bitstream that also encodes the picture to be reconstructed or that is an additional bitstream. A decoder may then parse the respective bitstream and obtain the value of methodIdx and/or SCIdx and/or RFIdx. Depending on the values, actions as indicated above may be taken.
The filter used for the interpolation for realizing the resizing can, for example be determined by the scaling ratio.
As indicated in the lower right of
In another example there might be 2 lookup tables, one for the case of upsizing and one for the case of downsizing. In this case LUT1(SCIdx) might indicate the resizing filter when downsizing is selected, and LUT2(SCIdx) might indicate the resizing filter for the upsizing case.
In general, the present disclosure is not limited to any particular way of signaling for RFIdx. It may be individual and independent from other elements or jointly signaled.
It is noted that the explanations that follow are only exemplarily and is not intended to limit the invention to specific kinds of padding operations. The straight vertical line indicates the border of the input (a picture, according to embodiments), right hand side of the border are the sample positions where the padding operation is applied to generate new samples. These parts are also referred below as “unavailable portions” which means that these do not exist in the original input but are added by means of padding during the rescaling operation for the further processing. The left side of the input border line represents the samples that are available and are part of the input. The three padding methods depicted in the figure are replication padding, reflection padding and filling with zeros. In the case of a downsampling operation that is to be performed in line with some embodiments, the input to the sub-network of the NN will be the padded information, i.e. the original input extended by the applied padding.
In the
Specifically, the padding type that is applied may depend on task to be performed. For example:
The padding or filling with zeros can be reasonable to be used for Computer Vision (CV) tasks such as recognition or detection tasks. Thereby, no information is added in order not to change the amount/value/importance of information already existing in the original input.
Reflection padding may be a computationally easy approach because the added values only need to be copied from existing values along a defined “reflection line” (i.e. the border of the original input).
The repetition padding (also referred to as repetition padding) may be preferred for compression tasks with Convolution Layers because most sample values and derivative continuity is reserved. The derivatives of the samples (including available and padded samples) are described on the right hand side of
In the examples shown, the replication padding has the smallest change in the derivatives. This is advantageous in view of video compression tasks but results in more redundant information being added at the border. With this, the information at the border may become more weight than intended for other tasks and, therefore, in some implementations, the overall performance of padding with zeros may supersede reflection padding.
The decoder 2020 has a corresponding structure of the upsampling layers 1 to N. One sub-network 2022 of the decoder 2020 comprises the upsampling layers N to M and the other sub-network 2021 comprises the upsampling layers 3 to 1 (here, in descending order so as to bring the numbering in line with the decoder when seen in the processing order of the respective input).
As indicated above, the rescaling applied to the input before the first sub-network 2011 of the encoder is correspondingly applied to the output of sub-network 2021 of the decoder. This means the size of the input to the first sub-network 2011 is the same as the size of the output of the sub-network 2021, as indicated above.
More generally, the rescaling applied to the input of a sub-network n of the encoder corresponds to the rescaling applied to the output of the sub-network n so that the size of the rescaled input is the same as the size of the rescaled output. The index n may denote the number of the sub-network in the order of processing an input through the encoder.
The neural network 2100 comprises, as such, a plurality of layers 2111, 2112, 2121 and 2122. These layers are provided to process an input they receive. The respective input to the respective layers is denoted with 2101, 2102, 2103 and 2104. In the end, with a dashed line, an output of the neural network 2105 is provided after the original input 2101 has been processed by each of the layers of the neural network.
The neural network 2100 according to
For further explanations, it will be assumed that the input 2101 has a given size in at least one dimension and may constitute an input having two dimensions which may, for example, be represented in the form of a matrix where each entry in the matrix constitutes a sample value of the input. In the sense of the input 2101 being a picture, the values in the matrix may correspond to values of samples of the picture, for example in a specific color channel. The picture may, as already explained above, be a still picture or a moving picture in the sense of a video sequence or a video. A picture of a video may also be referred to as an image or frame or the like.
During the processing of the input 2101 with the neural network 2100 and specifically by its respective layers, an output 2105 may be created that represents an encoded picture and may be provided in the form of a bitstream after binarization or encoding into the bitstream of an output from a NN layer. The binarization/encoding of the feature maps (channels) may be performed on the output of the NN. However, the binarization/encoding of the feature map may itself be considered a layer of the NN. Encoding may be, e.g. an entropy coding. The present disclosure encompasses that the size of the bitstream representing an encoded picture smaller than the size of the input picture.
This is achieved, according to some embodiments, by the layers 2111, 2112, 2121, 2122 comprising one or more downsampling layers. For ease of explanation, it will be assumed that each of the layers 2111, 2112, 2121, 2122 of the neural network 2100 depicted in
The downsampling encompasses that the size of the output of a downsampling layer multiplied by the downsampling ratio rm equals the size of the input provided to the downsampling layer.
The downsampling can be provided by applying a convolution to an input of the downsampling layer.
Such a convolution comprises the element-wise multiplication of entries in the original matrix of the input (for example, a matrix with 1024×512 entries, the entries being denoted with Mij) with a kernel K that is run (shifted) over this matrix and has a size that is typically smaller than the size of the input. The convolution operation of 2 discrete variables can be described as:
Therefore, calculation of the function (f*g) [n] for all possible values of n is equivalent to running (shifting) the kernel or filter f[ ] over the input array g[ ] and performing element-wise multiplication at each shifted position.
In the above example, the kernel K would be a 2×2 matrix that is run over the input by a stepping range of 2 so that the first entry D11 in the downsampled bitstream D is obtained by multiplying the kernel K with the entries M11, M12, M21, M22. The next entry D12 in the horizontal direction would then be obtained by calculating the inner product of the kernel with the entries or the reduced matrix with the entries M13, M14, M23, M24. In the vertical direction, this will be performed correspondingly so that, in the end, a matrix D is obtained that has entries Dij obtained from calculating the respective inner products of M with K and has only half as many entries per direction or dimension.
In other words the shifting amount, which is used to obtain the convolution output determines the downsampling ratio. If the kernel is shifted 2 samples between each computation steps, the output is downsampled by a factor of 2. The downsampling ratio of 2 can be expressed in the above formula as follows:
The transposed convolution operation (as may be applied during a decoding as explained in the following) can be expressed mathematically in a same manner as a convolution operation. The term “transposed” corresponds to the fact that the said transposed convolution operation corresponds to inverting of a specific convolution operation. However implementation-wise, the transposed convolution operation can be implemented similarly by using the formula above. An upsampling operation by using a transposed convolution can be implemented by using the function:
In the above formula the u corresponds to the upsampling ratio, and into function corresponds to conversion to an integer. The into operation for example can be implemented as a rounding operation.
In the above formula, the values m and n can be scalar indices when the convolution kernel or filter f( ) and the input variable array g( ) are one dimensional arrays. They can also be understood as multiple dimensional indices when the kernel and the input array are multi-dimensional.
The invention is not limited to downsampling or upsampling via convolution and deconvolution. Any possible way of downsampling or upsampling can be implemented in the layers of a neural network, NN.
In the context of the present disclosure, one or a plurality of the layers of the encoder 2100 are summarized in the form of a sub-network of the encoder. In
Moreover, one or more of the sub-networks may comprise even further layers that are no downsampling layers but perform different operations on the input. Additionally or alternative, the sub-networks may comprise further units, as was already exemplified above.
Furthermore, the layers of the neural network can comprise further units that perform other operations on the respective input and/or output of their corresponding layer of the neural network. For example, the layer 2111 of the sub-network 2110 may be a downsampling layer and, in the processing order of an input to this layer before the downsampling, there may be provided a rectifying linear unit (ReLu) and/or a batch normalizer.
Rectifying linear units are known to apply a rectification to the entries Pij of a matrix P so as to obtain modified entries P′ij in the form
Thereby, it is ensured that values in the modified matrix are all equal or greater than 0. This may be necessary or advantageous for some applications.
The batch normalizer is known to normalize the values of a matrix by firstly calculating a mean value from the entries Pij of a matrix P having a size M×N in the form of
With this mean value V, batch normalized matrix P′ with the entries P′ij is then obtained with by.
P′
ij
=P
ij
−V
Both, the calculations obtained by the batch normalizer and the calculations obtained by the rectified linear unit do not alter the number of entries (or the size) but only alter the values within the matrix.
Such units can be arranged before the respective downsampling layer or after the respective downsampling layer, depending on the circumstances. Specifically, as the downsampling layer reduces the number of entries in the matrix, it might be more appropriate to arrange the batch normalizer in the processing order of the bitstream after the respective downsampling layer. Thereby, the number of calculations necessary for obtaining V and P′ij can be reduced. As the rectified linear unit can simplify the multiplications to obtain the matrix of reduced size in the case of a convolution being used for the downsampling layer because some entries may be 0, it can advantageous to arrange the rectified linear unit before the application of the convolution in the downsampling layer.
However, the invention is not limited in this regard and the batch normalizer or the rectified linear unit may be arranged in another order with respect to the downsampling layer.
Furthermore, not each layer of the neural network necessarily has one of these further units or other further units may be used that perform other modifications or calculations.
While the provision of the sub-networks may be arbitrary in general with respect to the number of downsampling layers they comprise, two different sub-networks do have distinct layers in that no layer of the neural network (irrespective of whether it constitutes a downsampling layer or any other layer) is part of two sub-networks.
Furthermore, even though the association of specific layers of the neural network to a specific sub-network may be arbitrary, layers of a neural network may be summarized to a sub-network preferably for cases where they process an input they receive and provide, after the processing with the layers of the sub-network, a bitstream as output. In the context of
The size of the first sub-bitstream 2103 is preferably smaller than the size of the input 2101 and is larger than the size of the sub-bitstream 2105 in at least one dimension.
In order to ensure a reliable processing of an input (for example the input 2101) with the sub-network that processes this input, it is envisaged according to the present disclosure that, if necessary, a rescaling is applied to the input 2101 in at least one dimension. This rescaling encompasses a changing of the size of the input so that it matches an integer multiple of a combined downsampling ratio of all downsampling layers of the respective sub-network that is to process the input.
To explain this in more detail, it may be assumed that the sub-networks are numbered in the order they process an input, like the input 2101. The first sub-network that processes this input may be numbered 1, the second sub-network may be numbered 2 and so on up to the last sub-network K, where k is a natural number. Any sub-network may thus be denoted as the sub-network k, where k is a natural number. A downsampling layer within the sub-network k has, as explained above, an associated downsampling ratio. The sub-network k may comprise M downsampling layers, where M is a natural number. For reference, a downsampling layer m of a sub-network k then may be associated with a downsampling ratio denoted with rk,m where the index k associated this downsampling-ratio with the sub-network k and the index m indicates to which of the downsampling layers the downsampling ratio rk,m belongs.
Each of these sub-networks then has an associated combined downsampling ratio.
Specifically, the combined downsampling ratio Rk of a sub-network k may be obtained by calculating the product of the downsampling ratios rk,m of all downsampling layers of the sub-network k.
Referring back to the rescaling mentioned above, it may be preferred that the rescaling applied to the input of a sub-network k only depends on the combined downsampling ratio Rk of the respective sub-network k but does not depend on downsampling ratios of another sub-network l, where l is not equal k. Thereby, a rescaling is obtained that only changes the size of the input so that it can be reliably processed by the respective sub-network and its downsampling layers irrespective of whether the resulting output of this sub-network can reasonably be processed by another sub-network.
In the context of
More generally speaking, an input to a sub-network k may be considered. This input may be represented in the form of a matrix having, in at least one of its dimensions, a size Sk, where k denotes that this is the input to the sub-network k. As the input has the form of a matrix, Sk is an integer value that is at least 1. A reasonable processing of the input with the sub-network k is possible, if the size Sk of the input is an integer multiple of the combined downsampling ratio Rk already defined above, i.e. if Sk=nRk, where n is a natural number. If this is not the case, a rescaling may be applied to the input with the size Sk, thereby changing its size to a new size
The method 2200 begins with a first step 2201 where an input with a size Sk is received at the sub-network k. This input with a size Sk may be received, for example, from a preceding sub-network of the neural network and may thus not constitute a size that is identical to the input picture to be encoded. However, the size Sk can also constitute an input that corresponds to the original picture if the sub-network with index k is the first sub-network that processes the input picture.
In a subsequent step 2202, it may then be evaluated whether the size Sk corresponds to an integer multiple of the combined downsampling ratio Rk of the sub-network k that is to process the input with the size Sk.
This determination may comprise, for example, comparing the size Sk to a function depending on the combined downsampling ratio Rk and the size Sk. Specifically, the value
may be compared to the size Sk. Alternatively or additionally, the value
may be compared to the size Sk. This comparison may specifically comprise calculating the difference
If these values are 0, then Sk already is an integer multiple of the combined downsampling ratio Rk because both, the functions ceil and floor provide the closest integer of the result of the division
If this closest integer is multiplied with the combined downsampling ratio, it will only be equal to Sk if Sk already is an integer multiple of the combined downsampling ratio Rk.
Using the result of this comparison, it can then be determined whether a rescaling is to be applied to the input with the size Sk to change its size to a new size
In this situation, two cases can occur. If it is already determined in step 2202 that the size Sk is an integer multiple of the combined downsampling ratio Rk of the sub-network k and, therefore, corresponds to an allowed input size of the sub-network that allows for reasonably processing the input with this sub-network k, the determination in step 2210 can be made. In this case, in the subsequent step 2211, the downsampling operation can be performed on the input with the size Sk with the respective sub-network k. This encompasses reducing the size Sk to a size Sk+1 during the downsampling with the sub-network k, where Sk+1 is smaller than Sk due to the downsampling being applied to the input to the sub-network. In this case, the size Sk and the size Sk+1 are related by the combined downsampling ratio Rk of the sub-network k. Sk corresponds, in this case, to the product of Sk+1 and the combined downsampling ratio Rk.
Having performed this downsampling with the sub-network, an output with the size Sk+1 1 can be provided in step 2212.
For computational efficiency, it can be provided that even though it is determined that the size Sk already corresponds to an integer multiple of the combined downsampling ratio Rk of the respective sub-network k, a resizing of the original input with the size Sk is performed. This resizing will, however, not result in a change of the size Sk when applying because the size Sk already corresponds to the allowed input size.
In case it is determined in step 2202 that the size Sk does not correspond to an integer multiple of the combined downsampling ratio Rk, a rescaling that changes the size Sk to a size
In this context, in step 2221, a rescaling is applied to the input with the size Sk to change the input size to the allowed input size for the sub-network which may be considered to be
This rescaled input is then processed in step 2211 by applying the downsampling in the respective sub-network. The rescaling is preferably selected so that when applying the processing in step 2211, the downsampling applied by the sub-network nevertheless results in a reduced size Sk+1 that is still smaller than the input size Sk even though potentially the rescaling comprises increasing the size Sk to a size
This can be achieved by, for example, using a function
to obtain the closest larger integer multiple of the combined downsampling ratio. This value may be set or considered the allowed input size of the sub-network k and may be denoted with
If the size Sk is no integer multiple of the combined downsampling ratio Rk, this value will be smaller than Sk. The size Sk may then be rescaled to this value, thereby reducing the size Sk.
The determination whether to increase the size Sk to the closest larger integer multiple of the combined downsampling ratio or reducing the size Sk to the closest smaller integer multiple of the combined downsampling ratio may depend on further considerations.
For example, when encoding pictures, it is of importance to ensure that when decoding a bitstream constituting the encoded picture again, the quality of the decoded picture obtained from the bitstream is comparable to that of the picture originally input to the encoder. This can be achieved, for example, by only increasing size of an input to a sub-network k to the closest larger integer multiple of the combined downsampling ratio of this sub-network, thereby ensuring that no information is lost. This may encompass, as was already explained above with relation to, for example, the embodiments of
On the other hand, as this padding will result in information being added to the input which can have negative effects on the borders of a picture when it is decoded again, it can be envisaged to reduce the size Sk to the closest smaller integer multiple of the combined downsampling ratio Rk of the sub-network k by using either cropping or interpolation in a way that reduces the size. The cropping comprises deleting samples from the original input, thereby reducing its size. The interpolation to decrease the size may comprise calculating a mean value for one or more adjacent samples in the original input with the size Sk and using this mean value as a single sample instead of the original samples.
By applying this rescaling on a sub-network basis, a reduction in the size of the resulting bitstream that is finally output by the encoder is obtained. This will be explained in the following with respect to a numerical example that also makes use of the description associated with
In
Furthermore, as exemplified in
Coming back to the above numerical examples, the whole downsampling that is applied to an input to the neural network is actually independent from the separation of the neural network into sub-networks. It is obtained by calculating the whole downsampling ratio of the whole network, being the product of all downsampling ratios. This means, as there are six downsampling layers having a downsampling ratio of 2 each, the downsampling ratio of the whole neural network is 64. An input will thus be reduced in size by a factor of 64 after having it processed with the whole neural network.
In the prior art, a rescaling was applied to an input before processing it with the neural network so that it can be processed by the whole neural network. In other words, this requires that the input size is rescaled to a value that corresponds to an integer multiple of the overall downsampling ratio of the whole neural network. In the context of this example, this means that only an input size being an integer multiple of 64 would be allowed according to the prior art.
Taking as example an input size of 540. It is seen that this is no integer multiple of all downsampling ratio 64. For ensuring reliable processing in the prior art, a rescaling to either 576 or 512 is performed because these are integer multiples of the overall downsampling ratio 64 that come closest to the original size.
Assuming for the following discussion that the size of the input is increased to 576 and then processed by the downsampling layers according to the prior art and creating a first bitstream at the position 2103 (i.e. after having processed the input with two of the downsampling layers) and the second bitstream after having processed the input with all the downsampling layers. The first bitstream is obtained by processing the input with the rescaled size 576 with the first two downsampling layers. After the first downsampling layer, the input of the size 576 is reduced by the downsampling ratio 2 to the size 288. The next downsampling layer reduces this size to the value 144. The first output bitstream 2103 according to the prior art will thus have a size of 144.
This is then further processed by the remaining downsampling layers which, together, have a downsampling ratio of 16. By this, the size of the input 2103 to the subsequent downsampling layers according to the prior art will first be reduced to 72, then to 36, then to 18 and finally to 9. The second bitstream 2105 output after having processed the input with a neural network according to the prior art will thus have a size of 9.
Combining the first output 2103 and the second bitstream 2105 to a combined bitstream as output of the encoder, this will result in a size of 153=144+9.
According to the present disclosure, however, the situation is different.
As indicated above, the rescaling applied to the input is obtained such that a rescaling is applied in a way that changes the size of the input if the size Sk of the input does not equal an integer multiple of the combined downsampling ratio Rk of the sub-network k with which the respective input is to be processed.
In keeping with the above example, according to one embodiment, the first sub-network comprises two downsampling layers that each have a downsampling ratio of 2, resulting in a combined downsampling ratio R1=4. The input has a size of 540 as indicated above. 540 is an integer multiple of 4 (4×135=540). This means that when processing the input with the first sub-network, no rescaling is necessary and the output 2103 has a size of 135 after having been processed with the first sub-network 2110. Consequently, the first bitstream has a size that is smaller than the size of the first bitstream that was obtained with a method according to the prior art. There, the size of the first bitstream was 144.
In the next step, this intermediate result in the form of the output 2103 of the first sub-network is provided as input to the second sub-network which has a combined downsampling ratio R2=24=16 (4 downsampling layers with a downsampling ratio of 2 each). 135 is not an integer multiple of 16, thus requiring a rescaling of the input before processing the input 2103 with the second sub-network. Assuming again that the size will be increased to have a comparable result to the prior art, the closest larger integer multiple of the combined downsampling ratio R2=16 to the input size 135 is 144. After increasing the size of the of input 2103 to 144, the further processing results in a second bitstream 2105 obtained by applying the downsampling in the second sub-network. This second bitstream then has a size that equals 9 (144/16=9).
This means that, in this example, the bitstream output after having processed the input with the neural network comprising the first bitstream and the second bitstream according to embodiments of the present disclosure has a size of 135+9=144. This is approximately 5% smaller than the output size according to the prior art as explained above resulting in a significant reduction of the size of the bitstream while encoding the same information.
To provide a more specific example, the sub-networks 2110 and 2120 can, for example, be the networks that form the encoder 601 and the hyper encoder 603 according to
The method 2300 for determining whether the input with the size Sk needs to be rescaled begins with a step 2301 where an input with the size Sk not equal to lRk (l being a natural number and Rk being the combined downsampling ratio of the sub-network k) is received at the sub-network. In the next step 2302, a determination may be made what the closest smaller integer multiple of the combined downsampling ratio Rk and the closest larger integer multiple of the combined downsampling ratio Rk are. The step 2302 may comprise calculating the function
to obtain the value l that indicates the closest smaller integer multiple of the combined downsampling ratio to the size Sk. Alternatively or additionally, the value l+1 may be obtained by calculating
Instead of floor, also the function
may be used as both, floor and int, result in the closest smaller integer multiple of this division.
These calculations may be used instead of explicitly obtaining the values for l and l+1. Furthermore, it may be envisaged that only one of the functions floor, int and ceil is used in order to obtain the values l and l+1. For example, using ceil, the value l+1 can be obtained. From this, the value l can be obtained by subtracting 1. Likewise, by using either int or floor, the value l can be obtained and the value l+1 can be obtained by adding the value l.
Depending on a further condition, it may then be determined in step 2403, whether the size Sk is to be increased or decreased, depending on an evaluation of the condition and the corresponding result obtained in step 2310 or 2320. For example, the absolute value of the difference between lRk and Sk on the one side and the absolute value of the difference of (l+1)Rk and Sk may be determined, i.e. |Sk−lRk| and |Sk−Rk(l+1)| may be obtained. Depending on which of them is smaller (using for example the function Min that throws the smaller of two values), it can be determined that the size of the input Sk is closer to the closest smaller integer multiple of the combined downsampling ratio Rk or closer to the closest larger integer multiple of the combined downsampling ratio Rk.
If the condition 2403 comprises that as few as possible modifications to the original input with the size Sk are applied, the determination whether to increase or decrease the size may then be made by evaluating the outcome of the above-explained comparison. This means that if Sk is closer to the closest smaller integer multiple of the combined downsampling ratio than to the closest larger integer multiple of the combined downsampling ratio, then it may be determined in step 2320 that the size Sk is to be decreased to the closest smaller integer multiple of the downsampling ratio Rk, i.e. to the closest smaller integer multiple lRk in step 2321.
With this rescaled input that is decreased in size, the downsampling may be performed in step 2330 by the sub-network, thereby obtaining an output as already explained above.
Correspondingly, if the difference between the closest larger integer multiple (l+1)Rk and the input size Sk is smaller than the difference to the closest smaller integer multiple of the combined downsampling ratio Rk, the size may be increased depending on this result 2310 in the step 2311 to the size
Also after having increased the original input size Sk to the size
As already explained above, applying the rescaling may comprise (if rescaling to a larger size) applying for example padding or interpolation. If the size Sk is decreased to a size
As was already explained above with relation to the
The details explained above with respect to
As can be seen in
The input to the neural network 2400 is indicated with the item 2401. This may be a bitstream that encodes a picture or may be an input provided from a previous layer of the neural network or may be an input processed or pre-processed in any reasonable way.
In any case, the input may preferably be representable in the form of a two-dimensional matrix which has a size T in at least one dimension. The layers of the neural network 2400 and specifically the upsampling layers perform a processing on the input. This means that the input 2401 may be processed by the layer 2411 and an output 2402 of this layer may be provided to a subsequent layer 2412 and so on. Finally, an output 2405 of the neural network 2400 may be obtained. If this output 2405 is the output of the last layer of the neural network 2400, it may be considered to represent or be the decoded picture obtained from the bitstream.
According to the present disclosure, the neural network 2400 may be separated into sub-networks 2410 and 2420 in a manner corresponding to what was already described with respect to the encoder in
In line with some embodiments, it is envisaged that at least one of these sub-networks comprises at least two upsampling layers. In this context, for example the sub-network 2410 may comprise two upsampling layers 2411 and 2412. Apart from that, embodiments provided in this disclosure are not limited with respect to the number of upsampling layers or additional layers being provided in the respective sub-networks.
It is also encompassed by the present disclosure that there may be more than a single bitstream provided to the neural network. In this context, the input 2401 may be processed by all sub-networks 2410 and 2420 and potentially further sub-networks of the neural network while at least one further input bitstream, for example an input provided at the position 2403, may not be processed by all sub-networks of the neural network but may only be processed by the sub-network 2420 and potential subsequent sub-networks but not the sub-network 2410.
At the end of the processing of all inputs through the neural network 2400, an output 2405 for example with the size Toutput may be obtained, where this output may correspond to the decoded picture. In line with the present disclosure, the size Toutput of the output will generally be larger than the size T of the input. As the size T of the input may not be predefined and can vary depending on what information has been originally encoded by an encoder, for example, it can be advantageous to indicate the output size Toutput in the bitstream or in an additional bitstream so that the reconstruction of a picture that originally had the size Toutput can be performed reliably.
Based on such information, it is also encompassed that, after having processed an input with a sub-network, a potential rescaling is applied to the output obtained from this sub-network before processing the output (also encompassing a potentially rescaled output) with the next sub-network in the processing order of the neural network.
This information that may be used in order to determine a potential rescaling to be applied before the processing of an output of a sub-network by the subsequent sub-network may not only encompass the final target output size Toutput, but may also encompass additional information or may alternatively encompass additional information like, for example, an intended output size to be obtained after the processing with the respective sub-network or an intended input size for the input in the subsequent sub-network. This information can either be available at the decoder performing the decoding method or it can be provided in the bitstream provided to the decoder or an additional bitstream.
In line with embodiments of the present disclosure, each sub-network k (for example, the sub-networks 2410, 2420) has associated with it a combined upsampling ratio Uk where the index k enumerates the number of the sub-network in the processing order of the input through the neural network and may, as already explained above, be of integer value greater than 0 though other enumerations are also possible In the case k is an integer value beginning with 1 and running to the value K being the last sub-network, k may be considered to denote the position of a sub-network in the processing order of the bitstream through the neural network.
This enumeration may be chosen to be in line with the enumeration of sub-networks of an encoder as explained above. However, for matching of the processing performed by the respective sub-networks on the encoder and the decoder, respectively, it may be envisaged that the order of the indexing is different. This means that an inverse order is applied for the enumeration of the sub-networks in the decoder compared to what was applied at the encoder. For example, the first sub-network of the encoder may be denoted with the index k=1. The corresponding sub-network at the decoder that inverses the processing applied to the input at the encoder, is, in the processing order of the input of the neural network at the decoder, the last sub-network. This may be denoted with the index k=1 as well or it may be denoted with the index K where K denotes the number of all sub-networks of the neural network. In the first case, a mapping between the sub-networks of a decoder and the corresponding sub-networks of an encoder is possible. In the latter case, a transformation may be applied to obtain the respective mapping.
The method begins with a first step 2501 where an input having a size Tk is processed by a sub-network k. This processing encompasses an upscaling of the input with the size Tk to an output having a size
Preferably, the upsampling performed by a sub-network k is independent from an upsampling that may be performed by other sub-networks within the neural network.
Coming back to hat. The
can be preset or provided in the bitstream or in any other reasonable way to the decoder.
Alternatively or additionally, it can also be envisaged that the size is determined based on a formula depending on the combined upsampling ratio Uk+1 of the sub-network k+1 and a target output size of this sub-network where this target output size is then potentially denoted with
. The target output size may likewise constitute a target input size to the subsequent sub-network k+2.
For example, the target input size may be obtained using the target input size of the next sub-network k+2 and the combined upsampling ratio of the current sub-network k+1. For example, the target input size
may be obtained from the division of the target input size of the next sub-network k+2 by the combined upsampling ratio Uk+1 of the current sub-network k+1. Specifically, this may be represented as
Alternatively, the size may be obtained using either of
This ensures that the obtained value for always is an integer value.
Having determined , the rescaling of the size
can be applied by either increasing or decreasing the size
The rescaled output of the sub-network k (or, correspondingly, the rescaled input to the sub-network k+1) with the rescaled size matching the target input size of the sub-network k+1 is then processed in step 2504 by the sub-network k+1. Thereby, like for the sub-network k, an upsampling is applied to the input with the size
and an output with a size
. Even though in some embodiments it may be preferred to apply a rescaling that reduces the size
, this reduction in size may thus be provided in a way that, when processing the input with the size
in the sub-network k+1, the output obtained after the processing has a size
The output with the size that matches an intended target input size of the sub-network k+2. When the target input size
is calculated in the same way as indicated above and if the final output size Toutput is known, it is possible to iteratively obtain the target input size
for a general sub-network k (or
for the specific sub-network k+2) from the final output size Toutput and the combined upsampling ratios of the proceeding sub-networks, including the sub-network k (or k+2, respectively). Specifically, assuming that no rescaling would be necessary and processing the input by the neural network with all sub-network would immediately lead to a final output having a size Toutput In that case, the input size
to a sub-network k, multiplied with the combined upsampling ratios of all sub-networks that are still to process the input, will be equal to the target output size Toutput. This means
Πi,i=k . . . KUi=Toutput, where the index i for the combined upsampling ratios runs from k (for the current sub-network k) up to K, wherein K is the last sub-network of the neural network.
As mentioned, this holds true for the case that the input size to the sub-network k is of a size that can, without applying rescaling, be processed by the subsequent sub-networks, immediately resulting in the target output size Toutput. In any other case, the target input size
to a sub-network k may rather be obtained from
The actual input size
In general the target size the size that will be obtained by rescaling
=f(Toutput, Uk, Uk+1, Uk+2, . . . ), where Uk, Uk+1, Uk+2, . . . indicate the combined upsampling ratios of the sub-networks k, k+1, k+2, . . . respectively.
The target size at the output of a current sub-network might also be calculated according to a function as follows:
=f(Toutput,U,N)
where the U denotes a scalar (which might be a predefined number indicating an upsampling ratio) and N denotes the number of sub-networks including and following the kth subnetwork in the processing order. This function might especially be useful if the upsampling ratio of the sub-networks are all equal to U.
In another example the target size can be calculated according to a function such as
=f(Toutput, Scalek). The Scalek is a scalar number that might be pre-calculated or predefined. Generally the structure of decoder network, which consists of multiple sub-networks, is fixed during the design and cannot change later on. In such a case (when the decoder structure is fixed) all of the sub-networks that follow the current sub-network and their upsampling ratios are known during the design of the decoder. This means that the total upsampling ratio which depends on the combined upsampling ratios of individual sub-networks can be pre-calculated for each kth sub-network. In such a case, the obtaining of
might be performed according to
=f(Toutput,Scalek) where Scalek is the pre-calculated scalar corresponding to sub-network k that is determined (and stored as a constant-valued parameter) during the design of the decoder. In this example, instead of individual upsampling ratios of the sub-networks including and following the kth subnetwork, the pre-calculated scale ratio (Scalek) corresponding to the kth sub-network is used in the function.
For example in ) can be calculated according to the target output size Toutput (which is equal to the intended size of x) and the scalar factor 64 (16×4=64). The scalar factor 64 here corresponds to the total upsampling ratio of the combined upsampling ratios of the sub-networks 1004a and sub-network 1007a. In other words the Scale1007a corresponding to sub-network 1007a is equal to 64 in this case. According to this example one can calculate the
which is the target input size of sub-network 1007a according to formula:
=f(Toutput, Scale1007a)=f(Toutput, 64). Similarly the target input size of sub-network 1004a can be calculated according to
=f(Toutput, 16), where 16 is the Scale1004a of sub-network 1004a.
The target output size Toutput can be obtained from a bitstream. In the example of
The function f( ) may be ceil( ), floor( ), into etc.
Having processed the input or inputs received at the decoder 2400 according to
In this embodiment, additional information provided to the decoder comprises the target output size Toutput of the neural network where this target output size may be identical to the size of the picture originally encoded in the bitstream.
The method in
In a next step 2602, this size for the sub-network k+1. This comparing may comprise calculating a difference between
.
The target input size may be obtained in line with what was described already above. Specifically, the target input size
may be obtained in step 2610 using the target output size Toutput of the neural network. The target output size Toutput of the neural network may be identical to the size of the originally encoded picture. Having obtained the target input size
, it may be provided in step 2620 for the use in the comparison in step 2602.
Returning to the comparison step 2602, if it is determined (by, for example, explicitly calculating the difference between and
, then a rescaling may be applied so as to reduce the size
in step 2603 as part of performing the rescaling. This reduction in size may comprise a cropping or using interpolation to reduce the number of samples, thereby reducing the size of the input to the sub-network k+1, as was already explained above.
Alternatively, if it is determined that the size hat, a rescaling may be applied in step 2603 resulting in an increase of the size
.
After that, in step 2604, the upsampling of the input with the size may be performed with the sub-network k+1, thereby providing, as part of the step 2604, an output that has a size
It is noted that when using an iterative process to obtain the target input sizes by applying, for example
as explained above, the values for can either be provided as part of the bitstream or can be calculated at the decoder or can be provided for example in a lookup table where the index i of the sub-network that is to process the input may be used to derive the respective value for the product Πi,i=k . . . KUi (thus not explicitly calculating it for each sub-network) or, if the target output size Toutput has already a fixed value, even the values for
can be taken from a lookup table. In this context, the index k of the sub-network to process the input can be used as indicator to a value in the lookup-table. Additionally or alternatively to the lookup table, the pre-calculated values of the product Scalek=Πi,i=k . . . KUi corresponding to each sub-network k can be defined as constant value. Therefore the operation of obtaining a target size becomes
=f(Scalek,Toutput), wherein the f( ) may be a floor operation, a ceil operation, a rounding operation or like. The function f( ) for instance might be in the form of f(x,y)=(y+x−1)>>log 2(x). The equation presented here is equivalent to ceil(y/x) when x is a number that is power of 2. In other words when x is an integer number that can be represented as a power of 2, the function (y/x) can be implemented as equivalently (y+x−1)>>log 2(x). As another example the function f(x,y) might be y>>log 2(x). Here, “>>” indicates a downshift operation, also referred to as right shift operation, as explained below.
The encoder 2700 may, for this purpose, comprise a receiver 2701 for receiving a picture and potentially any additional information that pertains to how the encoding is to be performed as was already explained above. Furthermore, the encoder 2700 may comprise one or more processors denoted here with 2702 that are configured to implement a neural network wherein the neural network comprises, in the processing order of a picture through the neural network, at least two sub-networks wherein at least one of these sub-networks comprises at least two downsampling layers and the one or more processors are additionally further adapted to encode a picture by using the neural network by performing the following steps:
Additionally, the encoder may comprise a transmitter 2703 for providing an output like the bitstream and/or an additional bitstream or a plurality of bitstreams as was already discussed above. One of those bitstreams may comprise or represent the encoded picture whereas another bitstream may pertain to additional information as was already discussed above.
The decoder 2800 may, for this purpose, comprise a receiver 2801 for receiving a bitstream representing a picture (specifically representing an encoded picture). Furthermore, the decoder 2800 may comprise one or more processors 2802 that are configured to implement a neural network where this neural network comprises, in the processing order of a bitstream through the neural network, at least two sub-networks. One of these two sub-networks comprises at least two upsampling layers. Furthermore, the processors 2802, by using the neural network, are configured to apply an upsampling to an input representing the matrix (like the bitstream or something put off a preceding sub-network) where the matrix has a size T1 in at least one dimension and the processors and/or the encoder are further configured to decode a bitstream by:
Furthermore, the encoder 2800 or an additionally provided transmitter 2803 may be adapted to provide, after processing the bitstream using the neural network, a decoded picture as output of the neural network.
In embodiments of encoding methods or encoders described herein the bitstream output, e.g. output by the NN, may be, for example, the output or bitstream of the last subnetwork or network layer of the NN, e.g. bitstream 2105.
In further embodiments of encoding methods or encoders described herein the bitstream output, e.g. output by the NN, may be, for example, a bitstream formed by or comprising two sub-bitstreams, e.g. sub-bitstreams bitstream1 and bitstream2 (or 2103 and 2105), or more general, a first sub-bitstream and a second sub-bitstream (e.g. each sub-bitstream being generated and/or output by a respective sub-network of the NN). Both sub-bitstreams may be transmitted or stored separately or combined, e.g. multiplexed, as one bitstream.
In even further embodiments of encoding methods or encoders described herein the bitstream output, e.g. output by the NN, may be, for example, a bitstream formed by or comprising more than two sub-bitstreams, e.g. a first sub-bitstream, a second sub-bitstream, a third subbitstream, and optionally further sub-bitstreams (e.g. each sub-bitstream being generated and/or output by a respective sub-network of the NN). The sub-bitstreams may be transmitted or stored separately or combined, e.g. multiplexed, as one bitstream or more than one combined bitstream.
In embodiments of decoding methods or decoders described herein the received bitstream, e.g. received by the NN, may be, for example, used as input of the first subnetwork or network layer of the NN, e.g. bitstream 2401.
In further embodiments of decoding methods or decoders described herein the received bitstream may be, for example, a bitstream formed by or comprising two sub-bitstreams, e.g. sub-bitstreams bitstream1 and bitstream2 (or 2401 and 2403), or more general, a first sub-bitstream and a second sub-bitstream (e.g. each sub-bitstream being received and/or processed by a respective sub-network of the NN). Both sub-bitstreams may be received or stored separately or combined, e.g. multiplexed, as one bitstream, and demultiplexed to obtain the sub-bitstreams.
In even further embodiments of decoding methods or decoders described herein the received bitstream may be, for example, a bitstream formed by or comprising more than two sub-bitstreams, e.g. a first sub-bitstream, a second sub-bitstream, a third subbitstream, and optionally further sub-bitstreams (e.g. each sub-bitstream being received and/or processed by a respective sub-network of the NN). The sub-bitstreams may be received or stored separately or combined, e.g. multiplexed, as one bitstream or more than one combined bitstream, and demultiplexed to obtain the sub-bitstreams.
Mathematical Operators
The mathematical operators used in this application are similar to those used in the C programming language. However, the results of integer division and arithmetic shift operations are defined more precisely, and additional operations are defined, such as exponentiation and real-valued division. Numbering and counting conventions generally begin from 0, e.g., “the first” is equivalent to the 0-th, “the second” is equivalent to the 1-th, etc.
Arithmetic Operators
The following arithmetic operators are defined as follows:
Used to denote division in mathematical equations where no truncation or rounding is intended.
The summation of f(i) with i taking all integer values from x up to and including y.
Logical Operators
The following logical operators are defined as follows:
Relational Operators
The following relational operators are defined as follows:
When a relational operator is applied to a syntax element or variable that has been assigned the value “na” (not applicable), the value “na” is treated as a distinct value for the syntax element or variable. The value “na” is considered not to be equal to any other value.
Bit-Wise Operators
The following bit-wise operators are defined as follows:
Assignment Operators
The following arithmetic operators are defined as follows:
Range Notation
The following notation is used to specify a range of values:
Mathematical Functions
The following mathematical functions are defined:
Order of Operation Precedence
When an order of precedence in an expression is not indicated explicitly by use of parentheses, the following rules apply:
The table below specifies the precedence of operations from highest to lowest; a higher position in the table indicates a higher precedence.
For those operators that are also used in the C programming language, the order of precedence used in this Specification is the same as used in the C programming language.
Text Description of Logical Operations
In the text, a statement of logical operations as would be described mathematically in the following form:
may be described in the following manner:
Each “If . . . Otherwise, if . . . Otherwise, . . . ” statement in the text is introduced with “ . . . as follows” or “ . . . the following applies” immediately followed by “If . . . ”. The last condition of the “If . . . Otherwise, if . . . Otherwise, . . . ” is always an “Otherwise, . . . ”. Interleaved “If . . . Otherwise, if . . . Otherwise, . . . ” statements can be identified by matching “ . . . as follows” or “ . . . the following applies” with the ending “Otherwise, . . . ”.
In the text, a statement of logical operations as would be described mathematically in the following form:
may be described in the following manner:
In the text, a statement of logical operations as would be described mathematically in the following form:
Although embodiments of the invention have been primarily described based on video coding, it should be noted that embodiments of the coding system 10, encoder 20 and decoder 30 (and correspondingly the system 10) and the other embodiments described herein may also be configured for still picture processing or coding, i.e. the processing or coding of an individual picture independent of any preceding or consecutive picture as in video coding. In general only inter-prediction units 244 (encoder) and 344 (decoder) may not be available in case the picture processing coding is limited to a single picture 17. All other functionalities (also referred to as tools or technologies) of the video encoder 20 and video decoder 30 may equally be used for still picture processing, e.g. residual calculation 204/304, transform 206, quantization 208, inverse quantization 210/310, (inverse) transform 212/312, partitioning 262/362, intra-prediction 254/354, and/or loop filtering 220, 320, and entropy coding 270 and entropy decoding 304. In general, the embodiments of the present disclosure may be also applied to other source signals such as an audio signal or the like.
Embodiments, e.g. of the encoder 20 and the decoder 30, and functions described herein, e.g. with reference to the encoder 20 and the decoder 30, may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on a computer-readable medium or transmitted over communication media as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.
By way of example, and not limiting, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.
The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and/or firmware.
This application is a continuation of International Application No. PCT/EP2020/087334, filed on Dec. 18, 2020, the disclosure of which is hereby incorporated by reference in its entirety. The present disclosure relates to a method for encoding a picture using a neural network comprising at least two sub-networks and a method for decoding a picture using a neural network comprising at least two sub-networks. Furthermore, the disclosure presented here refers to an encoder implementing a neural network for encoding a picture and a decoder implementing a neural network for decoding a picture as well as a computer-readable storage medium with computer-executable instructions.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/EP2020/087334 | Dec 2020 | US |
| Child | 18338105 | US |
