Patent Application
20240244274
Embodiments of the present disclosure relate to the field of artificial intelligence (AI)-based video or picture compression technologies, and in particular, to context modelling using an attention layer within a neural network to process elements of a latent tensor.
Video coding (video encoding and decoding) is used in a wide range of digital video applications, for example broadcast digital TV, video transmission over internet and mobile networks, real-time conversational applications such as video chat, video conferencing, DVD and Blu-ray discs, video content acquisition and editing systems, and camcorders of security applications.
The amount of video data needed to depict even a relatively short video can be substantial, which may result in difficulties when the data is to be streamed or otherwise communicated across a communications network with limited bandwidth capacity. Thus, video data is generally compressed before being communicated across modern day telecommunications networks. The size of a video could also be an issue when the video is stored on a storage device because memory resources may be limited. Video compression devices often use software and/or hardware at the source to code the video data prior to transmission or storage, thereby decreasing the quantity of data needed to represent digital video pictures. The compressed data is then received at the destination by a video decompression device that decodes the video data. With limited network resources and ever increasing demands of higher video quality, improved compression and decompression techniques that improve compression ratio with little to no sacrifice in picture quality are desirable.
In recent years, deep learning is gaining popularity in the fields of picture and video encoding and decoding.
The embodiments of the present disclosure provide apparatuses and methods for entropy encoding and decoding of a latent tensor, which includes separating the latent tensor into segments and obtaining a probability model for the entropy encoding of a current element of the latent tensor by processing a set of elements by one or more layers of a neural network including an attention layer.
According to an embodiment a method is provided for entropy encoding of a latent tensor, comprising: separating the latent tensor into a plurality of segments in the spatial dimensions and in the channel dimension, each segment including at least one latent tensor element; processing an arrangement of the plurality of segments by one or more layers of a neural network, including at least one attention layer; and obtaining a probability model for the entropy encoding of a current element of the latent tensor based on the processed plurality of segments.
The method considers spatial correlations in the latent tensor and spatial adaptivity for the implicit entropy estimation. The attention mechanism adaptively weights the importance of the previously coded latent segments. The contribution of the segments to the entropy modeling of the current element corresponds to their respective importance. Thus, the performance of the entropy estimation is improved.
In an exemplary implementation, the separating the latent tensor comprises separating the latent tensor into two or more segments in the channel dimension.
Separating the latent tensor into segments in the channel dimension may enable the use of cross-channel correlations for the context modelling, and thus improving the performance of the entropy estimation.
For example, the processing of the arrangement of the plurality of segments further comprises: obtaining two or more groups of segments; and processing the segments within a group out of the two or more groups independently by the one or more layers of the neural network.
An independent processing of segments within a group facilitates a parallel processing, which may enable a more efficient use of hardware and may reduce the processing time.
In an exemplary implementation, for each group out of the two or more groups, the segments in said group have a same respective channel segment index, said channel segment index indicating the segments the channel dimension.
Segments in a group having a same respective channel segment index facilitate an independent processing of latent tensor elements having different coordinates in the spatial dimensions and a sequential processing of segments of different channels. Thus, spatial and cross-channel correlations are included in the obtained probability model.
For example, the segments having a same channel segment index are grouped into either two groups or four groups.
Obtaining two (four) groups per channel segment index enables a processing that requires at least two (four) sequential steps per channel segment index.
In an exemplary implementation, the segments having a same channel segment index are grouped according to a checkerboard pattern into either two groups or four groups in the spatial dimensions.
A checkerboard pattern causes a uniformly spatial distribution of segments of a group within a plurality of segments having a same channel segment index. This may enhance the spatial correlations in the processing of the segments.
For example, the processing of the arrangement of the plurality of segments further comprises: processing the segments in a first group in parallel, followed by processing the segments in a second group in parallel, wherein the segments in the second group have a same index in the channel dimension as the segments in the first group.
Such an arrangement may improve the performance of the entropy estimation by focusing on spatial correlations due to the related processing order.
In an exemplary implementation, wherein the processing of the arrangement of the plurality of segments further comprises processing the segments in a first group in parallel, followed by processing the segments in a second group in parallel, wherein the segments in the second group have a same spatial coordinate as the corresponding segments in the first group.
Such an arrangement may improve the performance of the entropy estimation by focusing on cross-channel correlations due to the related processing order.
For example, the method further comprises generating a bitstream; and including an indication into said bitstream indicating whether or not two or more groups of segments are obtained.
Such an indication may enable an efficient extraction from the bitstream during decoding.
In an exemplary implementation, the processing of the arrangement further of the plurality of segments comprises separating the segments into a plurality of patches, each patch including two or more segments; and processing each patch independently by the one or more layers of the neural network.
A separation into patches that are processed independently may improve the processing as this facilitates an adaption to hardware properties.
For example, each patch includes a K×M grid of spatially neighboring segments and L neighboring segments in the channel dimension, with L, K and M being positive integers, at least one of K and M being larger than one.
Patches including neighboring segments may enable a processing in a sliding window fashion, thus taking into account spatial correlation in the neighborhood of a currently processed segment.
In an exemplary implementation, the patches out of the plurality of patches are overlapping in the spatial dimensions and/or in the channel dimension.
Overlapping patches may improve the obtained probability model by taking into account additional spatial and/or cross-channel correlations.
For example, the method further comprises generating a bitstream; and including an indication into said bitstream indicating one or more of whether or not the patches out of the plurality of patches are overlapping, and an amount of overlap.
Such an indication may enable an efficient extraction from the bitstream during decoding.
In an exemplary implementation, the processing of a patch out of the plurality of patches by the one or more layers of the neural network further comprises obtaining two or more groups of segments within said patch; and processing the segments within a group out of the two or more groups independently by the one or more layers of the neural network.
A combination of an independent processing of patches and an independent processing of segments in a group within a patch may reduce the number of sequential steps in the processing of a patch and thus the overall processing time.
For example, the processing of the arrangement comprises arranging the segments in a predefined order, wherein segments with a same spatial coordinate are grouped together.
Such an arrangement may improve the performance of the entropy estimation by focusing on cross-channel correlations due to the related processing order.
In an exemplary implementation, the processing of the arrangement comprises arranging the segments, wherein segments, which have different spatial coordinates, are arranged consecutively in a predefined order.
Such an arrangement may improve the performance of the entropy estimation by focusing on spatial correlations due to the related processing order.
For example, the processing by the neural network comprises applying a first neural subnetwork to extract features of the plurality of segments, and providing an output of the first neural subnetwork as an input to a subsequent layer within the neural network.
Processing the input of the neural network to extract the features of the plurality of segments may enable a focus of the attention layer on independent deep features of the input.
In an exemplary implementation, the processing by the neural network further comprises: providing positional information of the plurality of segments as an input to the at least one attention layer.
The positional encodings may enable the attention layer to utilize the sequential order of the input sequence.
In an exemplary implementation, the processing an arrangement of the plurality of segments includes selecting a subset of segments from said plurality of segments; and said subset is provided as an input to a subsequent layer within the neural network.
Selecting a subset of segments may enable support for latent tensors of larger sizes by requiring a reduced size of memory and/or a reduced amount of processing.
For example, the processing by the at least one attention layer in the neural network further comprises: applying a mask, which masks elements in an attention tensor following the current element within a processing order of the latent tensor.
Applying a mask ensures, that only previously encoded elements may be processed and thus the coding order is preserved. The mask mirrors the availability of information at the decoding side to the encoding side.
In an exemplary implementation, the neural network includes a second neural subnetwork, the second neural subnetwork processing an output of the attention layer.
The neural subnetwork may process the features outputted by the attention layer to provide probabilities for the symbols used in the encoding and thus enabling an efficient encoding and/or decoding.
For example, at least one of the first neural subnetwork and the second neural subnetwork is a multilayer perceptron.
A multilayer perceptron may provide an efficient implementation of a neural network.
In an exemplary implementation, the at least one attention layer in the neural network is a multi-head attention layer.
A multi-head attention layer may improve the estimation of probabilities by processing different representations of the input in parallel and thus providing more projections and attention computations, which corresponds to various perspectives of the same input.
For example, the at least one attention layer in the neural network is included in a transformer subnetwork.
A transformer subnetwork may provide an efficient implementation of an attention mechanism.
In an exemplary implementation, the method is further comprising: padding the beginning of the arrangement of the plurality of segments with a zero segment before processing by the neural network.
A padding with zeros at the beginning of the arrangement mirrors the availability of information at the decoding side and thus causality in the coding order is preserved.
For example, the method is further comprising: entropy encoding the current element into a first bitstream using the obtained probability model.
Using the probability model obtained by processing the plurality of segments by a neural network including an attention layer may reduce the size of the bitstream.
In an exemplary implementation, the method is further comprising: quantizing the latent tensor before separating into segments.
A quantized latent tensor yields a simplified probability model, thus enabling a more efficient encoding process. Also, such latent tensor is compressed and can be processed with reduced complexity and represented more efficiently within the bitstream.
For example, the quantizing of the latent tensor comprises a Rate-Distortion Optimized Quantization (RDOQ) including the obtained probability model.
Optimizing the quantization including the obtained probability model may yield a more efficient representation of the latent tensor within the bitstream.
For example, the method is further comprising selecting the probability model for the entropy encoding according to: computational complexity and/or properties of the first bitstream.
Enabling the selection of the context modelling strategy may allow for better performance during the encoding process and may provide flexibility in adapting the encoded bitstream to the desired application.
In an exemplary implementation, the method is further comprising: hyper-encoding the latent tensor obtaining a hyper-latent tensor; entropy encoding the hyper-latent tensor into a second bitstream; entropy decoding the second bitstream; and obtaining a hyper-decoder output by hyper-decoding the hyper-latent tensor.
Introducing a hyper-prior model may further improve the probability model and thus the coding rate by determining further redundancy in the latent tensor.
For example, the method is further comprising: separating the hyper-decoder output into a plurality of hyper-decoder output segments, each hyper-decoder output segments including one or more hyper-decoder output elements; for each segment out of the plurality of segments, concatenating said segment and a set of hyper-decoder output segments out of the plurality of hyper-decoder output segments before obtaining the probability model.
The probability model may be further improved by concatenating the hyper-decoder output with a respective segment out of the plurality of segments.
In an exemplary implementation, the set of hyper-decoder output segments to be concatenated with a respective segment includes one or more of a hyper-decoder output segment corresponding to said respective segment, or a plurality of hyper-decoder output segments corresponding to a same channel as said respective segment, or a plurality of hyper-decoder output segments spatially neighboring said respective segment, or a plurality of hyper-decoder output segments including neighboring segments spatially neighboring said respective segment and segments corresponding to a same channel as said neighboring segment.
The probability model may be further improved by including a respective set of hyper-decoder output segments. The behavior for performance and complexity may depend on the set of hyper-decoder output segments and the content to be encoded.
For example, the method is further comprising adaptively selecting the set of hyper-decoder output segments according to: computational complexity and/or properties of the first bitstream.
Enabling the selection of additional hyper-prior modelling strategy may allow for better performance during the encoding process and may provide flexibility in adapting the encoded bitstream to the desired application.
In an exemplary implementation, one or more of the following steps are performed in parallel for each segment out of the plurality of segments: processing by the neural network, and entropy encoding the current element.
A parallel processing of the segments may result in a faster encoding into the bitstream.
According to an embodiment, a method is provided for encoding image data comprising: obtaining a latent tensor by processing the image data with an autoencoding convolutional neural network; and entropy encoding the latent tensor into a bitstream using a generated probability model according to any of the methods described above.
The entropy coding may be readily and advantageously applied to image encoding, to effectively reduce the data rate, e.g. when transmission or storage of pictures or videos is desired, as the latent tensors for image reconstruction may still have considerable size.
According to an embodiment, a method is provided for entropy decoding of a latent tensor, comprising: initializing the latent tensor with zeroes; separating the latent tensor into a plurality of segments in the spatial dimensions and in the channel dimension, each segment including at least one latent tensor element; processing an arrangement of the plurality of segments by one or more layers of a neural network, including at least one attention layer; and obtaining a probability model for the entropy decoding of a current element of the latent tensor based on the processed plurality of segments.
The method considers spatial correlations in the latent tensor and spatial adaptivity for the implicit entropy estimation. The attention mechanism adaptively weights the importance of the previously coded latent segments. The contribution of the segments to the entropy modeling of the current element corresponds to their respective importance. Thus, the performance of the entropy estimation is improved.
In an exemplary implementation, the separating the latent tensor comprises separating the latent tensor into two or more segments in the channel dimension.
Separating the latent tensor into segments in the channel dimension may enable the use of cross-channel correlations for the context modelling, and thus improving the performance of the entropy estimation.
For example, the processing of the arrangement of the plurality of segments further comprises: obtaining two or more groups of segments; and processing the segments within a group out of the two or more groups independently by the one or more layers of the neural network.
An independent processing of segments within a group facilitates a parallel processing, which may enable a more efficient use of hardware and may reduce the processing time.
In an exemplary implementation, for each group out of the two or more groups, the segments in said group have a same respective channel segment index, said channel segment index indicating the segments the channel dimension.
Segments in a group having a same respective channel segment index facilitate an independent processing of latent tensor elements having different coordinates in the spatial dimensions and a sequential processing of segments of different channels. Thus, spatial and cross-channel correlations are included in the obtained probability model.
For example, the segments having a same channel segment index are grouped into either two groups or four groups.
Obtaining two (four) groups per channel segment index enables a processing that requires at least two (four) sequential steps per channel segment index.
In an exemplary implementation, the segments having a same channel segment index are grouped according to a checkerboard pattern into either two groups or four groups in the spatial dimensions.
A checkerboard pattern causes a uniformly spatial distribution of segments of a group within a plurality of segments having a same channel segment index. This may enhance the spatial correlations in the processing of the segments.
For example, the processing of the arrangement of the plurality of segments further comprises: processing the segments in a first group in parallel, followed by processing the segments in a second group in parallel, wherein the segments in the second group have a same index in the channel dimension as the segments in the first group.
Such an arrangement may improve the performance of the entropy estimation by focusing on spatial correlations due to the related processing order.
In an exemplary implementation, wherein the processing of the arrangement of the plurality of segments further comprises processing the segments in a first group in parallel, followed by processing the segments in a second group in parallel, wherein the segments in the second group have a same spatial coordinate as the corresponding segments in the first group.
Such an arrangement may improve the performance of the entropy estimation by focusing on cross-channel correlations due to the related processing order.
For example, the method further comprises receiving a bitstream; and obtaining an indication from said bitstream indicating whether or not two or more groups of segments are obtained.
Such an indication may enable an efficient extraction from the bitstream during decoding.
In an exemplary implementation, the processing of the arrangement further of the plurality of segments comprises separating the segments into a plurality of patches, each patch including two or more segments; and processing each patch independently by the one or more layers of the neural network.
A separation into patches that are processed independently may improve the processing as this facilitates an adaption to hardware properties.
For example, each patch includes a K×M grid of spatially neighboring segments and L neighboring segments in the channel dimension, with L, K and M being positive integers, at least one of K and M being larger than one.
Patches including neighboring segments may enable a processing in a sliding window fashion, thus taking into account spatial correlation in the neighborhood of a currently processed segment.
In an exemplary implementation, the patches out of the plurality of patches are overlapping in the spatial dimensions and/or in the channel dimension.
Overlapping patches may improve the obtained probability model by taking into account additional spatial and/or cross-channel correlations.
For example, the method further comprises receiving a bitstream; and obtaining an indication from said bitstream indicating one or more of whether or not the patches out of the plurality of patches are overlapping, and an amount of overlap.
Such an indication may enable an efficient extraction from the bitstream during decoding.
In an exemplary implementation, the processing of a patch out of the plurality of patches by the one or more layers of the neural network further comprises obtaining two or more groups of segments within said patch; and processing the segments within a group out of the two or more groups independently by the one or more layers of the neural network.
A combination of an independent processing of patches and an independent processing of segments in a group within a patch may reduce the number of sequential steps in the processing of a patch and thus the overall processing time.
For example, the processing of the arrangement comprises arranging the segments in a predefined order, wherein segments with a same spatial coordinate are grouped together.
Such an arrangement may improve the performance of the entropy estimation by focusing on cross-channel correlations due to the related processing order.
In an exemplary implementation, the processing of the arrangement comprises arranging the segments, wherein segments, which have different spatial coordinates, are arranged consecutively in a predefined order.
Such an arrangement may improve the performance of the entropy estimation by focusing on spatial correlations due to the related processing order.
For example, the processing by the neural network comprises applying a first neural subnetwork to extract features of the plurality of segments, and providing an output of the first neural subnetwork as an input to a subsequent layer within the neural network.
Processing the input of the neural network to extract the features of the plurality of segments may enable a focus of the attention layer on independent deep features of the input.
In an exemplary implementation, the processing by the neural network further comprises: providing positional information of the plurality of segments as an input to the at least one attention layer.
The positional encodings may enable the attention layer to utilize the sequential order of the input sequence.
In an exemplary implementation, the processing an arrangement of the plurality of segments includes selecting a subset of segments from said plurality of segments; and said subset is provided as an input to a subsequent layer within the neural network.
Selecting a subset of segments may enable support for latent tensors of larger sizes by requiring a reduced size of memory and/or a reduced amount of processing.
In an exemplary implementation, the neural network includes a second neural subnetwork, the second neural subnetwork processing an output of the attention layer.
The neural subnetwork may process the features outputted by the attention layer to provide probabilities for the symbols used in the encoding and thus enabling an efficient encoding and/or decoding.
For example, at least one of the first neural subnetwork and the second neural subnetwork is a multilayer perceptron.
A multilayer perceptron may provide an efficient implementation of a neural network.
In an exemplary implementation, the at least one attention layer in the neural network is a multi-head attention layer.
A multi-head attention layer may improve the estimation of probabilities by processing different representations of the input in parallel and thus providing more projections and attention computations, which corresponds to various perspectives of the same input.
For example, the at least one attention layer in the neural network is included in a transformer subnetwork.
A transformer subnetwork may provide an efficient implementation of an attention mechanism.
In an exemplary implementation, the method is further comprising: padding the beginning of the arrangement of the plurality of segments with a zero segment before processing by the neural network.
A padding with zeros at the beginning of the arrangement mirrors the availability of information at the decoding side and thus causality in the coding order is preserved.
For example, the method is further comprising: entropy decoding the current element into a first bitstream using the obtained probability model.
Using the probability model obtained by processing the plurality of segments by a neural network including an attention layer may reduce the size of the bitstream.
For example, the method is further comprising selecting the probability model for the entropy encoding according to: computational complexity and/or properties of the first bitstream.
Enabling the selection of the context modelling strategy may allow for better performance during the encoding process and may provide flexibility in adapting the encoded bitstream to the desired application.
In an exemplary implementation, the method is further comprising: entropy decoding a hyper-latent tensor from a second bitstream; and obtaining a hyper-decoder output by hyper-decoding the hyper-latent tensor.
Introducing a hyper-prior model may further improve the probability model and thus the coding rate by determining further redundancy in the latent tensor.
For example, the method is further comprising: separating the hyper-decoder output into a plurality of hyper-decoder output segments, each hyper-decoder output segments including one or more hyper-decoder output elements; for each segment out of the plurality of segments, concatenating said segment and a set of hyper-decoder output segments out of the plurality of hyper-decoder output segments before obtaining the probability model.
The probability model may be further improved by concatenating the hyper-decoder output with a respective segment out of the plurality of segments.
In an exemplary implementation, the set of hyper-decoder output segments to be concatenated with a respective segment includes one or more of: a hyper-decoder output segment corresponding to said respective segment, or a plurality of hyper-decoder output segments corresponding to a same channel as said respective segment, or a plurality of hyper-decoder output segments spatially neighboring said respective segment, or a plurality of hyper-decoder output segments including neighboring segments spatially neighboring said respective segment and segments corresponding to a same channel as said neighboring segment.
The probability model may be further improved by including a respective set of hyper-decoder output segments. The behavior for performance and complexity may depend on the set of hyper-decoder output segments and the content to be encoded.
For example, the method is further comprising adaptively selecting the set of hyper-decoder output segments according to: computational complexity and/or properties of the first bitstream.
Enabling the selection of additional hyper-prior modelling strategy may allow for better performance during the encoding process and may provide flexibility in adapting the encoded bitstream to the desired application.
According to an embodiment, a method is provided for decoding image data comprising: entropy decoding a latent tensor from a bitstream according to any of the methods described above; and obtaining the image data by processing the latent tensor with an autodecoding convolutional neural network.
The entropy decoding may be readily and advantageously applied to image decoding, to effectively reduce the data rate, e.g. when transmission or storage of pictures or videos is desired, as the latent tensors for image reconstruction may still have considerable size.
In an exemplary implementation, a computer program stored on a non-transitory medium and including code instructions, which, when executed on one or more processors, cause the one or more processors to execute steps of the method according to any of the methods described above.
According to an embodiment, an apparatus is provided for entropy encoding of a latent tensor, comprising: processing circuitry configured to: separate the latent tensor into a plurality of segments in the spatial dimensions and in the channel dimension, each segment including at least one latent tensor element; process an arrangement of the plurality of segments by one or more layers of a neural network, including at least one attention layer; and obtain a probability model for the entropy encoding of a current element of the latent tensor based on the processed plurality of segments.
According to an embodiment, an apparatus is provided for entropy decoding of a latent tensor, comprising: processing circuitry configured to: initialize the latent tensor with zeroes; separate the latent tensor into a plurality of segments in the spatial dimensions and in the channel dimension, each segment including at least one latent tensor element; process an arrangement of the plurality of segments by one or more layers of a neural network, including at least one attention layer; and obtain a probability model for the entropy decoding of a current element of the latent tensor based on the processed plurality of segments.
The apparatuses provide the advantages of the methods described above.
Embodiments can be implemented in hardware (HW) and/or software (SW) or in any combination thereof. Moreover, HW-based implementations may be combined with SW-based implementations.
Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.
In the following, embodiments are described in more detail with reference to the attached figures and drawings, in which:
In the following description, reference is made to the accompanying figures, which form part of the disclosure, and which show, by way of illustration, specific aspects of embodiments or specific aspects in which embodiments may be used. It is understood that embodiments may be used in other aspects and comprise structural or logical changes not depicted in the figures. The following detailed description, therefore, is not to be taken in a limiting sense.
For instance, it is understood that a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa. For example, if one or a plurality of specific method steps is described, a corresponding device may include one or a plurality of units, e.g. functional units, to perform the described one or plurality of method steps (e.g. one unit performing the one or plurality of steps, or a plurality of units each performing one or more of the plurality of steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, for example, if a specific apparatus is described based on one or a plurality of units, e.g. functional units, a corresponding method may include one step to perform the functionality of the one or plurality of units (e.g. one step performing the functionality of the one or plurality of units, or a plurality of steps each performing the functionality of one or more of the plurality of units), even if such one or plurality of steps are not explicitly described or illustrated in the figures. Further, it is understood that the features of the various exemplary embodiments and/or aspects described herein may be combined with each other, unless specifically noted otherwise.
In image/video compression algorithms, entropy estimation is one of the components, which provides a significant gain. The entropy estimation includes, for example, explicit entropy estimation and/or implicit entropy estimation. Explicit entropy estimation may be realized by a hyper-prior that compresses the entropy estimation parameters and sends the side-information over a second bitstream. Implicit entropy estimation may use already decoded elements of a first bitstream and includes said decoded elements in the entropy estimation of the primary bitstream while considering the causality of the coding order. The implicit entropy estimation is usually called autoregressive context model and may be typically implemented by two-dimensional (2D) masked convolutions. However, the 2D masked convolutions provide a finite small support. This limits the performance of the implicit entropy model, since the long-distant dependencies are not considered.
Additionally, once trained, convolutional kernels are inherently not adaptive to the bitstream properties, i.e., latent tensor elements. The same kernel is applied to every position in the compressed bitstream, and therefore, convolutional kernels are location-agnostic. This limits the performance of the implicit model since only location-bounded dependencies can be learned. Even when the kernel size of the masked convolutions is increased, the performance of the implicit model increases marginally, since a fixed set of location-bounded internal relations between the previously coded elements can be exploited due to the non-adaptivity.
Furthermore, an implicit model with 2D masked convolutions encodes/decodes all channels of the latent tensor at once and does not exploit any cross-channel correlations. Since there is no channel-wise autoregression, no channel element of the currently coded latent element has access to the information of the other spatially-co-located elements with different channel index. The lack of channel-wise autoregression also causes performance degradation.
An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner. A schematic drawing thereof is shown in
This image h is usually referred to as code, latent variables, or latent representation. Here, σ is an element-wise activation function such as a sigmoid function or a rectified linear unit. W is a weight matrix b is a bias vector. Weights and biases are usually initialized randomly, and then updated iteratively during training through Backpropagation. After that, the decoder stage of the autoencoder maps h to the reconstruction x′ of the same shape as x:
where σ′, W′ and b′ for the decoder may be unrelated to the corresponding σ, W and b for the encoder.
Variational autoencoder models make strong assumptions concerning the distribution of latent variables. They use a variational approach for latent representation learning, which results in an additional loss component and a specific estimator for the training algorithm called the Stochastic Gradient Variational Bayes (SGVB) estimator. It assumes that the data is generated by a directed graphical model pθ(x|h) and that the encoder is learning an approximation qϕ(h|x) to the posterior distribution pθ(h|x) where ϕ and θ denote the parameters of the encoder (recognition model) and decoder (generative model) respectively. The probability distribution of the latent vector of a VAE typically matches that of the training data much closer than a standard autoencoder.
Recent progress in artificial neural networks area and especially in convolutional neural networks enables researchers' interest of applying neural networks based technologies to the task of image and video compression. For example, End-to-end Optimized Image Compression has been proposed, which uses a network based on a variational autoencoder.
Accordingly, data compression is considered as a fundamental and well-studied problem in engineering, and is commonly formulated with the goal of designing codes for a given discrete data ensemble with minimal entropy. The solution relies heavily on knowledge of the probabilistic structure of the data, and thus the problem is closely related to probabilistic source modeling. However, since all practical codes must have finite entropy, continuous-valued data (such as vectors of image pixel intensities) must be quantized to a finite set of discrete values, which introduces an error.
In this context, known as the lossy compression problem, a trade off between two competing costs is present: the entropy of the discretized representation (rate) and the error arising from the quantization (distortion). Different compression applications, such as data storage or transmission over limited-capacity channels, demand different rate-distortion trade-offs.
Joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in high-dimensional spaces is intractable. For this reason, most existing image compression methods operate by linearly transforming the data vector into a suitable continuous-valued representation, quantizing its elements independently, and then encoding the resulting discrete representation using a lossless entropy code. This scheme is called transform coding due to the central role of the transformation.
For example, JPEG uses a discrete cosine transform on blocks of pixels, and JPEG 2000 uses a multi-scale orthogonal wavelet decomposition. Typically, the three components of transform coding methods—transform, quantizer, and entropy code—are separately optimized (often through manual parameter adjustment). Modern video compression standards like HEVC, VVC and EVC also use transformed representation to code residual signal after prediction. The several transforms are used for that purpose such as discrete cosine and sine transforms (DCT, DST), as well as low frequency non-separable manually optimized transforms (LFNST).
Variable Auto-Encoder (VAE) framework can be considered as a nonlinear transforming coding model. The transforming process can be mainly divided into four parts:
The input image 311 to be compressed is represented as a 3D tensor with the size of H×W×C, where H and W are the height and width of the image and C is the number of color channels. In a first step, the input image is passed through the encoder 310. The encoder 310 down-samples the input image 311 by applying multiple convolutions and non-linear transformations, and produces a latent-space feature tensor (latent tensor in the following) y. (While this is not a re-sampling in the classical sense, in deep learning down and up-sampling are common terms for changing the size of height and width of the tensor). The latent tensor y 4020 corresponding to the input image 4010, shown exemplarily in
whereas De is the down-sampling factor of the encoder and Ce is the number of channels.
The difference between the pixels of an input/output image and the latent tensor are shown in
The latent space can be understood as a representation of compressed data in which similar data points are closer together in the latent space. Latent space is useful for learning data features and for finding simpler representations of data for analysis. The quantizer 330 transforms the latent representation y into the quantized latent representation ŷ with (discrete) values by ŷ=Q(y), with Q representing the quantizer function.
The entropy estimation of the latent tensor y may be improved by additionally applying an optional hyper-prior model.
In the first step of obtaining a hyper-prior model, a hyper-encoder 320 is applied to the latent tensor y, which down-samples the latent tensor with convolutions and non-linear transforms into a hyper-latent tensor z. The latent tensor z has the size of
In the next step, a quantization 331 may be performed on the latent tensor z. A factorized entropy model 342 produces an estimation of the statistical properties of the quantized hyper-latent tensor {circumflex over (z)}. An arithmetic encoder uses these statistical properties to create a bitstream representation 141 of the tensor {circumflex over (z)}. All elements of tensor {circumflex over (z)} are written into the bitstream without the need of an autoregressive process.
The factorized entropy model 342 works as a codebook whose parameters are available on the decoder side. An entropy decoder 343 recovers the quantized hyper-latent tensor from the bitstream 341 by using the factorized entropy model 342. The recovered quantized hyper-latent tensor is up-sampled in the hyper-decoder 350 by applying multiple convolution operations and non-linear transformations. The hyper-decoder output tensor 430 is denoted by ψ.
The hyper-encoder/decoder (also known as hyper-prior) 330-350 estimates the distribution of the quantized latent representation y to get the minimum rate achievable with a lossless entropy source coding. Furthermore, a decoder 380 is provided that transforms the quantized latent representation to the reconstructed image {circumflex over (x)} 381, {circumflex over (x)}=g(ŷ). The signal {circumflex over (x)} is the estimation of the input image x. It is desirable that x is as close to 5 as possible, in other words the reconstruction quality is as high as possible. However, the higher the similarity between {circumflex over (x)} and x, the higher the amount of side information necessary to be transmitted. The side information includes y bitstream and z bitstream shown in
In
The arithmetic decoding (AD) 372 is the process of reverting the binarization process, where binary digits are converted back to sample values. The arithmetic decoding is provided by the arithmetic decoding module 372.
In
The first subnetwork is responsible for:
The purpose of the second subnetwork is to obtain statistical properties (e.g. mean value, variance and correlations between samples of y bitstream) of the samples of “y bitstream”, such that the compressing of y bitstream by first subnetwork is more efficient. The second subnetwork generates a second bitstream “z bitstream”, which comprises the said information (e.g. mean value, variance and correlations between samples of y bitstream).
The second network includes an encoding part which comprises transforming 330 of the quantized latent representation ŷ into side information z, quantizing the side information z into quantized side information {circumflex over (z)}, and encoding (e.g. binarizing) 340 the quantized side information {circumflex over (z)} into z bitstream. In this example, the binarization is performed by an arithmetic encoding (AE). A decoding part of the second network includes arithmetic decoding (AD) 343, which transforms the input z bitstream into decoded quantized side information {circumflex over (z)}′. The {circumflex over (z)}′ might be identical to {circumflex over (z)}, since the arithmetic encoding and decoding operations are lossless compression methods. The decoded quantized side information {circumflex over (z)}′ is then transformed 350 into decoded side information ŷ′. ŷ′ represents the statistical properties of ŷ (e.g. mean value of samples of ŷ, or the variance of sample values or like). The decoded latent representation ŷ′ is then provided to the above-mentioned Arithmetic Encoder 370 and Arithmetic Decoder 372 to control the probability model of ŷ.
The
Similarly, in
Specifically, as is seen in
The output of the arithmetic encoding module is the y bitstream. The y bitstream and z bitstream are the output of the encoding of the signal, which are then provided (transmitted) to the decoding process. Although the unit 310 is called “encoder”, it is also possible to call the complete subnetwork described in
The remaining parts in the figure (quantization unit, hyper-encoder, hyper-decoder, arithmetic encoder/decoder) are all parts that either improve the efficiency of the encoding process or are responsible for converting the compressed output y into a series of bits (bitstream). Quantization may be provided to further compress the output of the NN encoder 310 by a lossy compression. The AE 370 in combination with the hyper-encoder 330 and hyper-decoder 350 used to configure the AE 370 may perform the binarization, which may further compress the quantized signal by a lossless compression. Therefore, it is also possible to call the whole subnetwork in
A majority of Deep Learning (DL) based image/video compression systems reduce dimensionality of the signal before converting the signal into binary digits (bits). In the VAE framework for example, the encoder, which is a non-linear transform, maps the input image x into y, where y has a smaller width and height than x. Since the y has a smaller width and height, hence a smaller size, the (size of the) dimension of the signal is reduced, and, hence, it is easier to compress the signal y. It is noted that in general, the encoder does not necessarily need to reduce the size in both (or in general all) dimensions. Rather, some exemplary implementations may provide an encoder which reduces size only in one (or in general a subset of) dimension.
The arithmetic encoder and decoder are specific implementations of entropy coding. AE and AD can be replaced by any other means of entropy coding. Also, the quantization operation and a corresponding quantization unit is not necessarily present and/or can be replaced with another unit.
Artificial neural networks (ANN) or connectionist systems are computing systems vaguely inspired by the biological neural networks that constitute animal brains. Such systems “learn” to perform tasks by considering examples, generally without being programmed with task-specific rules. For example, in image recognition, they might learn to identify images that contain cats by analyzing example images that have been manually labeled as “cat” or “no cat” and using the results to identify cats in other images. They do this without any prior knowledge of cats, for example, that they have fur, tails, whiskers and cat-like faces. Instead, they automatically generate identifying characteristics from the examples that they process.
An ANN is based on a collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain. Each connection, like the synapses in a biological brain, can transmit a signal to other neurons. An artificial neuron that receives a signal then processes it and can signal neurons connected to it.
In ANN implementations, the “signal” at a connection is a real number, and the output of each neuron is computed by some non-linear function of the sum of its inputs. The connections are called edges. Neurons and edges typically have a weight that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Neurons may have a threshold such that a signal is sent only if the aggregate signal crosses that threshold. Typically, neurons are aggregated into layers. Different layers may perform different transformations on their inputs. Signals travel from the first layer (the input layer), to the last layer (the output layer), possibly after traversing the layers multiple times.
The original goal of the ANN approach was to solve problems in the same way that a human brain would. Over time, attention moved to performing specific tasks, leading to deviations from biology. ANNs have been used on a variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, playing board and video games, medical diagnosis, and even in activities that have traditionally been considered as reserved to humans, like painting.
The name “convolutional neural network” (CNN) indicates that the network employs a mathematical operation called convolution. Convolution is a specialized kind of linear operation. Convolutional networks are neural networks that use convolution in place of a general matrix multiplication in at least one of their layers.
When programming a CNN for processing images, as shown in
In the past, traditional multilayer perceptron (MLP) models have been used for image recognition. However, due to the full connectivity between nodes, they suffered from high dimensionality, and did not scale well with higher resolution images. A 1000×1000-pixel image with RGB color channels has 3 million weights, which is too high to feasibly process efficiently at scale with full connectivity. In addition, such network architecture does not take into account the spatial structure of data, treating input pixels, which are far apart in the same way as pixels that are close together. This ignores locality of reference in image data, both computationally and semantically. Thus, full connectivity of neurons is wasteful for purposes such as image recognition that are dominated by spatially local input patterns.
Convolutional neural networks are biologically inspired variants of multilayer perceptrons that are specifically designed to emulate the behavior of a visual cortex. These models mitigate the challenges posed by the MLP architecture by exploiting the strong spatially local correlation present in natural images. The convolutional layer is the core building block of a CNN. The layer's parameters consist of a set of learnable filters (the above-mentioned kernels), which have a small receptive field, but extend through the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the entries of the filter and the input and producing a 2-dimensional activation map of that filter. As a result, the network learns filters that activate when it detects some specific type of feature at some spatial position in the input.
Stacking the activation maps for all filters along the depth dimension forms the full output volume of the convolution layer. Every entry in the output volume can thus also be interpreted as an output of a neuron that looks at a small region in the input and shares parameters with neurons in the same activation map. A feature map, or activation map, is the output activations for a given filter. Feature map and activation has same meaning. In some papers, it is called an activation map because it is a mapping that corresponds to the activation of different parts of the image, and also a feature map because it is also a mapping of where a certain kind of feature is found in the image. A high activation means that a certain feature was found.
Another important concept of CNNs is pooling, which is a form of non-linear down-sampling. There are several non-linear functions to implement pooling among which max pooling is the most common. It partitions the input image into a set of non-overlapping rectangles and, for each such sub-region, outputs the maximum.
Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. The pooling layer serves to progressively reduce the spatial size of the representation, to reduce the number of parameters, memory footprint and amount of computation in the network, and hence to also control overfitting. It is common to periodically insert a pooling layer between successive convolutional layers in a CNN architecture. The pooling operation provides another form of translation invariance.
The pooling layer operates independently on every depth slice of the input and resizes it spatially. The most common form is a pooling layer with filters of size 2×2 applied with a stride of 2 at every depth slice in the input by 2 along both width and height, discarding 75% of the activations. In this case, every max operation is over 4 numbers. The depth dimension remains unchanged. In addition to max pooling, pooling units can use other functions, such as average pooling or l2-norm pooling. Average pooling was often used historically but has recently fallen out of favor compared to max pooling, which often performs better in practice. Due to the aggressive reduction in the size of the representation, there is a recent trend towards using smaller filters or discarding pooling layers altogether. “Region of Interest” pooling (also known as ROI pooling) is a variant of max pooling, in which output size is fixed and input rectangle is a parameter. Pooling is an important component of convolutional neural networks for object detection based on Fast R-CNN architecture.
The above-mentioned ReLU is the abbreviation of rectified linear unit, which applies the non-saturating activation function. It effectively removes negative values from an activation map by setting them to zero. It increases the nonlinear properties of the decision function and of the overall network without affecting the receptive fields of the convolution layer. Other functions are also used to increase nonlinearity, for example the saturating hyperbolic tangent and the sigmoid function. ReLU is often preferred to other functions because it trains the neural network several times faster without a significant penalty to generalization accuracy.
After several convolutional and max pooling layers, the high-level reasoning in the neural network is done via fully connected layers. Neurons in a fully connected layer have connections to all activations in the previous layer, as seen in regular (non-convolutional) artificial neural networks. Their activations can thus be computed as an affine transformation, with matrix multiplication followed by a bias offset (vector addition of a learned or fixed bias term).
The “loss layer” (including calculating of a loss function) specifies how training penalizes the deviation between the predicted (output) and true labels and is normally the final layer of a neural network. Various loss functions appropriate for different tasks may be used. Softmax loss is used for predicting a single class of K mutually exclusive classes. Sigmoid cross-entropy loss is used for predicting K independent probability values in [0, 1]. Euclidean loss is used for regressing to real-valued labels.
In summary,
Attention mechanism is a deep learning technique, which allows neural network to enhance or to focus on important parts of the input and fade out unrelated parts. This simple yet powerful concept can be applied, for example, in any of the fields of natural language processing, recommendation, healthcare analytics, image processing, or speech recognition.
Originally, attention is computed over the entire input sequence (global attention). Despite its simplicity, such an approach may be computationally expensive. Using local attention may be a solution.
One exemplary implementation of the attention-mechanism is the so-called transformer model. A transformer model applies an attention layer followed by a feed-forward neural network. Two exemplary implementations of a transformer block are shown in
When the positional encoding 5020 is calculated, it is piece-wisely added to the embedded vectors 5010. Then the input vectors are prepared to enter the transformer block. The exemplary implementing of a transformer block in
In a self-attention module, all three vectors come from the same sequence and represent vectors with positional encoding built in.
A generic attention mechanism, which consists of the queries Q 620, keys K 621 and values V 622, is shown exemplarily in
After combining the embedding vector with the positional encodings pe, three different representations namely Queries Q, Keys K and Values V are obtained by a feed-forward neural network layers. Queries, Keys and Values have the size of S×dq, S×dk and S×dv respectively. Typically, Queries, Keys and Values may have the same dimensionality d. In order to calculate self-attention, first the scaled dot-product between Queries and Keys
may be calculated, and the softmax function may be applied to obtain attention scores. Next, these scores are multiplied with the Values to achieve self-attention. The self-attention mechanism can be formulated as:
where the dk is the dimensionality of the keys and softmax
has the size of S×S.
Afterwards, the calculated attention is added to the embedding vector by creating a form of residual connection, and normalized with a normalization layer. Finally, a multilayer feed-forward neural network (a.k.a. multilayer perceptron) with a residual connection is applied and final output is normalized. All the above mentioned steps (after creating the embedding tensor) describes one layer of the transformer which can be repeated for L times to create a transformer network with L layers.
In other words, an attention layer obtains a plurality of representations of an input sequence, for example the Keys, Queries and Values. To obtain a representation out of said plurality representations, the input sequence is processed by a respective set of weights. This set of weights may be obtained in a training phase. These set of weights may be learned jointly with the remaining parts of a neural network including such an attention layer. During inference, the output is computed as the weighted sum of the processed input sequence.
One extension to the above mentioned attention mechanism is the multi-head attention. In this version, the final dimension of the queries, keys and values are split into h sub-representations, and for each sub-representation the attention is calculated separately. The final attention is calculated by concatenating each sub-attention and feeding it to a feed forward neural network (FFN). The formulation of the multi-head attention is given as:
where Fi, Ki and Vi have the size of
respectively.
The multi-head attention enables parallelization and each embedding tensor to have multiple representations.
A single attention function is illustrated in
The exemplary single attention function in
The next step after Multi-Head attention in the transformer block is a simple position-wise fully connected feed-forward network. There is a residual connection around each block, which is followed by a layer normalization. The residual connections help the network to keep track of data it looks at. Layer normalization plays a role in reducing features variance.
There are several different architecture for transformers in the literature, which may differ in the order and type of its components. However, the fundamental logic is similar where some type of attention mechanism followed by another neural network encapsulates a layer of a transformer layer and multiple layers of this architecture forms a transformer network. Two examples are given in
The process of obtaining a context model by applying a neural network including an attention layer is shown exemplarily in
Image data to be compressed may be represented as a three-dimensional tensor 311 with the size of H×W×C where H and W are the height and width of the image and C is the number of color channels. The input image may be processed by an autoencoding convolutional neural network 310 as explained above with reference to
whereas De is the down-sampling factor of the encoder 310 and Ce is the number of channels. The obtained latent tensor may be encoded into a bitstream 371 using a probability model generated by attention based context modelling.
The latent tensor y may be quantized. The quantizing may be performed by the quantizing block 320. Such quantizing may be a Rate-Distortion Optimized Quantization (RDOQ) including the probability model generated by attention based context modelling.
A context model for an entropy encoding of a latent tensor may be determined by applying an attention layer 732. The latent-space feature tensor, which includes one or more elements, is separated 700 into a plurality of segments 820 in the spatial dimensions, as shown in
The latent tensor to be entropy encoded may arise from image data processed by an autoencoding neural network as explained above. However, the present disclosure is not limited to image data from autoencoders. The latent tensor to be entropy encoded may also arise during processing any other type of input date, such as of a multi-dimensional point clouds, audio data, video data or the like.
An arrangement 830 of the plurality of segments is processed by one or more layers of a neural network. Such an arrangement may be predefined, i.e. a specified order of scanning in the spatial and/or channel direction. Exemplary implementations for such arrangements are shown, for example, in
The neural network includes at least one attention layer. The attention mechanism is explained above in section Attention mechanism in Deep Learning with reference to
A probability model for the entropy encoding of a current element of the latent tensor is obtained based on the processed plurality of segments. The current element may be entropy encoded into a first bitstream, for example the y bitstream 371 of
The separating of the latent tensor may include a separation 701 of the latent tensor into two or more segments in the channel dimension as it is shown for example in
A separation in the channel dimension 920 is exemplarily illustrated in
The segments may be indicated by spatial coordinates in the spatial dimensions. The segments may further be indicted by a channel segment index in the channel dimension. In other words, the channel segment index is assigned to the segments in a one-to-one correspondence in the channel dimension. For example, when the latent tensor is split into Ncs segments in the channel dimension, a channel segment index in the range [0, Ncs−1] may be assigned.
The maximal number of channel segments Ncs is equal to the number of channels Ce of the latent tensor ŷ. This spatio-channel attention mechanism fully considers the cross-channel correlations. Any other value of Ncs<Ce may result in faster encoding and decoding, but may reduce the performance of the context model.
The processing of the arrangement of segments may include the obtaining of two or more groups of segments. A group may include one or more segments out of the plurality of segments.
In other words, each segment of the plurality of segments may be assigned to a group out of the two or more groups. For example, each segment is assigned to one group, respectively. This is exemplarily shown in
The segments within a group may be processed independently by the one or more layers of the neural network. Such independent processing of segments within a group facilitates a parallel processing of said segments. However, the present disclosure is not limited to parallel processing. The segments within a group may be processed sequentially or partially in parallel. The segments within a group may be processed simultaneously and independently in a given autoregressive step. Processing segments independently means that one segment may be processed without waiting for the processing of another segment.
In an exemplary implementation, for each group out of the two or more groups, the segments in said group have a same respective channel segment index. In other words, the segments in a group may differ in their respective spatial coordinates; however, said segments in the group have the same channel segment index in the channel dimension. This is shown, for example, in
The segments having a same channel segment index may be grouped, for example, into either two groups or four groups. An example for two groups per channel segment index is illustrated in
An example for four groups per channel segment index is illustrated in
The total number of autoregressive steps (which corresponds to the number of groups) are denoted by G, and may be calculated as
For example, segments having a same channel segment index are grouped according to a checkerboard pattern. In other words, for two groups per channel segment index, neighboring cuboids representing the segments, which share a face, may be assigned to different groups. Such a pattern is exemplarily shown for two groups per channel segment index in
In the processing of the arrangement of the plurality of segments, the segments in a first group may be processed in parallel followed by the processing the segments in a second group in parallel. Thus, in an encoding and/or in a decoding of segments of the second group, information of the segments of the first group is available.
In an exemplary implementation, segments in the second group may have a same index in the channel dimension as the segments in the first group. In other words, such a processing order of groups takes into account the spatial dimensions first, before processing groups of segments having a different channel segment index. Such a processing order is exemplarily shown in
For example, in
In another exemplary implementation, segments in the second group may have a same spatial coordinate as the corresponding segments in the first group. In other words, such a processing order of groups takes into account the channel dimensions first, before processing a group of segments that have different spatial coordinates than the corresponding segments in said first group. A group of segments having a same spatial coordinate as the corresponding segments in the first group includes segments, where for each segment in said group there is a segment in the first group that has the same spatial coordinates. In other words, there is a one-to one-correspondence of segments in said group and of segments within the first group regarding the spatial coordinates. Such a processing order is exemplarily shown in
For example, in
Note that
For example, if the segments of group “4” are currently processed, information of groups “1” to “3” is available for the processing of the segments of group “4”. The segments of groups “5” to “6” in this example are yet unavailable for processing.
Whether or not two or more groups of segments are obtained may be indicated in a bitstream. Such a bitstream may be generated and such an indication may be included. The indication may be included in the above-mentioned first bitstream, which includes the entropy-encoded elements of the latent tensor. In other words, such an indication indicates whether a subset (group) of segments may be processed independently or not. Such an indication may be, for example, a parameter, a flag or the like. The independent processing of segments within a group is described in the current section. An exemplary alternative, which may be indicated additionally by said indication, may be a sequential processing of an arrangement of the plurality of segments, which may be performed according to the section Arranging of segments below.
Alternatively or in addition to the processing of groups of segments explained above, the segments may be separated into a plurality of patches, each patch including two or more segments. Each patch may be processed independently by the one or more layers of the neural network. Such an independent processing of patches facilitates a parallel processing of said patches. In other words, the segments of the latent tensor are processed in a window fashion where the attention mechanism processes a sub-grid of segments.
A two-dimensional representation of a separation into patches in the spatial dimensions is exemplarily shown in
For example, each patch includes a K×M grid of spatially neighboring segments and L neighboring segments in the channel dimension, wherein L, K and M are positive integers, and at least one of K and M is larger than one. In other words, a patch includes all segments within a region in the channel dimension and the two spatial dimensions. A patch may include all segments in the channel dimension, i.e. L=Ncs, wherein Ncs is the number of segments in the channel dimension as defined in section Attention based Context Modelling or a subset of segments in the channel dimension i.e. L<Ncs. In this example, each patch is of a same size. However, the present disclosure is not limited to patches of a same size. Patches of a different size may also be used.
As already indicated, the patches out of the plurality of patches may be overlapping in the spatial dimensions and/or in the channel dimension. This is exemplarily shown in
An indication may be included into a bitstream, which indicates one or more of (i) whether or not the patches out of the plurality of patches are overlapping, and (ii) an amount of overlap. Such a bitstream may be generated and such an indication may be included. The indication may be included in the above-mentioned first bitstream, which includes the entropy-encoded elements of the latent tensor. The bitstream may be the same bitstream, which includes an indication regarding whether or not two or more groups of segments are obtained, as explained in section Processing of groups of segments.
The processing of a patch out of the plurality of patches by the one or more layers of the neural network may further include obtaining two or more groups of segments within said patch and processing the segments within a group out of the two or more groups independently by the said one or more layers. In other words, the processing of the patches of segments may be combined with the processing of groups of segments within a patch. The groups within each patch are processed as explained above in section Processing of groups of segments. For example,
An arrangement of a first exemplary embodiment may include a reshaping of the latent tensor into a sequential form 830. The reshaped tensor ŷseq may have dimension
where (pH,pW) corresponds to the size of a segment. This is exemplarily shown in
In a second exemplary embodiment, the segments may be arranged in a predefined order, wherein segments with a same spatial coordinate 931, 932 and 933 are grouped together. In other words, segments of a first spatial coordinate 930, which have different channel segment indices, for example, in the range [0, Ncs−1], may be grouped together. Consecutively, segments of a second spatial coordinate 931 having different channel indices may be grouped together. The arrangement 930 may include a reshaping of the latent tensor segments into a tensor ŷseq with a shape of
Such an arrangement 930 of the segments is exemplarily shown in
In a third exemplary embodiment, segments, which have different spatial coordinates 941, 942 and 943, may be arranged 701 consecutively in a predefined order. In other words, the segments corresponding to a first channel segment index 940 may be grouped together. Consecutively, segments corresponding to a second channel segment index 491 may be grouped together. The arrangement 940 may include a reshaping of the latent tensor segments into a tensor ŷseq with a shape of
Such an arrangement 940 is exemplarily shown in
For simplicity, the first dimension, which describes the number of sequential elements of ŷseq, may be denoted by S. In the above example S equals
The beginning of the arrangement of the plurality of segments of any of the above-mentioned exemplary embodiments may be padded 710, 711 with a zero segment 1000 before processing by the neural network. The zero segment may have the same dimensions as each segments within the plurality of segments. Each element within the zero segment may be zero.
elements being zero. Similarly, in
elements. The padding guarantees that the causality of the coding sequence is not disturbed, i.e. the decoder is able to decode data from the bitstream without additional prior knowledge.
As described above, in order to obtain a probability model, an arrangement of the plurality of segments is processed by one or more layers of a neural network, including at least one attention layer 732. The processing of the arrangement of segments may include any of the arrangements (i.e. processing orders) discussed in detail in sections Processing of groups of segments, Processing of patches of segments and Arranging of segments alone or in combination.
The (arrangement of the) plurality of segments of the latent tensor may be processed by a first neural subnetwork 720. Such a first neural subnetwork may extract features from the plurality of segments. Said features may be independent deep features (also called embeddings). Therefore, said first neural subnetwork 720 is a so-called embedding layer to extract contextual embedding in high-dimensional real-valued vector space. The first neural subnetwork 720 may be a fully-connected neural network such as a multilayer perceptron, which is explained above. For example, a convolutional network (CNN) or a recurrent neural network (RNN) may be used. The output of the first neural subnetwork 720, the so-called context embedding, may be provided as an input to a subsequent layer of the neural network.
Positional information 721 of the plurality of segments may be provided as an input to the attention layer. Such positional information 721 may be combined with the output of the first neural subnetwork 720, for example, by concatenation, addition, or the like. The context embedding, which may be combined with the positional information 721, may be normalized 731. The positional encoding contains the positional information, for example, coordinates in a linear space. The positional encodings enable an attention layer to understand the sequential order of the input sequence. For example, these encodings can be learned or pre-defined tensors representing the order of the sequence can be used.
In the processing by the attention layer 732, a mask may be applied, which masks subsequent elements in an attention tensor following the current element within a processing order of the latent tensor. The mask prevents the subsequent elements to be used in the calculation of the attention tensor. In other words, the attention mechanism may be adapted to the autoregressive task in order to ensure causality at the decoder side. Such a masked attention mechanism is an attention mechanism that is masked so that it does not process any data that is not at a position preceding the current position in the attention layer input sequence. The masking is exemplarily shown in
The attention mechanism by default is applied on the whole sequence S. That means each sequential element si in S applies attention to itself and every other element. This behavior is not desirable for an autoregressive task since the network cannot use any element, which is not processed yet. In order to cope with this problem, attention mechanism may be limited by masking the scaled dot product in the attention mechanism. The mask may be described with a S×S matrix, where its lower triangle (including the diagonal) contains 1 and the upper triangle part (excluding the diagonal) consists of minus infinity (softmax(−∞)=0). The masked attention may be formulated as:
where ⊙ is the elementwise multiplication and
is the mask.
In the case when groups of segments are obtained, the mask may take the form
wherein each block has dimensions
S is the number of segments in the latent tensor or in a patch of the latent tensor and G is the number of groups.
However, the definition of the mask M is not limited to the above mentioned triangular matrix. In general, the undesired or yet-to-be-processed parts of the sequence may be masked, for example by multiplying with −∞ while the rest may be multiplied with 1. The masking may be also applied for multi-head-attention, where each attention head is masked separately.
The masking of the present disclosure is not restricted to the application of this exemplary matrix M. Any other masking technique may be applied.
The output of the attention layer 732 may be processed by a second neural subnetwork 735. The second neural subnetwork 735 may be a multilayer perceptron. The output of the attention layer 732 may be normalized 734 before processing with the second neural subnetwork 735. The output of the attention layer 732 may be combined with the context embedding or with the combined representation of context embedding and positional information 721 by a residual connection 737.
The output of the attention based context model is denoted by ϕ.
The probability model 770 for the entropy encoding may be selected according to computational complexity and/or properties of the first bitstream. Properties of the first bitstream 731 may include a predefined target rate or a frame size. A set of rules, which option to use may be predefined. In this case, the rules may be known by the decoder, thus additional signaling is not required.
The selection may include selecting whether the separating of the latent tensor is performed in the channel dimension. The selection may include selecting between different ways how the arrangement is performed, for example, spatial dimensions first or channel dimension first.
For example, in the case when the separation in the channel dimension is not performed, the performance of the context model may be limited since cross-channel correlations are not considered for entropy modeling. However, this case may provide faster encoding and decoding due a reduced number of required autoregressive steps.
For example, in a case with Ncs>1, cross-channel correlations are considered, which may increase the performance of the context model. In the extreme case, where the number of channel segments Ncs is equal to the number of channels Ce of the latent tensor ŷ, the model fully considers the cross-channel correlations. Any other number of channel segments 1<Ncs<Ce provides a simplification of the extreme case for balancing the trade-off between the performance of the model and complexity.
In the first step of obtaining an optional hyper-prior model, a hyper-encoder 320 as shown in
Similar to the latent tensor, the output of the optional hyper-decoder 4 may be separated into a plurality of hyper-decoder output segments 740. Each hyper-decoder output segment may include one or more hyper-decoder output elements. For each segment out of the plurality of segments, said segment and a set of hyper-decoder output segments out of the plurality of hyper-decoder output segments may be concatenated 750 before the probability model 770 is obtained. In other words, the tensors ϕ and ψ may be concatenated in the channel dimension (last dimension) and may result in a concatenated two-dimensional tensor.
The hyper-decoder output segments may be arranged correspondingly to the arrangement of the plurality of the segments. The output of the hyper-decoder ψ may be brought into sequential format same as the sequential format of the latent tensor ŷseq.
Examples for the set of hyper-encoder output segments are illustrated in
In a fifth exemplary embodiment, the set of hyper-decoder output segments to be concatenated with a respective segment 1100 may include a plurality of hyper-decoder output segments corresponding to a same channel as said respective segment. In other words, said plurality ofhyper-decoder output segments may include hyper-decoder output segments, which have the same spatial coordinates as the respective segment, i.e. belong to a co-located channel. In the example in
In a sixth exemplary embodiment, the set of hyper-decoder output segments to be concatenated with a respective segment 1100 may include a plurality of hyper-decoder output segments 1130 spatially neighboring said respective segment 1100. Said plurality of hyper-decoder output segments 1130 spatially neighboring the respective segment 1100, which are exemplarily illustrated in
In a seventh exemplary embodiment, the set of hyper-decoder output segments to be concatenated with a respective segment may include a plurality of hyper-decoder output segments including neighboring segments 1140 spatially neighboring said respective segment and segments corresponding to a same channel 1141 and 1142 as said neighboring segments 1140. In other words, the set of hyper-decoder output segments may include hyper-decoder output segments 1140 spatially neighboring the respective segment 1100, which are exemplarily illustrated in
The set of hyper-decoder output segments to be concatenated to a respective segment is not limited to the above-mentioned examples. Any other set of hyper-decoder output segments may be concatenated with a respective segment of the latent tensor. For example, any combination of the above-mentioned fourth to seventh exemplary embodiments may be used. Any the above-mentioned fourth to seventh exemplary embodiments and any combination thereof may be combined with any of the arrangements of the second or third exemplary embodiments.
The concatenated tensor has the size of S×(Cϕ+Cψ′), where Cϕ and Cψ′ are the number of channels of the tensor ϕ and the number of channels from the tensor ψ, respectively. The result of the concatenation may be processed by a gathering process 760. For example, the gathering may be performed by a fully-connected neural network and a set of non-linear transformations on the last dimension. For example, the gathering may be implemented by one or several layers of convolutions with 1×1 kernel size and non-linear transformation(s). The entropy model 770 produces an estimation of the statistical properties of the quantized latent tensor 9. The entropy encoder 370 may use these statistical properties to create a bitstream representation 371 of the tensor 9.
Similar to the selection of the probability model for the entropy encoding, the set of hyper-decoder output segments may be selected adaptively according to computational complexity and/or properties of the first bitstream. Properties of the first bitstream may include a predefined target rate or a frame size. A set of rules, which option to use may be predefined. In this case, the rules may be known by the decoder, thus additional signaling is not required.
During the encoding, all elements of the latent tensor are available. Thus, the processing by the neural network and/or the entropy encoding the current element may be performed in parallel for each segment out of the plurality of segments.
The processing of the arrangement by the neural network may include selecting a subset of segments. Such a subset of segments is selected from the plurality of segments. The subset may be provided to a subsequent layer of the neural network. For example, the subset may be selected before applying the at least one attention layer. For example, the subset may be selected before applying the first neural subnetwork. Said subset of segments may include segments in a local neighborhood in the spatial dimensions. This is exemplarily shown in
For decoding of the latent-space feature tensor from a first bitstream the latent tensor is initialized with zeroes, as the decoder is agnostic to the latent tensor and its statistical properties. The latent-space feature tensor, which includes one or more elements, is separated 700 into a plurality of segments 820 in the spatial dimensions as shown in
which has been explained above for the encoding and is shown exemplarily in
The neural network includes at least one attention layer. The attention mechanism is explained above in section Attention mechanism in Deep Learning with reference to
A probability model for the entropy decoding of a current element of the latent tensor is obtained based on the processed plurality of segments. The current element may be decoded from the first bitstream, for example the y bitstream 371 of
The separating of the latent tensor may include a separation 701 of the latent tensor into two or more segments in the channel dimension as it is shown for example in
The processing of the arrangement of segments may include the obtaining of two or more groups of segments. Analogous to the encoding, a group may include one or more segments out of the plurality of segments. In other words, each segment of the plurality of segments may be assigned to a group out of the two or more groups.
The segments within a group may be processed independently by the one or more layers of the neural network. Such independent processing of segments within a group facilitates a parallel processing of said segments.
In an exemplary implementation, for each group out of the two or more groups, the segments in said group have a same respective channel segment index. The segments having a same channel segment index may grouped, for example, into either two groups or four groups. This is explained in detail above with reference to
Analogous to the encoding, segments having a same channel segment index may be grouped according to a checkerboard pattern. Such a pattern is exemplarily shown for two groups per channel segment index in
In the processing of the arrangement of the plurality of segments, the segments in a first group may be processed in parallel followed by the processing the segments in a second group in parallel.
In an exemplary implementation, segments in the second group may have a same index in the channel dimension as the segments in the first group. In other words, such a processing order of groups takes into account the spatial dimensions first. This is explained in detail in section Processing of groups of segments above with respect to
In another exemplary implementation, segments in the second group may have a same spatial coordinate as the corresponding segments in the first group. In other words, such a processing order of groups takes into account the channel dimensions first. This is explained in detail in section Processing of groups of segments above with respect to
Whether or not two or more groups of segments are obtained may be indicated in a bitstream. Such a bitstream may be received and such an indication may be obtained from the received bitstream. The indication may be included in the above-mentioned first bitstream, which includes the entropy-encoded elements of the latent tensor. Such an indication may be, for example, a parameter, a flag or the like. An exemplary alternative, which may be indicated additionally by said indication, may be a sequential processing of an arrangement of the plurality of segments, which may be performed analogously to the encoding as explained in section Arranging of segments below.
Alternatively or in addition to the processing of groups of segments explained above, the segments may be separated into a plurality of patches, each patch including two or more segments. Each patch may be processed independently by the one or more layers of the neural network. This is explained in detail above in section Processing of patches of segments with respect to
Analogously to the encoding, each patch may include a K×M grid of spatially neighboring segments and L neighboring segments in the channel dimension, wherein L, K and M are positive integers, and at least one of K and M is larger than one.
The patches out of the plurality of patches may be overlapping in the spatial dimensions and/or in the channel dimension. This is exemplarily shown in
An indication, which indicates one or more of (i) whether or not the patches out of the plurality of patches are overlapping, and (ii) an amount of overlap, may be obtained from a bitstream. Such a bitstream may be received in the decoding. The indication may be included in the above-mentioned first bitstream, which includes the entropy-encoded elements of the latent tensor. Such an indication may be, for example, a parameter, a flag or the like. The bitstream may be the same bitstream, which includes an indication regarding whether or not two or more groups of segments are obtained, as explained above.
The processing of a patch out of the plurality of patches by the one or more layers of the neural network may further include obtaining two or more groups of segments within said patch and processing the segments within a group out of the two or more groups independently by the said one or more layers. In other words, as explained above for the encoding, the processing of the patches of segments may be combined with the processing of groups of segments within a patch. The groups within each patch are processed as explained above. For example,
The segments may be arranged in a predefined order, wherein segments with a same spatial coordinate 931, 932 and 933 are grouped together. This arrangement 930 corresponds to the second exemplary embodiment, which is explained above in detail for the encoding in section Arrangement of segments.
Segments, which have different spatial coordinates 941, 942 and 943, may be arranged consecutively in a predefined order. Such an arrangement 940 is analogous to the third exemplary embodiment, which is explained above in detail for the encoding.
The beginning of the arrangement of the plurality of segments of any of the above-mentioned exemplary embodiments may be padded 710, 711 with a zero segment 1000 before processing by the neural network. The zero segment may have the same dimensions as each segments within the plurality of segments, which is illustrated exemplarily in
As described above, in order to obtain a probability model for decoding latent tensor elements, an arrangement of the plurality of segments is processed by one or more layers of a neural network, including at least one attention layer 732. The processing of the arrangement of segments may include any of the arrangements (i.e. processing orders) discussed in detail for the encoding in sections Processing of groups of segments, Processing of patches of segments and Arranging of segments alone or in combination.
According to the encoding side, the (arrangement of the) plurality of segments of the latent tensor may be processed by a first neural subnetwork 720. Such a first neural subnetwork may extract features from the plurality of segments. Said features may be independent deep features. The first neural subnetwork 720 may be a multilayer perceptron. Positional information 721 of the plurality of segments may be provided as an input to the attention layer. Such positional information 721 may be combined with the output of the first neural subnetwork 720, for example, by concatenation, addition, or the like.
The output of the attention layer 732 may be processed by a second neural subnetwork 735 analogous to the encoding side. The second neural subnetwork 735 may be a multilayer perceptron. The output of the attention layer 732 may be combined with the context embedding or with the combined representation of context embedding and positional information 721 by a residual connection 737.
Similar to the encoding, the probability model 770 for the entropy encoding may be selected according to computational complexity and/or properties of the first bitstream. Properties of the first bitstream 731 may include a predefined target rate or a frame size. A set of rules, which option to use may be predefined. In this case, the rules may be known by the decoder.
A hyper-latent tensor may be entropy decoded from a second bitstream 341. The obtained hyper-latent tensor may be hyper-decoded into a hyper-decoder output ψ.
Similar to the latent tensor, the output of the optional hyper-decoder ψ may be separated into a plurality of hyper-decoder output segments 740. Each hyper-decoder output segment may include one or more hyper-decoder output elements. For each segment out of the plurality of segments, said segment and a set of hyper-decoder output segments out of the plurality of hyper-decoder output segments may be concatenated 750 before the probability model 770 is obtained.
Examples for the set of hyper-encoder output segments are illustrated in
Similar to the selection of the probability model for the entropy encoding, the set of hyper-decoder output segments may be selected adaptively according to computational complexity and/or properties of the first bitstream. Properties of the first bitstream include, for example, a predefined target rate or a frame size. A set of rules, which option to use may be predefined. In this case, the rules may be known by the decoder.
The processing of the arrangement by the neural network may include selecting a subset of segments. Such a subset of segments is selected from the plurality of segments. The subset may be provided to a subsequent layer of the neural network. Examples are explained above with reference to
The probability model using an attention layer may be applied to the entropy decoding of a latent tensor that may be processed by an autodecoding convolutional neural network to obtain image data as discussed above.
Implementation within Picture Coding
The encoder 20 may be configured to receive a picture 17 (or picture data 17), e.g. picture of a sequence of pictures forming a video or video sequence. The received picture or picture data may also be a pre-processed picture 19 (or pre-processed picture data 19). For sake of simplicity the following description refers to the picture 17. The picture 17 may also be referred to as current picture or picture to be coded (in particular in video coding to distinguish the current picture from other pictures, e.g. previously encoded and/or decoded pictures of the same video sequence, i.e. the video sequence which also comprises the current picture).
A (digital) picture is or can be regarded as a two-dimensional array or matrix of samples with intensity values. A sample in the array may also be referred to as pixel (short form of picture element) or a pel. The number of samples in horizontal and vertical direction (or axis) of the array or picture define the size and/or resolution of the picture. For representation of color, typically three color components are employed, i.e. the picture may be represented or include three sample arrays. In RGB format or color space a picture comprises a corresponding red, green and blue sample array. However, in video coding each pixel is typically represented in a luminance and chrominance format or color space, e.g. YCbCr, which comprises a luminance component indicated by Y (sometimes also L is used instead) and two chrominance components indicated by Cb and Cr. The luminance (or short luma) component Y represents the brightness or grey level intensity (e.g. like in a grey-scale picture), while the two chrominance (or short chroma) components Cb and Cr represent the chromaticity or color information components. Accordingly, a picture in YCbCr format comprises a luminance sample array of luminance sample values (Y), and two chrominance sample arrays of chrominance values (Cb and Cr). Pictures in RGB format may be converted or transformed into YCbCr format and vice versa, the process is also known as color transformation or conversion. If a picture is monochrome, the picture may comprise only a luminance sample array. Accordingly, a picture may be, for example, an array of luma samples in monochrome format or an array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2, and 4:4:4 color format.
Some further implementations in hardware and software are described in the following.
Any of the encoding devices described with references to
The decoding devices in any of
Summarizing, methods and apparatuses are described for entropy encoding and decoding of a latent tensor, which includes separating the latent tensor into segments in the spatial dimensions, each segment including at least one latent tensor element. An arrangement of the segments is processed by a neural network; the neural network includes at least one attention layer. Based on the processed segment a probability model is obtained for entropy encoding or decoding of a latent tensor element.
In the following embodiments of a video coding system 10, a video encoder 20 and a video decoder 30 are described based on
As shown in
The source device 12 comprises an encoder 20, and may additionally, i.e. optionally, comprise a picture source 16, a pre-processor (or pre-processing unit) 18, e.g. a picture pre-processor 18, and a communication interface or communication unit 22.
The picture source 16 may comprise or be any kind of picture capturing device, for example a camera for capturing a real-world picture, and/or any kind of a picture generating device, for example a computer-graphics processor for generating a computer animated picture, or any kind of other device for obtaining and/or providing a real-world picture, a computer generated picture (e.g. a screen content, a virtual reality (VR) picture) and/or any combination thereof (e.g. an augmented reality (AR) picture). The picture source may be any kind of memory or storage storing any of the aforementioned pictures.
In distinction to the pre-processor 18 and the processing performed by the pre-processing unit 18, the picture or picture data 17 may also be referred to as raw picture or raw picture data 17.
Pre-processor 18 is configured to receive the (raw) picture data 17 and to perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. Pre-processing performed by the pre-processor 18 may, e.g., comprise trimming, color format conversion (e.g. from RGB to YCbCr), color correction, or de-noising. It can be understood that the pre-processing unit 18 may be optional component.
The video encoder 20 is configured to receive the pre-processed picture data 19 and provide encoded picture data 21.
Communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and to transmit the encoded picture data 21 (or any further processed version thereof) over communication channel 13 to another device, e.g. the destination device 14 or any other device, for storage or direct reconstruction.
The destination device 14 comprises a decoder 30 (e.g. a video decoder 30), and may additionally, i.e. optionally, comprise a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32) and a display device 34.
The communication interface 28 of the destination device 14 is configured receive the encoded picture data 21 (or any further processed version thereof), e.g. directly from the source device 12 or from any other source, e.g. a storage device, e.g. an encoded picture data storage device, and provide the encoded picture data 21 to the decoder 30.
The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoded picture data 21 or encoded data 13 via a direct communication link between the source device 12 and the destination device 14, e.g. a direct wired or wireless connection, or via any kind of network, e.g. a wired or wireless network or any combination thereof, or any kind of private and public network, or any kind of combination thereof.
The communication interface 22 may be, e.g., configured to package the encoded picture data 21 into an appropriate format, e.g. packets, and/or process the encoded picture data using any kind of transmission encoding or processing for transmission over a communication link or communication network.
The communication interface 28, forming the counterpart of the communication interface 22, may be, e.g., configured to receive the transmitted data and process the transmission data using any kind of corresponding transmission decoding or processing and/or de-packaging to obtain the encoded picture data 21.
Both, communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces as indicated by the arrow for the communication channel 13 in
The decoder 30 is configured to receive the encoded picture data 21 and provide decoded picture data 31 or a decoded picture 31.
The post-processor 32 of destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), e.g. the decoded picture 31, to obtain post-processed picture data 33, e.g. a post-processed picture 33. The post-processing performed by the post-processing unit 32 may comprise, e.g. color format conversion (e.g. from YCbCr to RGB), color correction, trimming, or re-sampling, or any other processing, e.g. for preparing the decoded picture data 31 for display, e.g. by display device 34.
The display device 34 of the destination device 14 is configured to receive the post-processed picture data 33 for displaying the picture, e.g. to a user or viewer. The display device 34 may be or comprise any kind of display for representing the reconstructed picture, e.g. an integrated or external display or monitor. The displays may, e.g. comprise liquid crystal displays (LCD), organic light emitting diodes (OLED) displays, plasma displays, projectors, micro LED displays, liquid crystal on silicon (LCoS), digital light processor (DLP) or any kind of other display.
The coding system 10 further includes a training engine 25. The training engine 25 is configured to train the encoder 20 (or modules within the encoder 20) or the decoder 30 (or modules within the decoder 30) to process an input picture or generate a probability model for entropy encoding as discussed above.
Although
As will be apparent for the skilled person based on the description, the existence and (exact) split of functionalities of the different units or functionalities within the source device 12 and/or destination device 14 as shown in
The encoder 20 (e.g. a video encoder 20) or the decoder 30 (e.g. a video decoder 30) or both encoder 20 and decoder 30 may be implemented via processing circuitry as shown in
Source device 12 and destination device 14 may comprise any of a wide range of devices, including any kind of handheld or stationary devices, e.g. notebook or laptop computers, mobile phones, smart phones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video gaming consoles, video streaming devices (such as content services servers or content delivery servers), broadcast receiver device, broadcast transmitter device, or the like and may use no or any kind of operating system. In some cases, the source device 12 and the destination device 14 may be equipped for wireless communication. Thus, the source device 12 and the destination device 14 may be wireless communication devices.
In some cases, video coding system 10 illustrated in
For convenience of description, embodiments of the disclosure are described herein, for example, by reference to High-Efficiency Video Coding (HEVC) or to the reference software of Versatile Video coding (VVC), the next generation video coding standard developed by the Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Motion Picture Experts Group (MPEG). One of ordinary skill in the art will understand that embodiments are not limited to HEVC or VVC.
The video coding device 400 comprises ingress ports 410 (or input ports 410) and receiver units (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 to process the data; transmitter units (Tx) 440 and egress ports 450 (or output ports 450) for transmitting the data; and a memory 460 for storing the data. The video coding device 400 may also comprise optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress ports 410, the receiver units 420, the transmitter units 440, and the egress ports 450 for egress or ingress of optical or electrical signals.
The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGAs, ASICs, and DSPs. The processor 430 is in communication with the ingress ports 410, receiver units 420, transmitter units 440, egress ports 450, and memory 460. The processor 430 comprises a coding module 470. The coding module 470 implements the disclosed embodiments described above. For instance, the coding module 470 implements, processes, prepares, or provides the various coding operations. The inclusion of the coding module 470 therefore provides a substantial improvement to the functionality of the video coding device 400 and effects a transformation of the video coding device 400 to a different state. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.
The memory 460 may comprise one or more disks, tape drives, and solid-state drives and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory 460 may be, for example, volatile and/or non-volatile and may be a read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and/or static random-access memory (SRAM).
A processor 502 in the apparatus 500 can be a central processing unit. Alternatively, the processor 502 can be any other type of device, or multiple devices, capable of manipulating or processing information now-existing or hereafter developed. Although the disclosed implementations can be practiced with a single processor as shown, e.g., the processor 502, advantages in speed and efficiency can be achieved using more than one processor.
A memory 504 in the apparatus 500 can be a read only memory (ROM) device or a random access memory (RAM) device in an implementation. Any other suitable type of storage device can be used as the memory 504. The memory 504 can include code and data 506 that is accessed by the processor 502 using a bus 512. The memory 504 can further include an operating system 508 and application programs 510, the application programs 510 including at least one program that permits the processor 502 to perform the methods described here. For example, the application programs 510 can include applications 1 through N, which further include a video coding application that performs the methods described herein, including the encoding and decoding using a neural network with a subset of partially updatable layers.
The apparatus 500 can also include one or more output devices, such as a display 518. The display 518 may be, in one example, a touch sensitive display that combines a display with a touch sensitive element that is operable to sense touch inputs. The display 518 can be coupled to the processor 502 via the bus 512.
Although depicted here as a single bus, the bus 512 of the apparatus 500 can be composed of multiple buses. Further, the secondary storage 514 can be directly coupled to the other components of the apparatus 500 or can be accessed via a network and can comprise a single integrated unit such as a memory card or multiple units such as multiple memory cards. The apparatus 500 can thus be implemented in a wide variety of configurations.
Although embodiments have been primarily described based on video coding, it should be noted that embodiments of the coding system 10, encoder 20 and decoder 30 (and correspondingly the system 10) and the other embodiments described herein may also be configured for still picture processing or coding, i.e. the processing or coding of an individual picture independent of any preceding or consecutive picture as in video coding. In general only inter-prediction units 244 (encoder) and 344 (decoder) may not be available in case the picture processing coding is limited to a single picture 17. All other functionalities (also referred to as tools or technologies) of the video encoder 20 and video decoder 30 may equally be used for still picture processing, e.g. residual calculation 204/304, transform 206, quantization 208, inverse quantization 210/310, (inverse) transform 212/312, partitioning 262/362, intra-prediction 254/354, and/or loop filtering 220, 320, and entropy coding 270 and entropy decoding 304.
Embodiments, e.g. of the encoder 20 and the decoder 30, and functions described herein, e.g. with reference to the encoder 20 and the decoder 30, may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on a computer-readable medium or transmitted over communication media as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media, which is non-transitory, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.
By way of example, and not limiting, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.
The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and/or firmware.
| Number | Date | Country | Kind |
|---|---|---|---|
| PCT/EP2021/079028 | Oct 2021 | WO | international |
This application is a continuation of International Application No. PCT/EP2022/069824, filed on Jul. 15, 2022, which claims priority to International Patent Application No. PCT/EP2021/079028, filed on Oct. 20, 2021. The disclosures of the aforementioned applications are hereby incorporated by reference in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/EP2022/069824 | Jul 2022 | WO |
| Child | 18620667 | US |
