Patent Application
20230252273
The domain technical field of the one or more embodiments of the present disclosure is related to the technical domain of data processing, like for data compression and/or decompression. For instance, at least some embodiments relate to data compression/decompression involving large volume of data, like compression and/or decompression of at least a part of an audio and/or video stream, or like compression and/or decompression of data in link with Deep Learning techniques, like at least some parameters of a Deep Neural Network (DNN).
At least some embodiments relate to improving compression efficiency compared to existing video compression systems such as HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2 described in “ITU-T H.265 Telecommunication standardization sector of ITU (October 2014), series H: audiovisual and multimedia systems, infrastructure of audiovisual services—coding of moving video, High efficiency video coding, Recommendation ITU-T H.265”), or compared to under development video compression systems such VVC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).
To achieve high compression efficiency, image and video coding schemes usually employ prediction, including spatial and/or motion vector prediction, and transforms to leverage spatial and temporal redundancy in the video content. Generally, intra or inter prediction is used to exploit the intra or inter frame correlation, then the differences between the original image and the predicted image, often denoted as prediction errors or prediction residuals, are transformed, quantized, and entropy coded. To reconstruct the video, the compressed data are decoded by inverse processes corresponding to the entropy coding, quantization, transform, and prediction.
At least some embodiments relate to improving compression efficiency compared to existing systems for compression a Deep Neural Network (DNN). such as some compression standard or draft standard like the current upcoming standard ISO/MPEG7 of neural networks for multimedia content description and analysis current developed by the International Organization for Standardization.
Generally, in an encoding process, parameters of a DNN are quantized and entropy coded to obtain compressed data. To reconstruct data, the compressed data are decoded, the decoding processes including entropy decoding and inverse quantization.
The present principles enable at least one of disadvantages of some known compression and/or decompression methods to be resolved by proposing a method and an apparatus for encoding or decoding data in at least one bitstream, data being one or more parameters of at least one tensor of at least one layer or sub-layer of at least one Deep Neural Network. It is to be pointed out that tensor of parameters associated to a layer can include weights and/or biases, even if sometimes simply called “weights” in the following for concision purpose.
According to an embodiment, a method for decoding at least one first tensor of at least one layer of at least one Deep Neural Network is provided. Such a method comprises responsive to a determination that at least one first tensor is decomposed into a second tensor and a third tensor whose parameters are encoded in a bitstream, decoding from the bitstream a size of at least one of the second tensor and the third tensor, and decoding the at least one of the second tensor and the third tensor from the bitstream based on the decoded size. According to another embodiment, an apparatus for decoding at least one first tensor of at least one layer of at least one Deep Neural Network is provided. The apparatus comprises one or more processors configured to determine that at least one first tensor of at least one layer of at least one Deep Neural Network is decomposed into a second tensor and a third tensor whose parameters are encoded in a bitstream, decode from the bitstream a size of at least one of the second tensor and the third tensor, decode the at least one of the second tensor and the third tensor from the bitstream based on the decoded size.
According to another embodiment, a method comprising encoding data representative of at least one first tensor of at least one layer of the Deep Neural Network in a bitstream is provided. The method comprises responsive to a determination that the at least one first tensor is decomposed into a second tensor and a third tensor, encoding a size of at least one of the second tensor and the third tensor, encoding parameters representative of the at least one of the second tensor and the third tensor.
According to another embodiment, an apparatus for encoding data representative of at least one first tensor of at least one layer of the Deep Neural Network in a bitstream is provided, wherein the apparatus comprises one or more processors, wherein the one or more processors are configured for determining that the at least one first tensor is decomposed into a second tensor and a third tensor, responsive to the determination, encoding a size of at least one of the second tensor and the third tensor, encoding the at least one of the second tensor and the third tensor.
One or more embodiments also provide a computer program comprising instructions which when executed by one or more processors cause the one or more processors to perform the encoding method or decoding method according to any of the embodiments described above. One or more of the present embodiments also provide a computer readable storage medium having stored thereon instructions for encoding or decoding data according to the methods described above. One or more embodiments also provide a computer readable storage medium having stored thereon a bitstream generated according to the methods described above. One or more embodiments also provide a method and apparatus for transmitting or receiving the bitstream generated according to the methods described above.
According to another general aspect of at least one embodiment, there is provided a device comprising an apparatus according to any one of the decoding embodiments; and at least one of (i) an antenna configured to receive a signal, the signal including the input data, (ii) a band limiter configured to limit the received signal to a band of frequencies that includes the input data, or (iii) a display configured to display an output representative of a video block.
While not explicitly described, the devices of the present disclosure can be adapted to perform the methods of the present disclosure in any of theirs embodiments.
While not explicitly described, the present embodiments related to the methods or to the corresponding signal, devices, and computer readable storage media can be employed in any combination or sub-combination.
Many technical fields can involve the processing, with computer means, of large volume of data. Such processing can involve data compression and/or decompression of data, for a purpose a storage or of transmission of at least a part of such data for instance. Examples of compression and/or decompression of streams containing large amount of data can be found in the technical field of video processing, or in technical fields involving Deep Learning techniques.
Embodiments of the present disclosure are detailed hereinafter in link with Deep Neural Networks (DNNs) as an exemplary and not limitative purpose. It is clear however that the present disclosure can also apply to the compression/decompression of other large amount of data, like in the technical field of video processing. For instance, the present disclosure can apply to the compression/decompression of a tensor obtained by a Deep Learning Algorithm from at least one image.
Deep Neural Networks (DNNs) have shown state of the art performance in variety of domains such as multimedia processing, computer vision, speech recognition, natural language processing, etc. This performance however can come at the cost of massive computational cost as DNNs tend to have a huge number of parameters often running into millions, and sometimes even billions.
This can lead for instance to prohibitively high inference complexity. In simple words, inference is the deployment of a DNN, once trained, for processing input data, in view of their classification for instance. Inference complexity can be defined as the computational cost of applying trained DNN to input data for inference. Inference complexity can be defined as the computational cost of applying trained DNN to test data for inference.
This high inference complexity can thus be an important challenge for using DNNs in environments involving an electronic device with limited hardware and/or software resource, for instance mobile or embedded devices with resource limitations like battery size, limited computational power, and memory capacity etc.
Deep Neural Networks are made up of several layers. A layer is associated with a set of parameters that can be obtained for instance during a training of the DNN. These parameters (like Weights and/or Biases) are stored as multi-dimensional arrays (also referred to herein as “tensors”). In the following, for simplicity purpose, the term “matrix can sometimes be used to denote a set of parameters (e.g. parameters of a given layer). It is to be understood, however, that some embodiments of the methods of the present disclosure can also be applied to tensors of parameters with more than two dimensions, such as 2D convolutional layers which usually contain 4D tensors of parameters. The huge number of parameters of DNNs can require a large bandwidth for deployment of DNNs (or solutions including DNNs) in distributed environments.
At least some embodiments of the present disclosure apply to the compression and/or decompression (decoding) of at least some parameters of at least one DNN (for instance a pre-trained DNN). Indeed, compression can facilitate the transmission and/or storage of the parameters of the at least one DNN. More precisely, at least some embodiments of the present disclosure apply to the compression of parameters of at least one tensor associated with at least one layer of at least one Deep Neural Network.
Depending upon embodiments of the present disclosure, the layers (or sub-layers) can be of different types. For instance, in some embodiments, all the at least one layer can be convolutional layer(s), or fully connecter layer(s), or the at least one layer can comprise at least one convolutional layer and/or at least one fully connecter layer.
Some embodiments of the present disclosure relate more specifically to compression solutions including, or at least that may include, decomposition of at least one tensor so as to improve the efficiency of the compression for instance, and/or to decoding solutions including, or at least that may include, reconstruction of at least one tensor. The decomposed at least one tensor can be for instance at least one tensor of one or more layer(s) to be compressed of one or more DNNs, The reconstructed at least one tensor can be for instance at least one tensor of the same shape than at least one tensor, that has been decomposed, of one or more layer(s) to be compressed of one or more DNNs,
According to some embodiments of the present disclosure, decomposition of a tensor can be obtained by using a Low Rank (LR) technology and/or a Low Displacement Rank (LDR) technology.
When tensor decomposition is used to compress large tensors of weights, at least two smaller tensors are produced and further compressed, quantized and entropy coded to be stored or transmitted within a bitstream.
As a non-limitative example, while some embodiments can be applied in non-standardized technologies, some embodiments can be used in contexts of standards for DNN compression/decompression, like the upcoming standard ISO/MPEG7 relating to compressed representations of neural networks for multimedia content description and analysis, which is denoted hereinafter more simply MPEG NNR.
At least some embodiments of the present disclosure propose a syntax structure as well as a mechanism for decomposing tensors and reconstructing tensors from multiple decoded tensors.
More precisely, according to some embodiments, when tensor decomposition is used to compress large tensors of weight, at least two smaller tensors can be produced (and input to quantization, at least some of the output of quantizing being then encoding for instance). For instance, in an exemplary embodiment where decomposition is based on Low Rank (LR), Low Rank approximations can represent an original matrix of weights as a product:
Ŵ
k
=G
k
H
k
T (1)
where Gk is a m×rk matrix and Hk is n×rk matrix that can be derived from a Single Value Decomposition (SVD).
At the decoder, two options can be envisioned when using this compression technique:
At least some embodiments of the present disclosure provide a syntax for enabling such condition as well as a mechanism for reconstructing the tensors in their original shape. Indeed, inventors have cleverly noticed that no solution has been proposed yet for reconstructing, for instance at a decoder at the time of the decoding of the model, an original shape of a tensor from the tensors obtained by decomposition of this tensor.
For instance, in the current specification draft of MPEG NNR, the current assumption is that the tensors resulting from decomposing an original tensor are output by the decoder.
This can be an issue when the original tensor shapes are needed by the inference engine as explained above.
At least some embodiments of the present disclosure invention help to address this issue. It is to be pointed out that the embodiments of the methods of the present disclosure detailed herein can be implemented in many compression solutions and are not limited to a specific standard, even if at least some of the embodiments can for instance apply in the context of some compression standards, like some draft standard developed by ISO/MPEG7.
As shown in the above equation (1), in an exemplary embodiment where the Tensor W is a 2D-Matrix, after decoding G and HT, a matrix multiplication of the two matrices is needed to be performed to obtain the original matrix shape.
In the above exemplary embodiments, a LR decomposition can be used for the original tensor. However, depending upon embodiments, or depending upon tensors of the one or more DNN, different decompositions can be performed. In the case of a tensor of a convolution or depth-wise convolution layers for instance, a tensor can be reshaped into a 2-dimension matrix, enabling LR/LDR methods.
This present disclosure describes necessary syntax and processes for permitting to reconstruct tensors.
Notably, some embodiments of the present disclosure, adapted for instance to reconstruct an original tensor from one or more tensor units, propose a mechanism involving a buffer of tensors to keep the previously decoded tensors in order to perform the reconstruction, like for instance (in the exemplary use case introduced above in link with equation (1)) the previously decoded G and/or H matrices in order to perform the reconstruction of W.
In the following, a Decoded Tensor Buffer (DTB) is introduced, which can contain multiple already decoded tensors in memory. For instance, in the above exemplary embodiment, the decoded tensors G and H are added to the buffer when they are the first of the two tensors (G and H) to be decoded for a given layer. More precisely, for a given layer, the decoded tensor G (respectively the decoded tensor H) is added to the buffer when the tensor H (respectively the tensor G) has not yet been decoded.
Then, when the corresponding tensor in the same layer is decoded, the reconstruction of a tensor having the shape of the original tensor can be triggered, and the memory taken by the saved tensor in the DTB can be freed.
In at least some embodiments of the present disclosure, once an original tensor has been decomposed in several tensors, the several resulting tensors can be encoded and decoded separately (in other words independently).
As illustrated by
In some embodiments, each obtained tensor can be decomposed.
In other embodiments, as in the embodiment of
In the exemplary embodiment of
If the decomposition mode is selected by the encoder (404), the method can comprise decomposing the tensors and encoding the resulting tensors (for instance encoding (405) the tensor G and encoding the tensor H (407)).
In the exemplary use case of
The output of the encoding is used to compose the bitstream.
This process can be iterated for several input tensors, for instance for all tensors in the model to quantize and/or encode (408).
In some embodiments, the method can further comprise, prior to the encoding, reducing the number of parameters (or Weights or Biases) of the Neural Network by utilizing the inherent redundancies in the Neural Network. For instance, original tensors of parameters of at least one layer of the DNN or tensors resulting from the decomposition of original tensors of parameters of at least one layer of the DNN can be made sparse. This reducing is optional and can be omitted in some embodiments and/or for some tensors of some layers
The encoding can comprise quantizing parameters (like Weights and Biases) of at least one tensor (e.g. either the tensors output by the decomposition of a tensor of a layer of the Neural Network, or the tensor itself when no decomposition is performed) and lossless entropy coding of the quantized information to represent them with a smaller number of bits.
In some embodiments, when several layers of a DNN are to be encoded, the method can be performed iteratively layer per layer, until the end of the encoding of parameters of the last layer to be encoded.
In some embodiments, as in the illustrated embodiment of
If the corresponding tensor is present, it is fetched from the DTB (505) and both current and fetched (507) tensors are used to reconstruct a tensor in the shape (508) of the original tensor (i.e. the obtained tensor of step 401). It is to be pointed out that in many embodiments, while having the same dimensions as the original tensor, the reconstructed tensor is however different from the original tensor.
If the corresponding tensor is not present in the DTB, the method 500 can comprise storing the current tensor in the DTB for future use (506).
In any cases (in other words after having decoded the current tensor unit) (step 509), the method can further comprise checking if the current tensor is the last in the bitstream (509) and the method outputs the model if it is the case or access the next tensor unit otherwise.
At the decoder, as illustrated by
When several tensors (of several layers for instance) are to be decoded, the method 500 can be performed until all the several tensors are decoded.
Some embodiments of the present disclosure can comprise transmitting/receiving signaling information between an encoder and a decoder. This signaling information is presented in the present disclosure in link with an exemplary, non-limitative, syntax. This exemplary syntax is mainly based, for the ease of explanation, on a syntax used in an exemplary MPEG NNR draft standard (like N19225—Working Draft 4 of Compression of neural networks for multimedia content description and analysis International Organization for Standardization ISO/IEC JTC1/SC29/WG11, April 2020.
The following syntax is just an exemplary syntax that does not limit the present disclosure. For instance, numbers of bits used for syntax elements are exemplary embodiments. For ease of understanding, in the exemplary syntax, the following identifiers and clauses according to embodiments of the present disclosure have been added with a numbering of sections and tables being kept aligned with the current exemplary working draft of the MPEG-NNR.
With this exemplary syntax, the following tensor operations can be added:
MatrixProd (array_name_1[ ], array_name_2[ ]) which returns the matrix product of array_name_1 by array_name_2.
TensorReshape (array_name[ ], tensor_dimension[ ]) which returns the reshaped tensor array_name[ ] with the specified tensor_dimension[ ], without changing its data.
Furthermore, definitions of the following terms are provided:
bin: One bit of a bin string.
binarization: A set of bin strings for all possible values of a syntax element.
binarization process: A unique mapping process of all possible values of a syntax element onto a set of bin strings.
bin string: An intermediate binary representation of values of syntax elements from the binarization of the syntax element.
bitstream: A sequence of bits, that forms the representation of coded units and associated data forming one or more coded Neural Network Models
decoded tensor buffer (DTB): A buffer holding decoded tensors/units for reference.
When decoding at least a part of a bitstream (simply said the “decoding process”), the following conditions can be applied with the exemplary detailed syntax:
According to some embodiments of the present disclosure, it is proposed to specify NNR tensor types in case of tensor decomposition. For instance, the following exemplary syntax can be used, in link with the exemplary MPEG NNR draft standard:
With the exemplary syntax detailed herein, a table can specify NNR tensor types in case of tensor decomposition.
At the decoder side, the tensor processing can be performed once per NNR compressed payload, after decoding the unit header (e.g. nnr_compressed_data_unit_header with the exemplary syntax) and the compressed payload.
In the exemplary embodiment detailed, the output of the processing of current tensor can be specified as follows:
The reconstruction of tensors having the shape of an original tensor can be performed after decoding all tensors resulting from the decomposition of the original tensor.
For instance, in embodiments where an original tensor has been decomposed in a tensor G and a tensor H, the reconstruction can happen after decoding a tensor (e.g. a tensor of type “TENSOR_G” or “TENSOR_H”) if the corresponding tensor (e.g. a tensor with an identifier (e.g. ref_id) specifying the same layer) is present in the DTB, as explained above.
In the exemplary syntax described herein, the inputs to this reconstruction can include
In the exemplary embodiment described, the output of this reconstruction is a current tensor array_w having the same shape as the original tensor (also called herein original shape). According to some embodiments of the present disclosure, the current tensor array w can be computed by taking account of a reconstruction mode of the tensor. In exemplary syntax detailed herein, the following table can be used for specifying the reconstruction mode of a tensor in the bitstream:
The current tensor array w can be computed as the following:
As for the High-Level Syntax, with the exemplary draft standard MPEG-NNR, some elements can be added to some tables of the exemplary draft standard as the following:
where
where
where
Some exemplary embodiments have been detailed above. The present disclosure also encompasses many variants of the above embodiments
For instance, some embodiments of the present disclosure can relate to the following variants
First Variant: Version without Output_Original_Graph
According to a first variant, the variable output_original_graph (introduced above) can be omitted. In this variant, the reconstruction depends on a topology_storage_format variable. With regards to the exemplary NNR tables introduced above, the table “NNR model parameter set payload syntax” can thus be modified (since the variable output_original_graph is not required).
With the exemplary syntax detailed herein, in link with the MPEG NNR standard, this leads to the following table
For the reconstruction method, the following applies:
This process can be invoked once per NNR compressed payload, after decoding of the unit header nnr_compressed_data_unit_header and the compressed payload.
The output of the current tensor can be specified as follows:
topology_storage_format specifies the format of the stored neural network topology information, as specified below:
In a second variant, information can be provided in the signaling regarding a performance of the decomposition process. For instance, such an information can be representative of a mapping between different Mean Square Error (MSE) values between the decomposed tensors and their original version and resulting Neural Network (NN) inference accuracies.
The resulting accuracies can be provided separately for different aspects or characteristics of the output of the NN. For a classifier NN, each MSE value (e.g. threshold) can be mapped to separate accuracies for each class, in addition to an overall accuracy which considers all classes. In some embodiments, classes can be ordered based on the neural network output order, i.e., the order specified during training.
With the exemplary syntax detailed herein, in link with the MPEG NNR standard, the following table can be used
where Decomposition_performance_map( ) can be defined for instance as follows:
where:
This third variant is detailed, for the ease of explanation, using an exemplary syntax compatible with an exemplary MPEG NNR draft standard (like N19225—Working Draft 4 of Compression of neural networks for multimedia content description and analysis». International Organization for Standardization ISO/IEC JTC1/SC29/WG11, April 2020), the exemplary syntax including units of type NNR_PT_BLOCK. With such an exemplary syntax, multiple parameters can be present within the unit, e.g. weight tensors that can be optionally decomposed, biases, batch norm parameters, etc. In the context of an NNR_PT_BLOCK unit, the output tensor of weights has to comply with the original shape that can be either transmitted or known by the decoder, through external topology information for instance. If tensor decomposition was applied at the encoder, the reconstruction of the tensor is performed at the decoder.
In this variant, it is proposed to simplify the process of reconstruction and the required syntax for specifying the sizes of the decomposed tensors G and H, sometimes also called at the decoder RecWeightG and RecWeightH, respectively.
The decoder needs to derive the sizes of G and H tensors, e.g. when the layer is of type convolutional (CONV) or depth-wise convolution (DWCONV). Some embodiments of the present disclosure thus propose to transmit information related to the size of G and/or H tensors. The size of the G and/or H tensor refers to the a size of a dimension of the G and/or H tensor, such as a number of rows, or a number of columns of the tensor.
According to an exemplary syntax, such information can be added to the High Level Syntax (HLS), for instance in a compressed data unit header. For instance, in an exemplary syntax (like an exemplary syntax compatible with some draft of MPEG NNR), the information related to the tensors size of G and/or H and the rank decomposition_rank can be transmitted to the decoder in the NNR compressed data unit header, as follows:
Where g_number_of_rows specifies the number of rows of matrix g in the case where the reconstruction is performed for decomposed tensors in an NNR unit of type NNR_PT_BLOCK. This information can enable the decoder to perform the reconstruction in the case of an NNR_PT_BLOCK as follows (with (block_parameter_types & 0x01)!=0 specifying that Low rank decomposed weights are present)
If (block_parameter_types & 0x01)!=0, the following applies:
The decoding process for an integer weight tensor can be invoked with input variable TensorDims set to [g_number_of_rows, decomposition_rank]. A variable RecWeightG can be set to the output variable RecParam.
The number of columns h_number_of_columns of matrix h in the case where the reconstruction is performed for decomposed tensors in an NNR unit of type NNR_PT_BLOCK can be obtained by
RecWeight=TensorReshape(RecWeightG*RecWeightH,tensor_dimensions)
With the above exemplary syntax, the variable g_number_of_rows can be now available from the unit header. Then, the variable h_number_of_columns can be derived since the dimensions of the output tensor_dimensions (tensor_dimensions) are known. Such an embodiment can thus enable for instance the decoder to separately decode tensors G and H and then reshape their product to obtain the reconstructed tensor RecWeight.
In other variants, the variable h_number_of_columns can be transmitted (for instance similarly to the g_number_of_rows in the third variant detailed above, for being available from the unit header), the g_number_of_rows being derived at the decoder side.
In still other variants, both the variables g_number_of_rows and h_number_of_columns can be transmitted (for instance similarly to the g_number_of_rows in the third variant detailed above, for being both available from the unit header), thus avoiding the corresponding computation at the decoder side for instance.
At 603, the first tensor is decoded based on the decoded size. At 604, a size of the second tensor is derived from the decoded size. For instance, when the second tensor is an H tensor, the size of the second tensor is a number of columns of the H tensor. At 605, the second tensor is decoded based on the derived size. At 606, the decoder can reconstruct the decomposed tensor from the decoded first and second tensors.
In some embodiments, the bitstream contains the size of the second tensor instead of the size of the first tensor, or both sizes.
This application describes a variety of aspects, including tools, features, embodiments, models, approaches, etc. Many of these aspects are described with specificity and, at least to show the individual characteristics, are often described in a manner that may sound limiting. However, this is for purposes of clarity in description, and does not limit the application or scope of those aspects. Indeed, all of the different aspects can be combined and interchanged to provide further aspects. Moreover, the aspects can be combined and interchanged with aspects described in earlier filings as well.
The aspects described and contemplated in this application can be implemented in many different forms.
In the present application, the terms “reconstructed” and “decoded” may be used interchangeably, the terms “pixel” and “sample” may be used interchangeably, the terms “image,” “picture” and “frame” may be used interchangeably. Usually, but not necessarily, the term “reconstructed” is used at the encoder side while “decoded” is used at the decoder side.
Various methods and other aspects described in this application can be used to modify modules, for example, the intra prediction, entropy coding, and/or decoding modules (160, 260, 145, 230), of an encoder 100 and decoder 200 as shown in
Moreover, the present aspects are not limited to VVC or HEVC, or even to video data, and can be applied to an encoder or decoder adapted to encode, respectively decode, at least one tensor of at least one layer of a neural network that can be used in many technical fields other than video (of course, in such embodiments, some modules like intra prediction module 160 can be optional)
Unless indicated otherwise, or technically precluded, the aspects described in this application can be used individually or in combination.
Various numeric values are used in the present application (for example tensor-reconstruction-modes). The specific values are for example purposes and the aspects described are not limited to these specific values.
Before being encoded, the sequence may go through pre-encoding processing (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0) in case of a video sequence, or performing a remapping of the input picture components in order to get a signal distribution more resilient to compression (for instance using a histogram equalization of one of the color components). Also, pre-encoding processing can include binarization as the exemplary binarization detailed above in link with CABAC.
Metadata can be associated with the pre-processing and attached to the bitstream.
In the encoder 100, in case of a video sequence, a picture is encoded by the encoder elements as described below. The picture to be encoded is partitioned (102) and processed in units of, for example, CUs. Each unit is encoded using, for example, either an intra or inter mode. When a unit is encoded in an intra mode, it performs intra prediction (160). In an inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which one of the intra mode or inter mode to use for encoding the unit, and indicates the intra/inter decision by, for example, a prediction mode flag. Prediction residuals are calculated, for example, by subtracting (110) the predicted block from the original image block.
The prediction residuals are then transformed (125) and quantized (130).
The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (145) to output a bitstream.
The encoder can skip the transform and apply quantization directly to the non-transformed residual signal. The encoder can bypass both transform and quantization, i.e., the residual is coded directly without the application of the transform or quantization processes.
The encoder decodes an encoded block to provide a reference for further predictions. The quantized transform coefficients are de-quantized (140) and inverse transformed (150) to decode prediction residuals. For instance, in case of a video sequence, combining (155) the decoded prediction residuals and the predicted block, an image block is reconstructed. In-loop filters (165) are applied to the reconstructed picture to perform, for example, deblocking/SAO (Sample Adaptive Offset) filtering to reduce encoding artifacts. The filtered image is stored at a reference picture buffer (180).
In particular, the input of the decoder 200 includes a bitstream, which can be generated by encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information.
In case of a video bitstream, the picture partition information indicates how the picture is partitioned. The decoder may therefore divide (235) the picture according to the decoded picture partitioning information. The transform coefficients are de-quantized (240) and inverse transformed (250) to decode the prediction residuals. Combining (255) the decoded prediction residuals and the predicted block, an image block is reconstructed. The predicted block can be obtained (270) from intra prediction (260) or motion-compensated prediction (i.e., inter prediction) (275). In-loop filters (265) are applied to the reconstructed image. The filtered image is stored at a reference picture buffer (280).
The decoded element (like the picture or the layer weights) can further go through post-decoding processing (285), for example, in case of a decoded image, an inverse color transform (e.g. conversion from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping performing the inverse of the remapping process performed in the pre-encoding processing (101). The post-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream.
The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this document. Processor 1010 can include embedded memory, input output interface, and various other circuitries as known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device, and/or a non-volatile memory device). System 1000 includes a storage device 1040, which can include non-volatile memory and/or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random-Access Memory (DRAM), Static Random-Access Memory (SRAM), flash, magnetic disk drive, and/or optical disk drive. The storage device 1040 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and/or a network accessible storage device, as non-limiting examples.
System 1000 includes an encoder/decoder module 1030 configured, for example, to process data to provide an encoded or decoded data stream (such a video stream and/or a stream representative of at least one weight of at least one layer of at least one DNN), and the encoder/decoder module 1030 can include its own processor and memory. The encoder/decoder module 1030 represents module(s) that can be included in a device to perform the encoding and/or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, encoder/decoder module 1030 can be implemented as a separate element of system 1000 or can be incorporated within processor 1010 as a combination of hardware and software as known to those skilled in the art.
Program code to be loaded onto processor 1010 or encoder/decoder 1030 to perform the various aspects described in this document can be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. In accordance with various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder/decoder module 1030 can store one or more of various items during the performance of the processes described in this document. Such stored items can include, but are not limited to, the input video, the decoded video or portions of the decoded video, data representative of at least one weight of at least one tensor of at least one layer of the at least one DNN, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.
In some embodiments, memory inside of the processor 1010 and/or the encoder/decoder module 1030 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device can be either the processor 1010 or the encoder/decoder module 1030) is used for one or more of these functions. The external memory can be the memory 1020 and/or the storage device 1040, for example, a dynamic volatile memory and/or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of, for example, a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for coding and decoding operations, such as for MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also referred to as ISO/IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).
The input to the elements of system 1000 can be provided through various input devices as indicated in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and/or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples, not shown in
In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) down converting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the down converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various of these functions, including, for example, down converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, down converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and/or add other elements performing similar or different functions. Adding elements can include inserting elements in between existing elements, such as, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.
Additionally, the USB and/or HDMI terminals can include respective interface processors for connecting system 1000 to other electronic devices across USB and/or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, can be implemented, for example, within a separate input processing IC or within processor 1010, as necessary. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface ICs or within processor 1010, as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 1010, and encoder/decoder 1030 operating in combination with the memory and storage elements to process the data stream as necessary for presentation on an output device.
Various elements of system 1000 can be provided within an integrated housing, Within the integrated housing, the various elements can be interconnected and transmit data therebetween using suitable connection arrangement 1140, for example, an internal bus as known in the art, including the Inter-IC (I2C) bus, wiring, and printed circuit boards.
The system 1000 includes communication interface 1050 that enables communication with other devices via communication channel 1060. The communication interface 1050 can include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 1060. The communication interface 1050 can include, but is not limited to, a modem or network card and the communication channel 1060 can be implemented, for example, within a wired and/or a wireless medium.
Data is streamed, or otherwise provided, to the system 1000, in various embodiments, using a wireless network such as a Wi-Fi network, for example IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal of these embodiments is received over the communications channel 1060 and the communications interface 1050 which are adapted for Wi-Fi communications. The communications channel 1060 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 1000 using a set-top box that delivers the data over the HDMI connection of the input block 1130. Still other embodiments provide streamed data to the system 1000 using the RF connection of the input block 1130. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth network.
The system 1000 can provide an output signal to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and/or a foldable display. The display 1100 can be for a television, a tablet, a laptop, a cell phone (mobile phone), or another device. The display 1100 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop). The other peripheral devices 1120 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and/or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide a function based on the output of the system 1000. For example, a disk player performs the function of playing the output of the system 1000.
In various embodiments, control signals are communicated between the system 1000 and the display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV. Link, Consumer Electronics Control (CEC), or other communications protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to system 1000 using the communications channel 1060 via the communications interface 1050. The display 1100 and speakers 1110 can be integrated in a single unit with the other components of system 1000 in an electronic device such as, for example, a television. In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (T Con) chip.
The display 1100 and speaker 1110 can alternatively be separate from one or more of the other components, for example, if the RF portion of input 1130 is part of a separate set-top box. In various embodiments in which the display 1100 and speakers 1110 are external components, the output signal can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.
The embodiments can be carried out by computer software implemented by the processor 1010 or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 1020 can be of any type appropriate to the technical environment and can be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 1010 can be of any type appropriate to the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.
Various implementations involve decoding. “Decoding”, as used in this application, can encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. In various embodiments, such processes also, or alternatively, include processes performed by a decoder of various implementations described in this application.
As further examples, in one embodiment “decoding” refers only to entropy decoding, in another embodiment “decoding” refers only to differential decoding, and in another embodiment “decoding” refers to a combination of entropy decoding and differential decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.
Various implementations involve encoding. In an analogous way to the above discussion about “decoding”, “encoding” as used in this application can encompass all or part of the processes performed, for example, on an input sequence in order to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, for example, partitioning, differential encoding, transformation, quantization, and entropy encoding. In various embodiments, such processes also, or alternatively, include processes performed by an encoder of various implementations described in this application.
As further examples, in one embodiment “encoding” refers only to entropy encoding, in another embodiment “encoding” refers only to differential encoding, and in another embodiment “encoding” refers to a combination of differential encoding and entropy encoding. Whether the phrase “encoding process” is intended to refer specifically to a subset of operations or generally to the broader encoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.
Note that the syntax elements as used herein, are descriptive terms. As such, they do not preclude the use of other syntax element names.
When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of a corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of a corresponding method/process.
Various embodiments refer to parametric models or rate distortion optimization. In particular, during the encoding process, the balance or trade-off between the rate and distortion is usually considered, often given the constraints of computational complexity. It can be measured through a Rate Distortion Optimization (RDO) metric, or through Least Mean Square (LMS), Mean of Absolute Errors (MAE), or other such measurements. The rate distortion optimization is usually formulated as minimizing a rate distortion function, which is a weighted sum of the rate and of the distortion. There are different approaches to solve the rate distortion optimization problem. For example, the approaches may be based on an extensive testing of all encoding options, including all considered modes or coding parameters values, with a complete evaluation of their coding cost and related distortion of the reconstructed signal after coding and decoding. Faster approaches may also be used, to save encoding complexity, in particular with computation of an approximated distortion based on the prediction or the prediction residual signal, not the reconstructed one. Mix of these two approaches can also be used, such as by using an approximated distortion for only some of the possible encoding options, and a complete distortion for other encoding options. Other approaches only evaluate a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a complete evaluation of both the coding cost and related distortion.
The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable/personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.
Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.
Additionally, this application may refer to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.
Further, this application may refer to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.
Additionally, this application may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.
It is to be appreciated that the use of any of the following “/”, “and/or”, and “at least one of”, for example, in the cases of “A/B”, “A and/or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and/or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.
Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments the encoder signals at least one of a plurality of transforms, coding modes or flags. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various embodiments. It is to be appreciated that signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.
As will be evident to one of ordinary skill in the art, implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.
We describe a number of embodiments. Features of these embodiments can be provided alone or in any combination, across various claim categories and types. Further, embodiments can include one or more of the following features, devices, or aspects, alone or in any combination, across various claim categories and types:
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2021/065522 | 6/9/2021 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63040048 | Jun 2020 | US | |
| 63050052 | Jul 2020 | US |
