Patent Application
20200366914
A significant fraction of Internet traffic involves the transmission of video content, and that fraction will likely continue to increase into the foreseeable future. Because image compression is at the core of substantially all video coding approaches, improvements in the compression of image data are expected to have a significant and beneficial impact on the transmission of video as well. Traditional approaches to performing image compression have utilized compression codecs that rely on hand-crafting of individual components. More recently, several neural network based approaches for image compression have been developed.
In conventional neural network based approaches to image compression, a rate-distortion objective function is typically optimized over a corpus of images in order to find functions for encoding and decoding that are parameterized by a neural network. Once this optimization is complete, the training phase for the neural network is concluded and the encoder function is stored at a sender, while the decoder function is stored at a receiver.
There are provided systems and methods for performing content adaptive optimization for neural data compression, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims.
The following description contains specific information pertaining to implementations in the present disclosure. One skilled in the art will recognize that the present disclosure may be implemented in a manner different from that specifically discussed herein. The drawings in the present application and their accompanying detailed description are directed to merely exemplary implementations. Unless noted otherwise, like or corresponding elements among the figures may be indicated by like or corresponding reference numerals. Moreover, the drawings and illustrations in the present application are generally not to scale, and are not intended to correspond to actual relative dimensions.
The present application discloses a neural data compression solution that overcomes the drawbacks and deficiencies in the conventional art. By contrast, the present application introduces an iterative procedure which adapts the latent representation encoded by a neural encoder to the specific content being compressed while keeping the parameters of the neural network and the predictive model fixed. The present solution delivers an overall increase in rate-distortion performance, independently of the specific architecture used to implement the neural network.
Thus, while conventional approaches to improving neural data compression have focused on more efficient architectures and predictive models, the content adaptive optimization approach disclosed in the present application refines, i.e., improves, the latent representations individually, on a per-image basis, during the encoding process. Due to this per-image adaptation, the refined latent image representation encoded using the present solution is more efficient in terms of rate-distortion performance compared to a latent representation obtained with a simple forward pass through the neural encoder.
The solution disclosed in the present application is general and, as such, can be applied to improve a number of different architectures for learned data compression. A significant advantage of the present neural data compression solution lies in its enablement of an improved compression performance while the neural compression network and the predictive model are kept fixed, and the computing time on the decoder side remains unchanged.
It is noted that, as defined in the present application, an artificial neural network (ANN), or simply neural network (NN) is a type of machine learning framework in which patterns or learned representations of observed data are processed using highly connected computational layers that map the relationship between inputs and outputs. A “deep neural network”, in the context of deep learning, may refer to a neural network that utilizes multiple hidden layers between input and output layers, which may allow for learning based on features not explicitly defined in raw data. “Online deep learning” may refer to a type of deep learning in which machine learning models are updated using incoming data streams, and are designed to progressively improve their performance of a specific task as new data is received and/or adapt to new patterns of a dynamic system. As such, various forms of ANNs may be used to make predictions about new data based on past examples or “training data”. In various implementations, ANNs may be utilized to perform image processing or natural-language processing.
As further shown in
It is noted that, in some implementations, data processing system 100 may include compressed data receiver 126, as well as computing platform 102. It is further noted that neural decoder 114b of compressed data receiver 126 is substantially identical to neural decoder 114a of data compression software code 110. That is to say, neural decoder 114b is parameterized by the same ANN having the same architecture used to parametrize neural decoder 114a. It is also noted that although data processing system 100 may receive series of compression input data 130 from data source 108 via communication network 120 and network communication links 122, in some implementations, data source 108 may be integrated with computing platform 102, or may be in direct communication with data processing system 100 as shown by dashed communication link 118.
Series of compression input data 130 may include compressible data in a variety of forms, including images, audio data, two-dimensional (2D) motion fields, and color residuals, to name a few examples. In implementations in which the compression input data corresponds to images, for example series of compression input data 130 may correspond to a video stream, for example, where each compression input data is an individual frame of video. However, in other implementations, each of compression input data 130 may be an individual color value or data describing a 2D motion.
Although the present application refers to data compression software code 110 as being stored in system memory 106 for conceptual clarity, more generally, data compression software code 110 may be stored on any computer-readable non-transitory storage medium. The expression “computer-readable non-transitory storage medium,” as used in the present application, refers to any medium, excluding a carrier wave or other transitory signal, capable of providing instructions to a hardware processor, such as hardware processor 104 of computing platform 102, for example. Thus, a computer-readable non-transitory medium may correspond to various types of media, such as volatile media and non-volatile media, for example. Volatile media may include dynamic memory, such as dynamic random access memory (dynamic RAM), while non-volatile memory may include optical, magnetic, or electrostatic storage devices. Common forms of computer-readable non-transitory media include, for example, optical discs, RAM, programmable read-only memory (PROM), erasable PROM (EPROM), and FLASH memory.
Moreover, although
According to the implementation shown by
Series of compression input data 230, data compression software code 210 including neural encoder 212 and neural decoder 214a, and compressed data bitstream 228 correspond respectively in general to series of compression input data 130, data compression software code 110 including neural encoder 112 and neural decoder 114a, and compressed data bitstream 128, in
It is noted that the goal of lossy data compression is to find a mapping or encoding function ψ:X→ from the input space X to a latent space representation
using neural encoder 112/212 and its reverse mapping or decoding function ϕ:
→X back to the original input space using neural decoder 114a/214a/114b. That mapping and reverse mapping are subject to the following competing constraints: (1) the latent representation should occupy as little storage as possible while, (2) the reconstructed image should closely resemble the original image.
In neural data compression, this mapping is realized with a neural encoder-decoder pair corresponding to neural encoder 112/212 and neural decoder 114a/214a/114b, where the bottleneck values constitute the latent representation. An image x is first mapped to its latent space representation y=ψ(x). After quantization, the resulting latents ŷ are coded losslessly to a bit stream that can be decoded into the image {circumflex over (x)}=ϕ(ŷ).
Data compression can be formally expressed as the minimization of both the expected length of the bitstream, as well as the expected distortion of the reconstructed image {acute over (x)} compared to the original x, which leads to the optimization of the following rate-distortion trade-off:
L(ψ,ϕ,pŷ)=x˜p
Here, −log2 pŷ(ŷ) is the rate and d(x, {circumflex over (x)}) is the distortion measure, e.g. mean squared error. The rate corresponds to the length of compressed data bitstream 128/228 needed to encode the quantized representation ŷ, based on a learned entropy model pŷ over the unknown distribution of natural compression input data px. The weight λ steers the rate distortion trade-off, e.g. reducing λ leads to a higher compression rate at the cost of a larger distortion of the reconstructed image.
Conventional approaches to improving compression results have attempted to optimize the neural network architecture to achieve more powerful encoder/decoder transformations by neural encoder 112/212 and neural decoder 114a/214a/114b. In existing approaches, Equation 1 is optimized over a corpus of potentially millions of compression input data in order to find optimal functions for encoding and decoding (ψ and ϕ), along with a suitable probability model pŷ for the latent space utilized by neural encoder 112/212 and neural decoder 114a/214a/114b.
Although neural encoder 112/212 and neural decoder 114a/214a/114b have been trained over a large corpus of compression input data to find what should ideally be an optimal encoding function over the whole data set, according to the present inventive concepts the encoding can still be improved by adapting to each individual image. According to various implementations of the present concepts, this per-image adaptation is performed without changing neural encoder 112/212 and neural decoder 114a/214a/114b or the parameters of their latent space probability model, but by changing the latent values themselves when sending compression input data. As such, the novel and inventive approach disclosed in the present application is effectively working to solve an optimization problem when sending compression input data, for each image individually. For example, for the single image x, the present approach works to solve the following:
The fact that the present data compression solution does not change the probability model utilized by neural encoder 112/212 and neural decoder 114a/214a/114b when performing content adaptive optimization is due to the assumption that neural decoder 114b and the probability model have been trained and deployed to compressed data receiver 126. Therefore, the present data compression solution seeks to find the best discrete latent representation for each image by varying only the latent values themselves.
There are several practical techniques for solving this problem, including both discrete and continuous optimization approaches. According to one exemplary implementation, the optimization problem described by Equation 2 is solved through an iterative procedure where gradient descent is applied on the latents according to:
y
t+1
=y
t−η∇yL(ψ,ϕ,pŷ,x) (Equation 3)
Here, L(ψ,ϕ,pŷ,x) is the rate-distortion objective for a particular image x:
L(ψ,ϕ,pŷ,x)=log2pŷ(ŷ)+λd(x,{circumflex over (x)}), (Equation 4)
and η is the weighting applied to the gradient. This procedure requires a differentiable approximation of the quantization operation performed in the bottleneck and, in one implementation, additive uniform noise is used for this purpose. Adopting the notation for an independent uniform noise of width 1, the density function pŷ of the random variable ŷ=y+
(−½,½) becomes a continuous differentiable relaxation of the probability mass function pŷ.
According to one implementation, a complete data compression pipeline is described by Algorithm 300, in
Referring now to
It is noted that neural encoder 112/212 and neural decoder 114a/214a/114b may be trained so as to be parameterized by an ANN, and neural decoder 114b and the probability model utilized by neural encoder 112/212 and neural decoder 114a/214a/114b may be deployed to compressed data receiver 126, prior to the start of the method outlined by flowchart 450.
Flowchart 450 begins with receiving series of compression input data 130/230 (action 451). As noted above, series of compression input data 130/230 may include compressible data in a variety of forms, including images, audio data, 2D motion fields, and color residuals, to name a few examples. In implementations in which the compression input data corresponds to images, for example series of compression input data 130/230 may correspond to a video stream, for example, where each of compression input data 130/230 is an individual frame of video. However, in other implementations, each of compression input data 130/230 may be an individual color value or data describing a 2D motion. Series of compression input data 130/230 may be received by neural encoder 112/212 of data compression software code 110/210, executed by hardware processor 104.
As noted above, in some implementations series of compression input data 130/230 may be received from data source 108 via communication network 120 and network communication links 122. However and as also noted above, in some implementations, data source 108 may be integrated with computing platform 102, or may be in direct communication with data processing system 100 as shown by dashed communication link 118. Thus, in some implementations, series of compression input data 130/230 may be received as transfer of data within data processing system 100.
Flowchart 450 continues with encoding first compression input data 232 of series of compression input data 130/230 to latent space representation 234 of first compression input data 232 (action 452). First compression input data 232 of series of compression input data 130/230 may be encoded to latent space representation 234 of first compression input data 232 by neural encoder 112/212 of data compression software code 110/210, executed by hardware processor 104. As discussed above, neural encoder 112/212 is parameterized during training using an ANN and is configured to encode first compression input data 232 to latent space representation 234 of first compression input data 232.
Flowchart 450 continues with decoding latent space representation 234 of first compression input data 232 to produce input space representation 236 of first compression input data 232 that corresponds to latent space representation 234 of first compression input data 232 (action 453). Latent space representation 234 of first compression input data 232 may be decoded to produce input space representation 236 of first compression input data 232 by neural decoder 114a/214a of data compression software code 110/210, executed by hardware processor 104. As discussed above, neural decoder 114a/214a, as well as neural decoder 114b deployed to compressed data receiver 126, are parameterized during training using an ANN and are configured to decode latent space representation 234 of first compression input data 232 to produce input space representation 236 of first compression input data 232.
Flowchart 450 continues with generating first compression input data refined latent values 238 for re-encoding first compression input data 232 based on a comparison of first compression input data 232 with input space representation 236 of first compression input data 232 (action 454). First compression input data refined latent values 238 may be generated by latent representation optimizer 240 of data compression software code 110/210, executed by hardware processor 104, in the manner described above. That is to say, first compression input data refined latent values 238 may be generated through comparison of first compression input data 232 with input space representation 236 of first compression input data 232 based on Equations 2, 3, and 4 above.
It is emphasized that first compression input data refined latent values 238 do not change any parameters of the latent space probability model of neural encoder 112/212, which is the same latent space probability model used by neural decoder 114a/214a/114b. Consequently, first compression input data refined latent values 238 do not change any parameters of latent space representation 234 of first compression input data 232, only the latent values applied to those parameters.
Flowchart 450 continues with, re-encoding first compression input data 232 using first compression input data refined latent values 238 to produce first compressed data ŷ1 corresponding to first compression input data 232 (action 455). First compression input data 232 may be re-encoded to produce first compressed data ŷ1 corresponding to first compression input data 232 by neural encoder 112/212 of data compression software code 110/210, executed by hardware processor 104.
Flowchart 450 can conclude with repeating actions 452, 452, 454, and 455 above on a per-image basis on second compression input data x2 and each of subsequent compression input data x3, . . . , xn of series of compression input data 130/230 to produce compressed data compression input data ŷ2, . . . , ŷn of compressed data bitstream 128/232 (action 456). Actions 452, 453, 454, and 455 may be repeated for each of compression input data x2, . . . , xn by data compression software code 110/210, executed by hardware processor 104, as described above.
For example, hardware processor 104 may execute data compression software code 110/210 to encode second compression input data x2 to latent space representation y2 of second compression input data x2, decode latent space representation y2 to produce input space representation {acute over (x)}2 of second compression input data x2 corresponding to latent space representation y2, generate second compression input data refined latent values for re-encoding second compression input data x2 based on a comparison of second compression input data x2 with input space representation {acute over (x)}2 of second compression input data x2 and re-encode second compression input data x2 using the second compression input data refined latent values to produce second compressed data ŷ2 corresponding to second compression input data x2, and so forth for compression input data x3, . . . , xn.
It is noted that, although not included in flowchart 450, in some implementations, the present method can include transmitting compressed data bitstream 128/228 including first compressed data ŷ1 to compressed data receiver 126 for decoding by neural decoder 114b and rendering by display 124. It is noted that display 124 may be implemented as a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or another suitable display screen that performs a physical transformation of signals to light. The transmitting of compressed data bitstream 128/228 to compressed data receiver 126 for decoding by neural decoder 114b and rendering by display 124 may be performed by data compression software code 110/210, executed by hardware processor 104 of computing platform 102.
Thus, the present application discloses a content adaptive optimization solution for neural data compression. The solution disclosed in the present application introduces an iterative procedure which adapts the latent representation encoded by a neural encoder to the specific content being compressed while advantageously keeping the parameters of the neural network and the predictive model fixed. The present solution delivers an overall increase in rate-distortion performance, independently of the specific architecture used to implement the neural network. Thus, the latent space adaptation techniques disclosed herein can be an effective strategy to make a given encoding process more powerful and content adaptive. This can be particularly advantageous in use cases such as content streaming, where the encoding complexity is not the limiting factor when compared to the transmission and decoding.
From the above description it is manifest that various techniques can be used for implementing the concepts described in the present application without departing from the scope of those concepts. Moreover, while the concepts have been described with specific reference to certain implementations, a person of ordinary skill in the art would recognize that changes can be made in form and detail without departing from the scope of those concepts. As such, the described implementations are to be considered in all respects as illustrative and not restrictive. It should also be understood that the present application is not limited to the particular implementations described herein, but many rearrangements, modifications, and substitutions are possible without departing from the scope of the present disclosure.
