Patent Application
20230353765
The present invention generally relates to data compression and more specifically to lossless compression utilizing probabilistic circuits.
Data compression, source coding, or bit-rate reduction may be described as a process of encoding information using fewer bits than an original representation. Generally, an encoder may perform compression and a decoder may perform decompression, which may be a reversal of the compression process to retrieve the original representation.
Typically, particular data compression processes may be either lossy or lossless. Lossy compression may result in some loss of information, where the reduction of bits is typically achieved by removing unnecessary or less important information. In contrast, lossless compression results in no loss of information, where reduction of bits may be provided by identifying and reducing statistical redundancy.
The various embodiments of the present lossless compression with probabilistic circuits (PCs) contain several features, no single one of which is solely responsible for their desirable attributes. Without limiting the scope of the present embodiments, their more prominent features will now be discussed below. In particular, the present systems, methods, and devices for lossless compression with PCs will be discussed in the context of compression of image data. However, the use of image data is merely exemplary and various other data sets may be used for lossless compress with PCs as appropriate to the requirements of a specific application in accordance with embodiments of the invention. After considering this discussion, and particularly after reading the section entitled “Detailed Description,” one will understand how the features of the present embodiments provide the advantages described here.
One aspect of the present embodiments includes the realization that despite extensive progress on image generation, common deep generative architectures may have certain restrictions when applied to lossless compression. For example, Variational Autoencoders (VAEs) suffer from a compression cost overhead due to their latent variables. This overhead can only be partially eliminated with elaborate schemes such as bits-back coding, often resulting in poor single-sample compression rates. To overcome such problems, the present embodiments establish a new class of tractable lossless compression models that permit efficient encoding and decoding: Probabilistic Circuits (PCs). PCs may be described as computation graphs of sum (representing probabilistic mixtures) and product (representing probabilistic factorization) units that may output likelihoods given input features. Further, in practice, the term PC may be referred to by other names such as, but not limited to, “tractable probabilistic models” and “sum-product networks.” PCs are a class of neural networks involving |p| computational units that may support efficient marginalization over arbitrary subsets of the D feature dimensions, enabling efficient arithmetic coding, as further described below. In many embodiments, efficient encoding and decoding schemes may be derived that both have time complexity (log(D)|p|), where a naive scheme would have linear costs in D and |p|, making the approach highly scalable. Empirically, PC-based (de)compression processes may run 40 times faster than neural compression methods that achieve similar bitrates. By scaling up the traditional PC structure learning pipeline, state-of-the-art results may be achieved on image datasets such as, but not limited to, Modified National Institute of Standards and Technology (MNIST). Furthermore, PCs may be naturally integrated with existing neural compression methods to improve the performance of these base models on various datasets including, but not limited to, natural image datasets. The present embodiments provide results that highlight the potential impact that non-standard learning architectures may have on neural data compression.
Therefore, good generative performance does not imply good compression performance for lossless compression, as the model needs to support efficient processes to encode and decode close to the model's theoretical rate estimate. While both Flow- and VAE-based compression methods support efficient and near-optimal compression under certain assumptions (e.g., the existence of an additional source of random bits), the present embodiments illustrate that PCs are suitable for lossless compression tasks. In various embodiments, this class of tractable models may have a particular structure that allows efficient marginalization of its random variables—a property that, as further described below, enables efficient conditional entropy coding. Thus, the present embodiments introduce PCs as backbone models and develop (de)compression processes that may achieve high compression ratios and high computational efficiency.
Similar to other neural compression methods, the present lossless compression approach may operate in two main phases—(i) learn good PC models that approximate the data distribution, and (ii) compress and decompress samples x with computationally efficient processes. The present lossless compression processes may have four main contributions:
Notation - Random variables are denoted by uppercase letters (e.g., X) and their assignments by lowercase letters (e.g., x). Analogously, bold uppercase (e.g., X) and lowercase (e.g., x) letters are used to denote sets of variables and their joint assignments, respectively. The set of all possible joint assignments to variables X is denoted val(X).
In a first aspect, a method for lossless compression using PCs is provided, the method comprising: receiving image data comprising a plurality of pixels, wherein each of the plurality of pixels is represented by a variable; sequentially compressing the variables one-by-one using conditional probabilities, wherein the conditional probabilities are computed by: calculating at least one marginal; initially setting to 1 a probability p(n) for every PC unit n; defining an evali for a set of PC units n that need to be evaluated in an ith iteration; and for i=1 to a dataset D, evaluating PC units n in evali using a bottom-up process and computing a target probability; and generating a bitstream using a streaming code for the compressed variables using the conditional probabilities
In an embodiment of the first aspect, the sequentially compressing the variable one-by-one using the conditional probabilities comprises encoding a next variable by defining a left and right side cumulative probabilities of the next variable given one or more already encoded variables
In another embodiment of the first aspect, the left and right side cumulative probabilities are defined using a 2D conditional probability that is a quotient of two marginals.
In another embodiment of the first aspect, the calculating the at least one marginal comprises: inputting a PC p having a variable instantiation x; outputting a marginal F(x)={p (x1, . . . , xi)}i=1D, wherein D is the dataset.
In another embodiment of the first aspect, each of D terms in F(x) is computed one-by-one.
In another embodiment of the first aspect, each iteration on average re-evaluates only log(D)/D of the PC p.
In another embodiment of the first aspect, the evaluating the PC units in evali is performed in a feedforward process to compute the target probability.
In another embodiment of the first aspect, wherein at iteration i, a set of PC units evali is selected that guarantees correctness of a target marginal, and contains a minimum number of PC units.
In another embodiment of the first aspect, the guarantee of correctness of the target marginal, and containing the minimum number of PC units is achieved by recognizing types of PC units that can be eliminated for evaluation.
In another embodiment of the first aspect, a root node's probability is equivalently computed using a weighted mixture of probabilities of PC units in evali.
In a second aspect, a method for lossless compression using PCs is provided, the method comprising non-transitory computer readable storage medium storing a program comprising instructions is provided, the program that, when executed by at least one processor of a computing device, cause the at least one processor to perform operations including: receiving image data comprising a plurality of pixels, wherein each of the plurality of pixels is represented by a variable; sequentially compressing the variables one-by-one using conditional probabilities, wherein the conditional probabilities are computed by: calculating at least one marginal; initially setting to 1 a probability p(n) for every PC unit n; defining an evali for a set of PC units n that need to be evaluated in an ith iteration; and for i=1 to a dataset D, evaluating PC units n in evali using a bottom-up process and computing a target probability; and generating a bitstream using a streaming code for the compressed variables using the conditional probabilities.
In an embodiment of the second aspect, the sequentially compressing the variable one-by-one using the conditional probabilities comprises encoding a next variable by defining a left and right side cumulative probabilities of the next variable given one or more already encoded variables.
In another embodiment of the second aspect, the left and right side cumulative probabilities are defined using a 2D conditional probability that is a quotient of two marginals.
In another embodiment of the second aspect, the calculating the at least one marginal comprises: inputting a PC p having a variable instantiation x; and outputting a marginal F(x)={p(x1, . . . , xi)}i=1D, wherein D is the dataset.
In another embodiment of the second aspect, each of D terms in F(x) is computed one-by-one.
In another embodiment of the second aspect, each iteration on average re-evaluates only log(D)/D of the PC p.
In another embodiment of the second aspect, the evaluating the PC units in evali is performed in a feedforward process to compute the target probability.
In another embodiment of the second aspect, at iteration i, a set of PC units evali is selected that guarantees correctness of a target marginal, and contains a minimum number of PC units.
In another embodiment of the second aspect, the guarantee of correctness of the target marginal, and containing the minimum number of PC units is achieved by recognizing types of PC units that can be eliminated for evaluation.
In another embodiment of the second aspect, a root node's probability is equivalently computed using a weighted mixture of probabilities of PC units in evali.
The various embodiments of the present lossless compression utilizing probabilistic circuits (PCs) will be discussed in detail with an emphasis on highlighting the advantageous features. These embodiments depict the novel and non-obvious features of lossless compression utilizing PCs shown in the accompanying drawings, which are for illustrative purposes only. These drawings include the following figures:
The following detailed description describes the present embodiments with reference to the drawings. In the drawings, reference numbers label elements of the present embodiments. These reference numbers are reproduced below in connection with the discussion of the corresponding drawing features. Turning now to the drawings, lossless compression utilizing PCs are provided. In many embodiments, one of the keys to lossless compression may be to reliably estimate how frequently any given “symbol” occurs that one desires to compress (e.g., words in language, patches in image data, or DNA sequences for compressing DNA data, etc.). Knowing these probabilities, one can convert frequent symbols into short bit representations, and rare symbols into longer bit representations. For complex data, probability models may be needed to estimate the probabilities of these symbols. The present embodiments utilize machine learning, and in particular generative modeling, since generative models try to assign probabilities to each possible data point.
Existing deep generative models for compression are suboptimal as they can estimate the data probabilities only indirectly and in a very costly fashion. To overcome such problems, the present embodiments analyze a new class of generative models and used them for data compression: Probabilistic Circuits (PCs). These models estimate all relevant probabilities needed for compression in a fast and direct fashion without the need of further approximations. In various embodiments, by scaling up the traditional PC learning pipeline, state-of-the-art compression results may be achieved. For example, compression results on image patch data are provided and, in particular, better results than both classical and recently proposed neural compression schemes are shown. In addition, a significant speed improvement is illustrated: the present embodiments of PC-based (de)compression processes run 40× faster than neural compression methods that achieve similar bitrates.
Probabilistic Circuits (PCs) are an umbrella term of a wide variety of Tractable Probabilistic Models (TPMs). They provide a set of succinct definitions for popular TPMs such as Sum-Product Networks, Arithmetic Circuits, and Probabilistic Sentential Decision Diagrams. The syntax and semantics of a PC are defined as follows.
Definition 1 (Probabilistic Circuits). A PC p(X) represents a probability distribution over X via a parametrized directed acyclic graph (DAG) with a single root node nr. Similar to neural networks, every node of the DAG defines a computational unit. Specifically, each leaf node corresponds to an input unit; each inner node n represents either a sum or a product unit that receives inputs from its children, denoted in (n). Each node n encodes a probability distribution pn, defined as follows:
where fn(·) is an univariate input distribution (e.g., Gaussian, Categorical), and θn,c denotes the parameter that corresponds to edge (n, c). Intuitively, sum and product units encode weighted mixtures and factorized distributions of their children's distributions, respectively. To ensure that a PC models a valid distribution, the present embodiments assume the parameters associated with any sum unit n are normalized: ∀n,Σc∈in(n)θn,c=1. Further, the present embodiments may assume w.l.o.g. that a PC alternates between sum and product units before reaching an input unit. The size of a PC p, denoted |p|, is the number of edges in its DAG.
The present embodiments may focus on PCs that may compute arbitrary marginal queries in time linear in their size, since this may be needed to unlock the efficient (de)compression process. In order to support efficient marginalization, PCs typically need to be decomposable (Def. 2), which is a property of the (variable) scope φ(n) of PC units n, that is, the collection of variables defined by all its descendent input units. Note—Another property called smoothness is may also be needed to compute marginals efficiently. However, since enforcing smoothness on any structured-decomposable PC typically imposes at most an almost-linear increase in its size the present embodiments omit introducing it here (all PCs described herein are structured-decomposable).
Definition 2 (Decomposability). A PC is decomposable if for every product unit n, its children have disjoint scopes: ∀c1, c2 ∈in(n) (c1≠c2), φ(c1)∩φ(c2)=Ø.
A diagram illustrating a structured decomposability (SD) PC in accordance with an embodiment of the invention is shown in
In reference to
Definition 3 (Structured decomposability). A PC is structured-decomposable if (i) it is decomposable and (ii) for every pair of product units (m, n) with identical scope (i.e., φ(m)=φ(n)), we have that |in(m)|=|in(n)| and the scopes of their children are pairwise identical: ∀i∈{1, . . . , |in(m)|}, φ(cmi)=φ(cni), where cmi and cni are the ith child unit of m and n.
In further reference to
As a key sub-routine in the present processes, how to compute marginal queries given a smooth and (structured-)decomposable PC in (|p|) time is provided. First, probabilities to every input unit may be assigned: for an input unit n defined on variable X, if evidence is provided for X in the query (e.g., X=x or X<x), we assign to n the corresponding probability (e.g., p(X=x), p(X<x)) according to fn in Eq. (2) (Eq. 2 is provided below); if evidence of X is not given, probability 1 is assigned to n. Next, a feedforward (children before parents) traverse of inner PC units and compute their probabilities following Eq. (2) may be conducted. The probability assigned to the root unit is the final answer of the marginal query. Concretely, consider computing p(x1,x2, x4) for the PC in
The goal of lossless compression processes is to map every input sample to an output codeword such that (i) the original input can be reconstructed from the codeword, and (ii) the expected length of the codewords is minimized. Practical (neural) lossless compression processes operate in two main phases—learning and compression. In the learning phase, a generative model p(X) may be learned from a dataset :={x(i}i=1N. According to Shannon's source coding theorem (Shannon, 1948), the expected codeword length is lower-bounded by the negative cross-entropy between the data distribution
and the model distribution p(X) (i.e., —Ex˜D[log p(x)]), rendering it a natural and widely used objective to optimize the model.
In the compression phase, compression processes take the learned model p and samples x as input and generate codewords whose expected length approaches the theoretical limit (i.e., the negative cross-entropy between and p). Although there exist various close-to-optimal compression schemes (e.g., Huffman Coding (Huffman, 1952) and Arithmetic Coding (Rissanen, 1976)), a natural question to ask is what are the requirements on the model p such that compression processes can utilize it for encoding/decoding in a computationally efficient manner? In the present embodiments, the advantages of tractable probabilistic models for lossless compression by introducing a concrete class of models that are expressive and support efficient encoding and decoding are highlighted.
To encode a sample x, a standard streaming code operates by sequentially encoding every symbol xi into a bitstream b, such that xi occupies approximately—log p(xi|x1, . . . , xi−1) bits in b. As a result, the length of b is approximately—log p(x). For example, Arithmetic Coding (AC) encodes the Symbols {xi}i=1D (define D :=|X| as the number of features) sequentially by successively refining an interval that represents the sample, starting from the initial interval [0,1). To encode xi, the process partitions the current interval [a,b) using the left and right side cumulative probability of xi:
l
i(xi):=p(Xi<xi|x1, . . . ,xi−1),hi(xi):=p(Xi≤xi|x1, . . . ,xi−1). (1)
Specifically, the process may update [a, b) to the following: [a+(b−a)·li(xi), a+(b−a)·hi(xi)), which is a sub-interval of [a, b). Finally, AC picks a number within the final interval that has the shortest binary representation. This number is encoded as a bitstream representing the codeword of x. Upon decoding, the symbols {xi}i=1D may be decoded sequentially: at iteration i, variable Xi may be decoded by looking up its value x such that its cumulative probability (i.e., li(x)) matches the subinterval specified by the codeword and x1, . . . , xi−1 (Rissanen, 1976); the decoded symbol xi may then be used to compute the following conditional probabilities (i.e., lj(x) for j>i). Despite implementation differences, computing the cumulative probabilities li(x) and hi(x) may be needed for many other streaming codes (e.g., rANS). For these streaming codes, the main computation cost of both the encoding and decoding process comes from calculating li(x) and hi(x).
The main challenge for the above (de)compression process may be to balance the expressiveness of p and the computation cost of {li(x), hi(x)}i=1D. On the one hand, highly expressive probability models such as energy-based models may potentially achieve high compression ratios at the cost of slow runtime, which is due to the requirement of estimating the model's normalizing constant. On the other hand, models that make strong independence assumptions (e.g., n-gram, fully-factorized) may be cheap to evaluate but lack the expressiveness to model complex distributions over structured data such as images. Note—Flow-model-based neural compression methods may adopt p defined on mutually independent latent variables (denoted Z), and may improve expressiveness by learning bijection functions between Z and X (i.e., the input space). This is orthogonal to the present embodiments of directly learn better p. Furthermore, one can naturally integrate the proposed expressive p with bijection functions and achieve better performance as further described below.
The present embodiments explore a middle-ground between the above two extremes. Specifically, it may be asked: are there probabilistic models that are both expressive and permit efficient computation of the conditional probabilities in Eq. (1)? This question may be answered in the affirmative by establishing a new class of tractable lossless compression processes using Probabilistic Circuits (PCs), which are neural networks that can compute various probabilistic queries efficiently. In the following, the empirical and theoretical results of the present (de)compression processes are further described below.
Theoretical findings may include: the present encoding and decoding processes enjoy time complexity (log(D)·|p|), where |p|≥D is the PC model size. The backbone of both processes, formally introduced below in the “Computationally Efficient (De)compression with PCs,” is a process that may compute the 2×D conditional probabilities {li(x), hi(x)}i=1D given any x efficiently, as justified by the following theorem.
Theorem 1 (informal). Let x be a D-dimensional sample, and let p be a PC model of size as proposed in the present embodiments. Thus, computing all quantities {li(xi), hi(xi)}i=1D takes (log(D)·|p|) time. Therefore, en- or decoding x with a streaming code (e.g., Arithmetic Coding) takes
(log(D)·|p|+D)=
(log(D)·|p|) time.
The properties of PCs that enable this efficient lossless compression process will be described below, and the backbone inference process with (log(D)·|p|) time complexity will later be shown as Process 1. Table 1 provides an (incomplete) summary of empirical results. First, the PC-based lossless compression process is fast and competitive. As shown in Table 1, the small PC model achieved a near-SoTA bitrate while being ˜15× faster than other neural compression processes with a similar bitrate. Next, PCs can be integrated with Flow-/VAE-based compression methods. As illustrated in Table 1 (reproduced above), the integrated model significantly improved performance on sub-sampled ImageNet compared to the base IDF model. Computationally efficient (de)compression with PCs in accordance with embodiments of the invention are further discussed below.
Computationally Efficient (De)Compression with PCs
As described above, the task of lossless compression may be considered as calculating conditional probabilities {li(x), hi(x)}i=1D given p and x. This section takes PCs into consideration and demonstrates how these queries can be computed efficiently. In the following, a relevant background on PCs and then the PC-based (de)compression processes are discussed. Then, the optimality and speed of the proposed compressor and decompressor are empirically evaluated.
Efficient (De-)Compression with Structured-Decomposable PCs
A diagram illustrating an overview of a PC-based (de)compressor in accordance with an embodiment of the invention is shown in
In reference to
Therefore, the main computation cost of the above en- and decoding procedures comes from calculating the 2D conditional probabilities {li(x), hi(x)}i=1D efficiently w.r.t. any x. Since every conditional probability can be represented as the quotient of two marginals, it is equivalent to compute the two following sets of marginals: F(x) :={p(x1, . . . , xi)}i=1D and G(x) :={p(xi, . . . , xi−1, Xi<xi)}i=1D.
As a direct application of the marginal process described in the “Background: Probabilistic Circuits” section above, for every x∈val(X), computing the 2D marginals {F(x), G(x)} take (D·|p|) time. However, the linear dependency on D could render compression and decompression extremely time-consuming.
One may significantly accelerate the en- and decoding times if the PC is structured-decomposable (see Definition 3). To this end, one may introduce a method that computes F(x) and G(x) in (log(D)·|p|) time (instead of
(D·|p|), given a smooth and structured-decomposable PC p. For ease of presentation, discussion of how to compute F(x) is provided—the values G(x) can be computed analogously. Note—the only difference between the computation of the ith term of F(x) and the ith term of G(x) is in the value assigned to the inputs for variable Xi (i.e., probabilities pn(Xi=x) vs. pn(Xi<x)).
Before proceeding with a formal argument, a high-level explanation of the acceleration is provided. In practice, one only needs to evaluate a small fraction of PC units to compute each of its D marginals.
This is different from regular neural networks and the key to speeding up the computation of F(x). In contrast to neural networks, changing the input only slightly will leave most activations unchanged for structured-decomposable PCs. One may make use of this property by observing that adjacent marginals in F(x) only differ in one variable—the ith term only adds evidence xi compared to the (i−1)th term. It may be shown that such similarities between the marginal queries will lead to a process that guarantees (log(D)·|p|) overall time complexity.
An informal version of the present method is shown in Process 1. In the main loop (lines 5-6), the D terms in F(x) are computed one-by-one. Although the D iterations seem to suggest that the process scales linearly with D, it may be highlighted that each iteration on average re-evaluates only log(D)/D of the PC. Therefore, the computation cost of Process 1 scales logarithmically w.r.t. D. The set of PC units need to be re-evaluated, evali, is identified in line 4, and lines 6 evaluates these units in a feedforward manner to compute the target probability (i.e., p(x1, . . . , xi)).
Specifically, to minimize computation cost, at iteration i, one may select a set of PC units evali that (i) guarantees the correctness of the target marginal, and (ii) contains the minimum number of units. This may be achieved by recognizing three types of PC units that can be safely eliminated for evaluation. Take the PC shown in
The idea of partially evaluating a PC originates from the Partial Propagation (PP) method. However, PP can only prune away units in Group #2. Thanks to the specific structure of the marginal queries, the present embodiments are able to also prune away units in Groups #1 and #3.
Finally, additional technical details to rigorously state the complexity of Process 1 are provided. First, one needs the variables X to have a specific order determined by the PC p. To reflect this change, F(x) to Fπ(x) :={p(xπ1, . . . , xπi)}i=1D may be generalized, where π defines some variable order over X, i.e., the ith variable in the order defined by π is Xπi. Next, a technical assumption is given and then the correctness and efficiency of Process 1 may be formally justified when using an optimal variable order π*.
Definition 4. For a smooth structured-decomposable PC p over D variables, for any scope φ, denote nodes(p, φ) as the set of PC units in p whose scope is φ. It may be said that p is balanced if for every scope φ′ that is equal to the scope of any unit n in p, we have |nodes(p, φ′)|=(|p|/D).
Theorem 1. For a smooth structured-decomposable balanced PC p over D variables X and a sample x, there exists a variable order π*, s.t. Process 3 correctly computes Fπ*(x) in (log(D)·|p|) time.
Proof. First note that Process 3 is a detailed version of Process 1. The high-level idea of the proof is to first show how to compute the optimal variable order π* for any smooth and structured-decomposable PC. Next, the correctness of Process 3 is justified by showing (i) one only needs to evaluate units that satisfy the criterion in line 6 of Process 3 and (ii) weighing the PC units with the top-down probabilities, as further described below, always give the correct result. Finally, one may use induction (on D) to demonstrate Process 3 computes (log(D)·|p|) PC units in total if π* is used (details provided further below).
While Def. 4 may seem restrictive at first glance, it may be highlighted that most existing PC structures such as EiNets, RAT-SPNs and HCLTs are balanced, as further discussed below. Once all marginal probabilities are calculated, samples x can be en- or decoded autoregressively with any streaming codes in time (log(D)·|p|). Specifically, the present implementation adopted the widely used streaming code rANS.
The present embodiments were compared with a Flow-model-based (IDF by Hoogeboom et al. (2019)) and VAE-based (BitSwap by Kingma et al. (2019)) neural compression methods using the MNIST dataset. First bitrates were evaluated. As shown in Table 2 (reproduced above), the PC (de)compressor achieved compression rates close to its theoretical rate estimate—codeword bpds only have ˜0.04 loss w.r.t. the corresponding theoretical bpds. It may be noted that PC and IDF have an additional advantage: their reported bitrates were achieved while compressing one sample at a time; however, BitSwap needs to compress sequences of 100 samples to achieve 1.31 codeword bpd.
Next, efficiency may be considered. While achieving a better codeword bpd (i.e., 1.30) compared to IDF and BitSwap, a relatively small PC model (i.e., HCLT, M=16) encodes (resp. decodes) images 30× (resp. 10×) faster than both baselines. Furthermore, a bigger PC model (M=32) with 7M parameters achieved codeword bpd 1.24, and is still 5× faster than BitSwap and IDF. Note that at the cost of increasing the bitrate, one can significantly improve the en- and decoding efficiency. For example, by using a small VAE model, others have managed to compress and decompress 10,000 binarized MNIST samples in 3.26 s and 2.82 s, respectively.
Related work As hinted by Tractability Matters in Lossless Compression section, the present embodiments seek to directly learn probability distributions p(X) that are expressive and support tractable (de)compression. In contrast, existing Flow-based and VAE-based neural lossless compression methods may be based on an orthogonal idea: they adopt simple (oftentimes fully factorized) distributions over a latent space Z to ensure the tractability of encoding and decoding latent codes z, and learn expressive neural networks that “transmit” probability mass from Z to the feature space X to compress samples x indirectly. It may be noted that both ideas can be integrated naturally: the simple latent distributions used by existing neural compression processes can be replaced by expressive PC models, as further described below.
Although specific PCs and PC-based (de)compressors are discussed above with respect to
Being equipped with an efficient (de)compressor, the next goal is to learn PC models that may achieve good generative performance on various datasets. Although recent breakthroughs have led to PCs that can generate CelebA and SVHN images, PCs have not been shown to have competitive (normalized) likelihoods on image datasets, which directly influence compression rates. In this section, it will be shown that Hidden Chow-Liu Trees (HCLTs), a PC model initially proposed for simple density estimation tasks containing binary features, may be scaled up to achieve state-of-the-art performance on various datasets such as, but not limited to, image datasets. An introduction of HCLTs and a demonstration of how to scale up their learning and inference (for compression), along with empirical evidence, is provided below.
Hidden Chow-Liu Trees (HCLTs) are smooth and structured-decomposable PCs that combine the ability of Chow-Liu Trees (CLTs) to capture feature correlations and the extra expressive power provided by latent variable models. Every HCLT can be equivalently represented as a Probabilistic Graphical Model (PGM) with latent variables. Diagrams illustrating a process for constructing a Hidden Chow-Liu Tree (HCLT) PC given a dataset D with 4 features in accordance with an embodiment of the invention are shown in , as further described below.
Specifically, containing 4 features X=X1, . . . , X4, a CLT w.r.t. X (
Recent advances in scaling up learning and inference of PCs largely rely on the regularity of the PC architectures they used—the layout of the PCs can be easily vectorized, allowing them to use well-developed deep learning packages such as PyTorch. However, due to the irregular structure of learned CLTs, HCLTs cannot be easily vectorized. To overcome this problem, customized GPU kernels for parameter learning and marginal query computation (i.e., Process 3) based on Juice.jl, an open-source Julia package, may be implemented. The kernels automatically segment PC units into layers such that the computation in every layer can be fully parallelized. As a result, PCs can be trained with millions of parameters in less than an hour and samples may be en- or decoded very efficiently. Implementation details are discussed further below.
Bringing together expressive PCs (i.e., HCLTs) and the present (de)compressors, the compression performance of the proposed PC-based processes is evaluated further below. A comparison with 5 competitive lossless compression processes: JPEG2000; WebP; IDF, a Flow-based lossless compression process; BitSwap, BB-ANS, and McBits, three VAE-based lossless compression methods. All 6 methods were tested on 4 datasets, which include MNIST, FashionMNIST, and two splits of EMNIST. As shown in Table 3 (reproduced above), the present methods out-performed all 5 baselines in 3 out of 4 datasets. On FashionMNIST, where the present approach did not achieve state-of-the-art result, it was only 0.02 bpd worse than BitSwap.
Although specific processes for constructing HCLT PCs are discussed above with respect to
As hinted by previous sections, PCs can be naturally integrated with existing neural compression processes: the simple latent variables distributions used by Flow- and VAE-based lossless compression methods can be replaced by more expressive distributions represented by PCs. In this section, IDF, a Flow-based lossless compression model, will be utilized as an example to demonstrate the effectiveness of such model integration. IDF was chosen because its authors provided an open-source implementation on GitHub. The use of IDF is only exemplary and PC may be integrated with various compression models including, but not limited to, any VAE- and Flow-based model.
A diagram illustrating using PCs as prior distributions of an integer discrete flow (IDF) model in accordance with an embodiment of the invention is shown in
The generative performance of the proposed PC+IDF model on 3 natural image datasets were evaluated: CIFAR10, ImageNet32, and ImageNet64. Results are shown in Table 4 (reproduced above). First, compared to 4 baselines (i.e., IDF, IDF++, Glow, and RealNVP) PC+IDF achieved the best bpd on ImageNet32 and ImageNet64. Next, PC+IDF improved over its base model IDF by 0.04, 0.16, and 0.19 bpd on three datasets, respectively. This shows the benefit of integrating PCs with IDFs. In various embodiments, the performance could be further improved by integrating PCs with better Flow models (e.g., IDF++).
Compression and decompression with the PC+IDF model may be done easily. In many embodiments, the high- level compression process of IDF and replace the parts of en- or decoding latent variables zi with the proposed PC (de)compressor may be adopted. The compression performance of these hybrid models may be improved utilizing the present embodiments. Note that Thm. 1 applies to the PC component, and the compression time still depends linearly on the size of the neural network.
Although specific PCs as prior distributions of an IDF model are discussed above with respect to
Additional details about the process used to compute the conditional probabilities Fπ(x) (i.e., Process 1) and the full proof of the theorems stated above are provided below.
This section provides additional technical details of Process 1. Specifically, (i) how to select the set of PC units evali (cf. Process 1 line 5) and (ii) how to compute p(x1, . . . , xi) as a weighted mixture of Pi (cf. Process 1 line 7) are demonstrated. Using the example in
The key to speeding up the naive marginalization method is the observation that we only need to evaluate a small fraction of PC units to compute each of the D marginals in Fπ(x). Diagrams illustrating impact of variable orders in computational efficiency of Fπ(x) in accordance with an embodiment of the invention are shown in ,
, and ⊚ denote sum, product, and input units, respectively. Model parameters are omitted for simplicity. Consider using the variable order π=(X1, X2, X3) 520 (as illustrated in
For ease of presentation, denote the two input units to compute 522 as n1 and n2, respectively. A key observation is that the probability of every ancestor unit of {n1, n2} (including the root unit) can be represented as a weighted mixture over pn1(x) and pn2(x), the probabilities assigned to n1 and n2, respectively. The reason is that for each decomposable product node m, only distributions defined on disjoint variables shall be multiplied. Since n1 and n2 have the same variable scope, their distributions will not be multiplied by any product node. Following the above intuition, the top-down probability pdown(n) of PC unit n is designed to represent the “weight” of n w.r.t. the probability of the root unit. Formally, pdown(n) is defined as the sum of the probabilities of every path from n to the root unit nr, where the probability of a path is the product of all edge parameters traversed by it. Back to our example, using the top-down probabilities, p(X1=x1)=Σi=12 pdown(ni)·pni(x1) can be computed without explicitly evaluating the ancestors of n1 and n2. The quantity pdown(n) of all PC units n can be computed by Process 2 in (|p|) time. Specifically, the method performs a top-down traversal over all PC units n, and updates the top-down probabilities of their children in(n) along the process.
Therefore, one may only need to compute the two PC units with scope {X1} in order to calculate p(X1=x1). Next, when computing the second term p(X1=x1, X2=x2) 523, as illustrated in
This procedure is formalized in Process 3, which adds additional technical details compared to Process 1. In the main loop (lines 5-9), the D terms in Fπ(x) are computed one-by-one. While computing each term, the PC units that need to be evaluated (line 6) may first be found. After computing their probabilities in a bottom-up manner (line 7), one may additionally use the pre-computed top-down probabilities to obtain the target marginal probability (lines 8-9).
E in(n) do pdown (c) + pdown (c) + pdown (10)
indicates data missing or illegible when filed
The previous example demonstrates that even without a careful choice of variable order, one can significantly lower the computation cost by only evaluating the necessary PC units. As further described below, with an optimal choice of variable order (denoted π*), the cost can be further reduced. For example, consider using order π*=(X3, X2, X1) 540, as shown in (log(D)·|p|) computation time. In the following, a technical assumption is first given and then a justification of the correctness and efficiency of Process 3 when using the optimal variable order π* is provided.
As hinted by the proof sketch provided above, this proof may comprise three main parts—(i) construction of the optimal variable order π* given a smooth and structured-decomposable PC, (ii) justification of the correctness of Process 3, and (iii) proof that Fπ*(x) can be computed by evaluating no more than (log(K)·|p|) PC units (i.e., analyze the time complexity of Process 3).
Construction of an optimal variable order For ease of illustration, the original smooth and structured-decomposable PC may first be transformed into an equivalent PC where every product node has two children. A diagram illustrating converting a product unit 600 with k children into an equivalent PC 610 in accordance with an embodiment of the invention is shown in
The variable tree (vtree) of a smooth and structured- decomposable PC may now be defined. Specifically, a vtree is a binary tree structure whose leaf nodes are labeled with a PC's input features/variables X (every leaf node is labeled with one variable). A PC conforms to a vtree if for every product unit n, there is a corresponding vtree node v such that children of n split the variable scope φ(n) in the same way as the children of the vtree node v. According to its definition, every smooth and structured-decomposable PC whose product units all have two children must conform to a vtree. Diagrams illustrating a structured-decomposable PC, vtrees, and a converted PC equivalent in accordance with an embodiment of the invention are shown in
One may say that a unit n in a smooth and structured-decomposable PC conforms to a node v in the PC's corresponding vtree if their scopes are identical. For ease of presentation, define ϕ(p, v) as the set of PC units that conform to vtree node v. Additionally, one may define ϕsum(p, v) and ϕprod(p, v) as the set of sum and product units in ϕ(p, v), respectively.
such that none of their parents are in eval
) ←Σ
pdown(n) ·
indicates data missing or illegible when filed
Next, an operation that changes a vtree into an ordered vtree may be defined, where for each inner node v, its left child has more descendent leaf nodes than its right child. See
Next, the optimal variable order may be defined. For a pair of ordered PC and ordered vtree, the optimal variable order π* is defined as the order the leaf vtree nodes (each corresponds to a variable) are accessed following an inorder traverse of the vtree (left child accessed before right child).
Correctness of Process 3 Assume we have access to a smooth, structured-decomposable, and ordered PC p and its corresponding vtree. Recall from the above construction, the optimal variable order π* is the order following an inorder traverse of the vtree.
It may be shown that it is sufficient to only evaluate the set of PC units stated in line 6 of Process 3. Using the new definition of vtrees, line 6 in the following equivalent way may be stated. At iteration i (i.e., we want to compute the ith term in Fπ(x):p(xπ1, . . . , xπi)), one needs to evaluate all PC units that conform to any vtree node in the set Tp,i. Here Tp,i is defined as the set of vtree nodes v that satisfy the following condition: Xπi∈φ(v) and there does not exist a child c of v such that {xπj}j=1i ∈φ(c). For ease of presentation, the present embodiments refers to evaluate PC units ϕ(p, v) when it is said “evaluate a vtree node v”.
First, one does not need to evaluate vtree units v where Xπi∉φ(v) because the probability of these PC units will be identical to that at iteration i−1 (i.e., when computing p(xπ1, . . . , xπi−1)). Therefore, one only needs to cache these probabilities computed in previous iterations.
Second, one does not need to evaluate vtree units v where at least one of its children c satisfy {xπj}j=1i=1 ∈φ(c) because one can obtain the target marginal probability p(xπ1, . . . , xπi) following lines 7-9 of Process 3. How this is done in the following is provided directly below.
Denote the “highest” in Tp,i as vr,i (i.e., the parent of vr,i is not in Tp,i). According to the variable order π*, vr,i uniquely exist for any i∈[D]. According to Process 2, the top-down probabilities of PC units is defined as follows
It may now be proved that
holds when v=vr,i.
where (a) reorders the terms for summation; (b) holds since ∀n∈ϕprod(p, vp), pn(x)=Πo∈in(n) po(x) and ∈in(n) and ∀o ∈in(n) such that {Xj}j=1i∩ϕ(o)=Ø, po(x)=1;8(c) holds because
Thus, it has been proved that Eq. (3) holds for v=vr,i and hence the probability p(xπ1, . . . , xπi) can be computed by weighting the probability of PC units ϕsum(p, vr,i) (line 8 in Process 3) with the corresponding top-down probabilities (line 9 in Process 3).
Efficiency of following the optimal variable order It may be shown that when using the optimal variable order π*, Process 3 evaluates no more than (log(D)·|p|) PC units.
According to the previous paragraphs, whenever Process 3 evaluates a PC unit n w.r.t. vtree node v, it will evaluate all PC units in ϕ(p, v). Therefore, one may instead count the total number of vtree nodes needed to be evaluated by Process 3. Since the PC is assumed to be balanced Def. 4, for every v, it follows ϕ(p, v)=(|p|/D). Therefore, one only needs to show that Process 3 evaluates
(D·log(D)) vtree nodes in total.
One may start with the base case, which is PCs correspond to a single vtree leaf node v. In this case, Fπ*(x) boils down to computing a single marginal probability p(xπ1*), which needs to evaluate PC units ϕ(p, v) once.
Define f(x) as the number of vtree nodes need to be evaluated given a PC corresponds to a vtree node with x descendent leaf nodes. From the base case, one knows that f(1)=1.
Next, consider the inductive case where v is an inner node that has x descendent leaf nodes. Define the left and right child node of v as c1 and c2, respectively. Let c1 and c2 have y and z descendent leaf nodes, respectively. One may want to compute Fπ*(x), which can be broken down into computing two following sets of marginals:
Set 1: {p(xπ*
Since π* follows the in-order traverse of v, to compute the first term, one only needs to evaluate c1 and its descendents, that is, one needs to evaluate f(y) vtree nodes. This is because the marginal probabilities in set 1 are only defined on variables in φ(c1). To compute the second term, in addition to evaluating PC units corresponding to c2 (that is f(z) vtree nodes in total), one also needs to re-evaluate the PC units ϕ(p, v) every time, which means one needs to evaluate z more vtree nodes. In summary, the following may be evaluated)
f(x)=f(y)+f(z)+z (y≥z,y+z=x)
vtree nodes.
To complete the proof, one may upper bound the number of vtree nodes needed to be evaluated. Define g(·) as follows:
It is not hard to verify that ∀x∈, g(x)≥f(x). Next, it may be proven that,
∀x∈(x≥2), g(x)≤3xlogx.
First, we can directly verify that g(2)≤3·2 log22≈4.1. Next, for x≥3,
where (a) holds since according to its derivative, h(y) obtains its maximum value at either y=1 or
For a structured-decomposable PC with D variables, g(D)≤3D log D vtree nodes need to be evaluated. Since each vtree node corresponds to
PC units, we need to evaluate (log(D)·|p|) PC units to compute Fπ*(x).
Consider the compilation from a PGM to an HCLT. It may be first noted that each PGM node g uniquely corresponds to a variable scope φ of the PC. That is, all PC units correspond to g have the same variable scope. Details on how to generate a HCLT given its PGM representation are further described below.
In the main loop of Process 4 (lines 5-10), for each PGM node g such that var(g) ∈Z, the number of computed PC units are the same (M product units compiled in line 9 and M sum units compiled in line 10). Therefore, for any variable scopes φ1 and φ2 possessed by some PC units, one has |nodes(p, φ(m))|≈|nodes(p, φ(n))|. Since there are in total Θ(D) different variable scopes in p, it may follow: for any scope φ′ exists in an HCLT p, nodes(p, φ′)=(|p|/D).
EiNets and RAT-SPNs are also balanced since they also have an equivalent PGM representation of their PCs. The main difference between these models and HCLTs is the different variable splitting strategy in the product units.
Although specific impacts of variable orders, conversion of product units into equivalent Pcs, structured-decomposable PCs, vtress, and conversion of PC equivalents are discussed above with respect to
Computing Mutual Information As mentioned above, computing the pairwise mutual information between variables X is the first step to compute the Chow-Liu Tree. Since the exemplary conditions are dealing with categorical data (e.g., 0-255 for pixels), one may compute mutual information by following its definition:
where CX and CY are the number of categories for variables X and Y, respectively. To lower the computation cost, for image data, one may truncate the data by only using 3 most-significant bits. That is, one may treat the variables as categorical variables with 23=8 categories during the construction of the CLT. Note that the full data when constructing/learning the PC is used.
Training pipeline Two types of EM updates may be adopted—mini-batch and full-batch. In mini-batch EM, parameters are updated according to a step size η: θ(k+1)←(1−η)θ(k)+ηθ(new), where θ(new) is the EM target computed with a batch of samples; full-batch EM updates the parameters by the EM target computed using the whole dataset. As described herein, HCLTs is trained by first running mini-batch EM with batch size 1024 and η changing linearly from 0.1 to 0.05; full-batch EM is then used to finetune the parameters.
)c)); hyperparameter M/
do
)] [0]
Generating PCs Following the HCLT Structure
After generating the PGM representation of a HCLT model, one may now be left with the final step of compiling the PGM representation of the model into an equivalent PC. Recall that one may define the latent variables {Zi}i=14 as categorical variables with M categories, where M is a hyperparameter. As demonstrated in Process 4, one may incrementally compile every PGM node into an equivalent PC unit though a bottom-up traverse (line 5) of the PGM. Specifically, leaf PGM nodes corresponding to observed variables Xi may be compiled into PC input units of Xi (line 6), and inner PGM nodes corresponding to latent variables are compiled by taking products and sums (implemented by product and sum units) of its child nodes' PC units (lines 8-10). Leaf units generated by PC leaf (X) can be any simple univariate distribution of X. Categorical leaf units in the HCLT experiments may be used.
The present embodiments adopted the EM parameter learning method which computes the EM update targets using expected flows. Further, the present embodiments may use a hybrid EM method, which uses mini-batch EM updates to initiate the training process, and switch to full-batch EM updates afterwards.
In our experiments, the HCLTs with 100 mini-batch EM epochs and 20 full-batch EM epochs were trained. During mini-batch EM updates, η was annealed linearly from 0.15 to 0.05.
Hardware specifications All experiments may be performed on a server with 72 GPUs, 512G Memory, and 2 TITAN RTX GPUs. In all experiments, only a single GPU on the server was used.
IDF All experiments were run with the code in the GitHub repo provided by the authors. We adopted an IDF model with the following hyperparameters: 8 flow layers per level; 2 levels; densenets with depth 6 and 512 channels; base learning rate 0.001; learning rate decay 0.999. The process adopts an CPU-based entropy coder rANS. For (de)compression, the following script may be used: https://github.com/jornpeters/integer_discrete_flows/blob/master/experiment_coding.py.
BitSwap All models were trained using the following author-provided script: https://github.com/fhkingma/bitswap/blob/master/model/mnist_train.py. The method adopts an CPU-based entropy coder rANS. And the following code for (de)compression may be used: https://github.com/fhkingma/bitswap/blob/master/mnist_compress.py.
BB-ANS All experiments may be performed using the following official code: https://github.com/bits-back/bits-back.
The adopted IDF architecture follows the original paper by Hoogeboom et al. For the PCs, we adopted EiNets with hyperparameters K=12 and R=4. Instead of using random binary trees to define the model architecture, we used binary trees where “closer” latent variables in z will be put closer in the binary tree.
Parameter learning was performed by the following steps. First, compute the average log-likelihood over a mini-batch of samples. The negative average log-likelihood is the loss we use. Second, compute the gradients w.r.t. all model parameters by backpropagating the loss. Finally, update the IDF and PCs using the gradients individually: for IDF the Adamax optimizer was used; for PCs, the gradients to compute the EM target of the parameters and performed mini-batch EM updates were used.
As illustrated, computing device 800 includes a one or more processor(s) 802, non-volatile memory 804, volatile memory 806, a network interface 808, and one or more input/output (I/O) interfaces 810. In the illustrated embodiment, processor(s) 802 retrieves and executes programming instructions stored in the non-volatile memory 804 and/or the volatile memory 806, as well as stores and retrieves data residing in the non-volatile memory 804 and/or the volatile memory 806. In certain embodiments, non-volatile memory 804 is configured to store instructions (e.g., computer-executable code, device application 816) that when executed by processor(s) 802, cause processor(s) 802 to perform the processes and/or operations described herein and/or illustrated in the figures. In certain embodiments, non-volatile memory 804 stores code for executing the functions of an encoder and/or a decoder. Note that the computing device 800 may be configured to perform the functions of only one of the encoder or the decoder, in which case additional system(s) may be used for performing the functions of the other.
Processor(s) 802 is generally representative of a single central processing unit (CPU) and/or graphics processing unit (GPU), multiple CPUs and/or GPUs, a single CPU and/or GPU having multiple processing cores, and the like. Volatile memory 806 is generally included to be representative of a random access memory (RAM). Non-volatile memory 804 may be any combination of disk drives, flash-based storage devices, and the like, and may include fixed and/or removable storage devices, such as fixed disk drives, removable memory cards, caches, optical storage, network attached storage (NAS), or storage area networks (SAN).
In some embodiments, I/O devices (such as keyboards, monitors, etc.) can be connected via the I/O interface(s) 810. Further, via network interface 808, computing device 800 can be communicatively coupled with one or more other devices and components, such as user database 801. In certain embodiments, computing device 800 is communicatively coupled with other devices via a network, which may include the Internet, local network(s), and the like. The network may include wired connections, wireless connections, or a combination of wired and wireless connections. As illustrated, processor(s) 802, non-volatile memory 804, volatile memory 806, network interface 808, and I/O interface(s) 810 are communicatively coupled by one or more bus interconnects 812. In certain embodiments, computing device 800 is a server executing in an on-premises data center or a cloud environment. In certain embodiments, the computing device 800 is a user's mobile device.
In the illustrated embodiment, the non-volatile memory 804 may include a device application 816 that configures the processor(s) 802 to perform various processes and/or operations in performing lossless compression using PCs, as described above. As described above, the computing device 800 may be configured to receive image data (e.g., image data 818) to be compressed. For example, the image data 818 may include a plurality of pixels and each pixel may be represented by variable data 820. In other examples, any data to be compressed may be represented by variable data 820. In some embodiments, the computing device 800 may also be configured to sequentially compress the variables one-by-one using conditional probabilities 832, as described herein. For example, in some embodiments, the conditional probabilities 832 may be computed by calculating at least one marginal 824 (e.g., F(x) and/or G(x)). Further, the computing device 800 may be configured to perform initialization steps such as, but not limited to, initially setting a probability p(n) 828 for every PC unit n 830 and defining an evali for a set of PC units n 830 that need to be evaluated in an ith iteration, for i=1 to a dataset D 826, as further described above. Moreover, the computing device 800 may be configured to evaluate PC units n 830 in evali using a bottom-up process and computing a target probability p(x1, . . . , xi) 834, as further described above. In addition, the computing device 800 may be configured to generate a bitstream 838 using various methods, including but not limited to, using a streaming code 836 using the conditional probabilities 832.
In further reference to
Further, although specific operations and data are described as being performed and/or stored by a specific computing device above with respect to
Each of these non-limiting examples can stand on its own or can be combined in various permutations or combinations with one or more of the other examples. The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as “examples.” Such examples can include elements in addition to those shown or described. However, the present inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the present inventors also contemplate examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.
In the event of inconsistent usages between this document and any documents so incorporated by reference, the usage in this document controls.
In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In this document, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” In this document, the term “set” or “a set of” a particular item is used to refer to one or more than one of the particular item.
Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, composition, formulation, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.
Method examples described herein can be machine or computer-implemented at least in part. Some examples can include a computer-readable medium or machine-readable medium encoded with instructions operable to configure an electronic device to perform methods as described in the above examples. An implementation of such methods can include code, such as microcode, assembly language code, a higher-level language code, or the like. Such code can include computer readable instructions for performing various methods. The code may form portions of computer program products. Further, in an example, the code can be tangibly stored on one or more volatile, non-transitory, or non-volatile tangible computer-readable media, such as during execution or at other times. Examples of these tangible computer-readable media can include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, random access memories (RAMs), read only memories (ROMs), and the like.
The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments can be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is provided to comply with 37 C.F.R. § 1.72(b), to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that such embodiments can be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.
While the above description contains many specific embodiments of the invention, these should not be construed as limitations on the scope of the invention, but rather as an example of one embodiment thereof. It is therefore to be understood that the present invention may be practiced otherwise than specifically described, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive.
The current application claims priority to U.S. Provisional Patent Application No. 63/336,544, filed on Apr. 29, 2022, the disclosure of which is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 63336544 | Apr 2022 | US |
