Patent Grant
11188616
The present invention relates to the electrical, electronic and computer arts, and, more particularly, to reduced order modeling and dimensionality reduction of physical models.
Much of real-world data is inherently multidimensional, and many operators and models are natively multi-way. By way of example, diagnostic medical imaging naturally involves three-dimensional (spatial) and four-dimensional (spatiotemporal), as does motion trajectory for self-driving cars. Other high-dimensional applications include machine vision (e.g., perspective, occlusions, illumination), natural language processing (e.g., latent semantic tensor indexing of co-occurring terms), and video processing (e.g., streaming, surveillance, motion signatures).
Dynamical models are instrumental in description, prediction, and design of complex systems and processes. Simulation of dynamics is instrumental for the description of a broad range of systems and phenomena, such as physical systems, biological systems, chemical reactions, finance, autonomous agents, optimal control, Internet of Things, neural networks, etc. Simulation is often embedded within end-goal objectives such as interface, design, control, or decision making under uncertainty. Simulation of large-scale, complex linear (or close to linear) systems is challenging computationally. In situations where simulation output is required instantaneously (e.g., real-time control systems) or at a rapid rate, such simulations often become intractable.
Model reduction techniques have been proposed to mitigate this fundamental problem. However, current methods for multi-dimensional model reduction are constrained by multiple factors. Data storage becomes excessively large, which could be particularly limiting if local processing upon a mobile device is essential, or communication of large volumes of data are required. Computation associated with derivation, simulation, control, or design of multi-dimensional dynamical systems is often a critical bottleneck, with scalability being a particular challenge. Many modeling approaches do not lend themselves to dynamic update. Parallelization is an issue: some procedures are more amenable than other to distributed (low communication) computation. Universality is also a problem: some model reduction mechanisms are efficient for specialized datasets or models, but do not apply for general dynamical systems.
Traditional model reduction techniques—including principal component analysis (PCA), singular vector decomposition (SVD), and proper orthogonal decomposition (POD)—are based on matrix computation and assume data vectorization (i.e., vectorized representation). These methods are agnostic to possible high-dimensional correlations associated with the behavior of the system, such as two-dimensional and/or three-dimensional objects evolving over time. Agnostic treatment of such high-dimensional correlations entails sub-optimal descriptions of the dynamics, and leads to low-fidelity simulations, excessive computation, and limited horizon predictions. Thus, existing frameworks for model reduction are sub-optimal for systems involving multi-dimensional correlations, which is problematic as data becomes increasingly complex and multidimensional.
An illustrative embodiment includes a method for solving a dynamical system. The method comprises: obtaining multidimensional snapshots representing respective discrete solutions of the dynamical system; storing the multidimensional snapshots within a snapshot tensor having an order of at least three; generating a basis for at least a subspace of a state space of the dynamical system at least in part by performing a decomposition of the snapshot tensor; deriving a reduced order model at least in part by using the basis to project the dynamical system from the state space onto the subspace; and solving the reduced order model of the dynamical system.
As used herein, “facilitating” an action includes performing the action, making the action easier, helping to carry the action out, or causing the action to be performed. Thus, by way of example and not limitation, instructions executing on one processor might facilitate an action carried out by instructions executing on a remote processor, by sending appropriate data or commands to cause or aid the action to be performed. For the avoidance of doubt, where an actor facilitates an action by other than performing the action, the action is nevertheless performed by some entity or combination of entities.
One or more embodiments of the invention or elements thereof can be implemented in the form of a computer program product including a computer readable storage medium with computer usable program code for performing the method steps indicated. Furthermore, one or more embodiments of the invention or elements thereof can be implemented in the form of a system (or apparatus) including a memory, and at least one processor that is coupled to the memory and operative to perform exemplary method steps. Yet further, in another aspect, one or more embodiments of the invention or elements thereof can be implemented in the form of means for carrying out one or more of the method steps described herein; the means can include (i) hardware module(s), (ii) software module(s) stored in a computer readable storage medium (or multiple such media) and implemented on a hardware processor, or (iii) a combination of (i) and (ii); any of (i)-(iii) implement the specific techniques set forth herein.
Illustrative embodiments of the present invention have practical applications and provide technological improvements. Illustrative embodiments provide a dynamical model mechanism which leverages the inherent structure of the data to account for high-dimensional correlations intrinsically, and honors the dimensional integrity of the system. In illustrative embodiments, reduced model fidelity can be prescriptively controlled (as a trade-off with storage) in a fine-grained manner, and reduced models are provably optimal. Illustrative embodiments provide a consistent matrix-mimetic multidimensional data compression framework to facilitate improved storage reduction and computation efficiency. Thus, illustrative embodiments provide efficient computation, even in a parallel and/or distributed environment, and are amenable for an efficient hardware-accelerated substrate.
An illustrative embodiment of the invention provides a multi-directional tensor dynamic model reduction framework that offers superior representations of the solution space as well as the flexibility to entertain various invertible transformations in the product definition beyond fast Fourier transform (FFT). An illustrative embodiment provides multi-directional tensor decomposition in a specialized framework to address the dynamical model reduction problem that allows for superior orientation invariant representation. An illustrative embodiment provides a multi-directional, tensor-tensor proper orthogonal decomposition for dynamical system model reduction which is capable of entertaining any invertible transformations.
An illustrative embodiment provides dimensionality reduction of a known physical model—of any type—using singular vectors. Instead of a projection matrix, an illustrative embodiment employs a projection tensor in a consistent tensor algebra under a tensor model that allows for an orthogonal projection framework, thus requiring less training data while producing better (provably optimal rather than merely heuristic) approximations. By employing a snapshot tensor rather than a snapshot matrix, an illustrative embodiment avoids artificial flattening of inherently multi-dimensional data, a lossy process that inadvertently devoid the data from multi-dimensional correlations. In an illustrative embodiment, the use of a tensor-based method to obtain the projection basis allows the preservation of any inherently multi-dimensional correlations in the spatial dimensions, as well as the temporal dimension. An illustrative embodiment natively addresses dynamical systems and natively performs all computations (including the projections) in a tensorial fashion.
An illustrative embodiment can incorporate various types of tensor-tensor products within the same framework, which is advantageous as various tensor-tensor products can extract different types of features more efficiently.
These and other features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.
shown in
A tensor is a multi-dimensional array, and the order of the tensor is defined as the number of dimensions of the array. As used herein, a tensor refers to a higher order generalization of a scalar (zeroth order tensor), a vector (first order tensor) and a matrix (second order tensor). More formally, a tensor ∈
n
Although embodiments of the present invention are primarily described herein with reference to third order tensors, one skilled in the art will understand that principles of the present invention may be applied to higher order tensors, e.g., if ordinary differential equations (ODEs) of more than two spatial dimensions are considered. A first order tensor will be referred to herein as a vector and denoted with lowercase bold and/or Fraktur letters such as
a second order tensor will be referred to herein as a matrix and denoted with uppercase bold and/or Fraktur letters such as
Mathematical (Euler) script letters such as
indicate third order tensors, or more generally higher order tensors (i.e., order ≥3). The present disclosure uses the MATLAB notation for entries, in which
(i, j, k) denotes the entry at row i and column j of the matrix going k inward. (MATLAB is a registered trademark of The MathWorks, Inc.)
. A fiber is a vector defined by varying only one index while fixing the rest (e.g., the other two indices for a third order tensor).
(:, j, k) where the i index is varied while the j and k indices are fixed.
(i, :, k) where the j index is varied while the i and k indices are fixed.
(i, j, :) where the k index is varied while the i and j indices are fixed.
A slice is a matrix defined by varying only two indices while fixing the rest (e.g., the other one index for a third order tensor). (i, :, :) where the i index is fixed while the j and k indices are varied.
(:, i, :) where the j index is fixed while the i and k indices are varied.
(:, :, k) where the k index is fixed while the i and j indices are varied. For convenience, frontal slices may also be denoted herein with the form A(k).
Tensor decompositions which attempt to mimic properties of the matrix singular value decomposition (SVD) have been proposed. The most widely recognized tensor decompositions are the CANDECOMP/PARAFAC (CP) decomposition and Tucker decompositions. Numerous variations of Tucker decomposition have been proposed, including the High Order Singular Value Decomposition (HOSVD) discussed in De Lathauwer et al., “A Multilinear Singular Value Decomposition,” SIAM Journal on Matrix Analysis and Applications, Volume 21, Issue 4, pp. 1253-1278, 2000, the entirety of which is incorporated by reference herein for all purposes. Insuasty et al., “Tensor-based reduced order modeling in reservoir engineering: An application to production optimization,” 2nd International Federation of Automatic Control (IFAC) Workshop on Automatic Control in Offshore Oil and Gas Production (OOGP), Vol. 48, No. 6, May 2015, pp. 254-259, the entirety of which is incorporated by reference herein for all purposes, attempts to generalize proper orthogonal decomposition (POD) to tensors using CP and Tucker decompositions.
However, CP decomposition is problematic as the definition of CP tensor rank is ill-posed: unlike the matrix case, the rank cannot generally be computed in polynomial time, and without certain restrictions, an optimal rank-k approximation may not even exist. Moreover, the CP decomposition need not be orthogonal, and fitting CP for a given number of components is NP hard with no perfect and/or consistent procedure. While a Tucker decomposition can be computed in polynomial time and an orthogonal one will always exist, unlike the matrix counterpart, truncating the Tucker decomposition does not give an optimal approximation. Also, the Tucker decomposition can have either a diagonal core or orthogonal columns in components, but not both.
A framework of tensor SVD (t-SVD) based on a new tensor multiplication (t-product) was proposed by Kilmer et al., “Factorization strategies for third-order tensors,” Linear Algebra and its Applications, Vol. 435, No. 3, Aug. 2011, pp. 641-658, the entirety of which is incorporated by reference herein for all purposes. The t-product was the first closed multiplicative operation between a pair of third order tensors of appropriate dimension.
If ∈
l×m×n with l×m frontal slices denoted A(1), then the corresponding block circulant matrix of size ln×mn is:
unfold() takes an l×m×n tensor and returns a block ln×m matrix corresponding to the first block column of bcirc(
), while fold(
) is the inverse:
If is a l×p×m tensor and
is a p×m×n tensor, then their t-product is the l×m×n tensor
*
=fold(bcirc(
)·unfold(
)).
Note that as shown in ∈
n
*
will be defined if the number of columns of the leftmost tensor equals the number of rows of the rightmost tensor, and their respective tube fibers have the same dimension. The t-product of two tube fibers is equivalent to the circular convolution between them, and we note that convolution is a commutative operation. Hence, the t-product behaves as the matrix product, with the scalar operation replaced by convolution upon tube fibers.
Moreover, the t-product is based on a convolution-like operation, which can be implemented efficiently using the Fast Fourier Transform (FFT). The t-product, and factorizations based on t-products, are computed efficiently (and can be implemented to take advantage of parallelism) in the Fourier Transform (FT) domain. and
are moved into the Fourier domain by taking FFTs along tube fibers. In lines 205-207, matrix-matrix products are computed to multiply frontal slices in parallel such that
=
Δ
where Δ is the face-wise product. In lines 202 and 208, output
is obtained by taking inverse FFTs along tube fibers.
The n1×n2×n3 identity tensor is the tensor whose first frontal slice
(1) is the n1×n2 identity matrix and whose other frontal slices are all zeroes. If
is a n1×n2×n3 tensor, then its conjugate transpose
T is a n2×n1×n3 tensor obtained by conjugate transposition of each of the frontal slices and then reversing the order of transposed frontal slices 2 through n3. Because this tensor algebra is a module over a complex ring, these definitions hold over the complex field: thus, the transpose notation (e.g.,
T) utilized herein for ease of presentation should be construed as encompassing complex conjugate transposes (e.g.,
H).
on the left includes frontal slices
(1) with a speckling pattern,
(2) with a square grid pattern, and
(3) with a diagonal grid pattern in order from front to back. In the conjugate transpose tensor, each of the frontal slices is transposed, but now the transposed frontal slices are arranged in order from front to back as
(1)T with a speckling pattern,
(3)T with a diagonal grid pattern, and
(2)T with a square grid pattern.
A n1×n2×n3 tensor is orthogonal if
T*
=
*
T. If
is n1×n2×n3 with n1≤n2 and
(:, i, :)T
(:,j, :) yields a tube of zeroes when i≠j but equals the (1,1) tube fiber of
when i=j, then the lateral slices of
are orthonormal, analogous to the columns of an orthonormal matrix.
If is a n1×n2×n3 tensor, then the (full) t-SVD of
is:
where ∈
n
∈
n
k∈
n
in
∈
n
∈
n
T∈
n
∈
n
From the above equation, if n1≥n2, a reduced t-SVD can be obtained by restricting to have only n2 orthonormal columns, and
to be n2×n2×n3. Truncating the t-SVD to k terms (i.e., where k is a target truncation term) provides a guaranteed optimal rank k approximation in the Frobenius norm, where the Frobenius norm of a tensor is the square root of the sum of the squares of all its entries. The Frobenius norm of a third order tensor
∈
n
The Eckart-Young theorem can be ended such that, given a tensor. ∈
n
where ={
=
*
|
∈
n
∈
k×n
where {circumflex over (σ)}j(i)=(j,j,i) is the i-th component of fft(
(j,j,:), [ ], 3).
Thus, for a tensor ∈
n
∈
n
Differential equations are ubiquitous in machine vision applications such as optical flow, stereo, denoising, enhancement, segmentation, registration, neural networks, etc. A dynamical system which describes the evolution of a field in the two-dimensional spatial domain may be of the form:
where =(x, y) are the spatial dimensions, g and f are the linear and nonlinear operators determining the evolution of the system, respectively, and q(
t) is a non-homogeneous source term. Solving the dynamical system means finding unknown state u(
t) that satisfies the equation.
The analytical (exact) solution can be difficult to obtain or not even exist in practice, so leveraging numerical methods to compute an approximated solution is necessary. Instead of providing the variation of dependent variables continuously throughout the domain as analytical solutions, the numerical solutions can only provide answers at discrete points on a grid. Thus, the system shown in (1) can be solved by discretizing u(t) both in space and time. The typical approach, independent of the dimension of the dynamical system, is to discretize in appropriate spatial dimensions, but then to recast the unknowns at a given time step as a vector. By discretizing the Partial Differential Equation (PDE) system (1) on a grid in the spatial domain (i.e., semi-discretization), the system (1) becomes an Ordinary Differential Equation (ODE) system. The ODE is then solved by discretizing it in the time domain and approximating
For the sake of completeness, implicitly assume a set of boundary and initial conditions are imposed: the specifics of such choice is application dependent and of no fundamental influence to the proposed methodology. For simplicity, the present disclosure primarily describes discretization on regular spatial and temporal grids, but the formulation applies to arbitrary higher dimensional structures. For the sake of expositional simplicity, assume that the spatial domain is two-dimensional with rectangular boundary: Ω=[as,ae]×[bs,be]. Although this illustrative embodiment employs a finite difference discretization scheme, other forms can be considered in the same framework. Let i,l=(xi,yl)=(as+ihx, bs+lhy) where:
Let (t)p(i,l)≡u(
i,l, t), where p is some scheme to map indices in
to indices in
. Thus,
(t) is a column vector constructed by raster scanning the semi-discretized u(
,t). In this illustrative embodiment, assume a lexicographical ordering first in x then in y: p=(i−1)Nx+j. Thus, a discretization, and more particularly a semi-discretization over space, of (1) is given by:
where ∈
n×n is the discretization of the operator g, where n≡NxNy, and
(t) is a discretization of q(
,t) under the same ordering scheme.
There are numerous options for approximating (t) in (2) at finite time points. The approaches are divided into explicit (e.g., forward Euler) and implicit (e.g., backward Euler, Newman, Runge-Kutta), per the assumption they take upon the availability of their support. Explicit Euler method uses forward approximation, implicit Euler method uses backward approximation, and Crank-Nicolson method uses the average of forward approximation and backward approximation. Implicit methods are often preferred for their unconditional stability, but such choice necessitates multiple solutions of systems of equations in order to update the solution, even when the non-linear term f is zero.
To appreciate this concern, it is sufficient to consider the implicit (i.e. backwards) Euler approach. Assuming a regular temporal grid and no non-linear term, tj=jht, j∈[0, ts], define j=
(tj). Each step in the use of backward Euler to evolve the system at the grid point requires solving the system
which can also be written as
It should be noted that this specific choice of temporal discretization (forward Euler for 1st order ODE) is a specific example, and other temporal discretization schemes can interchangeably used.
For a two-dimensional case, if the size of the spatial grid is nx×ny, then the size of
(t) is nxny×1, and the size of
is nxny×nxny. Note that this procedure can also be applied to higher-dimensional domains, such as three-dimensional, four-dimensional, etc. In order to obtain a highly accurate solution, the number of spatial grid points may need to be quite large. However, the order of this system is nx ny, so it can be expensive to solve it when the spatial grid
is large. As a consequence, the cost of solving the linear systems usually is the dominant cost for estimating the solution to the discrete ODE. Moreover, there are ts time steps, so the system has to be solved is times. If a non-linear term is present, the problem is further complicated, and solving the system requires the solution of quite a few additional linearized systems. The scale of the discrete problem is therefore often computationally challenging, which motivates the need for reduced space representation of the discrete problem.
Proper orthogonal decomposition (POD) is a well-established method that has been used for model reduction across a wide range of applications. POD is a projection-based approach in which the state of the system is composed of a linear combination of principal state representations. The main idea is to project the original large-scale system to a smaller system by forming a basis of the state space from a small number of snapshots of the solution, and more particularly by using the left singular basis obtained by singular value decomposition (SVD) at some number of time step instances. Thus, the POD method is designed to lower the computational cost by reducing the order of the system: projecting the original n-dimensional state space to some r<<n subspace (r is a parameter). More particularly, it constructs a smaller size approximation system of order r<<n by projecting the original large system with a reduced order orthonormal projection matrix. By forming an explicit basis for this r dimensional, all the computation can be performed in the reduced state space.
Snapshots are a small subset of the solution {1,
2, . . . ,
μ
∈
n
∈
n
∈
n
. The first k left singular vectors of a snapshot matrix
determined from discrete solutions of a dynamical system at previous steps is used to project the discretized dynamic system of equations to a lower dimension. Then, the lower dimensional system is evolved in time, resulting in substantial computational savings over evolution of the discretized system at the original scale. Estimates of the full scale solution are then computed through projections onto the space of the left singular vectors. Approximating
(t) by
(t) and plugging into the system (2) produces
Next, apply the Galerkin projection
where for this two-dimensional example T
is of size k×k, and k<<nxny. More generally, for a d-dimensional array, k<<n1 . . . nd independent of the original problem dimensions (e.g., for a three-dimensional example, k<<nxnynz).
Concretely, let 1, . . . ,
r∈
n be an orthogonal basis for the projected state space. The projection operator on the subspace is
r=
T where
=[
1, . . . ,
r]. Let
⊆
n be the set of state vectors observed from the dynamical system given the initial/boundary conditions of interest, and the timespan of interest. It would be desirable to have
≈
r
=
T
for
∈
However, generally
is a complex set for which an explicit description is lacking. This limitation is circumvented by using a set of snapshots,
(1),
(2), . . . ,
(s)∈
n, state solutions computed at different instants in time and/or different initial/boundary conditions. It is generally assumed that s<<n. If the parameter space is sampled well, the snapshots will represent
well. Thus, the aim is
(i)≈
T
(i) for i=1, . . . , s.
To turn this into an optimization problem and find the basis, a least squares approach can be used. Let =[
(1), . . . ,
(s)] denote the snapshot matrix of size n×s, and consider the following optimization problem:
Let =
T be the reduced SVD of the n×s matrix
. In this equation,
is a n×s matrix with orthonormal columns (the left singular vectors),
is a s×s diagonal matrix of real values ordered along the diagonal σ1≥σ2≥ . . . ≥σr (the singular values),
s is an s×s matrix with orthonormal columns (the right singular values). The optimal solution of (3) consists of the first r, r≤s, left singular vectors of the snapshot matrix, i.e., the first r columns of
, a matrix which is denoted as
r. (Note that, in general, POD assumes that r<s. As discussed below, this limitation is not shared by tensor-POD according to an embodiment of the invention.) It is easy to verify that the value of (3) at the minimum (i.e., the error of the approximation at the snapshots) is
The reduced basis leads to the approximation(t)≈
r(
rT
(j)), where (
rT
(j))=
(t) (4)
To get the reduced order problem, substitute (4) into (3) and apply a Galerkin projection. For simplicity, the embodiment discussed herein assumes that the Galerkin projection is to the same subspace, so rT is applied to both sides of the system. However, in general, a different subspace may be used. The resulting reduced order model is
where the size of r:=
rT
r is r×r, in contrast with the original n×n system.
Now a time evolution ODE solver can be applied to the reduced order model (5). For example, in the linear case when implicit Euler is used:
requiring a solve with r×r system, rather than an n×n system. As long as r<<n and r can be readily obtained, the solution of the reduced order system should be substantially cheaper. To obtain an estimate of
j from
j compute
j=
r
j.
Alternatively, consider a set of snapshots of the solution {(1),
(2), . . . ,
(μ
The optimal solution of (7) is given by the first that r left singular vectors of a matrix constructed using {
(j)}j=1μ
=
r
r
rT where r≤μs, and where the columns of
r∈
n×r are left singular vectors which are orthogonal. Therefore,
(j)≈
r
rT
(j) is the optimal approximation of
(j) (in the least square sense) in span of {
r(:,1),
r(:,2), . . . ,
r(:,
)}. The error of (7) is
where σ1, σ2, . . . , σr are the diagonal entries of r.
Let (j)=
rT
(j) and assume
(j)=
r
(j). Then, apply Galerkin projection: multiply from the left by
rT. The discretized system (2) is replaced by the reduced order model (5), which can then be solved at a decreased computational cost due to it being of order r<<n. At each time step, estimate
≈
r
The performance of POD depends on the availability of snapshots and the effectiveness of the basis. However, in practice, there may only be a limited number of snapshots available to use and the behavior of the POD based approximation may not be directly applicable to its “nearby” state space. Conventional POD techniques do not generate a basis that can capture enough information using this limited number of snapshot to approximate the solutions off the snapshots. Rather, the traditional projection basis is obtained from a snapshot matrix whose columns are the vectorized solutions at various time steps. However, in situations where high dimensional correlations are prevalent (e.g. spatio-temporal), and in particular when multi-linear relations are perceived by linear (matrix) algebra based approach as non-linear, matrix-based POD provides either unstable or inaccurate reduced models. Moreover, because of its reliance upon matrix SVD, conventional POD does not account for high dimensional correlations intrinsically.
Illustrative embodiments of the present invention provide a novel POD-based approach that principally employs a snapshot tensor, instead of a snapshot matrix, wherein each snapshot is a 2D or 3D solution to a system of partial differential equations. The use of a snapshot tensor avoids artificial flattening of inherently multi-dimensional data into a matrix, which is a lossy process that may inadvertently devoid the data from multi-dimensional correlations. Indeed, the use of a tensor-based method to obtain the projection basis according to embodiments of the invention allows the preservation of any inherently multi-dimensional correlations in the spatial dimensions, as well as the temporal dimension. Thus, unlike its matrix-based counterpart, the tensor formulation inherently accounts for intrinsic multi-dimensional correlations, and thereby provides a better basis of the snapshots and demonstrably superior quality reduced models.
As previously noted, dynamical system (1) describes the evolution of a field in the two-dimensional spatial domain. Accordingly, at each single time point, the solution to the discrete dynamical system is effectively a matrix, and is represented as such by an illustrative embodiment of the present invention. Instead of reshaping the snapshots and storing them in vectors, an illustrative embodiment stores the snapshots in the lateral slices of a tensor and obtains a basis from them. Thus, rather than form snapshot matrix œ
n
∈
n
determined from discrete solutions of a dynamical system at previous steps is used to project the discretized dynamic system of equations to a lower dimension. An illustrative embodiment may compute k-term truncated t-SVD
k=
k*
k*
kT, e.g., using the procedure discussed above with reference to
k∈
n
Suppose there are μs snapshots. Store them in a tensor ∈
n
(j)∈
n
This is equivalent to finding a small optimal tensor containing the basis {
1,
2, . . . ,
k
Based on the aforementioned extension of Eckart-Young theorem, the optimal solution is the first k1 left singular slices of
. The t-SVD of
is
*
*
T where the first k1 left singular slices are denoted
k
k
n
Alternatively, let (j)
N
(j) denotes the Nx1×Ny tensor representation obtained by taking that matrix and simply twisting into the third dimension: for instance,
(j)(3,1,4)=
(j)(3,4). It should be noted that this placement of the y axis in the third tensor dimension is somewhat arbitrary, and one could instead choose to place the x axis in the third dimension, leading to a completely different set of dimensions. In either case, the reshaping process of
(j) into
(j) should be such that
(i)=unfold(
(j)), which connects the tensor interpretation of the problem with the underlying dynamical system.
Mimicking the matrix-based POD discussion above, it is desirable to obtain an orthonormal tensor basis {1,
2, . . . ,
ρ} with
i∈
N
(1), . . . ,
(s). Let
=[
(1), . . . ,
(s)] denote the snapshot tensor. The desired optimization problem is to find a tensor
∈
N
Based on the aforementioned extended Eckart-Young theorem, the optimal solution is the left singular tensor obtained by t-SVD as discussed above. Namely, assume the data tensor
is Nx×s×Ny, with s≤Nx, and compute the reduced t-SVD
=
*
*
T with
of size Nx×s×Ny. Next, set
=
ρ=
(:,1:ρ,:), which is Nx×ρ×Ny with ρ≤s. Then assume
(j)≈
ρ*
ρT*
(j) for all time points, where
ρT*
(j)=
(j)∈
ρ×1×N
Building a reduced order model is simplest when the dynamical system can be written in terms of the t-product. As an example of this, consider the special case when the matrix is itself block circulant. This would be the case, for example, if
represented a diffusion operator with periodic boundary conditions in the appropriate direction. The matrix-vector product between a block circulant matrix
and vector
can be equivalently expressed using the t-product as
*
that is a third-order tensor
from the first block column of
, and a Nx×1×Ny representation of
obtained by reshaping the vector into a matrix and twisting the matrix into the third dimension. Thus, when
has block circulant structure, the discrete dynamical system (2) can be rewritten in tensor form:
where is such that
=circ(
) and
is a reshaping of
(t) into an Nx×1×Ny tensor, and likewise {tilde over (f)} is defined from f to account for the new shape dimensions.
It is now straightforward to write the reduced order problem for the above tensor system. For example, in the linear case, the reduced order model is
where ρ=
ρT*
*
ρ∈
ρ×ρ×N
∈
ρ×1×N
As discussed above, t-products and factorizations and inversions are all performed in the Fourier domain. To implement this (assuming =0) would take Ny solves of a ρ×ρ system, but all the solves could in theory be conducted in parallel assuming sufficient (e.g., Ny) processors. In that case, if working in the Fourier domain, the parallel cost of executing this would be the same as for executing a matrix POD with r=ρ.
However, it is not always the case that the matrix has the desired special structure. In that case, the reduced model can be used within the matrix formulation of the dynamical system. As noted above,
(j)=unfold(
(j)), so in terms of matrices and vectors, equation (8) can be rewritten as:
(j)≈circ(
ρ)(circ(
ρT)
(j)). (9)
At this point, one may proceed according to the matrix POD described above with reference to ρ:=circ(
ρ) and incorporating that projection into (6). Computation of
ρ=
ρT
ρ can be performed offline, leveraging any structure in
along with the block circulant structure in
ρ.
It is important to note that the size of ρ is ρNx×ρNy. On one hand, it should noted that while ρ is still required to be smaller than s, ρNy can in fact be larger than s, and thus this technique can extract a higher dimensional projection space than is possible using POD (which is limited to dimension s). As further discussed below, in a sense, more information can be extracted using tensor-POD than matrix-POD.
On the other hand, when Ny=Ny=n, the reduced model must be of order Ω(√{square root over (n)}). While this is clearly a substantial reduction from n, it is sometimes desirable to reduce to order to smaller sizes. The tensor-POD techniques discussed thus far are based on dimension reduction from one side only (one-sided tensor-POD). This desire for an even smaller model size, as well as the somewhat arbitrary choice of putting they axis in the third tensor dimension, suggests additional techniques which utilize multi-sided tensor-POD.
To set the stage, suppose is an m×t×n tensor. Consider the mapping
m×t×n→
n×t×m induced by matrix-transposing each of the t lateral slices. As previously noted, a lateral slice of X will be an n×m matrix oriented into the third dimension. In MATLAB, this could be obtained by using the command permute(
,[3,2,1]). To keep the notation succinct, the superscript P will denote a tensor that has been permuted in this manner. In other words,
P=permute(
,[3,2,1]) and (
P)P=
.
The definition of the t-product is tied directly to the orientation. In other words, given ∈
m×t×n,
could be viewed as an m×t matrix, with tube-fiber entries of length n. The elemental operation here is convolution on the tube-fibers. Let
n denote the collection of tube fibers of n components. The t-product establishes (
n′*) as a commutative ring with the identity element being the tube fiber e1 with a 1 in the (1,1,1) entry and zeros in the (1,1,i), i=2 . . . n entries. Because of this, (
n′m*) is a free module: not quite a vector space since (
n′*) is a ring rather than a field, but the next best algebraic construct.
On the other hand, the same could instead be viewed as an n×t matrix, with tubal elements entries of length m. Here the elemental operation is convolution of elements of length m. It is sometimes useful to distinguish between these two different rings/operations in the case that m≠n. Henceforth, *n indicates operation over the ring (
n′*) and *m indicates operation over the ring (
m′*).
If =
*n
typically there is no way to get a factorization of
P over *m. However, we have devised a novel compressed tensor approximation which permits such factorization. Suppose
=
*n
*n
T is m×t×n. Here, as earlier, a subscript on the tensor refers to the number of lateral slices of the tensor that are kept. For example,
k denotes
(:,1:k,:).
:
kT*n
=
∈
nk×t. In step 720, find the t-SVD of
P under *m:
P=
*m
*m
T. In step 730, use this t-SVD to compress further with q-term projection:
qT*m
P=:
where
q is k×q×m and
is now k×t×q.
This technique provides an (implicit) approximation to X that can be generated via the tuple (,
q,
k) and operators (*m,*n). Specifically, the approximation is
≈
where
:=
k*n(
q*m
)P=
k*n(
q*m
q*m(
kT*n
)P)P
In addition, since the compressed representations are optimal at each stage under their respective algebraic settings, the resulting approximation is optimal in the sense that this approximation is the best approximation in the Frobenius norm, with the error given as ∥
−
∥F2=∥
q+1:end,q+1:end,:∥F2+∥
k+1:end,k+1:end,:∥F2 This framework can be generalized to higher order tensor structures, incorporating additional directional ring operations. The same procedure can also be iteratively repeated over the rotated version of the tensor to offer further compression. Note that the error in this approximation differs from using the usual truncated t-SVD to k terms in the presence of the first term in the sum.
In order to use this new approximation scheme to build a projection and a reduced order model, one must first consider the action of permutation or transposition imposed upon lateral slices of the tensor. Let be a k×1×n tensor. Let
be the kn vector obtained by unwrapping
by columns, such that
(:,1,1) is the first subvector,
(:,1,2) is the second subvector, etc. Let
be the kn vector obtained by unwrapping
by tube fibers. Then there exists a permutation matrix
such that
=
and
T
=
To build the projection, the procedure discussed above with reference to ≈
k*n (
q*m
qT*m(
kT*n
)P)P. In a manner similar to that discussed above with reference to equation (9), this can be written in matrix form as
(j)≈circ(
k)
T(circ(
q)circ(
q)T
circ(
kT)
(j)). Accordingly, the desired matrix projector for the POD is the matrix with qk columns
qk:=circ(
k)
T circ(
q). The projected matrix
qkT
qk is qk×qk. If the product qk is suitably small, one can attain an efficient, orientation-independent reduced order model.
One can now set (j)≈
qk
qkT
(j) and proceed with POD as discussed above with reference to
An illustrative embodiment leverages the tensor t-product and related operators to formulate the dynamical system in the form of a tensor, and then uses the t-SVD to generate the basis of snapshots. As previously noted, building a reduced order model is simplest when the dynamical system can be written in terms of the t-product. However, one skilled in the art will appreciate that the t-product is a single instance of the *M (m-product) family of tensor-tensor products. Other tensor-tensor products from the *M family can be utilized as the corresponding linear algebraic framework allows one to elegantly generalize all classical numerical methods from linear algebra. See, e.g., Kilmer et al., “Third-Order Tensors as Operators on Matrices: A Theoretical and Computational Framework with Applications in Imaging, SIAM Journal on Matrix Analysis and Applications”, Volume 34, Issue 1, 2013, pp. 148-172, the entirety of which is incorporated by reference herein for all purposes. Other invertible linear transforms (e.g., cosine, wavelet, etc.) can be considered in a similar framework. See, e.g., Kernfeld et al., “Tensor-tensor products with invertible linear transforms”, Linear Algebra and its Applications, Vol. 485, Nov. 2015, pp. 545-570, the entirety of which is incorporated by reference herein for all purposes.
Given ∈
l×p×n,
∈
p×m×n, and an invertible n×n matrix
then
=
*M
=(
Δ
)x3
−1 where
∈
l×m×n and, as discussed above, Δ is a face-wise product multiplying the frontal slices in parallel. x3 denotes the mode-3 product, where k-mode Teawr, multiplication of a tensor
∈
n
∈
j×n
xk
and is of size n1× . . . nk−1×j×nk+1× . . . ×nd. Element-wise:
Hat notation is used to denote the tensor in the transform domain specified by . That is,
is the result of applying
along all tube fibers of
. However, it would not be efficient to compute
by looping over the tube fibers of
and applying
to each tube fiber.
=
x3
, where x3 denotes the aforementioned mode-3 product. Also,
=fold((
⊗
)unfold(
)) where ⊗ denotes the Kronecker product.
and
in the spatial domain. 820 moves tensors
and
into the transform domain by performing mode-3 multiplication of tensors
and
with matrix
, which generates
and
in the transform domain. 830 includes computing
=
Δ
in the transform domain. 840 computes the inverse transform b performing mode-3 multiplication of tensor
with the inverse of matrix
to produce the final
in the spatial domain shown in 850.
and
in the transform domain by performing mode-3 multiplication of tensors
and
with matrix
, corresponding to 810-820 in
=
Δ
where Δ is the face-wise product. Steps 902-904 correspond to 830 in
with the inverse of matrix
to produce
in the spatial domain, corresponding to 840-850 in
Although the illustrative embodiments are discussed herein primarily with reference to the t-product, an illustrative embodiment of the present invention may be adapted to utilize another tensor-tensor product instead of or in addition to the t-product. The ability to incorporate various types of tensor-tensor products within the same framework is advantageous as different tensor-tensor products can extract different types of features more efficiently. While approaches relying exclusively on the t-product may be limited to directional decomposition and to FFT-based t-linear operation, an illustrative embodiment of the invention provides a multi-directional (e.g., multi-linear) proper orthogonal decomposition capable of entertaining any invertible transformation.
By way of example, an illustrative embodiment of the present invention may be adapted for use with an *M-product as opposed to a t-product. This may involve, for example, replacing t-SVD discussed herein with reference to
If is a m×p×n tensor, then the (full) *M tensor SVD (t-SVDM) of
is:
where ∈
m×m×n,
∈
p×p×n are *M-unitary, and
k∈
m×p×n is a tensor whose frontal slices are diagonal (such a tensor is called f-diagonal). If m>p, one can derive a reduced t-SVDM by restricting
to have only p orthonormal columns, and
to be p×p×n as opposed to the full m×p×n. Similarly, if p>m, one needs only keep the m×m×n portion of
and the m columns of
to obtain the same representation.
the components of the t-SVDM are computed in transform space, as discussed above with reference to
is the (unnormalized) discrete Fourier transform (DFT) matrix, t-SVDM reduces to the t-product-based t-SVD discussed above with reference to
The Eckart-Young theorem discussed above with reference to t-SVD also extends to t-SVDM in cases where is a non-zero multiple of a unitary matrix. The multi-rank of
under *M is the vector ρ such that its i-th entry ρi denotes the rank of the i-th frontal slice of
Given the t-SVDM of
under *M, define
ρ to be approximation having multi-rank ρ:
ρ(i)=
:,1:ρ
1:ρ
:,1:ρi,:H. Then
ρ is the best multi-rank ρ approximation to
in the Frobenius norm. This in turn can be used to generalize the idea of the t-SVDII to the *M product when
is a scalar multiple of a unitary matrix.
There has been considerable skepticism regarding whether the inventive tensor-based approaches described herein can give gains when compared to conventional matrix-based approaches. After all, isn't the long-established basis given by the matrix SVD of the snapshot information already optimal? How can the tensor approach possibly do better, when, at the end of the day, the tensor-generated information is used to perform model order reduction in tensor space?However, both theoretical and empirical evidence demonstrates that illustrative embodiments of the present invention provide unexpected results relative to conventional techniques.
When comparing illustrative embodiments of the present invention to conventional techniques, this discussion focuses primarily on the single-sided t-SVD rather than the multi-sided techniques for ease of exposition. However, these findings can be extended to also include multi-sided techniques, which produce results at least comparable to, if not superior to, the single-sided approaches discussed herein.
In both conventional matrix POD and inventive tensor-based POD, the main computation cost is due to the computation of the basis and solving the reduced model. For the step of basis computation, using SVD and using t-SVD have a similar computation cost: O(nx2ny2μs). However, it is important to recall that, as discussed above with reference to
More particularly, the two-sided (or, more generally, multi-sided) approach allows for further reduction of the dimension of the tensor-based POD formulation while preserving the accuracy benefits of the full problem. In the two-sided tensor approach, the number of columns in the projection matrix qk is qk. The cost of a system solve using the reduced order model (ROM) based on the matrix-based POD and the cost of a system solve using the ROM using the ROM based on the two-sided tensor-based POD would be the same if qk=r.
Zhang at Section 3.2.1 gives theoretical proofs that show the basis obtained from t-SVD is more accurate than the basis obtained from conventional matrix-based SVD for the same k. If is a n1×n2×n3 tensor, then the first k terms of the t-SVD will be:
The matrix j that represents the j-th lateral slice of
can be approximated as
where i(j)=
(i,j,:) is the (i,j)-th tube fiber. If
(:,j)=vec(
j) and
then ∥−
k∥F≤∥
−
k∥F where k≤min(n1,n2,n3) and k≤min(n1n3,n2) simultaneously. Thus, the approximation error for tensor-based techniques is lower than the approximation error for matrix-based techniques. This conclusion not only works for the model reduction of dynamical systems, but also applies in general regarding to comparison of SVD and tensor SVD in accuracy.
Suppose again that ∈
m×s×n is the snapshot tensor, and note that
:=unfold(
) will give exactly the snapshot matrix. For ease of discussion,
j represents the m×n matrix corresponding to the j-th lateral slice of
. Note that the j-th column of the snapshot matrix,
j, is related to
j as
j=vec(
j). The truncated t-SVD solves the best tensor approximation problem:
As noted above, the solution can be written in terms of the first k terms of the tensor-SVD:
Set i∈
m×n as the matrix representation of
(:,i,:), e.g., using MATLAB's squeeze command. Let
i(j) be the representation in
n of the tube filter
(:,j,:). Observe that
The notation circ() for
∈
n means n×n circulant matrix generated using
as the first column.
These observations provide one way of comparing the approximation from the truncated matrix SVD of with the tensor approximation problem above. To begin, let
=
T be the SVD of
. Note
is mn×t, and assume that mn>s, so that the rank of
is less than s. Choose a truncation ρ≤s. Thus, the j-th column of
,
j, is approximated as
where dij is the j-th entry of the coefficient matrix Σ(1:ρ, 1:ρ)*(:,1:ρ)T. But recall
j is just vec(
j). So if
(:,i) is also reshaped into
i∈
m×n, the approximation given by the matrix SVD can be expressed according to
where j=1, . . . , t. This can be converted into tensor form:
where is a tensor with the matrix
on the front face and zeroes elsewhere. For ρ=k, and assuming k≤min(n,s) and ρ≤rank(
≤s≤min(m,n), and
k is the k-term best tensor approximation, then:
For a “skinny” data matrix, then, the tensor SVD gives an approximation to the same data that is at least as good, if not better, than the matrix SVD. Now, it is true that implicit storage of k costs more than storing
k, however, the amount of storage required for the k left singular tensor slices vs. the k left singular vectors is identical. Thus, the tensor left singular factors are in general more informative than the matrix left singular vectors, provided that one leverages the power of the t-product. Even though the reduced discrete dynamical system may have been intermediately expressed in matrix form, the representation of the snapshots and future states is always interpreted using the proposed tensor approach, so more information can be extracted out of the snapshot data with the use of tensors compared with matrices.
It is important to realize, that despite the measure similarity (Frobenius norm), the matrix truncation and the tensor truncation approximations correspond to two different optimization problems, as they address different algebraic structures. As mentioned above, the matrix problem is dealing with vector spaces and in that context with scalar multiplication as the elemental operation. In contrast, the tensor problem is dealing with algebraic modules, whose elements are tube fibers, and the respective elemental operation is the t-product between two tube fibers.
Since it is possible for a non-zero tube fiber to be non-invertible, these elements do not form a field, and hence, along with the elemental operation, they do not form vector spaces, but rather rings and modules as noted earlier. Yet, the notion of basis does exist. The action of on
∈
p×1×n when
∈
m×p×n is essentially t-linear (rather than linear),
*(
*c+
*d)=(
*
)*c+(
*
)*d, where c and d are any tube fibers of length n. Note that they cannot be put on the left because the t-product definition does not apply that way.
Another way to gain intuition into the superiority of the tensor decomposition is through observation of the range spaces. Let us assume for instance that the data tensor is n×n×n. Following the conventional matrix vectorization conduct, that implies a data matrix of n2×n. The range of the entailed data matrix cannot exceed n, and therefore the n left singular vectors do not span n
Experimental results generated using an illustrative embodiment of the present invention provide empirical evidence of unexpected results and significant advantages relative to conventional arrangements. These results were generated for an application involving a linear diffusion problem with Dirichlet boundary condition:
where u(t) is the state,
=(x,y) denotes the spatial coordinates, K is the diffusion coefficient, and q(
t) is the source term for experiments.
The gradient of u is defined as
Employing a second order finite difference discretization scheme, the components of the discrete gradients are as shown in x, and the (ny+1)×ny lower bidiagonal matrix in
y.
Let n=nxny, and (t) be a column vector with size n constructed by raster scanning u(
i,k,t) column by column. Then ∇
(t)≈
(t) where
is a n×n matrix with size s, and ⊗ denotes the Kronecker product.
The divergence of u(t) is
Therefore ∇·(κ(t)∇u(
t))≈
T
(t)
(t) where
(t) is a n×n diagonal matrix with entries of semi-discretized κ(
t). The two-dimensional diffusion system (10) becomes an ODE:
where T
(t)
corresponds to the matrix
in equation (2) and the size of
T
(t)
is n×n.
T
(t)
, like
is a representation of the discrete diffusion operator.
When using the implicit Euler method, equation (11) can be written as
Then, given a j at each time step j+1,
j+1 can be solved by
(−ht
T
(t)
)
j+1=
j+ht
j+1 (12)
Thus, the cost of each step of the implicit Euler method is the cost of solving the n×n system (12).
In the experimental setup, nx and ny, are each set at 150, indicating a grid size of 150×150. nt is also set at 150, indicating 150 total time steps until computing the final error. The diffusion coefficient κ(t) is set as κ(x,t)=5 and κ(y,t)=5. Other variables indicating granularity of space and time are dx=10/(nx−1); dy=10/(ny−1); and dt=0.01. The domain was of size 10 by 10, with each dimension subdivided into nx−1 or ny−1 subsections.
A set of snapshots of the solution are collected: {1,
2, . . . ,
μ
1),
3),
7),
9),
12), and
15). These sample snapshots allow for observation of the diffusion process. For example, because κx is 5 and κy is 2. the diffusion process rate along the x direction is faster than the diffusion process rate along they direction.
As previously mentioned, after a specified number of time steps, the final error is computed. The final error refers to the error relative to a solution computed without using POD. The final error of matrix based POD is defined as
and the final error of tensor based POD is defined as
where k and ρ are the number of snapshots used for formation of the projections, or more precisely, the truncation level used for the decomposition.
Therefore, from the same snapshot data, the tensor-POD gives a more powerful projection basis in terms of preserving fidelity of the reduced order model (ROM) to the full model. In order for the matrix POD to perform as well, one would need to increase the number of snapshots substantially. But, to increase the number of snapshots would mean many more full-order implicit steps, and a cost of a matrix SVD that has more columns and whose representation isn't structured—these added costs would need to be balanced against using the one-sided tensor POD approach for much less snapshot data: so one can start using the ROM much sooner in the process, even though each individual solve costs more.
Thus,
These theoretical and empirical results show that the proposed t-SVD approach must be at least as good as the celebrated matrix counterpart at representing the snapshot data, and is often better, for the same amount of basis element storage. The left t-SVD singular slices of snapshots arranged as a tensor are as, or more, informative basis elements compared to the left singular vectors of the same data arranged as a snapshot matrix. This important outcome indicates that the proposed framework may be valuable in applications beyond model reduction.
For a similar fidelity target of the reduced model, the proposed approach can learn the reduced subspace, using fewer snapshots. The ability to offer comparable fidelities, while supported by a smaller number of snapshots, is of great significance in situations where snapshots are formed by either computationally expensive, high fidelity simulation process, or by actual, time-consuming and often times expensive, experimentation.
An illustrative embodiment may provide a system and method for multi-dimensional multi-linear (e.g., t-linear) dynamical model reduction, for example, of a dynamical system modeled by a system of differential equations. Input data may include multi-dimensional snapshots of state representations, represented in native tensor structure.
For data of tensor degree n, the system may perform n−1 truncated tensor-tensor decompositions per tensor permutation upon the snapshot tensor. These tensor-tensor decompositions may include, for example, singular value decompositions or QR decompositions. These tensor-tensor decompositions may be performed with a predefined tensor-tensor product procedure, such as t-product or m-product. Truncation of the decompositions may be performed in accordance with user-defined parameters indicating truncation and/or fidelity properties, such as desired compression ratio, truncation indices, bounds on compression integrity, etc.
The decomposition may produce left singular tensors which form a reduced space basis for representation of the state of the dynamical system. For example, the state may be represented as a tensor-tensor product of the left singular tensor elements by a set of representation coefficients. The dynamical system could then be projected (e.g., Galerkin projection) by virtue of its product by the complex conjugated transpose of the left singular tensors to yield a reduced space-system of differential equations. For truncation indices smaller than the effective rank of the tensor, the resulting reduced-space system of differential equations is of smaller dimension than the original dynamical system, which allows for it to be solved more efficiently.
Illustrative embodiments offer compressed evolution of the state of a dynamical system of differential equations. Thus, illustrative embodiments enable approximated processing and/or simulation of dynamical systems. As discussed in, e.g., Haber et al., “Stable architectures for deep neural networks”, Inverse Problems, Vol. 34, No. 1, Art. 014004, January 2018, 22 pages; Newman et al., “Stable Tensor Neural Networks for Rapid Deep Learning”, Nov. 15, 2018, 20 pages; Niu et al., “Recurrent Neural Networks in the Eye of Differential Equations”, Apr. 29, 2019, 33 pages, each of which is incorporated by reference herein for all purposes, a neural network can be viewed as a system of discrete differential equations. Specifically, the weight matrices of a neural network may be regarded as analogous to the snapshot matrices of a dynamical system. Thus, illustrative embodiments of the invention can provide an effective means for compression of neural networks.
One or more embodiments of the invention, or elements thereof, can be implemented, at least in part, in the form of an apparatus including a memory and at least one processor that is coupled to the memory and operative to perform exemplary method steps.
One or more embodiments can make use of software running on a general purpose computer or workstation. With reference to
Accordingly, computer software including instructions or code for performing the methodologies of the invention, as described herein, may be stored in one or more of the associated memory devices (for example, ROM, fixed or removable memory) and, when ready to be utilized, loaded in part or in whole (for example, into RAM) and implemented by a CPU. Such software could include, but is not limited to, firmware, resident software, microcode, and the like.
A data processing system suitable for storing and/or executing program code will include at least one processor 102 coupled directly or indirectly to memory elements 104 through a system bus 110. The memory elements can include local memory employed during actual implementation of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during implementation.
Input/output or I/O devices (including but not limited to keyboards 108, displays 106, pointing devices, and the like) can be coupled to the system either directly (such as via bus 110) or through intervening I/O controllers (omitted for clarity).
Network adapters such as network interface 114 may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.
As used herein, including the claims, a “server” includes a physical data processing system (for example, system 112 as shown in
It should be noted that any of the methods described herein can include an additional step of providing a system comprising distinct software modules embodied on a computer readable storage medium; the modules can include, for example, any or all of the elements depicted in the block diagrams or other figures and/or described herein. The method steps can then be carried out using the distinct software modules and/or sub-modules of the system, as described above, executing on one or more hardware processors 102. Further, a computer program product can include a computer-readable storage medium with code adapted to be implemented to carry out one or more method steps described herein, including the provision of the system with the distinct software modules.
Exemplary System and Article of Manufacture Details
The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7379925 | Vasilescu et al. | May 2008 | B2 |
| 7529719 | Liu et al. | May 2009 | B2 |
| 10337315 | Horesh et al. | Jul 2019 | B2 |
| 20070208547 | Graglia et al. | Sep 2007 | A1 |
| 20170220925 | Alsharif et al. | Aug 2017 | A1 |
| 20190080210 | Owechko | Mar 2019 | A1 |
| 20190205696 | Owechko | Jul 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 102592309 | May 2014 | CN |
| 105654138 | Jun 2016 | CN |
| 107808166 | Mar 2018 | CN |
| 109255374 | Jan 2019 | CN |
| 105427345 | Apr 2019 | CN |
| 109885628 | Jun 2019 | CN |
| Entry |
|---|
| Willcox et al., Balanced model reduction via the proper orthogonal decomposition, AIAA Journal, vol. 40, No. 11, Nov. 2002, pp. 2323-2330, https://arc.aiaa.org/doi/abs/10.2514/2.1570. |
| Insuasty et al., Tensor-based reduced order modeling in reservoir engineering: An application to production optimization, 2nd International Federation of Automatic Control (IFAC) Workshop on Automatic Control in Offshore Oil and Gas Production (OOGP), vol. 48, No. 6, May 2015, pp. 254-259, https://www.sciencedirect.com/science/article/pii/S2405896315009076. |
| Bishop, Pattern recognition and machine learning, Springer, 2006, https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf. |
| Kilmer et al., Factorization strategies for third-order tensors, Linear Algebra and its Applications, vol. 435, No. 3, Aug. 2011, pp. 641-658, https://www.sciencedirect.com/science/article/pii/S0024379510004830. |
| Hao et al., Nonnegative tensor decomposition, Ch. 5, Compressed Sensing & Sparse Filtering, pp. 123-148, Springer, 2014, https://link.springer.com/chapter/10.1007/978-3-642-38398-4_5. |
| Soltani et al., A tensor-based dictionary learning approach to tomographic image reconstruction, BIT Numerical Mathematics 56, No. 4, 2016, pp. 1425-1454, https://link.springer.com/article/10.1007/s10543-016-0607-z. |
| Newman et al., Image classification using local tensor singular value decompositions, 2017 IEEE 7th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), Dec. 2017, 5 pages, https://ieeexplore.ieee.org/document/8313137. |
| Ivanov et al., Head pose estimation using multilinear subspace analysis for robot human awareness, 2009 IEEE 12th International Conference on Computer Vision (ICCV) Workshops, 2009, pp. 227-233, https://ieeexplore.ieee.org/document/5457694. |
| Shi et al., Multi-target tracking with motion context in tensor power iteration, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3518-3525, Jun. 2014, https://ieeexplore.ieee.org/document/6909845. |
| Kernfeld et al., Tensor-tensor products with invertible linear transforms. Linear Algebra and its Applications, vol. 485, Nov. 2015, pp. 545-570, https://www.sciencedirect.com/science/article/abs/pii/S0024379515004358. |
| De Silva et al., Tensor Rank and the Ill-Posedness of the Best Low-Rank Approximation Problem, SIAM Journal on Matrix Analysis and Applications, vol. 30, No. 3, 2008, pp. 1084-1127, https://epubs.siam.org/doi/abs/10.1137/06066518X. |
| Hitchcock, The Expression of a Tensor or a Polyadic as a Sum of Products, Journal of Mathematics and Physics, vol. 6, Issue 1-4, Apr. 1927, pp. 164-189, https://onlinelibrary.wiley.com/doi/abs/10.1002/sapm192761164. |
| Harshman, Foundations of the PARAFAC procedure: Models and conditions for an ìexplanatoryî multimodal factor analysis, UCLA Working Papers in Phonetics, 16, pp. 1-84. (University Microfilms, Ann Arbor, Michigan, No. 10,085). 1970. |
| Carroll et al., Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition, Psychometrika, Sep. 1970, vol. 35, Issue 3, pp. 283-319, https://link.springer.com/article/10.1007%2FBF02310791. |
| De Lathauwer et al., A Multilinear Singular Value Decomposition, SIAM Journal on Matrix Analysis and Applications, vol. 21, Issue 4, pp. 1253-1278, 2000, https://epubs.siam.org/doi/10.1137/S0895479896305696. |
| Cichocki et al., Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-Way Data Analysis and Blind Source Separation, John Wiley & Sons, 2009, https://onlinelibrary.wiley.com/doi/book/10.1002/9780470747278. |
| Tucker, Implications of Factor Analysis of Three-Way Matrices for Measurement of Change, Ch. 7, Problems in Measuring Change, pp. 122-137, University of Wisconsin Press, 1963. |
| Kilmer et al., Third-Order Tensors as Operators on Matrices: A Theoretical and Computational Framework with Applications in Imaging, SIAM Journal on Matrix Analysis and Applications, vol. 34, Issue 1, 2013, pp. 148-172, bttps://epubs.siam.org/doi/abs/10.1137/110837711. |
| Hao et al., Facial Recognition Using Tensor-Tensor Decompositions, SIAM Journal on Imaging Sciences, vol. 6, Issue 1, pp. 437-463, 2013, https://epubs.siam.org/doi/abs/10.1137/110842570. |
| Horesh et al., U.S. Appl. No. 16/289,064, filed Feb. 28, 2019, 66 pages. |
| Zhang, Design and Application of Tensor Decompositions to Problems in Model and Image Compression and Analysis, Ph.D. dissertation, Tufts University Department of Mathematics, May 2017, 114 pages, https://dl.tufts.edu/concern/pdfs/2b88qq41w. |
| Haber et al., Stable architectures for deep neural networks, Inverse Problems, vol. 34, No. 1, Art. 014004, Jan. 2018, 22 pages, https://iopscience.iop.org/article/10.1088/1361-6420/aa9a90/pdf. |
| Shekhawat et al., On the problem of low rank approximation of tensors, 21st International Symposium on Mathematical Theory of Networks and Systems (MTNS). Jul. 2014, pp. 1478-1485, http://fwn06.housing.rug.nl/mtns2014-papers/fullPapers/0386.pdf. |
| Bui-Thanh et al., Aerodynamic Data Reconstruction and Inverse Design Using Proper Orthogonal Decomposition, AIAA Journal, vol. 42, No. 8, Aug. 2004, pp. 1505-1516, https://arc.aiaa.org/doi/abs/10.2514/1.2159. |
| Roderick et al., Proper orthogonal decompositions in multifidelity uncertainty quantification of complex simulation models, International Journal of Computer Mathematics, vol. 91, No. 4, Apr. 2014, pp. 748-769, https://www.tandfonline.com/doi/abs/10.1080/00207160.2013.844431. |
| Newman et al., Stable Tensor Neural Networks for Rapid Deep Learning, Nov. 15, 2018, 20 pages, https://arxiv.org/pdf/1811.06569v1.pdf. |
| Niu et al., Recurrent Neural Networks in the Eye of Differential Equations, Apr. 29, 2019, 33 pages, https://arxiv.org/pdf/1904.12933v1.pdf. |
| Kilmer et al., Tensor-Tensor Products for Optimal Representation and Compression, Dec. 31, 2019, 27 pages, https://arxiv.org/pdf/2001.00046v1.pdf. |
| Number | Date | Country | |
|---|---|---|---|
| 20210271730 A1 | Sep 2021 | US |
