Method for Transforming Non-Stationary Signals Using a Dynamic Model

Abstract

An input signal, in the form of a sequence of feature vectors, is transformed to an output signal by first storing parameters of a model of the input signal in a memory. Using the vectors and the parameters, a sequence of vectors of hidden variables is inferred. There is at least one vector hn of hidden variables hi,n for each feature vector xn, and each hidden variable is nonnegative. The output signal is generated using the feature vectors, the vectors of hidden variables, and the parameters. Each feature vector xn is dependent on at least one of the hidden variables hi,n for the same n. The hidden variables are related according to

Description

FIELD OF THE INVENTION

This invention relates generally to signal processing, and more particularly to transforming an input signal to an output signal using a dynamic model, where the signal is an audio (speech) signal.

BACKGROUND OF THE INVENTION

A common framework for modeling dynamics in non-stationary signals is a hidden Markov model (HMM) with temporal dynamics. The HMM is the de facto standard for speech recognition. A discrete-time HMM models a sequence of N observed (acquired) random variables

$\left\{x_n\right\}\stackrel{\mathrm{def}}{=}x_{1:N}\stackrel{\mathrm{def}}{=}\left\{x_1,x_2,\dots ,x_N\right\},$

i.e., signal samples, by conditioning probability distributions on the sequence of unobserved random state variables {h_n}. Two constraints are typically defined on the HMM.

First, the state variables have first-order Markov dynamics. This means that p(h_n|h_l:n-1)=p(h_n|h_n−1), where the p(h_n|h_n−1) are known as transition probabilities. The transition probabilities are usually constrained to be time-invariant.

Second, each sample x_n, given the corresponding state h_n, is independent of all other hidden states h_n′, n′≠n, so that p(x_n|h_1:N)=p(x_n|h_n), where the p(x_n|h_n) are known as observation probabilities. In many speech applications, the states h_n are discrete, and observations x_n are F-dimensional vector-valued continuous acoustic features,

$x_n\stackrel{\mathrm{def}}{=}\left\{x_{f,\left(n\right)}\right\}\stackrel{\mathrm{def}}{=}\left\{x_{1n},x_{2n},\dots ,x_{\mathrm{Fn}}\right\},$

where the parentheses indicate that n is not iterated. Typical frequency features are short-time log power spectra, where f indicates a frequency bin.

Defining initial probabilities

$p\left(h_1h_0\right)\stackrel{\mathrm{def}}{=}p\left(h_1\right),$

the joint distribution of the random variables of the HMM is

$\begin{array}{cc}p\left(\left\{x_n\right\},\left\{h_n\right\}\right)=\prod _{n=1}^Np\left(x_nh_n\right)p\left(h_nh_{n-1}\right).& \left(1\right)\end{array}$

Linear Dynamical Systems

A related model is a linear dynamical system used in Kalman filters. The linear dynamical system is characterized by states and observations that are continuous, vector-valued, and jointly Gaussian distributed

h _n =Ah _n−1+ε_n,  (2)

v _n =Bh _n +v _n,  (3)

where h_nεR^K (or h_nεC^K) is the state at lime n, K the dimension of the state space, A is a state transition matrix, ε_n is additive Gaussian transition noise, v_nεR^F (or v_nεC^F) is the observation at time n, F is the dimension of the observation (or feature) space, B is an observation matrix, v_n is additive Gaussian noise, and R is real.

Non-Negative Matrix Factorization

In the context of audio signal processing, the signal is typically processed using a sliding window and a feature vector representation that is often a magnitude or power spectrum of the audio signal. The features are nonnegative. In order to discover repeating patterns in the signal in an unsupervised way, nonnegative matrix factorization (NMF) is extensively used.

For a nonnegative matrix V of dimensions F×N, a rank-reduced approximation is

V≈WH,

where W and H are nonnegative matrices of dimensions F×K and K×N, respectively. The approximation is typically obtained from a minimization

$\underset{W,H\ge 0}{\min }D\left(V\mathrm{WH}\right)=\sum _{\mathrm{fn}}^{}d\left(v_{\mathrm{fn}}{\left[\mathrm{WH}\right]}_{\mathrm{fn}}\right),$

where d(x|y) is a positive function scalar cost function with a unique minimum at x=y.

Itakura-Saito Nonnegative Matrix Factorization (IS-NMF)

For the audio signal, where the matrix V is the power spectrogram of a complex-valued short-time Fourier transform (STFT) matrix X, conventional methods have used the Itakura-Saito distance, which measures the difference between the actual and approximated spectrum, as the cost function, because the cost function implies a latent model of superimposed zero-mean. Gaussian components that is relevant for audio signals. More precisely, let x_fn be the complex-valued STFT coefficient at frame n and frequency f, and

$x_{\mathrm{fn}}=\sum _k^{}c_{\mathrm{fkn}},$

Where

c _fkn :N _c(0,w_fkh_kn).

Then,

$\begin{array}{cc}\begin{array}{c}-\log p\left(XW,H\right)=\sum _{\mathrm{fn}}^{}\frac{v_{\mathrm{fn}}}{\sum _k^{}w_{\mathrm{fk}}h_{\mathrm{kn}}}+\log \sum _k^{}w_{\mathrm{fk}}h_{\mathrm{kn}}\\ =D_{\mathrm{IS}}\left({X}^2\mathrm{WH}\right)+\mathrm{cst},\left(5\right)\end{array}\mathrm{where}v_{\mathrm{fn}}=x_{\mathrm{fn}}^2.& \left(4\right)\end{array}$

The model can also be expressed as

$x_{\mathrm{fn}}\text{:}N_c\left(0,\sum _k^{}w_{\mathrm{fk}}h_{\mathrm{kn}}\right).$

It is equivalent to assume that |x|_fn² is exponentially distributed with parameter Σ_kw_fkh_kn and uniform phase

$\begin{array}{cc}{x}_{\mathrm{fn}}^2\text{:}\mathrm{Exponential}\left(\sum _k^{}w_{\mathrm{fk}}h_{\mathrm{kn}}\right),& \left(6\right)\\ {\mathrm{\angle x}}_{\mathrm{fn}}\text{:}\mathrm{Uniform}\left(-\pi ,+\pi \right).& \left(7\right)\end{array}$

Smooth IS-NMF

In smooth variants of IS-NMF, an inverse-gamma or gamma random walk is assumed for independent rows of H. More precisely, the following model has been considered:

h _kn =h _k(n-1)∘ε_kn,

where ε_kn is nonnegative multiplicative innovation random variable with mode 1, such as

ε_kn:G(α,α−1), or

ε_kn:IG(α,α+1),

where by convention gamma and inverse-gamma are

$\begin{array}{cc}G\left(x\alpha ,\beta \right)=\frac{{\beta }^{\alpha }}{\Gamma \left(\alpha \right)}x^{\alpha -1}\exp -\beta x,\mathrm{and}& \left(8\right)\\ \mathrm{IG}\left(x\alpha ,\beta \right)=\frac{{\beta }^{\alpha }}{\Gamma \left(\alpha \right)}x^{-\left(\alpha +1\right)}\exp -\frac{\beta }{x}.& \left(9\right)\end{array}$

Models Combining HMMs and NMF

If HMMs and NMF are combined, then the restriction that only one discrete state can be active at a time is inherited from the HMMs. This means that multiple model are required for multiple source, leading to potential issues to computational tractability.

U.S. Pat. No. 7,047,047 describes denoising a speech signal using an estimate of a noise-reduced feature vector and a model of an acoustic environment. The model is based on a non-linear function that describes a relationship between the input feature vector, a clean feature vector, a noise feature vector and a phase relationship indicative of mixing of the clean feature vector and the noise feature vector.

U.S. Pat. No. 8,015,003 describes denoising a mixed signal, e.g., speech and noise, using a NMF constrained by a denoising model. The denoising model includes training basis matrices of a training acoustic signal and a training noise signal, and statistics of weights of the training basis matrices. A product of the weights of the basis matrix of the acoustic signal and the training basis matrices of the training acoustic signal and the training noise signal is used to reconstruct the acoustic signal.

In general, the prior art methods that focus on slow-changing noise, are inadequate for fast-changing nonstationary noise, such as experienced by using a mobile telephone in a noisy environment.

Although HMMs can handle speech dynamics, HMMs often lead to combinatorial issues due to the discrete state space, which is computationally complex, especially for mixed signals from several sources. In conventional HMM approaches it is also not straightforward to handle gain adaptation.

NMF solves both the computational and gain adaptation issues. However, NMF does not handle dynamic signals. Smooth IS-NMF attempts to handle dynamics. However, the independence assumption of the rows of H is not realistic, as the activation of a spectral pattern at frame n is likely to be correlated with the activation of other patterns at a previous frame n−1.

It is an object of the invention to solve inherent problems associated with signal and data processing using HMMs and NMF frameworks.

SUMMARY OF THE INVENTION

It is an object of the invention to transform an input signal to an output signal when the input signal is a non-stationary signal, and more specifically a mixture of signals. Therefore, the embodiments of the invention provide a non-negative linear dynamical system model for processing the input signal, particularly a speech signal that is mixed with noise. In the context of speech separation and speech denoising, our model adapts to signal dynamics on-line, and achieves better performance than conventional methods.

Conventional models for signal dynamics frequently use hidden Markov models (HMMs) or non-negative matrix factorization (NMF).

HMMs lead to combinatorial problems due to the discrete state space, are computationally complex, especially for mixed signals from several sources. In conventional HMM approaches it is also not straightforward to handle gain adaptation.

NMF solves both the computational complexity and gain adaptation problems. However, NMF does not take advantage of past observations of a signal to model future observations of that signal. For signals with predictable dynamics, this is likely to be suboptimal.

Our model has advantages of both the HMMs and the NMF. The model is characterized by a continuous non-negative state space. Gain adaptation is automatically handled during inference. The complexity of the inference is linear in the number of signal sources, and dynamics are modeled via a linear transition matrix.

Specifically the input signal, in the form of a sequence of feature vectors, is transformed to the output signal by first storing parameters of a model of the input signal in a memory.

Using the vectors and the parameters, a sequence of vectors of hidden variables is inferred. There is at least one vector h_n of hidden variables h_i,n for each feature vector x_n, and each hidden variable is nonnegative.

The output signal is generated using the feature vectors, the vectors of hidden variables, and the parameters. Each feature vector X_n is dependent on at least one of the hidden variables h_i,n or the same n. The hidden variables are related according to

$h_{i,n}=\sum _{j,l}^{}c_{i,j,l}{\varepsilon }_{l,n}h_{j,n-1,}$

where j and l are summation indices. The parameters include non-negative weights c_i,j,l, and ε_l,n are independent non-negative random variables.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram for transforming an input signal to an output signal;

FIG. 2 is a flow diagram of a method for determining parameters of a dynamic model according to embodiment of the invention; and

FIG. 3 is a flow diagram of a method for enhancing a speech signal using the dynamic model according to embodiments of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Introduction

The embodiments of our provide a model for transforming and processing dynamic (non-stationary) signal and data that has advantages of HMMs and NMF based models.

The model is characterized by a continuous non-negative state space. Gain adaptation is automatically handled on-line during inference. Dynamics of the signal are modeled using a linear transition matrix A. The model is a non-negative linear dynamical system with multiplicative non-negative innovation random variables ε_n. The signal can be a non-stationary linear signal, such as an audio or speech signal, or a multi-dimensional signal. The signal can be expressed in the digital domain as data. The innovation random variable is described in greater detail below.

The embodiments also provide applications for using the model. Specifically, the model can be used to process an audio signal acquired from several, sources, e.g., the signal is a mixture of speech and noise (or other acoustic interference) and the model is used to enhance the signal by, e.g., reducing noise. When we say “mixed,” we mean that the speech and noise are acquired by a single sensor (microphone).

However, it is understood that the model can also be used for other non-stationary signals and data that have characteristics that vary over time, such as economic or financial data, network data and signals, or signals, medical signals, or other signals acquired from natural phenomena. The parameters include non-negative weights c_i,j,l, and ε_l,n are independent non-negative random variables, the distributions of which also have parameters. The indices i,j,l, and n are described below.

General Method

As shown in FIG. 1, parameters 101 of a model of an input signal 102 are stored in a memory 103.

The input signal is received as a feature vectors x_n 104 of salient characteristics of the signal. The features are of course application and signal specific. For example, if the signal is an audio signal, the features can be log power spectra. It is understood that the different type of features that can be used is essentially unlimited for many types of different signals and data that can be processed by the method according to the invention.

The method infers 110 a sequence of vectors of hidden variables 111. The inference is based on the feature vector 104, the parameters, a hidden variable relationship 130, and a relationship 140 of observations to hidden variables. There is at least one vector h_n of hidden variables h_i,n for each feature vector x_n. Each hidden variable is nonnegative.

An output signal 122 corresponding to the input signal is generated 120 to form the feature vectors, the vectors of hidden variables, and the parameters.

General Method Details

In our method, each feature vector x_n is dependent on at least one of the hidden variables h_i,n for the same n. The hidden variables are related according to a hidden variable relationship

$\begin{array}{cc}{h}_{i,n}=\sum _{j,l}^{}c_{i,j,l}{\varepsilon }_{l,n}h_{j,n-1}& \end{array}$

130, where j and l are summation indices. The stored parameters include non-negative weights c_i,j,l, and ε_l,n are independent non-negative random variables. This formulation enables the model to represent statistical dependency over time in a structured way, so that the hidden variables for the current frame, n, are dependent on those of the previous frame, n−1 with a distribution that is determined by the combination of c_i,j,l, and the parameters of the distribution of the weights ε_l,n. The weight ε_l,n, for example, may be Gamma random variables with shape parameter α and inverse scale parameter β.

In one embodiment, c_i,j,l=δ(i,l)a_i,j, where a_i,j are non-negative scalars, so that

$h_{i,n}=\left(\sum _j^{}a_{i,j}h_{j,n-1}\right){\varepsilon }_{i,n},$

where δ is a Kronecker delta. In this case, if the weights ε_l,n, are Gamma random variables with shape parameter α and inverse scale parameter β, then the conditional distribution of h_i,n given {h_j,n-1}_j=1^K, where K is a number of elements in the hidden states vector, is

$p\left(h_{i,n}h_{j,n-1}\right)=\mathrm{Gamma}\left(h_{i,n}\alpha ,\frac{\beta }{\sum _j^{}a_{i,j}h_{j,n-1}}\right),$

where

$\mathrm{Gamma}\left(xa,b\right)=\frac{b^a}{\Gamma \left(a\right)}x^{a-1}{}^{-\mathrm{bx}}$

is the gamma distribution for random variable x with shape a, inverse scale b, and

$\Gamma \left(z\right)={\int }_0^{\infty }{}^{-t_tz-1}t$

is the gamma function. This embodiment is designed to conform to the simplicity of the basic structure of a conventional linear dynamical system, but differs from prior art by the non-negative structure of the model, and the multiplicative innovation random variables.

In another embodiment, c_i,j,l=α(m(i,j),l),a_i,j, where a_i,j are non-negative scalars, δ is the Kronecker delta, δ(a,b)={_{0: otherwise}^{1: if a=b}, and m(i,j) is a one-to-one mapping from each combination of i and j to an index corresponding to l, (e.g., m(i,j)=(i−1)K+j, where K is a number of elements in the hidden variable h_n) so that

$h_{i,n}=\sum _ja_{i,j}{\varepsilon }_m\left(i,j\right),n^hj,n-1.$

This embodiment enables flexibility in modeling the signal, because each transition, can be inferred independently.

Another embodiment that is important to modeling multiple sources comprises partitioning hidden variables h_i,n into S groups, where each group corresponds to one independent source in a mixture. Likewise, the non-negative random variables ε_l,n are partitioned according to the same S groups. This can be accomplished by a special case of the parameters c_i,j,l where c_i,j,l=0 when h_i,n, and h_j,n are not in the same group or when h_i,n and ε_l,n are not associated with the same group. When the hidden variables are ordered accordingly, this gives c_i,j,l block structure, where each block corresponds to the model for one of the signal sources.

In our embodiments, the hidden variables are related 140 to feature variables via a non-negative feature v_f,n, of the signal indexed by feature f and frame n. An observation model is based on

$v_{f,n}=\sum _jc_{f,i,l}^{\left(v\right)}h_{i,n}{\varepsilon }_{l,n}^{\left(v\right)},$

where c_f,i,l^(v) is a non-negative scalar, and ε_l,n^(v) are independent non-negative random variables, and j, and l are indices of different components.

In a more constrained embodiment c_f,i,l^(v)=δ(i,l)w_f,i, where w_f,i are non-negative scalars, where δ is the Kronecker delta, and ε_f,n^(v) are the Gamma distributed random variables, so that the observation model based, at least in part, on

$p\left(v_{f,n}h_n\right)=\mathrm{Gamma}\left(v_{f,n}{\alpha }^{\left(v\right)},{\beta }^{\left(v\right)}/\sum _iw_{f,i}h_{i,n}\right),$

where v_f,n is non-negative feature of the signal at frame n and frequency f, α^(v) and β^(v) are positive scalars, and w_f,i are non-negative scalars.

In applications where the features x_f,n are complex spectrogram values of the input signal, a frame n and frequency f, the observation model can use v_f,n=|x_f,n|², which is the power in frame n, and frequency f. Thus, an observation model can be formed based on

$x_{f,n}=\left({}^{{\theta }_{f,n}\sqrt{-1}}\right)\sqrt{v_{f,n}},$

where √{square root over (−1)} is the unit imaginary number, and θ_f,n=∠x_f,n is a phase for a frame n and frequency f.

In another embodiment, we select the parameter α^(v)=1, so that the gamma distribution reduces to an exponential distribution as a special case. In this case, if the phases θ_f,n are distributed uniformly, then we obtain the observation model

$p\left(x_{f,n}h_n\right)=N_c\left(0,\sum _iw_{f,i}h_{i,n}\right),$

where N_C is a complex Gaussian distribution. This observation model corresponds to the Itakura-Saito nonnegative matrix factorization described above, and is combined in our embodiments with the non-negative dynamical system model.

Another embodiment uses an observation model for v_f,n based on a cascade of transformations of the same type:

$u_{i^{\prime },n}=\sum _ic_{i^{\prime },i,l}^{\left(u\right)}h_{i,n}{\varepsilon }_{l^{\prime },n}^{\left(u\right)},\mathrm{and}$ $v_{f,n}=\sum _{i^{\prime }}c_{f,i^{\prime },l}^{\left(v\right)}u_{i^{\prime },n}{\varepsilon }_{l^{″},n}^{\left(v\right)},$

where c_i′,i,l′^(u) and c_f,i′,l″^(v) are non-negative scalars, and ε_l′,n^(u) and ε_l″,n^(v) are independent non-negative random variables, and i, i′, l′, l″ are indices.

The method for inferring the hidden variables depends on the model parameterization for each embodiment.

Model Parameters

As shown in FIG. 2, from the input signal 102, we obtain the model parameters 101 as follows. The input signal can be considered a training signal, although it should be understood that the method can be adaptive to the signal, and “learn” the parameters on-line. The input signal can also be in the form of a digital signal or data.

For example, the training signal is a speech signal, or a mixed signal from multiple acoustic sources, perhaps including non-stationary noise, or other acoustic interference. The signal is processed as frames of signal samples. The sampling rate and number of samples in each frame is application specific. It is noted that the updating 230 described below for processing the current frame n is dependent on a previous frame n−1. For each frame we determine 210 a feature vector x_n representation. For an audio input signal, frequency features such as log power spectra could be used.

Parameters of the model are initialized 220. The parameters can include basis functions W, a transition matrix A, activation matrix H, and a fixed shape parameter ca and an inverse scale parameter β of a continuous gamma distribution parameter, and various combinations of these parameters depending on the particular application. For example in some applications, updating H and β are optional. In a variational Bayes (VB) method, H is not used. Instead an estimate of the posterior distribution of H is used and updated. If a maximum a-posteriori (MAP) estimation, then updating β is optional.

During each iteration of the method, the activation matrix, the basis function, the transition matrix, and the gamma parameter are updated 231-134. It should again be noted that the set of parameters to be updated is also application specific.

A termination condition 260, e.g., convergence or a maximum number of iterations, is tested after the updating 230. If true, store the parameters in a memory, otherwise if false, repeat at step 230.

The above steps of the general method and the parameter determination can be performed in a processor connected to a memory and input/output interfaces as know. Specialized microprocessors, and the like can also be used. It is understood that the signals processed by the method, e.g., speech or financial data, can be extremely complex. The method transforms the input signal into features which can be stored in the memory. The method also stores the model parameters and inferred hidden variables in the memory.

Model Parameters Details

For simplicity of this description, we limit the notation to the embodiment where c_f,i,l^(v)=δ(i,l)w_f,i, the w_f,i are non-negative scalars δ is a Kronecker delta, and ε_f,n^(v) are gamma distributed random variables, with parameter α^(v)=1, and phases θ_f,n are distributed uniformly. In this case, our model is

$\begin{array}{cc}{x}_{\mathrm{fn}}:N_c\left(0,\sum _kw_{\mathrm{fk}}h_{\mathrm{kn}}\right),& \left(10\right)\\ h_n=\left({\mathrm{Ah}}_{n-1}\right)\circ {\varepsilon }_n,& \left(11\right)\end{array}$

where x_fn is the complex-valued STFT coefficient at frame n and frequency f, N_C is the complex Gaussian distribution, w_fk is the value of the k^th basis function for the power spectrum at frequency f, h_n and h_n−1 are the n^th and the (n−1)^th columns of the activation matrix H, respectively, A is the nonnegative K×K transition matrix that models the correlations between the different patterns in successive frames n−1 and n, ε_n is a nonnegative innovation random variable, e.g., a vector of dimension K, and ∘ denotes entry-wise multiplication. The smooth IS-NMF can be obtained as a particular case of our model by setting A=I_K, where I_K is the K×K identity matrix.

ADVANTAGES

A distinctive and advantageous property of our model is that more than one state dimension can be non-zero at a given time. This means that a signal simultaneously acquired from multiple sources by a single sensor can analyzed using a single model, unlike the prior art HMM which requires multiple models.

Gamma Model of Innovations

We use an independent gamma distribution for the innovation ε_kn, namely

p(ε_in|α,β)=G(α_i,β_i).

It follows that h_n is conditionally gamma distributed, such that

$p\left(h_n{\mathrm{Ah}}_{n-1}\right)=\prod _iG\left(h_{\mathrm{in}}{\alpha }_i,{\beta }_i/{\left[{\mathrm{Ah}}_{n-1}\right]}_i\right),$

and in particular

$\begin{array}{cc}E\left(h_{\mathrm{in}}{\mathrm{Ah}}_{n-1}\right)=\frac{{\alpha }_i}{{\beta }_i}\sum _ja_{\mathrm{ij}}h_{j\left(n-1\right)}.& \left(12\right)\end{array}$

For h₁, we use an independent scale-invariant noninformative Jeffreys prior i.e.,

$p\left(h_1\right)=\prod _kp\left(h_{k1}\right).$

In Bayesian probability, the Jeffreys prior is a non-informative (objective) prior distribution on a parameter space that is proportional to the square root of the determinant of Fisher information.

MAP Inference in the Gamma Innovation Model

The maximum a-posteriori (MAP) objective function is

$C\left(W,H,A,\beta \right)=\sum _{\mathrm{fn}}\left(\frac{v_{\mathrm{fn}}}{\sum _kw_{\mathrm{fk}}h_{\mathrm{kn}}}+\log \sum _kw_{\mathrm{fk}}h_{\mathrm{kn}}\right)+\sum _{i=1}^K\sum _{n=2}^N\left({\alpha }_i\log \sum _ja_{\mathrm{ij}}h_{j\left(n-1\right)}+{\beta }_i\frac{h_{\mathrm{in}}}{\sum _ja_{\mathrm{ij}}h_{j\left(n-1\right)}}+\left(1-{\alpha }_i\right)\log h_{\mathrm{in}}\right)+\left(N-1\right)\sum _i\left(\log \Gamma \left({\alpha }_i\right)-{\alpha }_i\log {\beta }_i\right)-\sum _i\log p\left(h_{i1}\right)$

Scales

Scale-Ambiguity Between A and β

A K×K nonnegative diagonal matrix with coefficients λ_i on its diagonal is Λ, thus,

C(W,H,ΛA,Λβ)=C(W,H,A,β),

which has a scale-ambiguity between A and β. When both A and β are estimated, the scale-ambiguity can be corrected in a number of ways, for example by fixing β to arbitrary values or by normalizing the rows of A at every iteration 230 and rescaling β accordingly. For example, we can normalize the rows of the transition matrix A such that the rows sum to 1, or so that the maximum coefficient in every row is 1. In some embodiments, β_i=α_i, i.e., the model expectation of the innovation random variable is 1.

Ill-Posedness of MAP

The scales of W and H are related by

$C\left(W{\Lambda }^{-1},\Lambda H,A,\beta \right)=C\left(W,H,{\Lambda }^{-1}A\Lambda ,\beta \right)+N\sum _i\log {\lambda }_i,$

where λ_i is the i-th element of the diagonal of Λ.

Without further constraints, the minimization of the MAP objective leads to a degenerate solution such that ∥W∥→∞ and ∥H∥→0. If we assume that all the diagonal elements of Λ are equal, such that Λ=λI_K, then

C(WΛ⁻¹,ΛH,A)=C(W,H,A)+KN log λ.

The MAP objective can be made arbitrarily small by decreasing the value of λ. Hence, the norm of W is controlled during optimization. This can be achieved by hard or soft constraints. The hard constraint is a regular constraint that must be satisfied, and the soft constraint is a cost functions expressing a preference.

Hard Constraint

We solve

minC(W,H,A)s·t W≧0,H≧0,∥|w_k∥_|1=1

using the change of variable W=WΛ⁻¹, H=ΛH with Λ=diag[λ₁, . . . , λ_K], and λ_k=Pw_kP₁, the norm-constraint can be relaxed by solving

minC(W, H,A)=D_IS(V|WH)+S(ΛH)s·t W≧0,H≧0.

Soft Constraint (Penalization)

Another way we can control the norm of W is to add an appropriate penalty to the objective function, e.g.,

minC(W,H,A)+λ∥W∥₁s·t W≧0,H≧0

The soft constraint is typically simpler to implement than the hard constraint, but requires the tuning of λ.

Learning and Inference Procedures for MAP Estimation

We describe a majorization-minimization (MM) procedure. The MM is an iterative optimization procedure that can be applied to a convex objective function to determine maximums. That is, MM is a way to construct the objective function. MM determines a surrogate function that majorizes the objective function by driving the function to a local optimum. In our embodiments, the matrices H, A, and W are updated conditionally on one and another. In the following, tildes ({tilde over ( )}) denote current parameter iterations.

Inequalities

For {φ_k} such that Σ_kφ_k=1, we have

$\frac{1}{\sum _kx_k}\le \sum _k\frac{{\phi }_k^2}{x_k},$

by Jensen s inequality. We can form an upper bound on log α by linearization, at any point φ,

$\log a\le \log \phi +\frac{\partial \log a}{\partial a}\left(a-\phi \right)=\left(\log \phi -1\right)+\frac{a}{\phi }.$

In particular,

$\log \sum _ka_kx_k\le \left(\log \sum _ka_k{\tilde{x}}_k-1\right)+\frac{1}{\sum _ja_j{\tilde{x}}_j}\sum _ka_kx_k,\mathrm{and}$ $\frac{1}{\sum _ka_kx_k}\le \frac{1}{{\left(\sum _ja_j{\tilde{x}}_j\right)}^2}\sum _ka_k\frac{{\tilde{x}}_k^2}{x_k}.$

Fit to Data

$D_{\mathrm{IS}}\left(V|\mathrm{WH}\right)\le \sum _{\mathrm{kn}}\left({\tilde{p}}_{\mathrm{kn}}\frac{{\tilde{h}}_{\mathrm{kn}}^2}{h_{\mathrm{kn}}}+{\tilde{q}}_{\mathrm{kn}}h_{\mathrm{kn}}\right)$ ${\tilde{p}}_{\mathrm{kn}}=\sum _fw_{\mathrm{fk}}\frac{v_{\mathrm{fn}}}{{\tilde{v}}_{\mathrm{fn}}^2}$ ${\tilde{q}}_{\mathrm{kn}}=\sum _f\frac{w_{\mathrm{fk}}}{{\tilde{v}}_{\mathrm{fn}}}$ ${\tilde{v}}_{\mathrm{fn}}={\left[W\tilde{H}\right]}_{\mathrm{fn}}$ $D_{\mathrm{IS}}\left(V|\mathrm{WH}\right)\le \sum _{\mathrm{fk}}\left({\tilde{p}}_{\mathrm{fk}}\frac{{\tilde{w}}_{\mathrm{fk}}^2}{w_{\mathrm{fk}}}+{\tilde{q}}_{\mathrm{fk}}w_{\mathrm{fk}}\right)$ ${\tilde{p}}_{\mathrm{fk}}=\sum _nh_{\mathrm{kn}}\frac{v_{\mathrm{fn}}}{{\tilde{v}}_{\mathrm{fn}}^2}$ ${\tilde{q}}_{\mathrm{fk}}=\sum _n\frac{h_{\mathrm{kn}}}{{\tilde{v}}_{\mathrm{fn}}}$ ${\tilde{v}}_{\mathrm{fn}}={\left[\tilde{W}H\right]}_{\mathrm{fn}}$

Penalty Terms

$\mathrm{Let}g_{in}=\sum _ja_{\mathrm{ij}}h_{j\left(n-1\right)}.\mathrm{Then},\text{}\log \left(g_{i\left(n+1\right)}\right)\le \log \left({\tilde{g}}_{i\left(n+1\right)}\right)+\frac{1}{{\tilde{g}}_{i\left(n+1\right)}}\sum _ja_{\mathrm{ij}}\left(h_{\mathrm{jn}}-{\tilde{h}}_{\mathrm{jn}}\right)$ $\log \left(g_{i\left(n+1\right)}\right)\le \log \left({\tilde{g}}_{i\left(n+1\right)}\right)+\frac{1}{{\tilde{g}}_{i\left(n+1\right)}}\sum _jh_{\mathrm{jn}}\left(a_{\mathrm{ij}}-{\tilde{a}}_{\mathrm{ij}}\right)$ $\frac{1}{g_{i\left(n+1\right)}}\le \frac{1}{{\tilde{g}}_{i\left(n+1\right)}^2}\sum _ja_{\mathrm{ij}}\frac{{\tilde{h}}_{\mathrm{jn}}^2}{h_{\mathrm{jn}}}$ $\frac{1}{g_{i\left(n+1\right)}}\le \frac{1}{{\tilde{g}}_{i\left(n+1\right)}^2}\sum _jh_{\mathrm{jn}}\frac{{\tilde{a}}_{\mathrm{ij}}^2}{a_{\mathrm{ij}}}$ $\left({\tilde{g}}_{in}\mathrm{is}\mathrm{either}\sum _ja_{\mathrm{ij}}{\tilde{h}}_{j\left(n-1\right)}\mathrm{or}\sum _j{\tilde{a}}_{\mathrm{ij}}h_{j\left(n-1\right)}\right)$

Update Rules

The MM framework includes majorizing the terms of the objective function with the previous inequalities, providing an upper bound of the objective function that is tight at the current parameters, and minimizing the upper bound instead of the original objective. This strategy applied to the minimization of the MAP objective with the soft constraint on the norm of W leads to the following updates 230 as shown in FIG. 2.

Update 231 Activation Matrix H

The columns of H are updated 231 sequentially. Left to right updates makes the update h_n⁽¹⁾ of h_n at iteration l dependent of h_n−1^(l) and h_n+1^(l−1). The update of h_kn involves rooting a polynomial of order 2, such that

$h_{\mathrm{kn}}=\frac{\sqrt{b^2-4\mathrm{ac}}-b}{2a}$

where the values of a, b, c are given in the next table.

n = 1
a q                        ~                                            kn                                        +                                                                  Σ                        i                                                                                          α                        i                                                                                                                    a                          ik                                                                                                      g                            ~                                                                                i                                                                                      (                                                              n                                +                                1                                                            )
b 1 (Jeffreys) or 0 (uniform)
c -                                                                                            h                          ~                                                kn                        2                                                                                          (                                                                                                            p                              ~                                                        kn                                                    +                                                                                    Σ                              i                                                                                                                  β                              i                                                                                                                                                                                    a                                  ik                                                                                                                                  h                                                                      i                                                                                                              (                                                                              n                                        +                                        1                                                                            )                                                                                                                                                                                                                                    g                                  ~                                                                                                  i                                                                                                        (                                                                          n                                      +                                      1                                                                        )                                                                                                  2                                                                                                                                    )
1 < n < N
a q                        ~                                            kn                                        +                                                                  Σ                        i                                                                                          α                        i                                                                                                                    a                          ik                                                                                                      g                            ~                                                                                i                                                                                      (                                                              n                                +                                1                                                            )                                                                                                                                            +                                                                  β                        k                                                                    g                        kn
b 1 − αk
c -                                                                                            h                          ~                                                kn                        2                                                                                          (                                                                                                            p                              ~                                                        kn                                                    +                                                                                    Σ                              i                                                                                                                  β                              i                                                                                                                                                                                    a                                  ik                                                                                                                                  h                                                                      i                                                                                                              (                                                                              n                                        +                                        1                                                                            )                                                                                                                                                                                                                                    g                                  ~                                                                                                  i                                                                                                        (                                                                          n                                      +                                      1                                                                        )                                                                                                  2                                                                                                                                    )
n = N
a q                        ~                                            kn                                        +                                                                  β                        k                                                                    g                        kn
b 1 − αk
c −{tilde over (h)}kn2 {tilde over (p)}kn

In particular, for the exponential innovation with expectation 1 (α_i=β_i=1), we obtain the following multiplicative updates:

$\mathrm{For}n=1,h_{\mathrm{kn}}={\tilde{h}}_{\mathrm{kn}}\sqrt{\frac{{\tilde{p}}_{\mathrm{kn}}+\sum _i\frac{a_{\mathrm{ik}}h_{i\left(n+1\right)}}{{\tilde{g}}_{i\left(n+1\right)}^2}}{{\tilde{q}}_{\mathrm{kn}}+\sum _i\frac{a_{\mathrm{ik}}}{{\tilde{g}}_{i\left(n+1\right)}}+\frac{1}{{\tilde{h}}_{\mathrm{kn}}}}}.\mathrm{For}1<n<N,h_{\mathrm{kn}}={\tilde{h}}_{\mathrm{kn}}\sqrt{\frac{{\tilde{p}}_{\mathrm{kn}}+\sum _i\frac{a_{\mathrm{ik}}h_{i\left(n+1\right)}}{{\tilde{g}}_{i\left(n+1\right)}^2}}{{\tilde{q}}_{\mathrm{kn}}+\sum _i\frac{a_{\mathrm{ik}}}{{\tilde{g}}_{i\left(n+1\right)}}+\frac{1}{g_{\mathrm{kn}}}}}$ $\mathrm{For}n=N,h_{\mathrm{kn}}={\tilde{h}}_{\mathrm{kn}}\sqrt{\frac{{\tilde{p}}_{\mathrm{kn}}}{{\tilde{q}}_{\mathrm{kn}}+\frac{1}{g_{\mathrm{kn}}}}}.$

Update 232 Basis Function W

$w_{\mathrm{fk}}={\tilde{w}}_{\mathrm{fk}}\sqrt{\frac{{\tilde{p}}_{\mathrm{fk}}}{{\tilde{q}}_{\mathrm{fk}}+{\lambda }_W}}$

Update 233 Transition Matrix A

$a_{\mathrm{ij}}={\tilde{a}}_{\mathrm{ij}}\sqrt{\frac{{\beta }_i\sum _{n=2}^N\frac{h_{in}h_{j\left(n-1\right)}}{g_{in}^2}}{{\alpha }_i\sum _{n=2}^N\frac{h_{j\left(n-1\right)}}{{\tilde{g}}_{in}}+{\lambda }_A}}$

Variational EM Procedure for Maximum Likelihood Estimation

The activation parameter H is a latent variable to integrate from the joint likelihood. For generality, we assume the gamma distribution parameters β={β_i} to be free. The shape parameters α_i are treated as fixed parameters. We minimize

C(W,A,β)=−log p(V|W,A,β)=−log∫_Hp(V|W,H)p(H|A,β)dH.

This yields a better posed estimation problem because the set of parameters is of fixed-dimensionality w.r.t to the number of samples N. Furthermore, the objective is now better posed in terms of scales. For any positive diagonal matrix Λ, we have

C(W,A,β)=C(WΛ⁻¹,ΛAΛ⁻¹,β)

so that the renormalization of solution W* only induces a renormalization of A*. This is not true for the MAP approach.

For minimizing C(W,A,β), the EM procedure can be based on the complete dataset (V,H), and on the iterative minimization of

Q(θ|{tilde over (θ)})=−∫_H log p(V,H|W)p(H|V,{tilde over (θ)})dH,

where θ={W,A,β}. We do not use the posterior probability p(H|V,θ). Instead, we use a variational EM procedure. For any probability density function q(H), the following inequality holds:

C(θ)≦−log p(V|WH)_q−log p(H|A)_q+log q(H)_q=B_q(θ),

where •_q denotes the expectation under q(H). Variational EM minimizes B_q(θ), instead of C(θ). At each iteration, the bound is first evaluated and tightened, given W and A by minimizing B_q(θ) over q, or more precisely, over the shape parameters of q, given a specific parameterized form, and then minimized with respect to (θ) given q. Variational EM coincides with EM when q(H)=p(H|θ), in which case C(θ) is decreased at every iteration. In other cases, variational EM conducts approximate inference. The validity depends on how well q(H) approximates the true posterior probability p(H|θ).

Derivation of the Bound

The expressions of log p(V|WH) and log p(H|A) show that the coefficients of H are coupled through ratios or logarithms of linear combinations Σ_kw_fkh_kn and Σ_ja_ijh_j(n-1). This makes expectations of log p(V|WH) and log p(H|A) very difficult to determine independently of the specific form of q(H).

Therefore, we majorize log p(V|WH) and log p(H|A), to obtain a tractable bound. Using the above inequalities and assuming a factored form of the variational distribution, such that

$q\left(H\right)=\prod _{\mathrm{kn}}q\left(h_{\mathrm{kn}}\right)$

is an upper bound of C(W,A,β), the function

$B_{q,\xi }\left(W,A,\beta \right)=\sum _{\mathrm{fkn}}\left({\phi }_{\mathrm{fkn}}^2\frac{v_{\mathrm{fn}}}{w_{\mathrm{fk}}}〈h_{\mathrm{kn}}^{-1}〉+\frac{w_{\mathrm{fk}}}{{\psi }_{\mathrm{fn}}}〈h_{\mathrm{kn}}〉\right)+\sum _{\mathrm{fn}}\left(\log {\psi }_{\mathrm{fn}}-1\right)+\sum _{n=2}^N\sum _{i=1}^K\left(\begin{array}{c}\left(1-{\alpha }_i\right)〈\log h_{in}〉+\\ \sum _{j=1}^K\left(\begin{array}{c}{\alpha }_i\frac{a_{\mathrm{ij}}}{{\rho }_{in}}〈h_{j\left(n-1\right)}〉+\\ {\beta }_i\frac{v_{\mathrm{ijn}}^2}{a_{\mathrm{ij}}}〈h_{in}〉〈h_{j\left(n-1\right)}^{-1}〉\end{array}\right)\end{array}\right)+\sum _{n=2}^N\sum _{i=1}^K{\alpha }_i\left(\log {\rho }_{in}-1\right)+\left(N-1\right)\sum _{i=1}^K\left(\log \Gamma \left({\alpha }_i\right)-{\alpha }_i\log {\beta }_i\right)+\sum _{i=1}^K〈\log h_{i1}〉+\sum _{\mathrm{kn}}〈\log q\left(h_{\mathrm{kn}}\right)〉$

Where φ_fkn are nonnegative coefficients such that Σ_kφ_fkn=1,v_ijn are nonnegative coefficients such that Σ_iv_ijn=1,ρ_in, ψ_fn are nonnegative coefficients,ξ denotes the set of all tuning parameters {φ_fkn, v_ijn, ρ_in, ψ_fn}^fknij,• denotes expectation w.r.t q, i.e., corresponds to •_q. We remove subscript q to alleviate notations.

The expression of the bound involves the expectation of h_kn, 1/h_kn and log h_kn. These expectations are precisely the sufficient statistics of the generalized inverse-Gaussian (GiG), which is a practical convenience for q(H). We use

$q\left(H\right)=\prod _{\mathrm{kn}}\mathrm{GIG}\left(h_{\mathrm{kn}}|{\stackrel{\_}{\alpha }}_{\mathrm{kn}},{\stackrel{\_}{\beta }}_{\mathrm{kn}},{\stackrel{\_}{\gamma }}_{\mathrm{kn}}\right),\mathrm{where}$ $\mathrm{GIG}\left(x|\alpha ,\beta ,\gamma \right)=\frac{{\left(\beta /\gamma \right)}^{\alpha /2}}{2K_{\alpha }\left(2\sqrt{\mathrm{\beta \gamma }}\right)}x^{\alpha -1}\exp -\left(\beta x+\frac{\gamma }{x}\right),$

and where K_α is a modified Bessel function of the second kind and x, β and γ are nonnegative scalars. Under the GIG distribution,

$\begin{array}{cc}〈x〉=\frac{K_{\alpha +1}\left(2\sqrt{\mathrm{\beta \gamma }}\right)}{K_{\alpha }\left(2\sqrt{\mathrm{\beta \gamma }}\right)}\sqrt{\frac{\gamma }{\beta }}& \left(13\right)\\ {〈x^{-1}〉}^{-1}=\frac{K_{\alpha }\left(2\sqrt{\mathrm{\beta \gamma }}\right)}{K_{\alpha -1}\left(2\sqrt{\mathrm{\beta \gamma }}\right)}\sqrt{\frac{\gamma }{\beta }}.& \left(14\right)\end{array}$

For any α, K_α+1(x)=2(α/x)K_α(x)+K_α−1(x), which leads to the alternative, implementation-efficient expression of

$\begin{array}{cc}〈x^{-1}〉=\frac{K_{\alpha +1}\left(2\sqrt{\mathrm{\beta \gamma }}\right)}{K_{\alpha }\left(2\sqrt{\mathrm{\beta \gamma }}\right)}\sqrt{\frac{\beta }{\gamma }}-\frac{\alpha }{\gamma }.& \left(15\right)\end{array}$

Optimization of the Bound

We give the conditional updates of the various parameters of the bound. Update orders are described below.

Updates

Tuning parameters v

$\begin{array}{cc}{\phi }_{\mathrm{fkn}}=\frac{w_{\mathrm{fk}}{〈h_{\mathrm{kn}}^{-1}〉}^{-1}}{\sum _jw_{\mathrm{fj}}{〈h_{\mathrm{jn}}^{-1}〉}^{-1}},& \left(16\right)\\ {\psi }_{\mathrm{fn}}=\sum _jw_{\mathrm{fj}}〈h_{\mathrm{jn}}〉,& \left(17\right)\\ v_{\mathrm{ijn}}=\frac{a_{\mathrm{ij}}{〈h_{j\left(n-1\right)}^{-1}〉}^{-1}}{\sum _ka_{\mathrm{ik}}{〈h_{k\left(n-1\right)}^{-1}〉}^{-1}},\mathrm{and}& \left(18\right)\\ {\rho }_{in}=\sum _ja_{\mathrm{ij}}〈h_{j\left(n-1\right)}〉& \left(19\right)\end{array}$

Variational distribution q

n = 1
α              kn 0 (Jeffreys) or 1 (uniform)
β              kn Σ                        f                                                                                                                    w                          fk                                                                          ψ                          fn                                                                                      +                                                                  Σ                        i                                                                                          α                        i                                                                                                                    a                          ik                                                                          ρ                                                      i                                                                                      (                                                              n                                +                                1                                                            )
γ              kn Σ                        f                                                                                          φ                        fkn                        2                                                                                                                    v                          fn                                                                          w                          fk                                                                                      +                                                                  Σ                        i                                                                                          β                        i                                                                                                                    v                                                      ik                                                                                      (                                                              n                                +                                1                                                            )                                                                                2                                                                          a                          ik                                                                                                                  〈                                                  h                                                      i                                                                                      (                                                              n                                +                                1                                                            )                                                                                                      〉
1 < n < N
α              kn αk
β              kn Σ                        f                                                                                                                    w                          fk                                                                          ψ                          fn                                                                                      +                                                                  Σ                        i                                                                                          α                        i                                                                                                                    a                          ik                                                                          ρ                                                      i                                                                                      (                                                              n                                +                                1                                                            )                                                                                                                                            +                                                                  β                        k                                                                                          Σ                        j                                                                                                                    v                          kjn                          2                                                                          a                          kj                                                                                                                  〈                                                  h                                                      j                                                                                      (                                                              n                                -                                1                                                            )                                                                                                            -                            1                                                                          〉
γ              kn Σ                        f                                                                                          φ                        fkn                        2                                                                                                                    v                          fn                                                                          w                          fk                                                                                      +                                                                  Σ                        i                                                                                          β                        i                                                                                                                    v                                                      ik                                                                                      (                                                              n                                +                                1                                                            )                                                                                2                                                                          a                          ik                                                                                                                  〈                                                  h                                                      i                                                                                      (                                                              n                                +                                1                                                            )                                                                                                      〉
n = N
α              kn αk
β              kn Σ                        f                                                                                                                    w                          fk                                                                          ψ                          fn                                                                                      +                                                                  β                        k                                                                                          Σ                        j                                                                                                                    v                          kjn                          2                                                                          a                          kj                                                                                                                  〈                                                  h                                                      j                                                                                      (                                                              n                                -                                1                                                            )                                                                                                            -                            1                                                                          〉
γ              kn Σ                      f                                                                                  φ                      fkn                      2                                                                                                          v                        fn                                                                    w                        fk

Parameters of Interest

$\begin{array}{cc}{w}_{\mathrm{fk}}=\sqrt{\frac{\sum _{n=1}^N{\phi }_{\mathrm{fkn}}^2v_{\mathrm{fn}}〈h_{\mathrm{kn}}^{-1}〉}{\sum _{n=1}^N{\psi }_{\mathrm{fn}}^{-1}〈h_{\mathrm{kn}}〉}}& \left(20\right)\\ a_{\mathrm{ij}}\sqrt{\frac{{\beta }_i\sum _{n=2}^Nv_{\mathrm{ijn}}^2〈h_{in}〉〈h_{j\left(n-1\right)}^{-1}〉}{{\alpha }_i\sum _{n=2}^N{\rho }_{in}^{-1}〈h_{j\left(n-1\right)}〉}}& \left(21\right)\\ {\beta }_i={\alpha }_i\left(N-1\right){\left(\sum _{n=2}^N\frac{〈h_{in}〉}{\sum _ja_{\mathrm{ij}}{〈h_{j\left(n-1\right)}^{-1}〉}^{-1}}\right)}^{-1}& \left(22\right)\end{array}$

Updating Order

We denote the set of tuning parameter for frame n by ξ_n, i.e., ξ_n{{φ_fkn}_fk,{v_ijn}_ij,{ρ_in}_i,{ψ_fn}_f}.

As shown in FIG. 2, the following order of updates 230 leads to an efficient implementation.

At iteration (l) do

For n=1, . . . , N,

Update 231 the activation parameters [q(h_n)]^(l) as a function of [q(h_n−1)]^(l), [q(h_n)]^(l-1), [q(h_n+1)]^(l-1), ξ_n^(2l-2), W^(l-1), A^(l-1), β^(l-1).

Update ξ_n^(2l-1).

Update 232 the basis function W^(l) as a function of W^(l-1), [q(H)]^(l), ξ^(2l-1).Update 233 the transition matrix A^(l) as a function of A^(l-1), β^(l-1), [q(H)]^(l), ξ^(2l-1),Update tuning parameters ξ^(2l) Update 234 gamma distribution parameters β^(l) as a function of the transition matrix A^(l) and the activation parameters, [q(H)]^(l).

Under this updating order, the VB-EM procedure is:

Update q(H).

n = 1
α              kn 0 (Jeffreys) or 1 (uniform)
β              kn Σ                        f                                                                                                                    w                          fk                                                                                                      Σ                            j                                                                                                          w                            fj                                                                                                          〈                                                          h                              jn                                                        〉                                                                                                                +                                                                  Σ                        i                                                                                          α                        i                                                                                                                    a                          ik                                                                                                      Σ                            j                                                                                                          a                            ij                                                                                                          〈                                                          h                              jn                                                        〉
γ              kn 〈                                                  h                          kn                                                      -                            1                                                                          〉                                                                    -                        2                                                                                                        (                                                                                                    Σ                            f                                                                                                          w                            fk                                                                                                                                        v                              fn                                                                                                                      (                                                                                                      Σ                                    j                                                                                                                                          w                                    fj                                                                                                                                                                                〈                                                                              h                                        jn                                                                                  -                                          1                                                                                                                    〉                                                                                                              -                                      1                                                                                                                                      )                                                            2                                                                                                      +                                                                              Σ                            i                                                                                                          β                            i                                                                                                          a                            ik                                                                                                                                        〈                                                              h                                                                  i                                                                                                        (                                                                          n                                      +                                      1                                                                        )                                                                                                                              〉                                                                                                                      (                                                                                                      Σ                                    j                                                                                                                                          a                                    ij                                                                                                                                                                                〈                                                                              h                                        jn                                                                                  -                                          1                                                                                                                    〉                                                                                                              -                                      1                                                                                                                                      )                                                            2                                                                                                                          )
1 < n < N
α              kn αk
β              kn Σ                        f                                                                                                                    w                          fk                                                                                                      Σ                            j                                                                                                          w                            fj                                                                                                          〈                                                          h                              jn                                                        〉                                                                                                                +                                                                  Σ                        i                                                                                          α                        i                                                                                                                    a                          ik                                                                                                      Σ                            j                                                                                                          a                            ij                                                                                                          〈                                                          h                              jn                                                        〉                                                                                                                +                                                                  β                        k                                                                                              Σ                          j                                                                                                  a                          kj                                                                                                                              〈                                                          h                                                              j                                                                                                  (                                                                      n                                    -                                    1                                                                    )                                                                                                                            -                                1                                                                                      〉                                                                                -                            1
γ              kn 〈                                                  h                          kn                                                      -                            1                                                                          〉                                                                    -                        2                                                                                                        (                                                                                                    Σ                            f                                                                                                          w                            fk                                                                                                                                        v                              fn                                                                                                                      (                                                                                                      Σ                                    j                                                                                                                                          w                                    fj                                                                                                                                                                                〈                                                                              h                                        jn                                                                                  -                                          1                                                                                                                    〉                                                                                                              -                                      1                                                                                                                                      )                                                            2                                                                                                      +                                                                              Σ                            i                                                                                                          β                            i                                                                                                          a                            ik                                                                                                                                        〈                                                              h                                                                  i                                                                                                        (                                                                          n                                      +                                      1                                                                        )                                                                                                                              〉                                                                                                                      (                                                                                                      Σ                                    j                                                                                                                                          a                                    ij                                                                                                                                                                                〈                                                                              h                                        jn                                                                                  -                                          1                                                                                                                    〉                                                                                                              -                                      1                                                                                                                                      )                                                            2                                                                                                                          )
n = N
α              kn αk
β              kn Σ                        f                                                                                                                    w                          fk                                                                                                      Σ                            j                                                                                                          w                            fj                                                                                                          〈                                                          h                              jn                                                        〉                                                                                                                +                                                                  β                        k                                                                                              Σ                          j                                                                                                  a                          kj                                                                                                                              〈                                                          h                                                              j                                                                                                  (                                                                      n                                    -                                    1                                                                    )                                                                                                                            -                                1                                                                                      〉                                                                                -                            1
γ              kn 〈                                                  h                          kn                                                      -                            1                                                                          〉                                                                    -                        2                                                                                                        Σ                      f                                                                                  w                      fk                                                                                                          v                        fn                                                                                              (                                                                                    Σ                              j                                                                                                                  w                              fj                                                                                                                                                  〈                                                                  h                                  jn                                                                      -                                    1                                                                                                  〉                                                                                            -                                1                                                                                                              )                                                2

Update W, A, β

$w_{\mathrm{fk}}=w_{\mathrm{fk}}\sqrt{\frac{\sum _{n=1}^N{〈h_{\mathrm{kn}}^{-1}〉}^{-1}{v_{\mathrm{fn}}\left[\sum _jw_{\mathrm{fj}}{〈h_{\mathrm{jn}}^{-1}〉}^{-1}\right]}^{-2}}{\sum _{n=1}^N{〈h_{\mathrm{kn}}〉\left[\sum _jw_{\mathrm{fj}}〈h_{\mathrm{jn}}〉\right]}^{-1}}}$ $a_{\mathrm{ij}}=a_{\mathrm{ij}}\sqrt{\frac{{\beta }_i\sum _{n=2}^N{〈h_{j\left(n-1\right)}^{-1}〉}^{-1}{〈h_{in}〉\left[\sum _ka_{\mathrm{ik}}{〈h_{k\left(n-1\right)}^{-1}〉}^{-1}\right]}^{-2}}{{\alpha }_i\sum _{n=2}^N{〈h_{j\left(n-1\right)}^{-1}〉\left[\sum _ka_{\mathrm{ik}}〈h_{k\left(n-1\right)}〉\right]}^{-1}}}$ ${\beta }_i={\alpha }_i\left(N-1\right){\left(\sum _{n=2}^N\frac{〈h_{in}〉}{\sum _ja_{\mathrm{ij}}{〈h_{j\left(n-1\right)}^{-1}〉}^{-1}}\right)}^{-1}$

Determine the Bound

$B_{q,\xi }\left(W,A,\beta \right)=\sum _{\mathrm{fn}}\left(\log \sum _jw_{\mathrm{fj}}〈h_{\mathrm{jn}}〉+\frac{v_{\mathrm{fn}}}{\sum _jw_{\mathrm{fj}}{〈h_{\mathrm{jn}}^{-1}〉}^{-1}}\right)+\sum _{n=2}^N\sum _{i=1}^K\left({\alpha }_i\log \sum _ja_{\mathrm{ij}}〈h_{j\left(n-1\right)}〉+{\beta }_i\frac{〈h_{\mathrm{in}}〉}{\sum _ja_{\mathrm{ij}}{〈h_{j\left(n-1\right)}^{-1}〉}^{-1}}\right)+\left(N-1\right)\sum _{i=1}^K\left(\log \Gamma \left({\alpha }_i\right)-{\alpha }_i\log {\beta }_i\right)-\sum _{n=1}^N\sum _{i=1}^K\left({\stackrel{\_}{\alpha }}_{\mathrm{in}}\log \sqrt{\frac{{\stackrel{\_}{\gamma }}_{\mathrm{in}}}{{\stackrel{\_}{\beta }}_{\mathrm{in}}}}+\log K_{\alpha }\left(2\sqrt{{\stackrel{\_}{\beta }}_{\mathrm{in}}{\stackrel{\_}{\gamma }}_{\mathrm{in}}}\right)+{\stackrel{\_}{\beta }}_{\mathrm{in}}〈h_{\mathrm{in}}〉+{\stackrel{\_}{\gamma }}_{\mathrm{in}}〈h_{\mathrm{in}}^{-1}〉\right)-\mathrm{KN}\log 2$

Speech Denoising with the Dynamic Model

As shown in FIG. 3 for one embodiment, we use our method and model for speech enhancement, e.g., denoising. We construct our model parameters 101 for speech 306 by estimating bases W and the transition matrix A on some speech (audio) training data 305 as described above. We denote the trained bases and transition matrix as W^(s) and A^(s), where (s) is speech.

Similarly, we construct a noise model 307 with bases W⁽ⁿ⁾ and transition matrix A⁽ⁿ⁾, and combining the two models 306-307 into the single model 300 by concatenating W^(s) and W⁽ⁿ⁾ into W=[W^(s),W⁽ⁿ⁾], and A^(s) and A⁽ⁿ⁾ into A, where A is a block-diagonal matrix with A^(s) and A⁽ⁿ⁾ on the diagonal.

We can also train for noise on some noise training data, or we can fix the speech part of the model, and train for the noise part on the test data, thus making the noise part a general model that collects parts of the signal that cannot be modeled by the speech model. The simplest version of the later model uses a single basis for the noise, and uses an identity matrix as the transition matrix A.

After the model 300 is constructed, we can use the model to enhance an input audio signal x 301. We determine 310 a time-frequency feature representation. We estimate 320 the parameters of the model 300 that vary, i.e., the activation matrix H^(s) for the speech and H⁽ⁿ⁾ for the noise (n), and the bases W⁽ⁿ⁾ and transition matrix A⁽ⁿ⁾ for the noise.

Thus, we obtain a single model that combines speech, W^(s)H^(s) and noise W⁽ⁿ⁾H⁽ⁿ⁾, which we then use to reconstruct 330 the complex STFT of the enhanced speech {circumflex over (x)} 340, using

$\begin{array}{cc}{\hat{x}}_{\mathrm{fn}}=\frac{\sum _kw_{\mathrm{fk}}^{\left(s\right)}H_{\mathrm{kn}}^{\left(s\right)}}{\sum _kW_{\mathrm{fk}}^{\left(s\right)}H_{\mathrm{kn}}^{\left(s\right)}+\sum _kW_{\mathrm{fk}}^{\left(n\right)}H_{\mathrm{kn}}^{\left(n\right)}}x_{\mathrm{fn}}.& \left(23\right)\end{array}$

The time-domain signal can be reconstructed using a conventional overlap-add method, which evaluates a discrete convolution of a very long input signal with a finite impulse response filter

Extensions

Other complex models can also generated based on the above embodiments.

Dirichlet Innovations

Instead of considering the innovation random variables ε_n to be gamma distributed, the innovation can be Dirichlet distributed, which is similar to a normalization of the activation parameter h_n.

HMM-Like Behavior

We can constrain h_n to be 1-sparse during inference.

Structured Variational Inference

Conventional variational inference assumes that the variational posterior probabilities q(h_n) are independent of each other, which, given a strong dependency relation between h_n and h_n−1, is likely to be very wrong. We can model the posterior probability in terms of q(h_n|h_n−1). One possibility for such a q distribution uses a GIG distribution with parameters dependent on Ah_n−1.

Gamma Distribution of Innovation

The complex Gaussian model on the complex STFT coefficients in Eqn. (6) is equivalent to assuming that the power is exponentially distributed with parameter WH. We can extend the model by assuming that the power is gamma distributed, thus leading to a donut-shaped distribution for the complex coefficients.

Full Covariance of Innovation Random Variables

In linear dynamical systems, the innovation random variables can have a full-covariance. For positive random variables, one way to include the correlations is to transform an independent random vector with a non-negative matrix. This leads to the model,

h _n=(Ah_n−1)∘(Bf_n),

where f_n is a nonnegative random vector of size J×1 and B is a nonnegative matrix of dimension K×J. When B=I_K×K, this simplifies to f_n=ε_n. This can be accomplished in the more general form of the model

$h_{i,n}=\sum _{j,l}c_{i,j,l}{\varepsilon }_{l,n}h_{j,n-1}$

by setting the parameters to a factorized form: c_i,j,l=a_i,jb_i,l, where a_i,j are the elements of A, and b_i,j are the elements of B.

Transition Innovations

It can also be useful to model the transition between each of the components of h_n and h_n−1 using separate innovation random variables. This is analogous to the use of Dirichlet prior probabilities in discrete Markov models. One method would admit h_n=(A∘E_n)h_n−1, where E_n is a nonnegative innovations matrix of dimension K×K. This can be accomplished in the more general form of the model

$h_{i,n}=\sum _{j,l}c_{i,j,l}{\varepsilon }_{l,n}h_{j,n-1}$

by setting the parameters c_i,j,l=δ(m(i,j),l)a_i,j, where a_i,j are the elements of A and m(i,j) is a one-to-one mapping from each combination of i and j to an index corresponding to l. Then, the i, j-th element of E_n is ε_m(i,j),n.

Considering Other Innovation Types Besides Gam Ma

A log-normal, Poisson distribution leads to yet different types of dynamical systems.

Considering Other Divergences

We so far only considered the Itakura-Saito divergence. We can also use the KL-divergence, and different divergences for h_n|h_n−1 and for v|h.

Online Procedure

For real-time applications, only the signal up to the current time is used, e.g., an application where only the activation matrix H are estimated, or another application where all parameters are optimized. In the later application, we can perform a “warm” start with pretrained bases W and transition matrix A.

Multi-Channel Version

Because our model relies on a generative model involving the complex STFT coefficients, the model can be extended to a multi-channel application. Optimization in this setting involves EM updates between mixing system and a source NMF procedure.

EFFECT OF THE INVENTION

The embodiments of the invention provide a non-negative linear dynamical system model for processing non-stationary signals, particularly speech signals mixed with noise. In the context of speech separation and speech denoising, our model adapts to signal dynamics on-line, and achieves better performance than conventional methods.

Conventional models for signal dynamics frequently use hidden Markov models (HMMs) or non-negative matrix factorization (NMF). HMMs lead to combinatorial problems due to the discrete state space, are computationally complex, especially for mixed signals from several sources, and make it difficult to handle gain adaptation. NMF solves both the computational complexity and gain adaptation problems. However, NMF does not take advantage of past observations of a signal to model future observations of that signal. For signals with predictable dynamics, this is likely to be suboptimal.

Our model has advantages of both the HMMs and the NMF. The model is characterized by a continuous non-negative state space. Gain adaptation is automatically handled during inference. The complexity of the inference is linear in the number of sources, and dynamics are modeled via a linear transition matrix.

Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.

Claims

1. A method for transforming an input signal, comprising the steps of: storing parameters of a model of the input signal in a memory;receiving the input signal as a sequence of feature vectors;inferring, using the sequence of feature vectors and the parameters, a sequence of vectors of hidden variables, wherein there is at least one vector hn of hidden variables hi,n for each feature vector xn, and wherein each hidden variable is nonnegative;generating an output signal corresponding to the input signal, using the feature vectors, the vectors of hidden variables, and the parameters,wherein each feature vector xn is dependent on at least one of the hidden variables hi,n for the same n, and the hidden variables are related according to

2. The method of claim 1, wherein ci,j,l=δ(i,l)ai,j, where ai,j are non-negative scalars, and where δ is a Kronecker delta, so that

3. The method of claim 1, wherein ci,j,l=δ(m(i,j),l)ai,j, where ai,j are non-negative scalars, δ is a Kronecker delta and, m(i,j) is a one-to-one mapping from each combination of i and j to an index corresponding to l, so that

4. The method of claim 1, wherein the random variables εl,n are gamma distributed.

5. The method of claim 1, wherein an observation model used during the inferring is based at least in part on

6. The method of claim 5, wherein cf,i,l(v)=δ(i,l)wf,j, where wf,j are non-negative scalars, where δ is a Kronecker delta, and εf,n(v) are Gamma distributed random variables, so that the observation model based at least in part on

7. The method of claim 5, further comprising: obtaining the feature vectors xf,n as a complex spectrogram of the input signal, where xf,n is a value of the complex spectrogram for a frame n and frequency f, anddetermining a non-negative feature vf,n=|xf,n|2 as a power in frame n and frequency f so that the observation model is based at least in part on

8. The method of claim 6, further comprising: setting the parameter α(v)=1, and where θf,n is a uniformly distributed random phase variable, so that

9. The method of claim 1, wherein the inferring uses a maximum a-posteriori estimation.

10. The method of claim 1, wherein the inferring uses a variational Bayes method.

11. The method of claim 1, wherein the inferring is adaptive and performed on-line on the input signal.

12. The method of claim 1, wherein the input signal is received simultaneously multiple channels.

13. The method of claim 1, wherein an observation model used during the inferring is based at least in part on

14. The method claim 1, where the hidden variables hi,n are partitioned into S groups, and the non-negative random variables εl,n are each associated with one of the groups, wherein ci,j,l=0 when hi,n, and hj,n, or hi,n and εl,n are in different groups.

15. The method of claim 1, wherein the model is dynamic, and the input signal is non-stationary.

16. The method of claim 1, further comprising: adapting to again of the input signal on-line during the inferring.

17. The method of claim 1, wherein the input signal is a mixed signal of speech and noise, and the output signal is an enhanced speech signal.

18. The method of claim 1, wherein the parameters include basis functions W, a transition matrix A, an activation matrix H, a fixed shape parameter α, an inverse scale parameter β of a continuous gamma distribution parameter, and various combinations thereof.

19. The method of claim 18 wherein updating H and β are optional.

20. The method of claim 18, wherein updating β is optional in a maximum a-posteriori estimation used by the inferring.

21. The method of claim 1, wherein the input signal is received simultaneously from multiple sources by a single sensor.

22. The method of claim 18, wherein a posterior distribution of H is used in a variational Bayes method.