METHOD AND DEVICE FOR ENCODING MULTIPLE AUDIO SIGNALS, AND METHOD AND DEVICE FOR DECODING A MIXTURE OF MULTIPLE AUDIO SIGNALS WITH IMPROVED SEPARATION

Abstract

A method for encoding multiple audio signals comprises random sampling and quantizing each of the multiple audio signals, and encoding the sampled and quantized multiple audio signals as side information that can be used for decoding and separating the multiple audio signals from a mixture of said multiple audio signals. A method for decoding a mixture of multiple audio signals comprises decoding and demultiplexing side information, the side information comprising quantized samples of each of the multiple audio signals, receiving or retrieving from any data source a mixture of said multiple audio signals, and generating multiple estimated audio signals that approximate said multiple audio signals, wherein said quantized samples of each of the multiple audio signals are used.

Description

FIELD OF THE INVENTION

This invention relates to a method and a device for encoding multiple audio signals, and to a method and a device for decoding a mixture of multiple audio signals with improved separation of the multiple audio signals.

BACKGROUND

The problem of audio source separation consists in estimating individual sources (e.g. speech, music instruments, noise, etc.) from their mixtures. In the context of audio, mixture means a recording of multiple sources by a single or multiple microphones. Informed source separation (ISS) for audio signals can be viewed as the problem of extracting individual audio sources from a mixture of the sources, given that some information on the sources is available. ISS relates also to compression of audio objects (sources) [6], i.e. encoding a multisource audio, given that a mixture of these sources is known on both the encoding and decoding stages. Both of these problems are interconnected. They are important for a wide range of applications.

Known solutions (e.g. [3], [4], [5]) rely on the assumption that the original sources are available during an encoding stage. Side-information is computed and transmitted along with the mixture, and both are processed in a decoding stage to recover the sources. While several ISS methods are known, in all these approaches the encoding stage is more complex and computationally expensive than the decoding stage. Therefore these approaches are not preferable in cases where the platform performing the encoding cannot handle the computational complexity demanded by the encoder. Finally, the known complex encoders are not usable for online encoding, i.e. progressively encoding the signal as it arrives, which is very important for some applications.

SUMMARY OF THE INVENTION

In view of the above, it is highly desirable to have a fully automatic and efficient solution for both the ISS problems. In particular, a solution would be desirable where the encoder requires considerably less processing than the decoder. The present invention provides a simple encoding scheme that shifts most of the processing load from the encoder side to the decoder side. The proposed simple way for generating the side-information enables not only low complexity encoding, but also an efficient recovery at the decoder. Finally, in contrast to some existing efficient methods that need the full signal to be known during encoding (which is called batch encoding), the proposed encoding scheme allows online encoding, i.e. the signal is progressively encoded as it arrives.

The encoder takes random samples from the audio sources with a random pattern. In one embodiment, it is a predefined pseudo-random pattern. The sampled values are quantized by a predefined quantizer and the resulting quantized samples are concatenated and losslessly compressed by an entropy coder to generate the side information. The mixture can also be produced at the encoding side, or it is already available through other ways at the decoding side. The decoder first recovers the quantized samples from the side information, and then estimates probabilistically the most likely sources within the mixture, given the quantized samples and the mixture.

In one embodiment, the present principles relate to a method for encoding multiple audio signals as disclosed in claim 1. In one embodiment, the present principles relate to a method for decoding a mixture of multiple audio signal as disclosed in claim 3.

In one embodiment, the present principles relate to an encoding device that comprises a plurality of separate hardware components, one for each step of the encoding method as described below. In one embodiment, the present principles relate to a decoding device that comprises a plurality of separate hardware components, one for each step of the decoding method as described below. In one embodiment, the present principles relate to a computer readable medium having executable instructions to cause a computer to perform an encoding method comprising steps as described below. In one embodiment, the present principles relate to a computer readable medium having executable instructions to cause a computer to perform a decoding method comprising steps as described below.

In one embodiment, the present principles relate to an encoding device for separating audio sources, comprising at least one hardware component, e.g. hardware processor, and a non-transitory, tangible, computer-readable, storage medium tangibly embodying at least one software component, and when executing on the at least one hardware processor, the software component causes steps of the encoding method as described below. In one embodiment, the present principles relate to an encoding device for separating audio sources, comprising at least one hardware component, e.g. hardware processor, and a non-transitory, tangible, computer-readable, storage medium tangibly embodying at least one software component, and when executing on the at least one hardware processor, the software component causes steps of the decoding method as described below.

Further objects, features and advantages of the present principles will become apparent from a consideration of the following description and the appended claims when taken in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments are described with reference to the accompanying drawings, which show in

FIG. 1 the structure of a transmission and/or storage system, comprising an encoder and a decoder;

FIG. 2 the simplified structure of an exemplary encoder;

FIG. 3 the simplified structure of an exemplary decoder; and

FIG. 4 a performance comparison between CS-ISS and classical ISS.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows the structure of a transmission and/or storage system, comprising an encoder and a decoder. Original sound sources s₁, s₂, . . . , s_J are input to an encoder, which provides a mixture x and side information. The decoder uses the mixture x and side information to recover the sound, wherein it is assumed that some information has been lost: therefore the decoder needs to estimate the sound sources, and provides estimated sound sources {tilde over (s)}₁, {tilde over (s)}₂, . . . , {tilde over (s)}_J. It is assumed that the original sources s₁, s₂, . . . , s_J are available at the encoder, and are processed by the encoder to generate the side information. The mixture can also be generated by the encoder, or it can be available by other means at the decoder. For example, for a known audio track available on the internet, side information generated from individual sources can be stored, e.g. by the authors of the audio track or others. One problem described herein is having single channel audio sources recorded with single microphones, which are added together to form the mixture. Other configurations, e.g. multichannel audio or recordings with multiple microphones, can easily be handled by extending the described methods in a straight forward manner.

One technical problem that is considered here within the above-described setting consists in: when having an encoder to generate the side information, design a decoder that can estimate sources {tilde over (s)}₁, {tilde over (s)}₂, . . . , {tilde over (s)}_J that are as close as possible to the original sources s₁, s₂, . . . , s_J. The decoder should use the side information and the known mixture x in an efficient manner so as to minimize the needed size of the side information for a given quality of the estimated sources. It is assumed that the decoder knows both the mixture and how it is formed using the sources. Therefore the invention comprises two parts: the encoder and the decoder.

FIG. 2 a) shows the simplified structure of an exemplary encoder. The encoder is designed to be computationally simple. It takes random samples from the audio sources. In one embodiment, it uses a predefined pseudo-random pattern. In another embodiment, it uses any random pattern. The sampled values are quantized by a (predefined) quantizer, and the resulting quantized samples y₁, y₂, . . . , y_J are concatenated and losslessly compressed by an entropy coder (e.g. Huffman coder or arithmetic coder) to generate the side information. The mixture is also produced, if not already available at the decoding side.

FIG. 2 b) shows, enlarged, exemplary signals within the encoder. A mixture signal x is obtained by overlaying or mixing different source signals s₁, s₂, . . . , s_J. Each of the source signals s₁, s₂, . . . , s_J is also random sampled in random sampling units, and the samples are quantized in one or more quantizers (in this embodiment, one quantizer for each signal) to obtain quantized samples y₁, y₂, . . . , y_J. The quantized samples are encoded to be used as side information. Note that, in other embodiments, the sequence order of sampling and quantizing may be swapped.

FIG. 3 shows the simplified structure of an exemplary decoder. The decoder first recovers the quantized samples y₁, y₂, . . . , y_J from the side information. It then estimates probabilistically the most likely sources {tilde over (s)}₁, {tilde over (s)}₂, . . . , {tilde over (s)}_J, given the observed samples y₁, y₂, . . . , y_J and the mixture x and exploiting the known structures and correlations among the sources.

Possible implementations of the encoder are very simple. One possible implementation of the decoder operates based on the following two assumptions:

• • (1) The sources are jointly Gaussian distributed in the Short-Time Fourier Transform (STFT) domain with window size F and number of windows N.
  • (2) The variance tensor of the Gaussian distribution VεR₊^F×N×J has a low rank Non-Negative Tensor Decomposition (NTF) of rank K such that

$V\left(f,n,j\right)=\sum _{k=1}^KH\left(n,k\right)W\left(f,k\right)Q\left(j,k\right),H\in R_{+}^{N\times K},W\in R_{+}^{F\times K},Q\in R_{+}^{J\times K}$

Following these two assumptions, the operation of the decoder can be summarized with the following steps:

• • 1. Initialize matrices HεR₊^N×K, WεR₊^F×K, QεR₊^J×K with random nonnegative values and compute the variance tensor VεR₊^F×N×J as:

$V\left(f,n,j\right)=\sum _{k=1}^KH\left(n,k\right)W\left(f,k\right)Q\left(j,k\right)$

• • 2. Until convergence or maximum number of iterations reached, repeat:
    • 2.1 Compute the conditional expectations of the source power spectra such that

P(f,n,j)=E{|S(f,n,j)|²|x,y₁,y₂, . . . ,y_J,V}

• • • • where SεC^F×N×J are the array of the STFT complex coefficients of the sources. More details on this conditional expectation computation are provided below.
    • 2.2 Re-estimate NTF model parameters HεR₊^N×K, WεR₊^F×K, QεR₊^J×K using the multiplicative update (MU) rules minimizing the IS divergence [15] between the 3-valence tensor of estimated source power spectra P(f,n,j) and the 3-valence tensor of the NTF model approximation V(f,n,j) such that:

$Q\left(j,k\right)\leftarrow Q\left(j,k\right)\left(\frac{{\sum }_{f,n}W\left(f,k\right)H\left(n,k\right)P\left(f,n,j\right){V\left(f,n,j\right)}^{-2}}{{\sum }_{f,n}W\left(f,k\right)H\left(n,k\right){V\left(f,n,j\right)}^{-1}}\right)$ $W\left(f,k\right)\leftarrow W\left(f,k\right)\left(\frac{{\sum }_{j,n}Q\left(j,k\right)H\left(n,k\right)P\left(f,n,j\right){V\left(f,n,j\right)}^{-2}}{{\sum }_{j,n}Q\left(j,k\right)H\left(n,k\right){V\left(f,n,j\right)}^{-1}}\right)$ $H\left(n,k\right)\leftarrow H\left(n,k\right)\left(\frac{{\sum }_{f,j}W\left(f,k\right)Q\left(j,k\right)P\left(f,n,j\right){V\left(f,n,j\right)}^{-2}}{{\sum }_{f,j}W\left(f,k\right)Q\left(j,k\right){V\left(f,n,j\right)}^{-1}}\right)$

• • • These updates can be iteratively repeated multiple times.
  • 3. Compute the array of STFT coefficients SεC^F×N×J as the posterior mean as

{circumflex over (S)}(f,n,j)=E{S(f,n,j)|x,y₁,y₂, . . . ,y_J,V}

and convert back into the time domain to recover the estimated sources {tilde over (s)}₁, {tilde over (s)}₂, . . . , {tilde over (s)}_J. More details on this posterior mean computation are provided below.

The following describes some mathematical basics on the above calculations. A tensor is a data structure that can be seen as a higher dimensional matrix. A matrix is 2-dimensional, whereas a tensor can be N-dimensional. In the present case, V is a 3-dimensional tensor (like a cube). It represents the covariance matrix of the jointly Gaussian distribution of the sources.

A matrix can be represented as the sum of few rank-1 matrices, each formed by multiplying two vectors, in the low rank model. In the present case, the tensor is similarly represented as the sum of K rank one tensors, where a rank one tensor is formed by multiplying three vectors, e.g. h_i, q_i and w_i. These vectors are put together to form the matrices H, Q and W. There are K sets of vectors for the K rank one tensors. Essentially, the tensor is represented by K components, and the matrices H, Q and W represent how the components are distributed along different frames, different frequencies of STFT and different sources respectively. Similar to a low rank model in matrices, K is kept small because a small K better defines the characteristics of the data, such as audio data, e.g. music. Hence it is possible to guess unknown characteristics of the signal by using the information that V should be a low rank tensor. This reduces the number of unknowns and defines an interrelation between different parts of the data.

The steps of the above-described iterative algorithm can be described as follows.

First, initialize the matrices H, Q and W and therefore V.

Given V, the probability distribution of the signal is known. And looking at the observed part of the signals (signals are observed only partially), it is possible to estimate the STFT coefficients Ŝ, e.g. by Wiener filtering. This is the posterior mean of the signal. Further, also a posterior covariance of the signal is computed, which will be used below. This step is performed independently for each window of the signal, and it is parallelizable. This is called the expectation step or E-step.

Once the posterior mean and covariance are computed, these are used to compute the posterior power spectra p. This is needed to update the earlier model parameters, ie. H, Q and W. It may be advantageous to repeat this step more than once in order to reach a better estimate (e.g. 2-10 times). This is called the maximization step or M-step.

Once the model parameters H, Q and W are updated, all the steps (from estimating the STFT coefficients Ŝ) can be repeated until some convergence is reached, in an embodiment. After the convergence is reached, in an embodiment the posterior mean of the STFT coefficients Ŝ is converted into the time domain to obtain an audio signal as final result.

One advantage of the invention is that it allows improved recovering of multiple audio source signals from a mixture thereof. This enables efficient storage and transmission of a multisource audio recording without the need for powerful devices. Mobile phones or tablets can easily be used to compress information regarding the multiple sources of an audio track without a heavy battery drain or processor utilization.

A further advantage is that the computational resources for encoding and decoding the sources are more efficiently utilized, since the compressed information on the individual sources are decoded only if they are needed. In some applications, such as music production, the information on the individual sources are always encoded and stored, however it is not always needed and accessed afterwards. Therefore, as opposed to an expensive encoder that performs high complexity processing on every encoded audio stream, a system with a low complexity encoder and a high complexity decoder has the benefit of utilizing the processing power only for those audio streams for which the individual sources are actually needed later.

A third advantage provided by the invention is the adaptability to new and better decoding methods. When a new and improved way of exploiting correlations within the data is discovered, a new method for decoding can be devised (a better method to estimate {tilde over (s)}₁, {tilde over (s)}₂, . . . , {tilde over (s)}_J given x, y₁, y₂, . . . , y_J), and it is possible to decode the older encoded bitstreams with better quality without the need to re-encode the sources. Whereas in traditional encoding-decoding paradigms, when an improved way of exploiting correlations within the data leads to a new method of encoding, it is necessary to decode and re-encode the sources in order to exploit the advantages of the new approach. Furthermore, the process of re-encoding an already encoded bitstream is known to introduce further errors with respect to the original sources.

A fourth advantage of the invention is the possibility to encode the sources in an online fashion, i.e. the sources are encoded as they arrive to the encoder, and the availability of the entire stream is not necessary for encoding.

A fifth advantage of the invention is that gaps in the separated audio source signals can be repaired, which is known as audio inpainting. Thus, the invention allows joint audio inpainting and source separation, as described in the following.

The approach disclosed herein is inspired by distributed source coding [9] and in particular distributed video coding [10] paradigms, where the goal is also to shift the complexity from the encoder to the decoder. The approach relies on the compressive sensing/sampling principles [11-13], since the sources are projected on a linear subspace spanned by a randomly selected subset of vectors of a basis that is incoherent [13] with a basis where the audio sources are sparse. The disclosed approach can be called compressive sampling-based ISS (CS-ISS). More specifically, it is proposed to encode the sources by a simple random selection of a subset of temporal samples of the sources, followed by a uniform quantization and an entropy encoder. In one embodiment, this is the only side-information transmitted to the decoder.

Note that the advantage of sampling in the time domain is double. First, it is faster than sampling in any transformed domain. Second, the temporal basis is incoherent enough with the short time Fourier transform (STFT) frame where audio signals are sparse and it is even more incoherent with the low rank NTF representation of STFT coefficients. It is shown in compressive sensing theory that the incoherency of the measurement and prior information domains is essential for the recovery of the sources [13].

To recover the sources at the decoder from the quantized source samples and the mixture, it is proposed to use a model-based approach that is in line with model-based compressive sensing [14]. Notably, in one embodiment the Itakura-Saito (IS) nonnegative tensor factorization (NTF) model of source spectrograms is used, as in [4,5]. Thanks to its Gaussian probabilistic formulation [15], this model may be estimated in the maximum-likelihood (ML) sense from the mixture and the transmitted quantized portion of source samples. To estimate the model, a new generalized expectation-maximization (GEM) algorithm [16] based on multiplicative update (MU) rules [15] can be used. Given the estimated model and all other observations, the sources can be estimated by Wiener filtering [17].

Overview of the CS-ISS Framework

The overall structure of the proposed CS-ISS encoder/decoder is depicted in FIG. 2, as already explained above. The encoder randomly subsamples the sources with a desired rate, using a predefined randomization pattern, and quantizes these samples. The quantized samples are then ordered in a single stream to be compressed with an entropy encoder to form the final encoded bitstream. The random sampling pattern (or a seed that generates the random pattern) is known by both the encoder and decoder and therefore needs not be transmitted, in one embodiment. In another embodiment, the random sampling pattern, or a seed that generates the random pattern, is transmitted to the decoder. The audio mixture is also assumed to be known by the decoder. The decoder performs entropy decoding to retrieve the quantized samples of the sources, followed by CS-ISS decoding as will be discussed in detail below. The proposed CS-ISS framework has several advantages over traditional ISS, which can be summarized as follows:

A first advantage is that the simple encoder in FIG. 2 can be used for low complexity encoding, as needed e.g. in low power devices. A low-complexity encoding scheme is also advantageous for applications where encoding is used frequently but only few encoded streams need to be decoded. An example of such an application is music production in a studio where the sources of each produced music are kept for future use, but are seldom needed. Hence, significant savings in terms of processing power and processing time is possible with CS-ISS.

A second advantage is that performing sampling in time domain (and not in a transformed domain) provides not only a simple sampling scheme, but also the possibility to perform the encoding in an online fashion when needed, which is not always as straight forward for other methods [4,5]. Furthermore, the independent encoding scheme enables the possibility of encoding sources in a distributed manner without compromising the decoding efficiency.

A third advantage is that the encoding step is performed without any assumptions on the decoding step. Therefore it is possible to use other decoders than the one proposed in this embodiment. This provides a significant advantage over classical ISS [2-5] in the sense that, when a better performing decoder is designed, the encoded sources can directly benefit from the improved decoding without the need for re-encoding. This is made possible by the random sampling used in the encoder. The compressive sensing theory shows that a random sampling scheme provides incoherency with a large number of domains, so that it becomes possible to design efficient decoders relying on different prior information on the data.

CS-ISS Decoder

Let us indicate the support of the random samples with Ω″, such that the source jε1,J is sampled at time indices tεΩ″_j⊂1,T. After the entropy decoding stage, the CS-ISS decoder has the subset of quantized samples of the sources y″_jt(Ω″_j), jε1,J, where the quantized samples are defined as

y″ _jt =s″ _jt +b″ _jt  (1)

where s″_jt indicates the true source signal and b″_jt is the quantization noise. Note that herein the time-domain signals are represented by letters with two primes, e.g. x″, while framed and windowed time-domain signals are denoted by letters with one prime, e.g. x′, and complex-valued short-time Fourier transform (STFT) coefficients are denoted by letters with no prime, e.g. x.

The mixture is assumed to be the sum of the original sources such that

x″ _t=Σ_j=1^Js″_jt,tε1,T,jε1,J  (2)

The mixture is assumed to be known at the decoder. Note that the mixture is assumed to be noise free and without quantization herein. However, the disclosed algorithm can as well easily be extended to include noise in the mixture.

In order to compute the STFT coefficients, the mixture and the sources are first converted to a windowed time domain with a window length M and a total of N windows. Resulting coefficients denoted by y′_jmn, s′_jmn and x′_mn represent the quantized sources, the original sources and the mixture in windowed-time domain respectively for j=1, . . . , J, n=1, . . . , N and m=1, . . . , M (only form in appropriate subset Ω′_jn in case of quantized source samples). The STFT coefficients of the sources, s_jfn, and of the mixture, x_fn, are computed by applying the unitary Fourier transform Uε^F×M (F=M), to each window of the windowed-time domain counterparts. For example, [x_1n, . . . , x_Fn]^T=U[x′_1n, . . . , x′_Mn]^T.

The sources are modelled in the STFT domain with a normal distribution (s_jfn˜N_c(0, v_ifn) where the variance tensor V=[v_ifn]_i,f,n has the following low-rank NTF structure [18]:

v _ifn=Σ_k=1^Kq_jkw_fkh_nk,K<max(J,F,N)  (3)

The model is parameterized by ⊕={Q, W, H}, with Q=[q_jk]ε₊^J×K, W=[w_fk]ε₊^F×K and H=[h_nk]ε₊^N×K.

According to an embodiment of the present principles, the source signals are recovered with a generalized expectation-maximization algorithm that is briefly described in Algorithm 1. The algorithm estimates the sources and source statistics from the observations using a given model ⊕ via Wiener filtering at the expectation step, and then updates the model using the posterior source statistics at the maximization step. The details on each step of the algorithm are given below.

Algorithm 1 GEM algorithm for CS-ISS Decoding using the NTF
model
1: procedure CS-IS|S DECODING(x′, {y′j}1J, {Ω′j}1J, K)
2:  Initialize non-negative Q, W, H randomly
3:  repeat
4:    Estimate ŝ (sources) and {circumflex over (P)} (posterior power spectra),
given Q, W, H, x′, {y′j}1J, {Ω′j}1J     E-step, see section 3.1
5:    Update Q, W, H given P    M-step, see section 3.2
6:  until convergence criteria met
7: end procedure

Estimating the Sources

Since all the underlying distributions are Gaussian and all the relations between the sources and the observations are linear, the sources may be estimated in the minimum mean square error (MMSE) sense via the Wiener filter [17], given the covariance tensor V defined in (3) by the model parameters Q,W,H.

Let the observed data vector for the n-th frame ō′_n be defined as ō′_n [y′_1n^T, . . . y′_jn^T. x′_n^T]^T, where x′_n [x′_1n, . . . , x′_Mn]^T and y′_jn [y′_jmn, mεΩ′_jn]^T.

Given the corresponding observed data ō′_n and the NTF model ⊕, the posterior distribution of each source frame s_jn can be written as s_jn|ō′_n; ⊕˜N_c(ŝ_jn, Σ_sjnsjn) with ŝ_jn and {circumflex over (Σ)}_sjnsjn being, respectively, posterior mean and posterior covariance matrix. Each of them can be computed by Wiener filtering as

$\begin{array}{cc}{\hat{s}}_{\mathrm{jn}}=\sum _{{\stackrel{\_}{o}}_n^{\prime }s_{\mathrm{jn}}}^H\sum _{{\stackrel{\_}{o}}_n^{\prime }{\stackrel{\_}{o}}_n^{\prime }}^{-1}{\stackrel{\_}{o}}_n^{\prime },& \left(4\right)\\ {\hat{\sum }}_{s_{\mathrm{jn}}s_n}={\sum }_{s_{\mathrm{jn}}s_{\mathrm{jn}}}-\sum _{{\stackrel{\_}{o}}_n^{\prime }s_{\mathrm{jn}}}^H\stackrel{-1}{\sum _{{\stackrel{\_}{o}}_n^{\prime }{\stackrel{\_}{o}}_n^{\prime }}}{\sum }_{{\stackrel{\_}{o}}_n^{\prime }s_{\mathrm{jn}}},& \left(5\right)\end{array}$

given the definitions

$\begin{array}{cc}{\sum }_{{\stackrel{\_}{o}}_n^{\prime }{\stackrel{\_}{o}}_n^{\prime }}=\left[\begin{array}{cccc}{\sum }_{{\stackrel{\_}{y}}_{1n}^{\prime }{\stackrel{\_}{y}}_{1n}^{\prime }}& \dots & \underset{\_}{0}& \sum _{{\stackrel{\_}{x}}_n^{\prime }{\stackrel{\_}{y}}_{1n}^{\prime }}^H\\ ⋮& \ddots & ⋮& ⋮\\ \underset{\_}{0}& \dots & {\sum }_{{\stackrel{\_}{y}}_{\mathrm{Jn}}^{\prime }{\stackrel{\_}{y}}_{\mathrm{Jn}}^{\prime }}& \underset{{\stackrel{\_}{x}}_n^{\prime }{\stackrel{\_}{y}}_{\mathrm{Jn}}^{\prime }}{\sum ^H}\\ {\sum }_{{\stackrel{\_}{x}}_n^{\prime }{\stackrel{\_}{y}}_{1n}^{\prime }}& \dots & {\sum }_{{\stackrel{\_}{x}}_n^{\prime }{\stackrel{\_}{y}}_{\mathrm{Jn}}^{\prime }}& {\sum }_{{\stackrel{\_}{x}}_n^{\prime }{\stackrel{\_}{x}}_n^{\prime }}\end{array}\right],& \left(6\right)\\ {\sum }_{{\stackrel{\_}{o}}_n^{\prime }s_{\mathrm{jn}}}={\left[{\underset{\_}{0}}_{S_1\times F}^T,\sum _{{\stackrel{\_}{y}}_{\mathrm{jn}}^{\prime }s_{\mathrm{jn}}}^T,{\underset{\_}{0}}_{S_2\times F}^T,\sum _{{\underset{\_}{x}}_n^{\prime }s_{\mathrm{jn}}}^T\right]}^T,S_1\stackrel{\Delta }{=}\sum _{\hat{j}=1}^{j-1}{\Omega }_{\hat{j}n}^{\prime },S_2\stackrel{\Delta }{=}\sum _{\hat{j}=j+1}^J{\Omega }_{\hat{j}n}^{\prime },& \left(7\right)\\ {\sum }_{s_{\mathrm{jn}}s_{\mathrm{jn}}}=\mathrm{diag}\left({\left[v_{\mathrm{jfn}}\right]}_f\right),& \left(8\right)\\ {\sum }_{{\stackrel{\_}{y}}_{\mathrm{jn}}^{\prime }{\stackrel{\_}{y}}_{\mathrm{jn}}^{\prime }}={U\left({\Omega }_{\mathrm{jn}}^{\prime }\right)}^H\mathrm{diag}\left({\left[v_{\mathrm{jfn}}\right]}_f\right)U\left({\Omega }_{\mathrm{jn}}^{\prime }\right),& \left(9\right)\\ {\sum }_{{\stackrel{\_}{y}}_{\mathrm{jn}}^{\prime }s_{\mathrm{jn}}}={U\left({\Omega }_{\mathrm{jn}}^{\prime }\right)}^H\mathrm{diag}\left({\left[v_{\mathrm{jfn}}\right]}_f\right),& \left(10\right)\\ {\sum }_{{\stackrel{\_}{x}}_n^{\prime }s_{\mathrm{jn}}}=U^H\mathrm{diag}\left({\left[v_{\mathrm{jfn}}\right]}_f\right),& \left(11\right)\\ {\sum }_{{\stackrel{\_}{x}}_n^{\prime }{\stackrel{\_}{x}}_n^{\prime }}=U^H\mathrm{diag}\left({\left[{\sum }_jv_{\mathrm{jfn}}\right]}_f\right)U,& \left(12\right)\\ {\sum }_{{\stackrel{\_}{x}}_n^{\prime }{\stackrel{\_}{y}}_{\mathrm{jn}}^{\prime }}=U^H\mathrm{diag}\left({\left[v_{\mathrm{jfn}}\right]}_f\right)U\left({\Omega }_{\mathrm{jn}}^{\prime }\right),& \left(13\right)\end{array}$

where U(Ω′_jn) is the F×|Ω′_jn| matrix of columns from U with index in Ω′_jn.

Therefore the posterior power spectra {circumflex over (P)}=[{circumflex over (p)}_jfn] that will be used to update the NTF model as described below, can be computed as

{circumflex over (p)} _jfn = [|s _jfn|²|ō′_n;θ]=|ŝ_jfn|²+{circumflex over (Σ)}_sjnsjn(f,f).  (14)

Updating the Model

NTF model parameters can be re-estimated using the multiplicative update (MU) rules minimizing the IS divergence [15] between the 3-valence tensor of estimated source power spectra {tilde over (P)} and the 3-valence tensor of the NTF model approximation V defined as D_IS({tilde over (P)}∥V)=Σ_j,f,nd_IS({tilde over (p)}_jfn∥v_jfn), where

$d_{\mathrm{IS}}\left(x||y\right)=\frac{x}{y}-\log \left(\frac{x}{y}\right)-1$

is the IS divergence; and {tilde over (p)}_jfn and v_jfn are specified by (14) and (3). As a result, Q,W,H can be updated with the MU rules presented in [18]. These MU rules can be repeated several times to improve the model estimate.

Further, in source separation applications using the NTF/NMF model it is often necessary to have some prior information on the individual sources. This information can be some samples from the sources, or knowledge about which source is “inactive” at which instant of time. However, when such information is to be enforced, it has always been the case that the algorithms needed to predefine how many components each source is composed of. This is often enforced by initializing the model parameters Wε₊^M×K, Hε₊^N×K, Qε₊^J×K, so that certain parts of Q and H are set to zero, and each component is assigned to a specific source. In one embodiment, the computation of the model is modified such that, given the total number of components K, each source is assigned automatically to the components rather than manually. This is achieved by enforcing the “silence” of the sources not through STFT domain model parameters, but through time domain samples (with a constrain to have time domain samples of zeros) and by relaxing the initial conditions on the model parameters so that they are automatically adjusted. A further modification to enforce a sparse structure on the source component distribution (defined by Q) is also possible by slightly modifying the multiplicative update equations above. This results in an automatic assignment of sources to components.

Thus, in one embodiment the matrices H and Q are determined automatically when side information I_S of the form of silence periods of the sources are present. The side information I_S may include the information which source is silent at which time periods. In the presence of such specific information, a classical way to utilize NMF is to initialize H and Q in such a way that predefined k_i components are assigned to each source. The improved solution removes the need for such initialization, and learns H and Q so that k_i needs not to be known in advance. This is made possible by 1) using time domain samples as input, so that STFT domain manipulation is not mandatory, and 2) constraining the matrix Q to have a sparse structure. This is achieved by modifying the multiplicative update equations for Q, as described above.

Results

In order to assess the performance of the present approach, three sources of a music signal at 16 kHz are encoded and then decoded using the proposed CS-ISS with different levels of quantization (16 bits, 11 bits, 6 bits and 1 bit) and different sampling bitrates per source (0.64, 1.28, 2.56, 5.12 and 10.24 kbps/source). In this example, it is assumed that the random sampling pattern is pre-defined and known during both encoding and decoding. The quantized samples are truncated and compressed using an arithmetic encoder with a zero mean Gaussian distribution assumption. At the decoder side, following the arithmetic decoder, the sources are decoded from the quantized samples using 50 iterations of the GEM algorithm with STFT computed using a half-overlapping sine window of 1024 samples (64 ms) with a Gaussian window function and the number of components fixed at K=18, i.e. 6 components per source. The quality of the reconstructed samples is measured in signal to distortion ratio (SDR) as described in [19]. The resulting encoded bitrates and SDR of decoded signals are presented in Tab.1 along with the percentage of the encoded samples in parentheses. Note that the compressed rates in Tab.1 differ from the corresponding raw bitrates due to the variable performance of the entropy coding stage, which is expected.

TABLE 1
The final bitrates (in kbps per source) after the entropy coding stage of
CS-ISS with corresponding SDR (in dBs) for different (uniform) quantization levels
and different raw bitrates before entropy coding. The percentage of the samples
kept is also provided for each case in parentheses. Results corresponding to the
best rate-distortion compromise are in bold.
	Compressed Rate/SDR (% of Samples Kept)
	Raw rate (kbps/source)
Bits per Sample	0.64	1.28	2.56	5.12	10.24
16 bits	0.50/−1.64 (0.25)	1.00/4.28 (0.50)	2.00/9.54 (1.00)	4.01/16.17 (2.00)	8.00/21.87 (4.00)
11 bits	0.43/1.30 (0.36)	0.87/6.54 (0.73)	1.75/13.30 (1.45)	3.50/19.47 (2.91)	7.00/24.66 (5.82)
6 bits	0.27/4.17 (0.67)	0.54/7.62 (1.33)	1.08/12.09 (2.67)	2.18/14.55 (5.33)	4.37/16.55 (10.67)
1 bit	0.64/−5.06 (4.00)	1.28/−2.57 (8.00)	2.56/1.08 (16.00)	5.12/1.59 (32.00)	10.24/1.56 (64.00)

The performance of CS-ISS is compared to the classical ISS approach with a more complicated encoder and a simpler decoder presented in [4]. The ISS algorithm is used with NTF model quantization and encoding as in [5], i.e., NTF coefficients are uniformly quantized in logarithmic domain, quantization step sizes of different NTF matrices are computed using equations (31)-(33) from [5] and the indices are encoded using an arithmetic coder based on a two states Gaussian mixture model (GMM) (see FIG. 5 of [5]). The approach is evaluated for different quantization step sizes and different numbers of NTF components, i.e. Δ=2⁻², 2^−1.5, 2⁻¹, . . . , 2⁴ and K=4, 6, . . . , 30. The results are generated with 250 iterations of model update. The performance of both CS-ISS and classical ISS are shown in FIG. 4, in which CS-ISS clearly outperforms the ISS approach, even though the ISS approach can use optimized number of components and quantization as opposed to our decoder which uses a fixed number of components (the encoder is very simple and does not compute this value). The performance difference is due to the high efficiency achieved by the CS-ISS decoder thanks to the incoherency of random sampled time domain and of low rank NTF domain. Also, the ISS approach is unable to perform beyond an SDR of 10 dBs due to the lack of fidelity in the encoder structure as explained in [5]. Even though it was not possible to compare to the ISS algorithm presented in [5] in this paper due to time constraints, the results indicate that the rate distortion performance exhibits a similar behavior. It should be reminded that the proposed approach distinguishes itself by it low complexity encoder and hence can still be advantageous against other ISS approaches with better rate distortion performance.

The performance of CS-ISS in Tab.1 and FIG. 4 indicates that different levels of quantization may be preferable in different rates. Even though neither 16 bits nor 1 bit quantization seem well performing, the performance indicates that 16 bits quantization may be superior to other schemes when a much higher bitrate is available. Similarly coarser quantization such as 1 bit may be beneficial when considering significantly low bitrates. The choice of quantization can be performed in the encoder with a simple look up table as a reference. One must also note that even though the encoder in CS-ISS is very simple, the proposed decoder is significantly high complexity, typically higher than the encoders of traditional ISS methods. However, this can also be overcome by exploiting the independence of Wiener filtering among the frames in the proposed decoder with parallel processing, e.g. using graphical processing units (GPUs).

The disclosed solution usually leads to the fact that a low-rank tensor structure appears in the power spectrogram of the reconstructed signals.

It is to be noted that the use of the verb “comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in a claim.

Furthermore, the use of the article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. Several “means” may be represented by the same item of hardware. Furthermore, the invention resides in each and every novel feature or combination of features. As used herein, a “digital audio signal” or “audio signal” does not describe a mere mathematical abstraction, but instead denotes information embodied in or carried by a physical medium capable of detection by a machine or apparatus. This term includes recorded or transmitted signals, and should be understood to include conveyance by any form of encoding, including pulse code modulation (PCM), but not limited to PCM.

While there has been shown, described, and pointed out fundamental novel features of the present invention as applied to preferred embodiments thereof, it will be understood that various omissions and substitutions and changes in the apparatus and method described, in the form and details of the devices disclosed, and in their operation, may be made by those skilled in the art without departing from the spirit of the present invention. It is expressly intended that all combinations of those elements that perform substantially the same function in substantially the same way to achieve the same results are within the scope of the invention. Substitutions of elements from one described embodiment to another are also fully intended and contemplated. Each feature disclosed in the description and (where appropriate) the claims and drawings may be provided independently or in any appropriate combination. Features may, where appropriate be implemented in hardware, software, or a combination of the two. Connections may, where applicable, be implemented as wireless connections or wired, not necessarily direct or dedicated, connections.

CITED REFERENCES

• [1] E. Vincent, S. Araki, F. J. Theis, G. Nolte, P. Bofill, H. Sawada, A. Ozerov, B. V. Gowreesunker, D. Lutter, and N. Q. K. Duong, “The signal separation evaluation campaign (2007-2010): Achievements and remaining challenges,” Signal Processing, vol. 92, no. 8, pp. 1928-1936, 2012.
• [2] M. Parvaix, L. Girin, and J.-M. Brossier, “A watermarkingbased method for informed source separation of audio signals with a single sensor,” IEEE Trans. Audio, Speech, Language Process., vol. 18, no. 6, pp. 1464-1475, 2010.
• [3] M. Parvaix and L. Girin, “Informed source separation of linear instantaneous under-determined audio mixtures by source index embedding,” IEEE Trans. Audio, Speech, Language Process., vol. 19, no. 6, pp. 1721-1733, 2011.
• [4] A. Liutkus, J. Pinel, R. Badeau, L. Girin, and G. Richard, “Informed source separation through spectrogram coding and data embedding,” Signal Processing, vol. 92, no. 8, pp. 1937-1949, 2012.
• [5] A. Ozerov, A. Liutkus, R. Badeau, and G. Richard, “Coding-based informed source separation: Nonnegative tensor factorization approach,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 21, no. 8, pp. 1699-1712, August 2013.
• [6] J. Engdegard, B. Resch, C. Falch, O. Hellmuth, J. Hilpert, A. H{umlaut over ( )}olzer, L. Terentiev, J. Breebaart, J. Koppens, E. Schuijers, and W. Oomen, “Spatial audio object coding (SAOC)—The upcoming MPEG standard on parametric object based audio coding,” in 124th Audio Engineering Society Convention (AES 2008), Amsterdam, Netherlands, May 2008.
• [7] A. Ozerov, A. Liutkus, R. Badeau, and G. Richard, “Informed source separation: source coding meets source separation,” in IEEE Workshop Applications of Signal Processing to Audio and Acoustics (WASPAA'11), New Paltz, New York, USA, October 2011, pp. 257-260.
• [8] S. Kirbiz, A. Ozerov, A. Liutkus, and L. Girin, “Perceptual coding-based informed source separation,” in Proc. 22nd European Signal Processing Conference (EUSIPCO), 2014, pp. 959-963.
• [9] Z. Xiong, A. D. Liveris, and S. Cheng, “Distributed source coding for sensor networks,” IEEE Signal Processing Magazine, vol. 21, no. 5, pp. 80-94, September 2004.
• [10] B. Girod, A. Aaron, S. Rane, and D. Rebollo-Monedero, “Distributed video coding,” Proceedings of the IEEE, vol. 93, no. 1, pp. 71-83, January 2005.
• [11] D. Donoho, “Compressed sensing,” IEEE Trans. Inform. Theory, vol. 52, no. 4, pp. 1289-1306, April 2006.
• [12] R. G. Baraniuk, “Compressive sensing,” IEEE Signal Processing Mag., vol. 24, no. 4, pp. 118-120, July 2007.
• [13] E. J. Candes and M. B. Wakin, “An introduction to compressive sampling,” IEEE Signal Processing Magazine, vol. 25, pp. 21-30, 2008.
• [14] R. G. Baraniuk, V. Cevher, M. F. Duarte, and C. Hegde, “Model-based compressive sensing,” IEEE Trans. Info. Theory, vol. 56, no. 4, pp. 1982-2001, April 2010.
• [15] C. Fevotte, N. Bertin, and J.-L. Durrieu, “Nonnegative matrix factorization with the Itakura-Saito divergence. With application to music analysis,” Neural Computation, vol. 21, no. 3, pp. 793-830, March 2009.
• [16] A. P. Dempster, N. M. Laird, and D. B. Rubin., “Maximum likelihood from incomplete data via the EM algorithm,” Journal of the Royal Statistical Society. Series B (Methodological), vol. 39, pp. 1-38, 1977.
• [17] S. M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory. Englewood Cliffs, N.J.: Prentice Hall, 1993.
• [18] A. Ozerov, C. Fevotte, R. Blouet, and J.-L. Durrieu, “Multichannel nonnegative tensor factorization with structured constraints for user-guided audio source separation,” in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'11), Prague, May 2011, pp. 257-260.
• [19] V. Emiya, E. Vincent, N. Harlander, and V. Hohmann, “Subjective and objective quality assessment of audio source separation,” IEEE Trans. Audio, Speech, Language Process., vol. 19, no. 7, pp. 2046-2057, 2011.

Claims

1. A method for encoding multiple time-domain audio signals as side information that can be used for decoding and separating the multiple time-domain audio signals from a mixture of said multiple time-domain audio signals, said method comprising: random sampling and quantizing each of the multiple time-domain audio signals; andencoding the sampled and quantized multiple time-domain audio signals as said side information.

2. The method according to claim 1, wherein the random sampling uses a predefined pseudo-random pattern.

3. The method according to claim 1, wherein the mixture of multiple time-domain audio signal is progressively encoded as it arrives.

4. The method according to claim 1, further comprising steps of determining which source is silent at which time periods, and encoding the determined information in said side information.

5. A method for decoding a mixture of multiple audio signals, comprising receiving or retrieving, from storage or any data source, a mixture of said multiple audio signals; andgenerating multiple estimated audio signals that approximate said multiple audio signals from side information associated with said mixture of multiple audio signals,

6. The method according to claim 5, wherein generating multiple estimated audio signals comprises: computing a variance tensor V from random nonnegative values;computing conditional expectations of the source power spectra of the quantized samples of the multiple audio signals, wherein estimated source power spectra P(f,n,j) are obtained and wherein the variance tensor V and complex Short-Time Fourier Transform (STFT) coefficients of the multiple audio signals are used;iteratively re-calculating the variance tensor V from the estimated source power spectra P(f,n,j);computing an array of STFT coefficients Ŝ from the resulting variance tensor V; andconverting the array of STFT coefficients Ŝ to the time domain,

7. The method according to claim 5, further comprising audio inpainting for at least one of the multiple audio signals.

8. The method according to claim 5, wherein said side information further comprises information defining which audio source is silent at which time periods, further comprising determining automatically matrices H and Q that define the variance tensor V.

9. An apparatus for encoding multiple audio signals as side information that can be used for decoding and separating the multiple time-domain audio signals from a mixture of said multiple time-domain audio signals, comprising at least one processor configured for causing the apparatus to perform a method for encoding multiple time-domain audio signals, wherein said at least one processor is configured for causing the apparatus to random sampling and quantizing each of the multiple time-domain audio signals; andencoding the sampled and quantized multiple time-domain audio signals as said side information.

10. The apparatus according to claim 9, wherein the random sampling uses a predefined pseudo-random pattern.

11. An apparatus for decoding a mixture of multiple audio signals, comprising at least one processor configured for causing the apparatus to perform a method for decoding a mixture of multiple audio signals that comprises receiving or retrieving, from storage or any data source, a mixture of said multiple audio signals; andgenerating multiple estimated audio signals that approximate said multiple audio signals from side information associated with said mixture of multiple audio signals, wherein said at least one processor is configured for:decoding and demultiplexing the side information comprising randomly sampled quantized time-domain samples of each of the multiple audio signals;generating said multiple estimated audio signals using said quantized samples of each of the multiple audio signals.

12. The apparatus according to claim 11, wherein generating multiple estimated audio signals comprises: computing a variance tensor V from random nonnegative values;computing conditional expectations of the source power spectra of the quantized samples of the multiple audio signals, wherein estimated source power spectra P(f,n,j) are obtained and wherein the variance tensor V and complex Short-Time Fourier Transform (STFT) coefficients of the multiple audio signals are used;iteratively re-calculating the variance tensor V from the estimated source power spectra P(f,n,j);computing an array of STFT coefficients Ŝ from the resulting variance tensor V; andconverting the array of STFT coefficients Ŝ to the time domain, wherein the multiple estimated audio signals are obtained.

13. The apparatus according to claim 11, wherein said at least one processor is further configured for audio inpainting for at least one of the multiple time-domain audio signals.

14. A non-transitory program storage device, readable by a computer, tangibly embodying a program of instruction executable by the computer to perform a method for encoding multiple time-domain audio signals as side information that can be used for decoding and separating the multiple time-domain audio signals from a mixture of said multiple time-domain audio signals, said method comprising: random sampling and quantizing each of the multiple time-domain audio signals; andencoding the sampled and quantized multiple time-domain audio signals as said side information.

15. A non-transitory program storage device, readable by a computer, tangibly embodying a program of instruction executable by the computer to perform a method for decoding a mixture of multiple audio signals, comprising: receiving or retrieving, from storage or any data source, a mixture of said multiple audio signals; andgenerating multiple estimated audio signals that approximate said multiple audio signals from side information associated with said mixture of multiple audio signals,

16. A storage medium tangibly embodying a signal comprising side information configured for decoding a mixture of multiple audio signals, wherein said side information comprises randomly sampled quantized time-domain samples of each of the multiple audio signals.