Multi-channel speech enhancement system and method based on psychoacoustic masking effects

Abstract

The present invention is generally directed to a system and method for enhancing speech using a multi-channel noise filtering process that is based on psychoacoustic masking effects. A speech enhancement/noise reduction scheme according to the present invention is designed to satisfy the psychoacoustic masking principle and to minimize the signal total distortion by exploiting multiple microphone signals to enhance the useful speech signal at reduced level of artifacts.

Description

TECHNICAL FIELD

[0002] The present invention relates generally to a system and method for enhancing speech signals for speech processing systems (e.g., speech recognition). More particularly, the invention relates to a system and method for enhancing speech signals using a psychoacoustic noise reduction process that filters noise based on a multi-channel recording of the speech signal to thereby enhance the useful speech signal at a reduced level of artifacts.

BACKGROUND

[0003] In speech processing systems such as speech recognition, for example, it is desirable to remove noise from speech signals to thereby obtain accurate speech processing results. There are various techniques that have been developed to filter noise from an audio signal to obtain an enhanced signal for speech processing. Many of the known techniques use a single microphone solution (see, e.g., “Advanced Digital Signal Processing and Noise Reduction”, by S. V. Vaseghi, John Wiley & Sons, 2nd Edition, 2000).

[0004] For example, one approach for speech enhancement, which is based on psychoacoustic masking effects, is proposed in the article by S. Gustafsson, et al., A Novel Psychoacoustically Motivated Audio Enhancement Algorithm Preserving Background Noise Characteristics, ICASSP, pp. 397-400, 1998, which is incorporated herein by reference. Briefly, this method uses an observation from human hearing studies known as “tonal masking”, wherein a given tone becomes inaudible by a listener if another tone (the masking tone) having a similar or slightly different frequency is simultaneously presented to the listener. A detailed discussion of “tonal masking” can be found, for example, in the reference by W. Yost, Fundamentals of Hearing—An Introduction, 4th Ed., Academic Press, 2000.

[0005] More specifically, for a given speech signal (or more particular, for a given spectral power density), there is a psychoacoustic spectral threshold such that any interferer of spectral power below such threshold becomes unnoticed. In most de-noising schemes, there is a trade off between speech intelligibility (e.g., as measured by an “articulation index” defined in the reference by J. R. Deller, et al., Discrete-Time Processing of Speech Signals, IEEE Press, 2000) and the amount of removed noise as measured by SNR (signal-to-noise ratio) (see, the above-incorporated Gustafsson, et al. reference). Therefore, the entire removal of the noise from the speech signal is not necessarily desirable or even feasible.

[0006] Other noise reduction schemes that are known in the art employ two or more microphones to provide increased signal to noise ratio of the estimated speech signal. Theoretically, multi-channel techniques provide more information about the acoustic environment and therefore, should offer the possibility for improvement, especially in the case of reverberant environments due to multi-path effects and severe noise conditions known to affect the performance of known single channel techniques. However, the effectiveness of multiple channel techniques for a few microphones is yet to be proven.

[0007] For example, known beamforming techniques and, in general, conventional approaches that are based on microphone arrays, may achieve relatively small SNR improvements in the case of a small number of microphones. In addition, some multi-channel techniques may result in reduced intelligibility of the speech signal due to artifacts in the speech signal that are generated as a result of the particular processing algorithm.

[0008] Therefore, a speech enhancement system and method that would provide significant reduction of noise in a speech signal while maintaining the intelligibility of such speech signal for purposes of improved speech processing (e.g., speech recognition) would be highly desirable.

SUMMARY OF THE INVENTION

[0009] The present invention is generally directed to a system and method for enhancing speech using a multi-channel noise filtering process that is based on psychoacoustic masking effects. A speech enhancement/noise reduction scheme according to the present invention is designed to satisfy the psychoacoustic masking principle and to minimize the signal total distortion by exploiting the multiple microphone signals to enhance the useful speech signal at reduced level of artifacts.

[0010] A noise reduction system and method according to the present invention utilizes a noise filtering method that processes a multi-channel recording of the speech signal to filter noise from an input audio/speech signal. A preferred noise filtering method is based on a psychoacoustic masking threshold and calibration parameter (e.g., relative impulse response between the channels). Preferably, the noise is reduced down to the psychoacoustic threshold, but not below such threshold, which results in an estimated filtered (enhanced) speech signal that comprises a reduced level of artifacts. Advantageously, the present invention provides enhanced, intelligible speech signals that may be further processed (e.g., speech recognition) with improved accuracy.

[0011] In one aspect of the invention, a method for filtering noise from an audio signal comprises obtaining a multi-channel recording of an audio signal, determining a psychoacoustic masking threshold for the audio signal, determining a filter for filtering noise from the audio signal using the multi-channel recording, wherein the filter is determined using the masking threshold, and filtering the multi-channel recording using the filter to generate an enhanced audio signal.

[0012] The method further comprises determining a calibration parameter for the input channels. Preferably, the calibration parameter comprises a ratio of the impulse response of different channels. The calibration parameter is used to compute the filter.

[0013] In another aspect, the calibration parameter is determined by processing a speech signal recorded in the different channels under quiet conditions. For example, in one embodiment, the calibration parameter is determined by processing channel noise recorded in the different channels to determine a long-term spectral covariance matrix, and then determining an eigenvector of the long-term spectral covariance matrix corresponding to a desired eigenvalue.

[0014] In yet another aspect, the calibration parameter is determined using an adaptive process. In one embodiment, the adaptive process comprises a blind adaptive process. In other embodiments, the adaptive process comprises a non-parametric estimation process using a gradient algorithm or a model-based estimation process using a gradient algorithm.

[0015] In another aspect, a noise spectral power matrix is determined using the multi-channel recording, and the signal spectral power is determined using the noise spectral power matrix. The signal spectral power is used to determine the masking threshold, and the noise spectral power matrix is used to determine the filter.

[0016] In yet another aspect, the method comprises detecting speech activity in the audio signal, and updating the noise spectral power matrix at times when speech activity is not detected in the audio signal.

[0017] These and other objects, features and advantages of the present invention will be described or become apparent from the following detailed description of preferred embodiments, which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

[0018] FIG. 1 is a block diagram of a speech enhancement system according to an embodiment of the present invention.

[0019] FIG. 2 is a flow diagram of a speech enhancement method according to one aspect of the present invention.

[0020] FIGS. 3 a and 3b are diagram illustrating exemplary input waveforms of a first and second channel, respectively, in a two-channel speech enhancement system according to the present invention.

[0021] FIG. 3 c is an exemplary diagram of the output waveform of a two-channel speech enhancement system according to the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0022] The present invention is generally directed to a system and method for enhancing speech using a multi-channel noise filtering process that is based on psychoacoustic masking effects. A speech enhancement system and method according to the present invention utilizes a noise filtering method that processes a multi-channel recording of an audio signal comprising speech to filter the input audio signal to generate a speech enhanced (filtered) signal. A preferred noise filtering method utilizes a psychoacoustic masking threshold and a calibration parameter (e.g., ratio of the impulse response of different channels) to enhance the speech signal. Preferably, the noise is reduced down to the psychoacoustic threshold, but not below such threshold, which results in an estimated (enhanced) speech signal that comprises a reduced and minimal level of artifacts.

[0023] It is to be understood that the systems and methods described herein in accordance with the present invention may be implemented in various forms of hardware, software, firmware, special purpose processors, or a combination thereof. Preferably, the present invention is implemented in software as an application comprising program instructions that are tangibly embodied on one or more program storage devices (e.g., magnetic floppy disk, RAM, CD ROM, ROM and Flash memory), and executable by any device or machine comprising suitable architecture.

[0024] It is to be further understood that since the constituent system modules and method steps depicted in the accompanying Figures are preferably implemented in software, the actual connections between the system components (or the flow of the process steps) may differ depending upon the manner in which the present invention is programmed. Given the teachings herein, one of ordinary skill in the related art will be able to contemplate these and similar implementations or configurations of the present invention.

[0025] FIG. 1 is a block diagram of a speech enhancement system 10 according to an embodiment of the present invention. The system 10 comprises an input microphone array 11 and a speech enhancement processor 12. For purposes of illustration, the exemplary psychoacoustic noise reduction system 10 comprises a two-channel scheme, wherein a second microphone signal is used to further enhance the useful speech signal at reduced level of artifacts. It is to be understood, however, that FIG. 1 should not be construed as any limitation because a speech enhancement and noise filtering method according to this invention may comprise a multi-channel framework having 3 or more channels. Various embodiments for multi-channel schemes will be described herein.

[0026] A multi-channel speech enhancement/noise reduction system (e.g., the dual-channel scheme of FIG. 1) can be used, for example, in real office or car environments. The system can be implemented as a front-end processing component for voice enhancement and noise reduction in a voice communication or speech recognition device. Preferably, a source of interest S is localized, wherein it is assumed that the microphones of microphone array 11 are placed at substantially fixed locations with respect to the speech source S (e.g., the user (speaker) is assumed to be static with respect to the microphones while using the speech processing device). However, adaptive mechanisms according to the present invention can be used to account for, e.g., movement of the source S during use of the system.

[0027] The signal processing front-end 12 comprises a sampling module 13 that samples the input signals received from the microphone array 11. In a preferred embodiment, the sampling module 13 samples the input signals in the frequency domain by computing the DFT (Discrete Fourier Transform) for each input channel. The speech processor 12 further comprises a calibration module 14 for determining a calibration parameter K that is used for filtering the input audio signal. In one preferred embodiment, K is an estimate of the transfer function ratios between channels. As explained in further detail below, K may be a static parameter that is determined or set (default parameter) only at initialization, or K may be a dynamic parameter that is determined/set at initialization and then adapted during use of the system 10.

[0028] In a speech enhancement/noise reduction system comprising a two-channel framework (wherein a second microphone signal is used to further enhance the useful speech signal at reduced level of artifacts), a mixing model according to an embodiment of the invention is given by:

x 1 (t)=s(t)+n1(t)   (1)

x 2 (t)=k*s(t)+n2(t)   (2)

[0029] where x1(t) and x2(t) are the measured input signals, s(t)is the speech signal as measured by the first microphone in the absence of the ambient noise, and n1(t) and n2(t) are the ambient nose signals, all sampled at moment t.

[0030] The sequence k represents the relative impulse response between the two channels and is defined in the frequency domain by the ratio of the measured input signals X1o, X2o in the absence of noise: 1$\begin{array}{cc}K\left(w\right)=\frac{X_2^o}{X_1^o}& \left(3\right)\end{array}$

[0031] Since a speech enhancement method according to the present invention is preferably applied in the frequency domain, the sequence k(t) is defined as the function K(w). Accordingly, in the frequency domain, the mixing model (equations 1 and 2) becomes:

X 1 (w)=S(w)+N1(w)   (4)

X 2 (w)=K(w)S(w)+N2(w)   (5)

[0032] The speech processor 12 further comprises a VAD (voice activity detection) module 15 for detecting whether voice is present in a current frame of data of the recorded audio signal. Although any suitable multi-channel voice detection method may be used, a preferred voice detection method is described in the publication by J. Rosca, et al., “Multi-channel Source Activity Detection”, In Proceedings of the European Signal Processing Conference, EUSIPCO, 2002, Toulouse, France, which is fully incorporated herein by reference.

[0033] Further, in the illustrative embodiment, the voice activity detector module 15 determines a noise spectral power matrix Rn, which is used in a noise filtering process. In one embodiment, the noise spectral power matrix Rn is dynamically computed and updated. In accordance with the present invention, an ideal noise spectral power matrix (for a two channel framework) is defined by: 2$\begin{array}{cc}{\hat{R}}_n=E\left[\begin{array}{c}{N}_1\\ N_2\end{array}\right]\left[\begin{array}{cc}{\stackrel{\_}{N}}_1& {\stackrel{\_}{N}}_2\end{array}\right]& \left(6\right)\end{array}$

[0034] where E is the expectation operator. In one embodiment of the invention, the ideal noise spectral power matrix is estimated using the frequency domain representation of the input signals X1(w)and X2(w) as follows: 3$\begin{array}{cc}{R}_n^{\mathrm{new}}=\left(1-\alpha \right)R_n^{\mathrm{old}}+\alpha \left[\begin{array}{c}{X}_1\\ X_2\end{array}\right]\left[\stackrel{\_}{X_1{X}_2}\right]& \text{(6a)}\end{array}$

[0035] wherein Rnnew denotes an updated noise spectral power matrix that is estimated using the old (last computed) noise spectral power matrix Rnold, and wherein denotes a learning rate, which is a predefined experimental constant that is determined based on the system design. In a two-channel system such as depicted in FIG. 1, a preferred value is =0.1.

[0036] When voice is not detected in the current frame of data, the VAD module 15 will update the noise spectral power matrix Rn using equation (6a), for example. Other methods for determining the noise spectral power matrix are described below.

[0037] The speech enhancement processor 12 further comprises a filter parameter module 16, which determines filter parameters that are used by filter module 17 to generate an enhanced/filtered signal S(w) in the frequency domain. An IDFT (inverse discrete Fourier transform) module 18, transforms the frequency domain representation of the enhanced signal S(w) into a time domain representation s(t). Various methods according to the invention for filtering a multi-channel recording using estimated filter parameters will be described in detail below.

[0038] FIG. 2 is a flow diagram of a speech enhancement method according to one aspect of the present invention. For purposes of illustration, the method of FIG. 2 will be described with reference to a two-channel system, but the method of FIG. 2 is equally applicable to a multi-channel system with 3 or more channels.

[0039] In general, the method of FIG. 2 comprises two processes: (i) a calibration process whereby noise reduction parameters are estimated or set (default parameters) upon initialization of the multi-channel system; and (ii) a signal estimation process whereby the input signals in each channel are filtered to generate an enhanced signal.

[0040] During use of the speech system, a two-channel speech enhancement process according to the invention uses X1(w), X2(w), the DFT on current time frame of x1(t), x2(t) windowed by w, and an estimate of the noise spectral power matrix Rn (e.g., a 2×2 matrix Rn=R11R12,R21R22) to filter the input signal and generate an enhanced speech signal.

[0041] More specifically, referring now to FIG. 2, during initialization of the speech system, a calibration parameter K is determined (step 20). In one preferred embodiment, K is an estimate of the transfer function ratios between channels. K is used for filtering the input audio signal. As explained in further detail below, K may be a static parameter that is determined or set (default parameter) only at initialization, or K may be a dynamic parameter that is determined/set at initialization and then adapted during use of the system.

[0042] In particular, a calibration process can be initially performed to estimate the calibration parameter (e.g., estimate the ratio of the transfer functions of the channels). In one embodiment, this calibration process is performed by the user speaking a sentence in the absence (or a low level) of noise. Based on the two recordings, x1c(t),x2c(t), in accordance with one embodiment of the present invention, the constant K(w) is estimated by: 4$\begin{array}{cc}K\left(w\right)=\frac{\sum _{l=1}^F{X}_2^c\left(l,w\right)\stackrel{\_}{X_1^c\left(l,w\right)}}{\sum _{l=1}^F{\&LeftBracketingBar;X_1^c\left(l,w\right)\&RightBracketingBar;}^2}& \left(7\right)\end{array}$

[0043] where X1c(l,w),X2c(l,w) represents the discrete windowed Fourier transform at frequency w, and time-frame index l of the signals x1c(t),x2c(t), windowed by a Hamming window w(.) of size 512 samples, for example. Other methods for performing a calibration to estimate K are described below.

[0044] Alternatively, a default parameter K may be set upon initialization of the system. In this embodiment, the calibration parameter K is predetermined based on the system design and intended use, for example. Moreover, as noted above, the calibration parameter K may be determined once at initialization and remain constant during use of the system, or an adaptive protocol may be implemented to dynamically adapt the calibration to account for, e.g., possible movement of the speech source (user) with respect to the microphone array during use of the system.

[0045] In addition, upon initialization, an initial noise spectral power matrix is determined (step 21). In one embodiment of the present invention, this initial value is preferably computed using equation (6a) with =1, i.e., 5$R_n^{\mathrm{initial}}=\left[\begin{array}{c}{X}_1\\ X_2\end{array}\right]\left[\stackrel{\_}{X_1{X}_2}\right].$

[0046] Other methods for determining the initial noise spectral power matrix are described below.

[0047] After initialization of the system (e.g., steps 20 and 21), a signal estimation process is performed to enhance the user's voice signal during use of the speech system. The system samples the input signal in each channel in the frequency domain (step 22). More specifically, in the exemplary embodiment, X1 and X2 are computed using a windowed Fourier transform of current data x1, x2. During operation of the speech system, whenever voice activity is not detected in the input signal (negative determination in step 23) the noise spectral power matrix Rn is updated (step 24). In accordance with one embodiment of the present invention, this update process is performed using equation (6a) (other methods for updating the noise spectral power matrix are described below). By updating Rn on such basis, the efficiency of noise filtering process will be maintained at an optimal level.

[0048] In addition, if adaptive estimation of K is desired (affirmative result in step 25), the calibration parameter K will be adapted (step 26). K is dynamically updated using, for example, any of the methods described herein.

[0049] As the input signal is received and sampled (and the noise parameters updated), the signal spectral power ρS is determined (step 27), preferably using spectral subtraction on channel one. By way of example, according to one embodiment of the present invention, the signal spectral power is determined by estimating the signal spectral power for a two-channel system as follows: 6$\begin{array}{cc}{\rho }_s=\theta \left({\&LeftBracketingBar;X_1\&RightBracketingBar;}^2-R_{11}\right),\theta \left(x\right)=\{\begin{array}{cc}x,& \mathrm{if}x>0\\ 0,& \mathrm{otherwise}\end{array}& \left(8\right)\end{array}$

[0050] Other methods for determining the signal spectral power are described below.

[0051] Next, the psychoacoustic masking threshold RT is determined using the signal spectral power, ρS (step 28). In a preferred embodiment, the masking threshold RT is computed ousing the known ISO/IEC standard (see, e.g., International Standard. Information Technology—Coding of moving pictures and associated audio for digital media up to about 1.5 Mbits/s—Part 3: Audio. ISO/IEC, 1993).

[0052] Next, the filter parameters are determined (step 29) using the masking threshold, RT, the noise spectral power matrix Rn, and the calibration parameter K. In a two-channel system, one method for estimating filter parameters A,B, is as follows: 7$\begin{array}{cc}{A}_o=\zeta +\left(R_{22}-R_{21}\stackrel{\_}{K}\right)\sqrt{\frac{R_T}{\left(R_{11}{R}_{22}-{\&LeftBracketingBar;R_{12}\&RightBracketingBar;}^2\right)\left(R_{22}+R_{11}{\&LeftBracketingBar;K\&RightBracketingBar;}^2-R_{12}K-R_{21}\stackrel{\_}{K}\right)}}& \left(9\right)\\ B_o=\left(R_{11}\stackrel{\_}{K}-R_{12}\right)\sqrt{\frac{R_T}{\left(R_{11}{R}_{22}-{\&LeftBracketingBar;R_{12}\&RightBracketingBar;}^2\right)\left(R_{22}+R_{11}{\&LeftBracketingBar;K\&RightBracketingBar;}^2-R_{12}K-R_{21}\stackrel{\_}{K}\right)}}\text{and then:}& \left(10\right)\\ \left(A,B\right)=\{\begin{array}{cc}\left(1,0\right),& \mathrm{if}\&LeftBracketingBar;A_o+B_{o}K\&RightBracketingBar;>1\\ \left(A_o,B_o\right),& \mathrm{otherwise}.\end{array}& \left(11\right)\end{array}$

[0053] Further details of various embodiments of the filter parameter estimation process will be described hereafter.

[0054] Next, the input signals are filtered using the filter parameters to compute an enhanced signal (step 30). For example, in the exemplary two-channel framework using the above filter parameters A,B, a filtering process is as follows:

S=AX 1 +BX 2   (12)

[0055] The signal S is then preferably transformed into the time domain using an overlap-add procedure using a windowed inverse discrete Fourier transform process to thus obtain an estimate for the signal s(t) (step 31).

[0056] A detailed discussion regarding the filtering process will now be presented by explaining the basis for equations 9, 10 and 11. In a preferred embodiment for a two-channel framework as described herein, a linear filter [A,B] is preferably applied on the measurements X1, X2. The output (estimated signal S) is computed as:

S=AX 1 +BX 2 =(A+BK)S+AN1+BN2

[0057] Preferably, we would like to obtain an estimate of S that contains a small amount of noise. Let 0≦ζ1, ζ2≦1 be two given constants such that the desired signal is w=S+ζ1N1+ζ2N2. Then the error e=s−w has the variance: 8$R_e={\&LeftBracketingBar;A+\mathrm{BK}-1\&RightBracketingBar;}^2{\rho }_s+\left[\begin{array}{cc}A-{\zeta }_1& B-{\zeta }_2\end{array}\right]R_n\left[\begin{array}{c}\stackrel{\_}{A}-{\zeta }_1\\ \stackrel{\_}{B}-{\zeta }_2\end{array}\right]$

[0058] Preferably, the filter(s) are designed such that the distortion term due to noise achieves a preset value RT, the threshold masking, depending solely on the signal spectral power ps. The idea is that any noise whose spectral power is below the threshold RT is unnoticed and consequently, such noise should not be completely canceled. Furthermore, by doing less noise removal, the artifacts would be smaller as well. Thus, following this premise, it is preferred that the filter achieve a noise distortion level of RT. Yet, we have two unknowns (one for each channel) and one constraint (RT) so far. This leaves us with one degree of freedom. We can use this degree of freedom to choose A,B that minimizes the total distortion. In one embodiment of the invention, an optimization problem for the two-channel system is: 9$\begin{array}{cc}\arg {\min }_{A,B}{R}_e,s\mathrm{ubject}\mathrm{to}\left[\begin{array}{cc}A-{\zeta }_1& B-{\zeta }_2\end{array}\right]R_n\left[\begin{array}{c}\stackrel{\_}{A}-{\zeta }_1\\ \stackrel{\_}{B}-{\zeta }_2\end{array}\right]=R_T& \left(14\right)\end{array}$

[0059] Suppose (Ao, Bo) is the optimal solution. Then we validate it by checking whether |Ao+BoK|≦1. If not, we choose not to do any processing (perhaps the noise level is already lower than the threshold, so there is no need to amplify it).

[0060] Hence: 10$\begin{array}{cc}\left(A,B\right)=\left\{\begin{array}{cc}\left(A_o,B_o\right)& \mathrm{if}\&LeftBracketingBar;A_o+B_{o}K\&RightBracketingBar;\le 1\\ \left(1,0\right)& \mathrm{if}\mathrm{otherwise}\end{array}\right\}& \left(15\right)\end{array}$

[0061] Let M(A,B) denote the expression in A, B subject to the constraint. Using the Lagrange multiplier theorem, for the lagrangian:

L (A,B,λ)=|A+BK−1|2 ρS+Φ(A,B)+λ(RT−Φ(A,B))

[0062] we obtain the system: 11$\begin{array}{cc}\left(p_s\left[\begin{array}{cc}1& \stackrel{\_}{K}\\ K& {\&LeftBracketingBar;K\&RightBracketingBar;}^2\end{array}\right]-\lambda R_n\right)\left[\begin{array}{c}\stackrel{\_}{A}-{\zeta }_1\\ \stackrel{\_}{B}-{\zeta }_2\end{array}\right]-p_s\left(1-{\zeta }_1-{\zeta }_2\stackrel{\_}{K}\right)\left[\begin{array}{c}1\\ K\end{array}\right]=0& \left(i\right)\end{array}$

[0063] (ii) M(A,B) =RT

[0064] Solving for (A,B) in the first equation (i) and inserting the expression into the second equation (ii), we obtain for 8: 12$\left[\begin{array}{cc}1& \stackrel{\_}{K}\end{array}\right]{\left({\rho }_s\left[\begin{array}{cc}1& \stackrel{\_}{K}\\ K& {\&LeftBracketingBar;K\&RightBracketingBar;}^2\end{array}\right]-\lambda R_n\right)}^{-1}$${R_n\left({\rho }_s\left[\begin{array}{cc}1& \stackrel{\_}{K}\\ K& {\&LeftBracketingBar;K\&RightBracketingBar;}^2\end{array}\right]-\lambda R_n\right)}^{-1}\left[\begin{array}{c}1\\ K\end{array}\right]=\frac{R_T}{{\rho }_s^2{\&LeftBracketingBar;1-{\zeta }_1-{\zeta }_{2}K\&RightBracketingBar;}^2}$

[0065] Using the Matrix Inversion Lemma (see, e.g., D. G. Manolakis, et al., “Statistical and Adaptive Signal Processing”, McGraw Hill Series in Electrical and Computer Engineering, Appendix A, 2000), the equation in 8 becomes: 13$\begin{array}{cc}\begin{array}{c}\lambda ={\rho }_s{R}_{22}+R_{11}{\&LeftBracketingBar;K\&RightBracketingBar;}^2-\frac{R_{12}K{R}_{21}\stackrel{\_}{K}}{R_{11}{R}_{22}-{\&LeftBracketingBar;R_{12}\&RightBracketingBar;}^2}\pm \\ {\rho }_s\&LeftBracketingBar;1-{\zeta }_1-{\zeta }_{2}K\&RightBracketingBar;\\ \sqrt{\frac{R_{22}+R_{11}{\&LeftBracketingBar;K\&RightBracketingBar;}^2-R_{12}K-R_{21}\stackrel{\_}{K}}{R_T\left(R_{11}{R}_{22}-{\&LeftBracketingBar;R_{12}\&RightBracketingBar;}^2\right)}}.\end{array}& \left(16\right)\end{array}$

[0066] Replacing in Re, we obtain: 14$\begin{array}{c}{R}_e=R_T+{\rho }_s{\&LeftBracketingBar;1-{\zeta }_1-{\zeta }_{2}K\&RightBracketingBar;}^2\\ {\&LeftBracketingBar;1\pm \frac{1}{\&LeftBracketingBar;1-{\zeta }_1-{\zeta }_{2}K\&RightBracketingBar;}\sqrt{\frac{R_T(R_{22}+R_{11}{\&LeftBracketingBar;K\&RightBracketingBar;}^2-R_{12}K-R_{21}K\stackrel{\_}{)}}{R_{11}{R}_{22}-{\&LeftBracketingBar;R_{12}\&RightBracketingBar;}^2}}\&RightBracketingBar;}^2\end{array}$

[0067] Hence the optimal solution is the one with—in equation (16). Consequently, the optimizer becomes: 15$\begin{array}{cc}\begin{array}{c}{A}_o={\zeta }_1-\left(R_{22}-R_{21}\stackrel{\_}{K}\right)\arg \left({\zeta }_1+{\zeta }_{2}K-1\right)\\ \sqrt{\frac{R_T}{\left(R_{11}{R}_{22}-{\&LeftBracketingBar;R_{12}\&RightBracketingBar;}^2\right)\left(R_{22}+R_{11}{\&LeftBracketingBar;K\&RightBracketingBar;}^2-R_{12}K-R_{21\stackrel{\_}{K}}\right)}}\end{array}& \left(17\right)\\ \begin{array}{c}{B}_o={\zeta }_2-\left(R_{11}\stackrel{\_}{K}-R_{12}\right)\arg \left({\zeta }_1+{\zeta }_{2}K-1\right)\\ \sqrt{\frac{R_T}{\left(R_{11}{R}_{22}-{\&LeftBracketingBar;R_{12}\&RightBracketingBar;}^2\right)\left(R_{22}+R_{11}{\&LeftBracketingBar;K\&RightBracketingBar;}^2\right)-R_{12}K-R_{21\stackrel{\_}{K}})}}\end{array}& \left(18\right)\end{array}$

[0068] The more practical form is obtained for ζ1=ζ and ζ210. Then: 16$\begin{array}{cc}\begin{array}{c}{A}_o=\zeta +\left(R_{22}-R_{21}\stackrel{\_}{K}\right)\\ \sqrt{\frac{R_T}{\left(R_{11}{R}_{22}-{\&LeftBracketingBar;R_{12}\&RightBracketingBar;}^2\right)\left(R_{22}+R_{11}{\&LeftBracketingBar;K\&RightBracketingBar;}^2-R_{12}K-R_{21\stackrel{\_}{K}}\right)}}\end{array}\mathrm{and}& \left(19\right)\\ \begin{array}{c}{B}_o=\left(R_{11}\stackrel{\_}{K}-R_{12}\right)\\ \sqrt{\frac{R_T}{\left(R_{11}{R}_{22}-{\&LeftBracketingBar;R_{12}\&RightBracketingBar;}^2\right)(R_{22}+R_{11}{\&LeftBracketingBar;K\&RightBracketingBar;}^2-R_{12}K-R_{21\stackrel{\_}{K}})}}\end{array}& \left(20\right)\end{array}$

[0069] which are exactly equations 9-11.

[0070] Further embodiments of a multi-channel noise reduction system according to the present invention will now be described in detail. In a D-channel framework wherein D microphone signals, x1(t), . . . , xD(t), record a source s(t) and noise signal, n1(t), . . . , xD(t), a mixing model according to another embodiment of the present invention is preferably defined as follows: 17$\begin{array}{cc}{x}_1\left(t\right)=\sum _{k=0}^{L_1}{a}_k^{1}s\left(t-{\tau }_k^1\right)+n_1\left(t\right)x_D\left(t\right)=\sum _{k=0}^{L_D}{a}_k^{D}s\left(t-{\tau }_k^D\right)+n_D\left(t\right)& \left(21\right)\end{array}$

[0071] where the terms (ak1, τk1) denote the attenuation and delay on the kth path to microphone L. In the frequency domain, the convolutions become multiplications. Furthermore, since we are not interested in balancing the channels, we redefine the source so that the first channel becomes unity:

X 1 (k,w)=S(k,w)+N1(k,w)

X 2 (k,w)=K2(w)S(k,w)+N2(k,w)   (22)

X D (k,w)=KD(w)S(k,w)+ND(k,w)

[0072] wherein k denotes the frame index and w denotes the frequency index. More compactly, the model can be rewritten as:

X=KS+N   (23)

[0073] where X, K, S. and N are D-complex vectors. With this model, the following assumptions are made:

[0074] 1. The transfer function ratios K1 are known;

[0075] 2. S(w) are zero-mean stochastic processes with spectral power ρS(w)=E[|S|2];

[0076] 3. (N1,N2, . . . , ND) is a zero-mean stochastic signal with the following spectral covariance matrix: 18$\begin{array}{cc}{R}_n\left(w\right)=\left[\begin{array}{c}E[{\&LeftBracketingBar;N_1\&RightBracketingBar;}^2,E\left[N_1\stackrel{\_}{N_2}\right],\dots ,E\left[N_1\stackrel{\_}{N_D}\right]\\ E\left[N_2\stackrel{\_}{N_1}\right],E[{\&LeftBracketingBar;N_2\&RightBracketingBar;}^2,\dots ,E\left[N_2\stackrel{\_}{N_D}\right]\\ \dots \\ E\left[N_D\stackrel{\_}{N_1}\right],E\left[N_D\stackrel{\_}{N_2}\right],\dots ,E\left[{\&LeftBracketingBar;N_D\&RightBracketingBar;}^2\right]\end{array}\right];\mathrm{and}& \left(24\right)\end{array}$

[0077] 4. S is independent of n.

[0078] A detailed discussion of methods for estimating K, ΔS and Rn according to embodiments of the invention will be described below.

[0079] In the multi-channel embodiment with D channels, preferably, a linear filter:

A=[A 1 A2 AD]  (25)

[0080] is applied to the measured signals X1, X2, . .. XD. The output of the filter is: 19$\begin{array}{cc}Y=\sum _{l=1}^D{A}_l{X}_l=\mathrm{AKS}+\mathrm{AN}.& \left(26\right)\end{array}$

[0081] The goal is to obtain an estimate of S that contains a small amount of noise. Assume that 0≦ζ1, . . . ,ζD≦1 are constants such that the desired signal is w=S+ζ1N1+ζ2N2+. . . +ζDND. Then the error e=s−w has the variance Re=|AK−1|2ρS+(A−ζ)Rn(A*−ζT) where ζ=[ζ1, . . . , ζM] is a 1×M vector of desired levels of noise. As explained above, it is preferable that the filter achieve a noise distortion level of RT. The D-1 degrees of freedom are used to choose A that minimizes the total distortion. Preferably, the optimization problems becomes:

arg minARe, subject to (A−ζ)Rn(A*−ζT)=RT   (27)

[0082] Assuming Ao denotes an optimal solution, then we validate it by checking whether |AoK|≦1. If not, no processing is performed because the noise level is lower than the threshold and there is no reason to amplify it.

[0083] Therefore: 20$\begin{array}{cc}A=\{\begin{array}{ccc}{A}_o& \mathrm{if}& \&LeftBracketingBar;A_{o}K\&RightBracketingBar;\le 1\\ \left(1,0,\dots ,0\right)& \mathrm{if}& \mathrm{otherwise}.\end{array}& \left(28\right)\end{array}$

[0084] Setting B=A−ζ, and constructing the Lagrangian: L(B,λ)=|BK+ζK−1|2ρS+BRnB*+λ(BRnB*−RT), we obtain the system:

K* (BK+ζK−1)ρS+BRn+λBRn=0

K (K*B*+B*ζT−1)ρS+RnB*+λRnB*=0

BR n B*−R T =0

[0085] Solving for B in the first equation and inserting the expression into the second equation, we obtain with μ=(1+λ)/ρS, the threshold:

RT=| 1−ζK|2K*(μRn+KK*)−1Rn(μRn+KK*)−1K

[0086] Using the Inversion Lemma (see, e.g., S. V. Vaseghi, Advanced Digital Signal Processing and Noise Reduction, John Wiley & sons, 2nd Edition, 2000), the equation in becomes: 21$\begin{array}{cc}\mu =-K^{*}{R}_n^{-1}K\pm \&LeftBracketingBar;1-\zeta K\&RightBracketingBar;\sqrt{\frac{K^{*}{R}_n^{-1}K}{R_T}}.& \left(29\right)\end{array}$

[0087] Replacing in Re, we obtain:

Re=RT+ρS|±{square root}{square root over (RT(K*Rn−1K))}−|1−ζK||2.

[0088] Hence, the optimal solution is the solution with “+” in equation (29). Consequently, the optimizer becomes: 22$\begin{array}{cc}{A}_o=\zeta +\frac{1-\zeta K}{\&LeftBracketingBar;1-\zeta K\&RightBracketingBar;}\sqrt{\frac{R_T}{K^{*}{R}_n^{-1}K}}K*R_n^{-1}.& \left(30\right)\end{array}$

[0089] A more practical form is obtained for ζ1=ζ and ζk=0, k>1. 23$\mathrm{Then}\text{:}$$\begin{array}{cc}{A}_o=\left(\zeta ,0,\dots ,0\right)+\sqrt{\frac{R_T}{K^{*}{R}_n^{-1}K}}K*R_n^{-1}& \left(31\right)\end{array}$

[0090] and

{A0K{=ζ+{square root}{square root over (rT(K*Rn−1K))}.

[0091] The following is a detailed description of other preferred methods for estimating the transfer function ratios K and spectral power densities ΔS and Rn according to the invention. It is assumed that an ideal VAD signal is available. For example, in accordance with the present invention, there are various methods for estimating K that may be implemented: (i) an ideal estimator of K done through a subspace method; (ii) a non-parametric estimator using a gradient algorithm; and (iii) a model-based estimator using a gradient algorithm. The ideal estimator can be thought of as an initialization of an adaptive procedure, whereas the non-parametric and model-based estimators can be used to adapt K blindly.

[0092] Ideal Estimator of K: Assume that a set of measurements are made under quiet conditions with the user speaking, wherein x1(t), . . . , xD(t) denotes such measurements and wherein X1(k,w), . . . , XD(k,w) denote the time-frequency domain transform of such signals. Assuming that the only noise is microphone noise (hence independence among channels) is recorded, the noise spectral power covariance in equation (24) is Rn(w)=σn2(w)ID which turns the measured signal long-term spectral power density (i.e., time-averaged) into:

R x (w)=ρS(w)KK*+σn2(w)ID.   (32)

[0093] This suggest a subspace method to estimate K. Indeed, K is the eigenvector of Rx corresponding to the largest eigenvalue λmax=ρS∥K∥2+σn2. Thus, K is preferably estimated by first computing the long term spectral covariance matrix Rx, and then determining K as the eigenvector corresponding to the largest eigenvalue of Rx.

Adaptive Non-Parametric Estimator of K

[0094] Assuming that the measurements x1 . . . , xD contain signal and noise (equation (21)). Assume further that we have estimates of the noise spectral power Rn, the signal spectral power ΔS, and an estimate of K′ that we want to update. The measured signal (short-time) spectral power Rx(k,w) is:

R x (k,w)=ρS(k,w)KK*+Rn(k,w)   (33)

[0095] We want to update K to K′=K+ΔK constrained by ∥ΔK∥ small, and ΔK=[0Λ]T, where Λ=[ΔK2 . . . ΔKD], which best fits equation (33) in some norm, preferably the Frobenius norm, ∥A∥F2=trace{AA*}. Then the criterion to minimize becomes:

J (X)=tracer{(Rx−Rn−ρS(K+[0Λ]T)(K+[0Λ]T)*)2}  (34)

[0096] The gradient at Λ=0 is: 24$\begin{array}{cc}{\frac{\partial J}{\partial \Lambda }\&RightBracketingBar;}_0=-2{{\rho }_s\left(K^{*}E\right)}_r& \left(35\right)\end{array}$

[0097] where the index r truncates the vector by cutting out the first component: for ν=[ν1ν2 . . . νD], νr=[ν2 . . . νD], and E=Rx−Rn−ρSKK*. Thus the gradient algorithm for K gives the following adaptation rule:

K′=K+[ 0ζ]T, ζ=αρS(K*E)r   (36)

[0098] where 0<α<1 is the learning rate.

Adaptive Model-based Estimator of K

[0099] Another adaptive estimator according to the present invention makes use of a particular mixing model, thus reducing the number of parameters. The simplest but fairly efficient model is a direct path model:

K l (s)=aliwδ1, l≦2   (37)

[0100] In this case, a similar criterion to equation (34) is to be minimized, in particular: 25$\begin{array}{cc}I\left(\mathrm{a2},\dots ,\mathrm{aD},\delta 2,\dots ,\delta D\right)=\sum _w\mathrm{trace}\left\{{\left(R_x-R_n-{\rho }_s{\mathrm{KK}}^{*}\right)}^2\right\}& \left(38\right)\end{array}$

[0101] Note the summation across the frequencies because the same parameters (al,δl)2≦l≦D have to explain all the frequencies. The gradient of I evaluated on the current estimate (al,δl)2≦l≦D is: 26$\begin{array}{cc}\frac{\partial I}{\partial a_l}=-4\sum _w{\rho }_s·\mathrm{real}\left(K*{\mathrm{Ev}}_l\right)& \left(39\right)\\ \frac{\partial I}{\partial a_l}=-2a_l\sum _{w}w{\rho }_s·\mathrm{imag}\left(K*{\mathrm{Ev}}_l\right)& \left(40\right)\end{array}$

[0102] where E=Rx−Rn−ρSKK* and νl the D-vector of zeros everywhere except on the lth entry where it is eiwδl, νl=[0 . . . 0eiwδl0 . . . 0]T. Then, the preferred updating rule is given by: 27$\begin{array}{cc}{a}_l^{\prime }=a_l-\alpha \frac{\partial I}{\partial a_l}& \left(41\right)\\ {\delta }_l^{\prime }={\delta }_l-\alpha \frac{\partial I}{\partial {\delta }_l}& \left(42\right)\end{array}$

[0103] where 0]∀]1;

[0104] Estimation of Spectral Power Densities

[0105] In accordance with another embodiment of the present invention, the estimation of Rn is computed based on the VAD signal as follows: 28$\begin{array}{cc}{R}_n^{\mathrm{new}}=\{\begin{array}{cc}\left(1-\beta \right)R_n^{\mathrm{old}}+\beta {\mathrm{XX}}^{*}& \mathrm{if}\mathrm{voice}\mathrm{not}\mathrm{present}\\ R_n^{\mathrm{old}}& \mathrm{if}\mathrm{otherwise}\end{array}& \left(43\right)\end{array}$

[0106] where is a learning curve (equation 43 is similar to equation (6a)).

[0107] The measured signal spectral power Rx is then estimated from the measured input signals as follows:

R x new =(1−α)Rxadd+αXX*

[0108] where is a learning rate, preferably equal to 0.9.

[0109] Preferably, the signal spectral power, ΔS, is estimated through spectral subtraction, which is sufficient for psychoacoustic filtering. Indeed, the signal spectral power, ΔS, is not used directly in the signal estimation (e.g., Y in equation (26)), but rather in the threshold RT evaluation and K updating rule. As for the K update, experiments have shown that a simple model, such as the adaptive model-based estimator of equation (37) yields good results, where ΔS plays a relatively less significant role. Accordingly, according to another embodiment of the present invention, the spectral signal power is estimated by: 29$\begin{array}{cc}{\rho }_s=\{\begin{array}{cc}{R}_{x;11}-R_{n;11}& \mathrm{if}{R}_{x;11}>{\beta }_{\mathrm{ss}}{R}_{n;11}\\ \left({\beta }_{\mathrm{ss}}-1\right)R_{n;11}& \mathrm{if}\mathrm{otherwise}\end{array}& \left(44\right)\end{array}$

[0110] where ∃SS>1 is a floor-dependent constant. By using ∃SS, even when voice is not present, we still determine the signal spectral power to avoid clipping of the voice, for example. In a preferred embodiment, ∃SS=1.1.

EXEMPLARY EMBODIMENT

[0111] To assess the performance of a two-channel framework using the algorithms described herein, stereo recordings for two microphones were captured in noisy car environment (−6.5 dB overall SNR on average), at a sampling frequency of 8 HHz. Exemplary waveforms for a two-channel system are shown in FIGS. 3a, 3b and 3c. FIG. 3a illustrates the first channel waveform and FIG. 3b illustrates the second channel waveform with the VAD decision superimposed thereon. FIG. 3c illustrates the filter output.

[0112] For the experiment, a time-frequency analysis was performed by using a Hamming window of size 512 samples with 50% overlap, and the synthesis by overlap-add procedure. Rx was estimated by a first-order filter with learning rate =0.9 (equation (43a)). In addition, the following parameters were applied: ∃SS=1.1 (equation (44)); ∃=0.2 (equation (43)); .=0.001 (equation (30)); and ∀=0.01 (equations 36, or 42).

[0113] The two-channel psychoacoustic noise reduction algorithm was applied on a set of two voices (one male, one female) in various combinations with noise segments from two noise files.

[0114] Two-channel experiments show considerably lower distortion on average as compared to the single-channel system (as in Gustafsson et al., idem), while still reducing noise. Informal listening tests have confirmed these results. The two-channel system output signal had little speech distortion and noise artifacts as compared to the mono system. In addition, the blind identification algorithms performed fairly well with no noticeable extra degradation of the signal.

[0115] In conclusion, the present invention provides a multi-channel speech enhancement/noise reduction system and method based on psychoacoustic masking principles. The optimality criterion satisfies the psychoacoustic masking principle and minimizes the total signal distortion. The experimental results obtained in a dual channel framework on very noisy data in a car environment illustrate the capabilities and advantages of the multi-channel psychoacoustic system with respect to SNR gain and artifacts.

[0116] Although illustrative embodiments of the present invention have been described herein with reference to the accompanying drawings, it is to be understood that the invention is not limited to those precise embodiments, and that various other changes and modifications may be affected therein by one skilled in the art without departing from the scope or spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as defined by the appended claims.

Claims

1. A method for filtering noise from an audio signal, comprising the steps of: obtaining a multi-channel recording of an audio signal; determining a psychoacoustic masking threshold for the audio signal; determining a filter for filtering noise from the audio signal using the multi-channel recording, wherein the filter is determined using the psychoacoustic masking threshold; and filtering the multi-channel recording using the filter to generate an enhanced audio signal.

2. The method of claim 1, further comprising the steps of determining a calibration parameter for the input channels, wherein the calibration parameter comprises a ratio of the impulse response of different channels, and wherein the calibration parameter is used to determine the filter.

3. The method of claim 2, wherein the calibration parameter is determined by processing a speech signal recorded in the different channels under quiet conditions.

4. The method of claim 2, wherein the step of estimating the calibration parameter comprises processing channel noise recorded in the different channels to determine a long-term spectral covariance matrix, and determining an eigenvector of the long-term spectral covariance matrix corresponding to a desired eigenvalue.

5. The method of claim 2, wherein the step of determining the calibration parameter is performed using an adaptive process.

6. The method of claim 5, wherein the adaptive process comprises a blind adaptive process.

7. The method of claim 57 wherein the adaptive process comprises a non-parametric estimation process using a gradient algorithm.

8. The method of claim 5, wherein the adaptive process comprises a model-based estimation process using a gradient algorithm.

9. The method of claim 2, wherein the step of determining the calibration parameter comprises setting a default calibration parameter.

10. The method of claim 1, further comprising the steps of: determining a noise spectral power matrix using the multi-channel recording; and determining the signal spectral power using the noise spectral power matrix, wherein the signal spectral power is used to determine the masking threshold, and wherein the noise spectral power matrix is used to determine the filter.

11. The method of claim 10, further comprising the steps of: detecting speech activity in the audio signal; and updating the noise spectral power matrix at times when speech activity is not detected in the audio signal.

12. The method of claim 1 wherein the filter comprises a linear filter.

13. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform method steps for filtering noise from an audio signal, the method steps comprising: obtaining a multi-channel recording of an audio signal; determining a psychoacoustic masking threshold for the audio signal; determining a filter for filtering noise from the audio signal using the multi-channel recording, wherein the filter is determined using the psychoacoustic masking threshold; and filtering the multi-channel recording using the filter to generate an enhanced audio signal.

14. The program storage device of claim 13, further comprising instructions for performing the steps of determining a calibration parameter for the input channels, wherein the calibration parameter comprises a ratio of the impulse response of different channels, and wherein the calibration parameter is used to determine the filter.

15. The program storage device of claim 14, wherein the calibration parameter is determined by processing a speech signal recorded in the different channels under quiet conditions.

16. The program storage device of claim 14, wherein the instructions for determining the calibration parameter comprise instructions for performing the steps of processing channel noise recorded in the different channels to determine a long-term spectral covariance matrix, and determining an eigenvector of the long-term spectral covariance matrix corresponding to a desired eigenvalue.

17. The program storage device of claim 14, wherein the instructions for determining the calibration parameter comprise instructions for determining the calibration parameter using an adaptive process.

18. The program storage device of claim 17, wherein the adaptive process comprises a blind adaptive process.

19. The program storage device of claim 17, wherein the adaptive process comprises a non-parametric estimation process using a gradient algorithm.

20. The program storage device of claim 17, wherein the adaptive process comprises a model-based estimation process using a gradient algorithm.

21. The program storage device of claim 14, wherein the instructions for determining the calibration parameter comprise instructions for setting a default calibration parameter.

22. The program storage device of claim 13, further comprising instructions for performing the steps of: determining a noise spectral power matrix using the multi-channel recording; and determining the signal spectral power using the noise spectral power matrix, wherein the signal spectral power is used to determine the masking threshold, and wherein the noise spectral power matrix is used to determine the filter.

23. The program storage device of claim 22, further comprising instructions for performing the steps of: detecting speech activity in the audio signal; and updating the noise spectral power matrix at times when speech activity is not detected in the audio signal.

24. The program storage device of claim 13, wherein the filter comprises a linear filter.

25. A system for reducing noise of an audio signal, comprising: an audio capture system comprising a microphone array, for capturing and recording an audio signal in each input channel of the microphone array; and a front-end speech processor that determines a psychoacoustic masking threshold of the audio signal and generates an enhanced speech signal of the audio signal by filtering noise from the speech signal using the psychoacoustic masking threshold.

26. The system of claim 25, wherein the front-end speech processor comprises: a sampling module for generating a time-frequency representation of an audio signal in each channel; a calibration module for determining a calibration parameter, the calibration parameter comprising a ratio of the transfer functions between different channels; a voice activity detection module for detecting a speech signal in the input audio signal; a filter module for determining filter parameters using the psychoacoustic masking threshold and the calibration parameter; a filter for filtering the multi-channel recording using the filter parameters to generate an enhanced signal; and a conversion module for converting the enhanced signal into a time domain representation.

27. The system of claim 26, further comprising: a noise spectral power module for determining a noise spectral power matrix using the multi-channel recording; and a signal spectral power module for determining the signal spectral power using the noise spectral power matrix, wherein the signal spectral power is used to determine the masking threshold, and wherein the noise spectral power matrix is used to determine the filter parameters.