The present invention relates to a technique that uses several microphones to perform clear sound collection of a sound source signal coming from a target direction.
Firstly, a framework of basic signal processing will be described.
It is assumed that an array formed of M microphones is used. M is an integer equal to or larger than 2. For example, it is assumed that M is on the order of 2 to 4. M may be on the order of 100. In an observation signal Xm(ω, τ) (m=1, 2, . . . , M) at a frequency ω and a frame time τ, one target sound S0(ω, τ) and K interference noises Sk(ω, τ) (k=1, 2, . . . , K) that are coherent and non-stationary and an incoherent stationary noise Nm(ω, τ) are included. K is to be a predetermined positive integer. m is the number for each microphone, and the observation signal Xm(ω, τ) is a signal obtained by converting a time domain signal collected using the microphone m into a frequency domain.
A target sound is a sound coming from a predetermined target area. A target area is an area in which a sound source desired to be collected is included. The number of the sound sources desired to be collected and the position of the sound source desired to be collected in the target area may be unknown. For example, it is assumed that an area in which six speakers and three microphones are arranged is divided into three areas (an area 1, an area 2, and an area 3), as illustrated in
The target sound may contain a reflected sound from a sound source outside the target area. For example, when the target area is the area 1, among sounds generated from sound sources included in the area 2 and the area 3, a sound coming to a microphone in the direction of the area 1 due to reflection may be contained in the target sound.
The target area may be an area within a predetermined distance from the microphone. In other words, the target area may be an area including a finite area. Furthermore, a plurality of target areas may be present.
An area including a sound source generating a noise is also referred to as a noise area. In the example in
When a transfer characteristic from the m-th microphone to a target sound S0(ω, τ) is described as Am0(ω) and a transfer characteristic from the m-th microphone to a k-th interference noise is described as Amk(ω), the observation signal Xm(ω, τ) is modeled as below.
When the number of microphones is small, that is, M<K, for example, a framework in which a minimum variance distortionless response (MVDR) beamforming approach and a post-filter are combined is thought to be effective for suppressing noises (see Non-patent Literature 1, for example).
With x being an optional vector or matrix, xT represents a transpose of x and xH represents a complex conjugate transpose of x. h0(ω)=[H0,1(ω), . . . , H0,M(ω)]T is an array manifold vector in the target sound direction. The array manifold vector is a transfer characteristic H0,m(ω) from the sound source to the microphone, the transfer characteristic H0,m(ω) represented by a vector h0(ω). The transfer characteristic H0,m(ω) from the sound source to the microphone includes a transfer characteristic with which only a direct sound that can be theoretically calculated from the sound source and the microphone position is assumed, a transfer characteristic actually measured, and a transfer characteristic estimated by calculator simulation such as a mirror method and a finite element method. When it is assumed that source signals are uncorrelated with each other, a spatial correlation matrix R(ω) can be modeled as below.
hk(ω) here is an array manifold vector of the k-th interference noise. An output signal Y0(ω, τ) of beamforming is obtained with the formula below.
Y
0(ω,τ)=w0H(ω)x(ω,τ) (4)
x(ω, τ)=[X1(ω, τ), . . . , XM(ω, τ)]T holds. To suppress a noise signal included in Y0(ω, τ), a post-filter G(ω, τ) is multiplied.
Z(ω,τ)=G(ω,τ)Y0(ω,τ) (5)
Finally, Z(ω, τ) is subjected to inverse fast Fourier transforming (IFFT), whereby the output signal is obtained.
Next, a post-filter designing method based on Non-patent Literature 2 will be described.
Non-patent Literature 2 proposes a method of designing a post-filter based on a power spectrum density (PSD) of each area estimated using multiple beamforming (see Non-patent Literature 2, for example). Hereinafter, this method is referred to as an LPSD method (local PSD-based post-filter design).
When the post-filter is designed based on a Wiener method, G(ω, τ) is calculated as below.
φS(ω, τ) represents the power spectrum density of the target area and φN(ω, τ) represents the power spectrum density of the noise area. The power spectrum density of a certain area means the power spectrum density of a sound coming from that area. More specifically, the power spectrum density of a target area is the power spectrum density of a sound coming from the target area, for example, and the power spectrum density of a noise area is the power spectrum density of a sound coming from the noise area. Although there are various methods of estimating φS(ω, τ) and φN(ω, τ) from Xm(ω, τ), the LPSD method is used because it is assumed that the observation signal contains an interference noise.
With the LPSD method, it is assumed that the observation signal contains a target sound and an interference noise, which are sparse in the time-frequency domain. To analyze the power spectrum density of each area positioned in various directions, L+1 beamforming filters wu(ω) (u=0, 1, . . . , L) are designed. The relation among a sensitivity |Duk(ω)|2 in the direction of the k-th area of a filter wu(ω), the power |Yu(ω, τ)|2 of the u-th output signal, and the power spectrum density |Sk(ω, τ)|2 of each area can be modeled as below. For |Duk(ω)|2, |Duk(ω)|2=|wuH(ω)hk(ω)|2 holds, for example. As |Duk(ω)|2, a measured value may be used.
The index of each symbol is here omitted. More specifically, Yu=Yu(ω, τ), Duk=Duk(ω), and Su=Su(ω, τ) hold. Furthermore, φY(ω, τ)=[|Y0(ω, τ)|2, |Y1(ω, τ)|2, . . . , |YL(ω, τ)|2]T and φS(ω, τ)=[|S0(ω, τ)|2, |S1(ω, τ)|2, . . . , |SK(ω, τ)|2]T hold.
For example, the power spectrum density of each area is calculated by solving the inverse problem of formula (7).
{circumflex over (Φ)}S(ω,τ)=D+(ω)ΦY(ω,τ) (8)
With b being an optional matrix, b+ represents a pseudo inverse matrix calculation for b. A local PSD estimation unit 11 uses the observation signal Xm(ω, τ) (m=1, 2, . . . , M) as an input to output a local power spectrum density ̂φS(ω, τ) defined by formula (8), for example. “̂” indicates that the density is from estimation.
Local indicates an area. In the example in
A target area/noise area PSD estimation unit 12 uses the local power spectrum density ̂φS(ω, τ) estimated based on formula (8) for each frequency ω and frame τ as an input to calculate ̂φS(ω, τ) and ̂φN(ω, τ) which are defined by the formula below.
Finally, a Wiener gain calculation unit 13 uses ̂φS(ω, τ) and ̂φN(ω, τ) as an input to calculate the post-filter G(ω, τ) defined by formula (6) and outputs the calculated post-filter G(ω, τ). Specifically, the Wiener gain calculation unit 13 inputs ̂φS(ω, τ) and ̂φN(ω, τ) as φS(ω, τ) and φN(ω, τ) of formula (6) to calculate G(ω, τ) and outputs the calculated G(ω, τ).
Two main advantages of the LPSD method are described below. (i) In a power spectrum domain, the relation between an output of beamforming and each sound source is formulated, whereby flexibility of control surpassing the number of microphones can be achieved and noises thus can be effectively suppressed. (ii) By calculating in advance L beamforming filters wu(ω) (u=0, 1, . . . , L) and D(ω) of formula (7), the merit of (i) can be implemented with low-complexity.
With an LPSD method, a problem has been formulated assuming that a target sound and an interference noise are mixed. However, in an actual problem, not only a coherent interference noise but also a stationary noise being highly incoherent (such as air-conditioning noise and microphone's internal noise) is often mixed. In such a case, estimation errors of φS(ω, τ) and φN(ω, τ) become large and the noise suppressing performance is lowered in some cases.
An object of the present invention is to provide a signal processing apparatus, a method, and a program whose noise suppressing performances are more improved than conventional ones.
A signal processing apparatus according to an aspect of the present invention includes a local PSD estimation unit, a target area/noise area PSD estimation unit, a first component extraction unit, a second component extraction unit, and a various noise responding gain calculation unit. The local PSD estimation unit estimates each of a local power spectrum density of a target area and that of at least one noise area different from the target area based on an observation signal of a frequency domain obtained from a signal collected with M microphones forming a microphone array. The target area/noise area PSD estimation unit estimates a power spectrum density ̂φS(ω, τ) of the target area and a power spectrum density ̂φN(ω, τ) of the noise area based on the estimated local power spectrum density, w being a frequency and r being an index of a frame. The first component extraction unit extracts a non-stationary component ̂φS(A)(ω, τ) derived from a sound coming from the target area and a stationary component ̂φS(B)(ω, τ) derived from an incoherent noise from the power spectrum density ̂φS(ω, τ) of the target area. The second component extraction unit extracts a non-stationary component ̂φN(A)(ω, τ) derived from an interference noise from a power spectrum density ̂φN(ω, τ) of the noise area. The various noise responding gain calculation unit uses at least the non-stationary component ̂φS(A)(ω, τ) derived from a sound coming from the target area, the stationary component ̂φS(B)(ω, τ) derived from an incoherent noise, and the non-stationary component ̂φN(A)(ω, τ) derived from an interference noise to calculate a post-filter {tilde over ( )}G(ω, τ) emphasizing the non-stationary component of the sound coming from the target area.
The present invention can improve the noise suppressing performance compared with a conventional case.
With a signal processing apparatus and a method described below, an LPSD method is expanded to robustly estimate a post-filter with respect to various noise environments. Specifically, a power spectrum density is estimated in a divided manner for each noise type, whereby an estimation error of the ratio of the power of a target sound to that of other noise is reduced.
The signal processing apparatus includes, as illustrated in
Each step of signal processing implemented by this signal processing apparatus, for example, is illustrated in
Details of an embodiment of the signal processing apparatus and the method will be described below. It should be noted that the basic signal processing framework, definition of terms, and the like are similar to those described in [BACKGROUND ART]. A repeated explanation thereof thus will be omitted.
<Local PSD Estimation Unit 11>
The local PSD estimation unit 11 is similar to a conventional local PSD estimation unit 11.
More specifically, the local PSD estimation unit 11 estimates a local power spectrum density ̂φS(ω, τ) of each of a target area and a noise area based on an observation signal Xm(ω, τ) (m=1, 2, . . . , M) of a frequency domain obtained from a signal collected with M microphones forming a microphone array (Step S1). ω is a frequency and τ is an index of a frame. M is an integer equal to or larger than 2. For example, M is on the order of 2 to 4. M may be on the order of 100.
The estimated local power spectrum density ̂φS(ω, τ) is output to the target area/noise area PSD estimation unit 12.
Examples of specific processing of estimating the local power spectrum density are similar to those described in [BACKGROUND ART]. The explanation thereof thus will be omitted here.
It should be noted that a beamforming filters wu(ω) and a sensitivity |Duk(ω)|2 are to be set in advance, prior to the processing performed by the local PSD estimation unit 11. Furthermore, when the direction of the target area is changed to some degrees, the local PSD estimation unit 11 may prepare a plurality of filter sets and select the filter with which the power is the maximum.
It should be noted that the local PSD estimation unit 11 may estimate the local power spectrum density ̂φS(ω, τ) based not on Yu(ω, τ) (u=0, 1, . . . , L) obtained by beamforming, but on Yu(ω, τ) (u=0, 1, . . . , L) collected with microphones, each one of which has directionality in the direction of each area.
<Target Area/Noise Area PSD Estimation Unit 12>
The target area/noise area PSD estimation unit 12 is similar to a conventional target area/noise area PSD estimation unit 12.
More specifically, the target area/noise area PSD estimation unit 12 estimates the power spectrum density ̂φS(ω, τ) of the target area and the power spectrum density ̂φN(ω, τ) of the noise area based on the estimated local power spectrum density (Step S2).
The estimated power spectrum density ̂φS(ω, τ) of the target area is output to the first component extraction unit 14. The estimated power spectrum density ̂φN(ω, τ) of the noise area is output to the second component extraction unit 15.
Examples of specific processing of estimating the power spectrum density ̂φS(ω, τ) of the target area and the power spectrum density ̂φN(ω, τ) of the noise area are similar to those described in [BACKGROUND ART]. The explanation thereof thus will be omitted here.
<First Component Extraction Unit 14>
For example, in ̂φS(ω, τ) defined by formula (9), a non-stationary component ̂φS(A)(ω, τ) derived from a sound coming from the target area and a stationary component ̂φS(B)(ω, τ) derived from an incoherent noise are included. In this case, the stationary component is a component the temporal change of which is small and the non-stationary component is a component the temporal change of which is large.
In this case, the noise includes two types of noises, an interference noise and an incoherent noise. The interference noise is a noise emitted from a noise sound source arranged in the noise area. The incoherent noise is not a noise emitted from the target area or the noise area, but a noise emitted from a place other than these areas and being regularly present.
The first component extraction unit 14 extracts the non-stationary component ̂φS(A)(ω, τ) derived from a sound coming from the target area and the stationary component ̂φS(B)(ω, τ) derived from an incoherent noise from the power spectrum density ̂φS(ω, τ) of the target area through smoothing processing (Step S3). For example, the smoothing processing is implemented by processing of exponential moving average, time average, and weighted average as in formulas (11) and (12).
The extracted non-stationary component ̂φS(A)(ω, τ) derived from a sound coming from the target area and stationary component ̂φS(B)(ω, τ) derived from an incoherent noise are output to the various noise responding gain calculation unit 16.
For example, the first component extraction unit 14 performs processing of exponential moving average as in formulas (11) and (12), thereby calculating ̂φS(B)(ω, τ) from ̂φS(ω, τ).
αS here is a smoothing coefficient and a predetermined positive actual number. For example, 0<αS<1 holds. Furthermore, with αS=time length/time constant of a frame, as may be set such that the time constant is on the order of 150 ms. YS is a set of indexes of frames for a predetermined interval. For example, YS is set such that the predetermined interval is on the order of 3 to 4 seconds. min is a function that outputs the minimum value.
̂φS(B)(ω, τ) thus is a component obtained by smoothing ̂φS(ω, τ) by formulas (11) and (12), for example. More specifically, ̂φS(B)(ω, τ) is the minimum value in a predetermined time interval of a value obtained by smoothing ̂φS(ω, τ) by formula (11), for example.
The first component extraction unit 14 subtracts ̂φS(B)(ω, τ) from ̂φS(ω, τ), thereby calculating ̂φS(A)(ω, τ), as in formula (13).
{circumflex over (φ)}S(A)(ω,τ)={circumflex over (φ)}S(ω,τ)−βS(ω){circumflex over (φ)}S(B)(ω,τ) (13)
βS(ω) here is a weighted coefficient and a predetermined positive actual number. βS(ω) is set to an actual number on the order of 1 to 3, for example.
φS(A)(ω, τ) thus is a component obtained by removing ̂φS(B)(ω, τ) from ̂φS(ω, τ).
It should be noted that ̂φS(A)(ω, τ) may be subjected to flooring processing such that a condition of ̂φS(A)(ω, τ)≧0 is satisfied. This flooring processing is performed by the first component extraction unit 14, for example.
<Second Component Extraction Unit 15>
For example, in ̂φN(ω, τ) defined by formula (10), a non-stationary component ̂φN(A)(ω, τ) derived from an interference noise and a stationary component ̂φN(B)(ω, τ) derived from an incoherent noise are included.
The second component extraction unit 15 extracts the non-stationary component ̂φN(A)(ω, τ) derived from an interference noise and the stationary component ̂φN(B)(ω, τ) derived from an incoherent noise from the power spectrum density ̂φN(ω, τ) of the noise area through smoothing processing (Step S4). For example, the smoothing processing is implemented by processing of exponential moving average, time average, and weighted average as in formulas (14) and (15).
The extracted non-stationary component ̂φN(A)(ω, τ) derived from an interference noise and stationary component ̂φN(B)(ω, τ) derived from an incoherent noise are output to the various noise responding gain calculation unit 16.
For example, the second component extraction unit 15 performs processing of exponential moving average as in formulas (14) and (15), thereby calculating ̂φN(B)(ω, τ) from ̂φN(ω, τ).
αN here is a smoothing coefficient and a predetermined positive actual number. For example, 0<αN<1 holds. Furthermore, with αN=time length/time constant of a frame, αN may be set such that the time constant is on the order of 150 ms. YN is a set of indexes of frames for a predetermined interval. For example, YN is set such that the predetermined interval is on the order of 3 to 4 seconds.
φN(B)(ω, τ) thus is a component obtained by smoothing ̂φN(ω, τ) by formulas (14) and (15), for example. More specifically, ̂φN(B)(ω, τ) is the minimum value in a predetermined time interval of a value obtained by smoothing ̂φN(ω, τ) by formula (14), for example.
The second component extraction unit 15 subtracts ̂φN(B)(ω, τ) from ̂φN(ω, τ), thereby calculating ̂φN(A)(ω, τ), as in formula (16).
{circumflex over (φ)}N(A)(ω,τ)={circumflex over (φ)}N(ω,τ)−βN(ω){circumflex over (φ)}N(B)(ω,τ) (16)
βN(ω) here is a weighted coefficient and a predetermined positive actual number. βN(ω) is set to an actual number on the order of 1 to 3, for example.
φN(A)(ω, τ) thus is a component obtained by removing ̂φN(B)(ω, τ) from ̂φN(ω, τ).
It should be noted that ̂φN(A)(ω, τ) may be subjected to flooring processing such that a condition of ̂φN(A)(ω, τ)≧0 is satisfied. This flooring processing is performed by the second component extraction unit 15, for example.
αN may be the same as αS and may be different from αS. YN may be the same as YS and may be different from YS. βN(ω) may be the same as βS(ω) and may be different from βS(ω).
It should be noted that when ̂φN(B)(ω, τ) is not used in the various noise responding gain calculation unit 16, the second component extraction unit 15 does not have to obtain ̂φN(B)(ω, τ). In other words, the second component extraction unit 15 may obtain only ̂φN(A)(ω, τ) from ̂φN(ω, τ) in this case.
<Various Noise Responding Gain Calculation Unit 16>
The various noise responding gain calculation unit 16 uses at least the non-stationary component ̂φS(A)(ω, τ) derived from a sound coming from the target area, the stationary component ̂φS(B)(ω, τ) derived from an incoherent noise, and the non-stationary component ̂φN(A)(ω, τ) derived from an interference noise to calculate a post-filter {tilde over ( )}G(ω, τ) emphasizing the non-stationary component of the sound coming from the target area (Step S5).
The calculated post-filter {tilde over ( )}G(ω, τ) is output to the time frequency averaging unit 17.
Because power spectrum density estimation is performed for each noise type (in other words, for each of the noise types, incoherent noise and coherent noise), the various noise responding gain calculation unit 16 calculates the post-filter {tilde over ( )} G(ω, τ) defined by formula (17) below, for example.
When there is a difference between the behavior of the value of ̂φS(B)(ω, τ) and that of the value of ̂φN(B)(ω, τ) and the assumption of the incoherence has been destroyed, the various noise responding gain calculation unit 16 may calculate the post-filter {tilde over ( )}G(ω, τ) defined by formula (18) below.
<Time Frequency Averaging Unit 17>
The time frequency averaging unit 17 performs smoothing processing in at least one of the time direction and the frequency direction with respect to the post-filter {tilde over ( )}G(ω, τ) (Step S6).
The post-filter {tilde over ( )}G(ω, τ) subjected to the smoothing processing is output to the gain shaping unit 18.
When the smoothing processing is performed in the time direction, with τ0 and τ1 being integers equal to or larger than 0, the time frequency averaging unit 17 may perform additional average with respect to {tilde over ( )}G(ω, τ−τ0), . . . , {tilde over ( )}G(ω, τ+τ1) being a post-filter in the vicinity of the post-filter {tilde over ( )}G(ω, τ) in the time direction, for example. The time frequency averaging unit 17 may perform weighted addition with respect to {tilde over ( )}G(ω, τ−τ0), . . . , {tilde over ( )}G(ω, τ+τ1).
Furthermore, when the smoothing processing is performed in the frequency direction, with ω0 and ω1 being actual numbers equal to or larger than 0, the time frequency averaging unit 17 may perform additional average with respect to {tilde over ( )}G(ω−ω0, τ), . . . , {tilde over ( )}G(ω+ω1, τ) being a post-filter in the vicinity of the post-filter {tilde over ( )}G(ω, τ) in the frequency direction, for example. The time frequency averaging unit 17 may perform weighted addition with respect to {tilde over ( )} G(ω−ω0, τ), . . . , {tilde over ( )}G(ω+ω1, τ).
<Gain Shaping Unit 18>
The gain shaping unit 18 performs gain shaping with respect to the post-filter {tilde over ( )}G(ω, τ) subjected to the smoothing processing, thereby generating the post-filter G(ω, τ) (Step S7). The gain shaping unit 18 generates the post-filter G(ω, τ) defined by formula (19) below, for example.
G(ω,τ)=γ({tilde over (G)}(ω,τ)−0.5)+0.5 (19)
γ here is a weighted coefficient and a positive actual number. γ may be set to an actual number on the order of 1 to 1.3, for example.
The gain shaping unit 18 may perform flooring processing with respect to the post-filter G(ω, τ) such that A≦G(ω, τ)≦1 is satisfied. A is an actual number from 0 to 0.3 and normally on the order of 0.1. When G(ω, τ) is larger than 1, too much emphasis may be caused. When G(ω, τ) is too small, a musical noise may be generated. With appropriate flooring processing performed, the emphasis and generation of a musical noise can be prevented.
A function f the domain and the range of which are actual numbers is considered. The function f is a non-decreasing function, for example. Gain shaping means an operation for obtaining an output value when {tilde over ( )}G(ω, τ) before gain shaping is input to the function f. In other words, an output value when {tilde over ( )}G(ω, τ) is input to the function f is G(ω, τ). An example of the function f is formula (19). With the function f in accordance with formula (19), f(x)=γ(x−0.5)+0.5 holds.
Another example of other function f will be described with reference to
The graph of the function f is not limited to that illustrated in
According to the above-described signal processing apparatus and method, a post-filter for robustly suppressing noises can be designed with respect to an environment in which noises having various properties are present. Furthermore, such a post-filter can be designed with processing with real-time property.
With the LPSD method as a conventional method, an experiment for verifying the effect of the proposed method has been performed. As illustrated in
Under these conditions, the noise suppressing performance has been evaluated through spectral distortion (SD) defined by the formula below
Ψ and |Ψ| here represent a set of indexes of the frame and the total number thereof, respectively. Ω and |Ω| represent an index of a frequency bin and the total number thereof. The smaller the SD value, the higher the noise suppressing performance. The SD is calculated with respect to 650 sentences of speech of a man and a woman to be 14.0 with the conventional method and 11.5 with the proposed method. This indicates that the SD is reduced. Especially, the suppressing effect is increased with respect to the background noises outside the speech section.
Processing performed by the time frequency averaging unit 17 and the gain shaping unit 18 is performed to suppress what is called musical noises. The processing performed by the time frequency averaging unit 17 and the gain shaping unit 18 does not have to be performed.
Calculation of ̂φS(B)(ω, τ) and ̂φS(A)(ω, τ) through processing of exponential moving average is an example of the processing performed by the first component extraction unit 14. The first component extraction unit 14 may extract ̂φS(B)(ω, τ) and ̂φS(A)(ω, τ) through other processing.
Similarly, the calculation of ̂φN(B)(ω, τ) and ̂φN(A)(ω, τ) through processing of exponential moving average is an example of the processing performed by the second component extraction unit 15. The second component extraction unit 15 may extract ̂φN(B)(ω, τ) and ̂φN(A)(ω, τ) through other processing.
The processing explained with respect to the signal processing apparatus and method described above may be performed not only in time series in accordance with the described order but also in parallel or individually in accordance with the processing capacity of the apparatus performing the processing or the need.
Furthermore, when each unit in the signal processing apparatus is implemented by a computer, the processing content of the function that has to be included in each unit in the signal processing apparatus is written in a program. With this program executed on the computer, the unit is implemented on the computer.
This program with the processing content written thereinto can be stored in a computer-readable recording medium. Examples of such a computer-readable recording medium include a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory, and any type of computer-readable recording medium is acceptable.
Furthermore, it may be configured such that each processing means is implemented with a predetermined program executed on the computer, and at least part of the processing contents thereof may be implemented in a hardware manner.
Needless to say, modifications also can be added as appropriate within the scope of the present invention.
Voice recognition has come to be generally used as a command input to a smartphone. In a noisy environment such as in a vehicle or in a factory, it is conceivable that there is a high demand for operating the device in a hands-free manner or making a call to a remote area.
The present invention can be utilized in such a case, for example.
Number | Date | Country | Kind |
---|---|---|---|
2014-037820 | Feb 2014 | JP | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/JP2015/055442 | 2/25/2015 | WO | 00 |