Patent Application
20120121100
This application relates generally to noise detection and suppression and, more particularly to, wind noise detection and suppression.
A speech signal picked up by a microphone can be corrupted by acoustic noise present in the environment surrounding the microphone as well as by certain system-introduced noise, such as noise introduced by quantization and channel interference. If no attempt is made to mitigate the impact of the noise, the corruption of the speech signal will result in a degradation of its perceived quality and intelligibility when played back to a listener. The corruption of the speech signal can also adversely impact the performance of speech coding and recognition algorithms.
One additional source of noise that can corrupt the speech signal picked up by the microphone is wind. Wind causes turbulence in air flow and, if this turbulence impacts the microphone, it can result in the microphone picking up sound referred to as “wind noise.” In general, wind noise is bursty in nature and can last from a few milliseconds up to a few hundred milliseconds or more. Because wind noise is impulsive and can exceed the nominal amplitude of the speech signal, the presence of such noise will degrade the perceived quality and intelligibility of the speech signal. Furthermore, because wind noise is non-stationary in nature, it is typically not attenuated by noise suppression schemes conventionally used to suppress acoustic noise or system-introduced noise.
Therefore, what is needed is a method and apparatus that can effectively detect and suppress wind noise.
The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the present invention and, together with the description, further serve to explain the principles of the invention and to enable a person skilled in the pertinent art to make and use the invention.
The present invention will be described with reference to the accompanying drawings. The drawing in which an element first appears is typically indicated by the leftmost digit(s) in the corresponding reference number.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention, including structures, systems, and methods, may be practiced without these specific details. The description and representation herein are the common means used by those experienced or skilled in the art to most effectively convey the substance of their work to others skilled in the art. In other instances, well-known methods, procedures, components, and circuitry have not been described in detail to avoid unnecessarily obscuring aspects of the invention.
References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
Unlike sound based pressure waves that go everywhere, air turbulence caused by wind is usually a fairly local event. Therefore, in a system that utilizes two or more spatially separated microphones to pick up sound signals (e.g., speech), wind noise picked up by one of the microphones often will not be picked up (or at least not to the same extent) by the other microphone(s).
Described below are methods and apparatuses that utilize this fact and others to effectively detect and suppress wind noise using multiple microphones that are spatially separated. Before describing particular aspects of these methods and apparatuses, the discussion below begins by providing an example communication device and multi-microphone noise suppression system in which embodiments of the present invention can be implemented.
As shown in
By positioning primary microphone 104 so that it is closer to the user's mouth than reference microphone 106 during regular use, a magnitude of the user's speech that is detected by primary microphone 104 is likely to be greater than a magnitude of the user's speech that is detected by reference microphone 106. This can be exploited to effectively suppress acoustic background noise as will be described further below in regard to
In addition, because the two microphones 104 and 106 are spatially separated, wind noise picked up by one of the two microphones often will not be picked up (or at least not to the same extent) by the other microphone. This fact can be exploited to detect and suppress wind noise.
It should be noted that primary microphone 104 and reference microphone 106 are shown to be positioned on the respective front and back portions of wireless communication device 102 for illustrative purposes only and is not intended to be limiting. Persons skilled in the relevant art(s) will recognize that primary microphone 104 and reference microphone 106 can be positioned in any suitable locations on wireless communication device 102.
It should be further noted that a single reference microphone 106 is shown in
Moreover, primary microphone 104 and reference microphone 106 are respectively shown in
Referring now to
Ignoring the operational details of wind noise detection and suppression module 305 for the moment, acoustic noise suppression module 310 is configured to process a wind noise suppressed primary signal {circumflex over (P)}(f) and a wind noise suppressed reference signal {circumflex over (R)}(f) to remove acoustic background noise from {circumflex over (P)}(f). In general, {circumflex over (P)}(f) and {circumflex over (R)}(f) respectively represent the residual signals of P(f) and R(f) after having undergone wind noise detection and, potentially, wind noise suppression by wind noise detection and suppression module 305. Both {circumflex over (P)}(f) and {circumflex over (R)}(f) contain components of the user's speech and acoustic background noise. However, because of the positioning of primary microphone 104 and reference microphone 106 on wireless communication device 102, the magnitude of the user's speech S1(f) in {circumflex over (P)}(f) is likely to be greater than a magnitude of the user's speech S2(f) in {circumflex over (R)}(f).
Acoustic noise suppression module 310 is configured to exploit this difference in magnitude to filter the wind noise suppressed primary signal {circumflex over (P)}(f) using wind noise suppressed reference signal {circumflex over (R)}(f) to provide, as output, speech signal Ŝ1(f), which represents the acoustic and wind noise suppressed speech signal. As illustrated, acoustic noise suppression module 310 specifically includes a time-varying blocking matrix (BM) 315 and a time-varying active noise canceller (ANC) 320.
Time-varying BM 315 is configured to estimate and remove the undesirable speech component S2(f) in {circumflex over (R)}(f) to get a “cleaner” noise reference signal. More specifically, time-varying BM 315 includes a first filter module 325 configured to filter {circumflex over (P)}(f) to provide an estimate of the speech signal S2(f) in {circumflex over (R)}(f). The estimated speech signal Ŝ2(f) is then subtracted from {circumflex over (R)}(f) by adder 335 to provide the cleaner noise reference signal {circumflex over (N)}2(f).
After the cleaner noise reference signal {circumflex over (N)}2(f) has been obtained, time-varying ANC 320 is configured to estimate and remove the undesirable acoustic background noise component N1(f) in {circumflex over (P)}(f) to provide Ŝ1(f). More specifically, time-varying ANC 320 includes a second filter module 330 configured to filter the cleaner noise reference signal {circumflex over (N)}2(f) to provide an estimate of the acoustic background noise N1(f) in {circumflex over (P)}(f). The estimated background noise {circumflex over (N)}1(f) is then subtracted from {circumflex over (P)}(f) by adder 340 to provide the acoustic and wind noise suppressed speech signal Ŝ1(f).
Acoustic noise suppression module 310 further includes an adaptation control module 345 configured to update tap coefficients of first filter module 325 and second filter module 330 to provide the desired filter functionality described above. In an embodiment, first filter module 325 and second filter module 330 are configured to respectively filter {circumflex over (P)}(f) and {circumflex over (N)}2(f) in the frequency domain using one or more taps per frequency component in signals {circumflex over (P)}(f) and {circumflex over (N)}2(f). In another embodiment, first filter module 325 and second filter module 330 are configured to respectively filter these two signals in the time domain.
In at least one embodiment, and as further shown in
Beyond being used to improve the effectiveness of acoustic noise suppression module 310, wind noise suppression and detection module 305, by detecting and suppressing wind noise in primary signal P(f), more generally improves the perceptual quality and intelligibility of the speech component of primary signal P(f) when played back to a listener.
The following sections describe two different implementations of wind noise detection and suppression module 305. It should be noted that these two implementations are described as being implemented in noise suppression system 300 for illustration purposes only and are not intended to be limiting. Persons skilled in the relevant art(s) will recognize that these implementations of wind noise detection and suppression module 305 can be implemented in a wide number of different multi-microphone devices and noise suppression systems, including noise suppression systems that do not perform acoustic noise suppression. For example, these implementations can be used in a wireless communication device such as a cellular telephone, a PDA, a tablet computer, a non-wireless communication device, a Bluetooth® headset, a hearing aid, a personal recorder, a video recorder, and a sound pick-up system for public speech or on-stage performances.
In operation, primary signal P(f) and reference signal R(f) are first processed by multi-method wind noise detection module 405 on a frame-by-frame basis. In general, multi-method wind noise detection module 405 is configured to detect the presence or absence of wind noise in primary signal P(f) using two or more wind noise detection methods and to detect the presence or absence of wind noise in reference signal R(f) using two or more wind noise detection methods. Each wind noise detection method produces a wind noise detection signal that indicates whether wind noise is present or absent. These detection signals are labeled as intermediate wind noise detection signals 420 in
In an embodiment, one or more of intermediate wind noise detection signals 420 represent hard decisions that simply indicate whether wind noise is present or absent in primary signal P(f) or reference signal R(f). In other words, these hard decisions do not indicate how much wind noise there is or the likelihood that wind noise is present or absent. In another embodiment, one or more of intermediate wind noise detection signals 420 represent soft decisions that indicate how much wind noise there is or the likelihood that wind noise is present or absent in primary signal P(f) or reference signal R(f).
In yet another embodiment, one or more of intermediate wind noise detection signals 420, corresponding to primary signal P(f), are generated based on both primary signal P(f) and reference signal R(f). In other words, the joint information contained in primary signal P(f) and reference signal R(f) is used to determine whether wind noise is present or absent in primary signal P(f). In another embodiment, one or more of intermediate wind noise detection signals 420, corresponding to reference signal R(f), are generated based on both primary signal P(f) and reference signal R(f).
After intermediate wind noise detection signals 420 are generated, wind noise detection signal combining module 410 is configured to combine them, in some logical manner, to provide primary microphone wind noise detection signal 425 and reference microphone wind noise detection signal 430. Primary microphone wind noise detection signal 425 indicates whether wind noise is present or absent in primary signal P(f), and reference microphone wind noise detection signal 430 indicates whether wind noise is present or absent in reference signal R(f). By combining intermediate wind noise detection signals 420, primary microphone wind noise detection signal 425 and reference microphone wind noise detection signal 430 can more precisely or more accurately indicate whether wind noise is present or absent in primary signal P(f) and reference signal R(f) than any of intermediate wind noise detection signals 420 taken individually.
In an embodiment, wind noise detection signal combining module 410 performs a logical “AND” operation to combine intermediate wind noise detection signals 420 that correspond to primary signal P(f). In this embodiment, primary microphone wind noise detection signal 425 indicates wind noise is present in primary signal P(f) only if each intermediate wind noise detection signal 420, corresponding to primary signal P(f), indicates that wind noise is present or above some threshold value. Otherwise, primary microphone wind noise detection signal 425 indicates wind noise is not present in primary signal P(f). This same scheme can be used to determine reference microphone wind noise detection signal 430 using intermediate wind noise detection signals 420 that correspond to reference signal R(f).
In another embodiment, wind noise detection signal combining module 410 performs a majority vote operation and indicates, through primary microphone wind noise detection signal 425, that wind noise is present in primary signal P(f) if a majority of intermediate wind noise detection signals 420, corresponding to primary signal P(f), indicate wind noise is present or above some threshold value. This same scheme can be used to determine reference microphone wind noise detection signal 430 using intermediate wind noise detection signals 420 that correspond to reference signal R(f).
After wind noise detection signals 425 and 430 have been generated, primary wind noise suppression module 415A and reference wind noise suppression module 415B perform wind noise suppression. More specifically, primary microphone wind noise suppression module 415A performs wind noise suppression on the frame of samples of primary signal P(f) for which wind noise detection took place, and reference wind noise suppression module 415B performs wind noise suppression on the frame of samples of reference signal R(f) for which wind noise detection took place. Wind noise suppression modules 415A and 415B are described further below in regard to
Turning now to the description of SD-WND module 505, it can be shown that the expected spectrum of wind noise has an envelope that decays in a roughly linear fashion with frequency. SD-WND module 505 is configured to exploit this characteristic of wind noise to detect its presence or absence in primary signal P(f). More specifically, SD-WND module 505 is configured to compare the spectrum of a frame of primary signal P(f) with an expected wind noise spectrum having the characteristics noted above (i.e., a spectrum with a magnitude that decreases with frequency and an overall spectral shape that is close to linear). If a difference in the spectrums is greater than a certain threshold, SD-WND module 505 determines that wind noise is absent in primary signal P(f). Otherwise, SD-WND module 505 determines that wind noise is present in primary signal P(f).
In one embodiment, SD-WND module 505 is configured to compare the magnitude or energy of certain frequencies of a frame of primary signal P(f) to corresponding magnitudes or energies of an expected wind noise spectrum. For example, because wind noise is often concentrated in the lower frequency range of speech (e.g., <2250 Hz), SD-WND module 505 can compare the magnitude or energies of only those frequencies of primary signal P(f), within the lower frequency range of speech, to corresponding magnitudes or energies of the expected wind noise spectrum. If a difference in magnitude or energy between the spectrums is greater than a certain threshold, then SD-WND module 505 determines that wind noise is absent in primary signal P(f). Otherwise, SD-WND module 505 determines that wind noise is present in primary signal P(f). Primary microphone SD-WND module 505 provides, as output, primary microphone SD-WND signal that indicates whether wind noise is present or absent in the frame of primary signal P(f).
SD-WND module 510 is configured to operate in a similar manner as described above in regard to SD-WND module 505. However, SD-WND module 510 is configured to detect the presence or absence of wind noise in a frame of reference signal R(f). SD-WND module 510 provides, as output, reference microphone SD-WND signal that indicates whether wind noise is present or absent in the frame of samples of reference signal R(f).
It should be noted that spectral derivation based wind noise detection is a single channel method and is applied on primary signal P(f) and reference signal R(f) separately (i.e., without using the information contained in the other signal). In addition, it should be noted that the thresholds used by SD-WND modules 505 and 510, to determine whether wind noise is present or absent in primary signal P(f) and reference signal R(f), can be different in value.
Turning now to the description of C-WND module 515, the following three facts are exploited by C-WND module 515 to detect whether wind noise is present or absent in primary signal P(f) and reference signal R(f): (1) wind noise typically does not correlate well with acoustic sounds (e.g., speech or background noise); (2) acoustic sounds picked up by a first microphone (e.g., primary microphone 104 illustrated in
Using the three facts noted above, C-WND module 515 detects whether wind noise is present or absent in primary signal P(f) and reference signal R(f), on a frame-by-frame basis, by examining the relationship between: (i) the maximum normalized correlation of primary signal P(f) in an estimated pitch period range; (ii) the maximum normalized correlation of reference signal R(f) in an estimated pitch period range; and (iii) the cross-channel normalized correlation between primary signal P(f) and reference signal R(f).
In one embodiment, if all three of these correlation values are above some defined threshold, it is assumed that primary signal P(f) and reference signal R(f) include voiced speech and wind noise is not present in either primary signal P(f) or reference signal R(f).
In another embodiment, if the cross-channel correlation value in (iii) is above the defined threshold and the same-channel correlation values in (i) and (ii) are below the defined threshold, it is assumed that primary signal P(f) and reference signal R(f) include unvoiced speech and/or background noise and wind noise is not present in either primary signal P(f) or reference signal R(f).
In yet another embodiment, if one of the two same-channel correlation values determined in (i) and (ii) is above the defined threshold and the other is below the defined threshold, and the cross-channel correlation value in (iii) is also below the defined threshold, then it is assumed that wind noise is present in the signal with the same-channel correlation value below the defined threshold and that wind noise is not present in the signal with the same-channel correlation value above the defined threshold (or at least to a much less extent).
It should be noted that different defined thresholds can be used for comparison against each correlation value. It should be further noted that the relative differences, between the three correlation values, can be further used to detect whether wind noise is present or absent in primary signal P(f) and reference signal R(f). For example, in addition to requiring all three correlation values be above some defined threshold in order to assume that wind noise is not present in either primary signal P(f) or reference signal R(f), it can be further required that the relative difference in value between one or more of the correlation values be within some defined range. In addition, it should be further noted that the three correlation values can be non-normalized in other embodiments.
C-WND module 515 provides, as output, two intermediate wind noise detection signals 420 based on the relationship between the correlation values as outlined above. More specifically, C-WND module 515 provides a primary microphone C-WND signal and a reference microphone C-WND signal, as output, to respectively indicate whether wind noise is present or absent in primary signal P(f) and reference signal R(f).
The method of flowchart 600 begins at step 605 and transitions to step 610. At step 610, the maximum normalized correlation of primary signal P(f) in the pitch period range is calculated (labeled as prim. mic. single channel correlation (SCC) in
During decision step 615, if the three calculated correlation values (i.e., CCC, prim. mic. SCC, and ref. mic. SCC) are all above a defined threshold, output signal primary microphone C-WND is set to a value that indicates wind noise is not present in primary signal P(f) and output signal reference microphone C-WND is set to a value that indicates wind noise is not present in reference signal R(f) as shown in step 620. In general, if the three calculated correlation values are all above the defined threshold, it is assumed that primary signal P(f) and reference signal R(f) include voiced speech and wind noise is not present in either primary signal P(f) or reference signal R(f). On the other hand, if the three conditions in step 615 are not all true, flowchart 600 proceeds to step 625.
During decision step 625, if CCC is above the defined threshold and primary microphone SCC and reference microphone SCC are below the defined threshold, primary microphone C-WND signal is set to a value that indicates wind noise is not present in primary signal P(f) and reference microphone C-WND signal is set to a value that indicates wind noise is not present in reference signal R(t) as shown in step 620. In general, if CCC is above the defined threshold and primary microphone SCC and reference microphone SCC are below the defined threshold, it is assumed that primary signal P(f) and reference signal R(f) include unvoiced speech and/or background noise and wind noise is not present in either primary signal P(f) or reference signal R(f). On the other hand, if the three conditions in step 625 are not all true, flowchart 600 proceeds to step 630.
During decision step 630, if CCC and reference microphone SCC are below the defined threshold and primary microphone SCC is above the defined threshold, primary microphone C-WND signal is set to a value that indicates wind noise is not present in primary signal P(f) and reference microphone C-WND signal is set to a value that indicates wind noise is present in reference signal R(f) as shown in step 635. On the other hand, if the three conditions in step 630 are not all true, flowchart 600 proceeds to step 640.
During decision step 640, if CCC and primary microphone SCC are below the defined threshold and reference microphone SCC is above the defined threshold, primary microphone C-WND signal is set to a value that indicates wind noise is present in primary signal P(f) and reference microphone C-WND signal is set to a value that indicates wind noise is not present in reference signal R(f) as shown in step 645. On the other hand, if the three conditions in step 640 are not all true, flowchart 600 proceeds to step 650.
At step 650, flowchart 600 ends and if primary microphone C-WND signal and reference microphone C-WND signal are not set (i.e., they do not indicate, either way, whether wind noise is present or not) then the subsequent processing logic can deal with the indeterminate values of primary microphone C-WND signal and reference microphone C-WND signal. In another embodiment, rather than simply ending flowchart 600 at step 650 and leaving the values of primary microphone C-WND signal and reference microphone C-WND undetermined, these two values can be set to a default value.
It should be noted, in regard to flowchart 600, that different defined thresholds can be used for comparison against each correlation value (i.e., CCC, prim. mic. SCC, and ref. mic. SCC). It should be further noted, in regard to flowchart 600, that the relative differences between the three correlation values can be further used to detect whether wind noise is present or absent in primary signal P(f) and reference signal R(f). For example, in addition to requiring all three correlation values be above some defined threshold in step 615 in order to assume that wind noise is not present in either primary signal P(f) or reference signal R(f), it can be further required that the relative difference in value between one or more of the correlation values be within some defined range. In addition, it should be further noted that the three correlation values calculated in step 610 can be non-normalized in other embodiments.
Referring now to
As discussed above, wind noise picked up by primary microphone 104 or reference microphone 106 often will not be picked up (or at least not to the same extent) by the other microphone because air turbulence caused by wind is usually a fairly local event. Therefore, a difference in energy between corresponding sub-bands of primary signal P(f) and reference signal R(f) can provide a good indication as to how much wind noise, if any, is present in each signal and, thereby, how much wind noise to suppress in each signal. However, in some instances, a difference in energy between corresponding sub-bands of primary signal P(f) and reference signal R(f) can be misrepresentative of the actual amount of wind noise present in each signal. Therefore, primary microphone wind noise suppression module 415A is further configured to utilize primary microphone wind noise detection signal 425 to improve suppression results, the generation of which was described above in regard to
As illustrated in
In operation, sub-band analysis module 705 is configured to process primary signal P(f) on a frame-by-frame basis, where a frame includes a set of consecutive samples taken from primary signal P(f) in the time domain. In one embodiment, sub-band analysis module 705 is configured to receive each frame of primary signal P(f) already transformed into the frequency domain. In another embodiment, sub-band analysis module 705 is configured to receive each frame of primary signal P(f) in the time domain and is configured to calculate the discrete Fourier transform (DFT) of each frame to transform the frames into the frequency domain. Sub-band analysis module 705 can calculate the DFT using, for example, the Fast Fourier Transform (FFT). In general, the resulting frequency domain signal describes the magnitudes and phases of component cosine waves (also referred to as component frequencies) that make up the time domain frame, where each component cosine wave corresponds to a particular frequency between DC and one-half the sampling rate used to obtain the samples of the time domain frame.
For example, and in one embodiment, each time domain frame of primary signal P(f) includes 128 samples and is transformed into the frequency domain using a 128-point DFT by sub-band analysis module 705 or some other module not shown. The 128-point DFT provides 64 values that represent the magnitudes of the component cosine waves that make up the time domain frame. In another embodiment, each time domain frame of primary signal P(f) includes N samples and is transformed into the frequency domain using an M-point DFT by sub-band analysis module 705 or some other module not shown, where N and M are integer numbers and M is greater than or equal to N. When M is larger than N, the N samples of primary signal P(f) can be padded with M-N zeroes.
Once the magnitudes of the component cosine waves are obtained for a frame of primary signal P(f), sub-band analysis module 705 is configured to group the cosine wave components into sub-bands, where a sub-band can include one or more cosine wave components. In one embodiment, sub-band analysis module 705 is configured to group the cosine wave components into sub-bands based on the Bark frequency scale. As is well known, the Bark frequency scale ranges from 1 to 24 Barks and each Bark corresponds to one of the first 24 critical bands of hearing.
Table 1 below provides an example grouping of 62 component cosine waves (i.e., component cosine waves 3 through 64) into 16 sub-bands based on the Bark frequency scale. Each of the 62 component cosine waves has a corresponding magnitude obtained using a 128-point DFT (the first two component cosine waves 1-2, and their corresponding magnitudes, are ignored). The 128-point DFT is specifically calculated over a frame of 128 time-domain samples of primary signal P(f) obtained at a sampling rate of 8000 Hz.
In one embodiment, the cosine wave components are grouped into each sub-band by adding their corresponding squared magnitudes together. For example, the 3rd and 4th cosine wave components are grouped into the first sub-band, as indicated by table 1 above, by adding their corresponding squared magnitudes together. The resulting sum represents an estimated energy of the first sub-band. Extending the exemplary sub-band grouping provided in table 1 to the illustration of
It should be noted that table 1 is for illustration purposes only and is not intended to be limiting. Persons skilled in the relevant art(s) will recognize that other groupings can be used, for example, based on different sampling rates and DFT sizes. It should be further noted that the cosine wave components can be grouped using other methods that provide a reasonable estimate of the energy of the sub-band to which they belong.
In one embodiment, sub-band analysis module 705 is configured to provide estimated sub-band energies for sub-bands corresponding only to a lower frequency range of speech. For example, and as shown in
Sub-band analysis module 710 is configured to provide estimated energies for sub-bands corresponding to frames in reference signal R(f) in a similar manner as sub-band analysis module 705 described above. The estimated energies are provided as output in a two dimensional array Y2(k,i) indexed by frame number (k) and sub-band number (i).
Energy ratio calculation module 715 is configured to determine a difference in energy between corresponding sub-bands of primary signal P(f) and reference signal R(f). In one embodiment, energy ratio calculation module 715 is configured to divide the sub-band energies of primary signal P(f), provided by sub-band analysis module 705, by corresponding sub-band energies of reference signal R(f), provided by sub-band analysis module 710, to determine differences in energy. For example, energy ratio calculation module 715 is configured to divide the sub-band energy Y1(k,1) by the sub-band energy Y2(k,1) and provide the resulting quotient as output R(k,1), where R(k,i) is a two dimensional array indexed by frame number (k) and sub-band number (i). Thus, R(k,1) represents the difference in energy between the first sub-band of the kth frame of primary signal P(f) and the first sub-band of the kth frame of reference signal R(f).
In another embodiment, energy ratio calculation module 715 is configured to subtract the sub-band energies of primary signal P(f), provided by sub-band analysis module 705, from corresponding sub-band energies of reference signal R(f), provided by sub-band analysis module 710, to determine differences in energy. The resulting values of each subtraction are provided as output R(k,i), in this embodiment, energy ratio calculation module 714 may be more aptly referred to as an energy difference calculation module 714.
Threshold calculation module 720 is configured to calculate threshold values for the sub-bands of primary signal P(f) that are to be used to determine how much wind noise suppression to apply to a particular sub-band. In one embodiment, threshold calculation module 720 is configured to calculate threshold values for the sub-bands of primary signal P(f) based on the differences in energy between corresponding sub-bands of primary signal P(f) and reference signal R(f), represented by two dimensional array R(k,i), and based on previously calculated threshold values. For example, and in one embodiment, threshold calculation module 720 is configured to calculate a threshold value for the ith sub-band of the kth frame of primary signal P(f), represented by Tnew(k,i), according to the following equations:
T
new(k,i)=α×Told(k,i)+(1−α)×R(k,i)
T
old(k+1,i)=Tnew(k,i)
where Told(k,i) represents the threshold value calculated for the ith sub-band of the previous frame (i.e., frame k−1) and α is a smoothing factor with a value between 0 and 1. As illustrated in
Suppression gain calculation module 725 is configured to determine suppression gains for the sub-bands of primary signal P(f) based on the calculated threshold values (i.e., Tnew(k,i)) and the differences in energy between corresponding sub-bands of primary signal P(f) and reference signal R(f) (i.e., R(k,i)). In one embodiment, suppression gain calculation module 725 multiplies each calculated threshold value for the kth frame of primary signal P(f), represented by Tnew(k,i), by two constant values: a speech constant with a value ‘s’, and a wind noise constant with a value ‘w’.
For example, for the first sub-band of the kth frame of primary signal P(f), the threshold value represented by Tnew(k,1) is multiplied by the speech constant ‘s’, to obtain a speech threshold T1, and by the wind noise constant ‘w’, to obtain a wind noise threshold T2. Suppression gain calculation module 725 is then configured to use these two threshold values, T1 and T2, to construct a suppression gain function.
In the instant example, plot 800 was constructed using threshold values T1 and T2 calculated for the first sub-band of the kth frame of primary signal P(f). Therefore, plot 800 (and the function it represents) would be used to determine a suppression gain for the first sub-band of the kth frame of primary signal P(f). More specifically, suppression gain calculation module 725 would use the difference in energy between the first sub-band of the kth frame of primary signal P(f) and the first sub-band of the kth frame of reference signal R(f), represented by R(k,1), as the independent variable of the function represented by plot 800 to determine a suppression gain.
For example, if the difference in energy represented by R(k,1) is less than T1, the function represented by plot 800 would return a suppression gain of 0 dB. The threshold T1 is referred to as the speech threshold because it is assumed that primary signal P(f) is substantially wind noise free when the calculated difference in energy between corresponding sub-bands of primary signal P(f) and reference signal R(f) is below T1. Therefore, the function represented by plot 800 returns 0 dB (i.e. no suppression), for the first sub-band of primary signal P(f) if the difference in energy represented by R(k,1) is less than T1.
If the difference in energy represented by R(k,1) is greater than T2, the function represented by plot 800 would return a suppression gain of −20 dB. T2 is referred to as the wind noise threshold because it is assumed that primary signal P(f) contains a substantial amount of wind noise when the difference in energy between corresponding sub-bands of primary signal P(f) and reference signal R(f) is greater than T2. Therefore, the function represented by plot 800 returns a large amount of suppression, such as −20 dB, for the first sub-band of primary signal P(f) if the difference in energy represented by R(k,1) is greater than T2.
If the difference in energy represented by R(k,1) is greater than T1 and less than T2, the function represented by plot 800 would return a suppression gain between 0 dB and −20 dB. As specifically illustrated in plot 800, the suppression gain changes from 0 dB to −20 dB as the difference in energy between sub-bands increases from T1 to T2. If the difference in energy represented by R(k,1) falls between T1 and T2 it is assumed that primary signal P(f) contains some wind noise and some speech.
In an embodiment, suppression gain calculation module 725 is configured to smooth the suppression gains determined for each sub-band across adjacent sub-bands and/or frames (or time).
It should be noted that, in
In general, suppression gain calculation module 725 is configured to construct a suppression gain function that provides: a constant suppression gain of 0 dB for differences in energy that are less than T1; a large, constant amount of suppression (e.g., −20 dB) for differences in energy that are greater than T2; and a suppression amount that increases as the difference in energy between sub-bands increases from T1 to T2.
Referring back to
Gain mapping module 730 is configured to map the suppression gains for each sub-band of primary signal P(f) (i.e., g0(k,i)) to the component cosine waves (also referred to as component frequencies) of primary signal P(f). For example, gain mapping module 730 is configured to map the suppression gain for the first sub-band of the kth frame of primary signal P(f), represented by g0(k,1), to the component cosine waves grouped into the first sub-band. In an embodiment, gain mapping module 730 is configured to map the suppression gain for the first sub-band of the kth frame of primary signal P(f), represented by g0(k,1), to the component cosine waves grouped into the first sub-band by interpolating between the suppression gain of the first sub-band and the suppression gain of the second sub-band, represented by g0(k,2).
In another embodiment, for higher frequency sub-bands in which a suppression gain was not calculated, gain mapping module 730 is configured to set the suppression gain for the component cosine waves belonging to these sub-bands to a value of 0 dB.
In yet another embodiment, gain mapping module 730 is configured to set the suppression gain for each component cosine wave of primary signal P(f) to 0 dB if primary microphone wind noise detection signal 425 indicates that wind noise is not present in primary signal P(f).
Referring now to
Referring now to
The method of flowchart 900 begins at step 905 and transitions to step 910. At step 910, wind noise detection is performed on primary signal P(f) and reference signal R(f) using multiple methods. More specifically, at step 910 wind noise detection is performed to detect the presence or absence of wind noise in primary signal P(f) using two or more wind noise detection methods and to detect the presence or absence of wind noise in reference signal R(f) using two or more wind noise detection methods. Each wind noise detection method produces a wind noise detection signal that indicates whether wind noise is present or absent. In an embodiment, spectral-deviation based wind noise detection and correlation based wind noise detection are performed to determine if primary signal P(f) or reference signal R(f) contain wind noise. The resulting wind noise detection signals corresponding to primary signal P(f) are combined to produce a single wind noise detection signal for primary signal P(f), and the resulting wind noise detection signals corresponding to reference signal R(f) are combined to produce a single wind noise detection signal for reference signal R(f). Further details regarding wind noise detection using multiple methods were described above in regard to
At step 915, based on the combined wind noise detection signals for primary signal P(f) and reference signal R(f), produced at step 910, flowchart 900 proceeds to step 920 or step 925. For example, if the combined wind noise detection signal for primary signal P(f) indicates that wind noise is not present in primary signal P(f), flowchart 900 proceeds to step 920 for primary signal P(f). Otherwise, flowchart 900 proceeds to step 925 for primary signal P(f). Similarly, if the combined wind noise detection signal for reference signal R(f) indicates that wind noise is not present in reference signal R(f), flowchart 900 proceeds to step 920 for reference signal R(f). Otherwise, flowchart 900 proceeds to step 925 for reference signal R(f).
At step 920, suppression gains for sub-bands of primary signal P(f) and/or reference signal R(f) are set to 0 dB (or some other low suppression gain).
At step 925, suppression gains are calculated for the sub-bands of primary signal P(f) and/or reference signal R(f) based on differences in energy between corresponding sub-bands of primary signal P(f) and reference signal R(f). Details regarding the calculation of suppression gains for the sub-bands of primary signal P(f) and/or reference signal R(f) were described above in regard to
At step 930 the suppression gains for the sub-bands of primary signal P(f) and reference signal R(f) are mapped and applied to the component cosine waves (also referred to as component frequencies) of primary signal P(f) and reference signal R(f). Details regarding the mapping of suppression gains were described above in regard to
In operation, primary signal P(f) and reference signal R(f) are first processed by multi-method wind noise detection module 1005 on a frame-by-frame basis. In general, multi-method wind noise detection module 1005 is configured to detect the presence or absence of wind noise in primary signal P(f) using two or more wind noise detection methods and to detect the presence or absence of wind noise in reference signal R(f) using two or more wind noise detection methods. Each wind noise detection method produces a wind noise detection signal that indicates whether wind noise is present or absent. These detection signals are labeled as intermediate wind noise detection signals 1020 in
in an embodiment, one or more of intermediate wind noise detection signals 1020 represent hard decisions that simply indicate whether wind noise is present or absent in primary signal P(f) or reference signal R(f). In other words, these hard decisions do not indicate how much wind noise there is or the likelihood that wind noise is present or absent. In another embodiment, one or more of intermediate wind noise detection signals 1020 represent soft decisions that indicate how much wind noise there is or the likelihood that wind noise is present or absent in primary signal P(f) or reference signal R(f).
In another embodiment, one or more of intermediate wind noise detection signals 1020, corresponding to primary signal P(f), are generated based on both primary signal P(f) and reference signal R(f). In other words, the joint information contained in primary signal P(f) and reference signal R(f) is used to determine whether wind noise is present or absent in primary signal P(f). In yet another embodiment, one or more of intermediate wind noise detection signals 1020, corresponding to reference signal R(f), are generated based on both primary signal P(f) and reference signal R(f).
After intermediate wind noise detection signals 1020 are generated, wind noise detection signal combining module 1010 is configured to combine them, in some logical manner, to provide primary microphone wind noise detection signal 1025 and reference microphone wind noise detection signal 1030. Primary microphone wind noise detection signal 1025 indicates whether wind noise is present or absent in primary signal P(f), and reference microphone wind noise detection signal 1030 indicates whether wind noise is present or absent in reference signal R(f). By combining intermediate wind noise detection signals 1020, primary microphone wind noise detection signal 1025 and reference microphone wind noise detection signal 1030 can more precisely or more accurately indicate whether wind noise is present or absent in primary signal P(f) and reference signal R(f) than any of intermediate wind noise detection signals 1020 taken individually.
In an embodiment, wind noise detection signal combining module 1010 performs a logical “AND” operation to combine intermediate wind noise detection signals 1020 that correspond to primary signal P(f). In this embodiment, primary microphone wind noise detection signal 1025 indicates wind noise is present in primary signal P(f) only if each intermediate wind noise detection signal 1020, corresponding to primary signal P(f), indicates that wind noise is present or above some threshold value. Otherwise, primary microphone wind noise detection signal 1025 indicates wind noise is not present in primary signal P(f). This same scheme can be used to determine reference microphone wind noise detection signal 1030 using intermediate wind noise detection signals 1020 that correspond to reference signal R(f).
In another embodiment, wind noise detection signal combining module 1010 performs a majority vote operation and indicates, through primary microphone wind noise detection signal 1025, that wind noise is present in primary signal P(f) if a majority of intermediate wind noise detection signals 1020, corresponding to primary signal P(f), indicate wind noise is present or above some threshold value. This same scheme can be used to determine reference microphone wind noise detection signal 1030 using intermediate wind noise detection signals 1020 that correspond to reference signal R(f).
After wind noise detection signals 1025 and 1030 have been generated, primary microphone wind noise suppression module 1015 is configured to perform wind noise suppression on primary signal P(f). More specifically, primary microphone wind noise suppression module 1015 performs wind noise suppression on the frame of samples of primary signal P(f) for which wind noise detection took place.
In general, when wind noise is present or above some threshold in the frame of primary signal P(f), as indicated by primary microphone wind noise detection signal 1020, and wind noise is absent or below some threshold in the corresponding frame of reference signal R(f), as indicated by reference microphone wind noise detection signal 1030, primary microphone wind noise suppression module 1015 is configured to replace (at least a portion of) the wind noise corrupted frame of primary signal P(f) with (at least a portion of) the comparatively cleaner frame of reference signal R(f). The “at least a portion of” above can mean some portion of the time domain samples or some portion of the DFT coefficients that are corrupted by wind noise more than the remaining portions. The same applies to this term when used in the description that follows.
To maintain consistent speech characteristics of primary signal P(f), reference signal adjustment module 1035 is configured to adjust one or more of the delay, gain, spectral shape, and background noise level of reference signal R(f) to match those of primary signal P(f) before portions of primary signal P(f) are replaced with portions of reference signal R(f). Reference signal adjustment module 1035 is configured to provide the adjusted reference signal R(f) to primary microphone wind noise suppression module 1015 via adjusted reference signal 1040.
In one embodiment, reference signal adjustment module 1035 separately estimates the difference in one or more of delay, gain, spectral shape, and background noise level between primary signal P(f) and reference signal R(f) and separately adjusts the one or more parameters of reference signal R(f), based on the estimates, to more closely match the corresponding parameters of primary signal P(f).
In another embodiment, reference signal adjustment module 1035 adjusts one or more of the delay, gain, and spectral shape of reference signal R(f) using a single adaptive filter in the time domain. Such an adaptive filter can filter reference signal R(f) to adjust one or more of these parameters to better match the corresponding parameters of primary signal P(f). For example, the adaptive filter can filter reference signal R(f) to adjust its delay and gain to better match the delay and gain of primary signal P(f).
In an embodiment, the filter taps of the adaptive filter are adapted only when wind noise is absent or below some threshold in primary signal P(f) and is absent or below some threshold in reference signal R(f) such that the filtered reference signal effectively tracks the speech characteristics of primary signal P(f). In this embodiment, when wind noise is present or above some threshold in primary signal P(f) and absent or below some threshold in reference signal R(f), adaptation of the filter taps is stopped, and the adaptive filter is used to filter reference signal R(f). The adaptively filtered reference signal R(f) is then used by primary microphone wind noise suppression module 1015 to replace (at least a portion of) the wind noise corrupted frame of primary signal P(f).
In yet another embodiment, if wind noise detection and suppression module 305 is implemented in a noise suppression system that implements a time-varying blocking matrix in the frequency domain, similar to blocking matrix 315 described in
Because the first filter module 325 in the time-varying blocking matrix filters the primary signal P(f) to approximate the speech signal in the reference signal R(f), the inverse filter function of the first filter module 325 should achieve the opposite effect. In other words, filtering the speech component in reference signal R(f), using the inverse filter function of the first filter module 325, should provide an approximation of primary signal P(f).
Furthermore, in a noise suppression system that implements a time-varying blocking matrix in the frequency domain, such as noise suppression system 300 illustrated in
One issue that has not yet been discussed in detail is how reference signal adjustment module 1035 can adjust the acoustic background noise level of reference signal R(f) to better match the acoustic background noise level of primary signal P(f). In general, in a typical dual-microphone configuration with a primary microphone (e.g., primary microphone 104) and a noise reference microphone (e.g., reference microphone 106), the signal-to-noise ratio (SNR) of the signal picked up by the noise reference microphone is usually lower than the SNR of the signal picked up by the primary microphone. In other words, the background noise level is usually higher relative to the speech signal level in the signal picked up by the noise reference microphone than it is in the signal picked up by the primary microphone. Thus, for example, after the gain of reference signal R(f) is adjusted to better match the speech signal level of primary signal P(f), the acoustic background noise level in the adjusted reference signal 1040 will typically be higher than the acoustic background noise level in primary signal P(f). The adaptive filtering approaches described above (whether implemented in the time-domain or frequency-domain) generally only adjust the delay, gain, and spectral shape of reference signal R(f) and do not adjust the acoustic background noise level to compensate for this anticipated difference.
In one embodiment, to adjust the acoustic background noise level of reference signal R(f) to better match the acoustic background noise level of primary signal P(f), reference signal adjustment module 1035 estimates the long-term average acoustic background noise levels and speech levels in both primary signal P(f) and reference signal R(f). From these estimated levels, reference signal adjustment module 1035 calculates a long-term signal-to-noise ratio (SNR) for each signal.
Reference signal adjustment module 1035 is then configured to use these calculated, long-term SNR values in combination with a single-channel noise suppression technique to suppress the acoustic background noise level in reference signal R(f) to better match the acoustic background noise level in primary signal P(f). Any suitable single-channel noise suppression technique can be used, but to make the acoustic background noise level of reference signal R(f) roughly the same as the acoustic background noise level of primary signal P(f), the target amount of noise suppression can be set to (or determined based on) the difference between the calculated, long-term SNR of primary signal P(f) and the calculated, long-term SNR of reference signal R(f).
After applying single-channel noise suppression to reference signal R(f) with the amount of noise suppression set to (or at least based on) SNR1-SNR2, where SNR1 and SNR2 are the calculated, long-term SNR values of primary signal P(f) and reference signal R(f), respectively, the resulting noise-suppressed reference signal R(f) should have roughly the same level of acoustic background noise as primary signal P(f). This is important for maintaining a consistent level of acoustic background noise in the final wind noise suppressed primary signal P(f). Without such background noise level matching, the wind noise suppressed primary signal P(f) can have an acoustic background noise level that modulates with the application of waveform substitution performed by primary microphone wind noise suppression module 1015.
To avoid potential waveform discontinuities at the boundaries between primary signal P(f) and a substituted waveform, primary microphone wind noise suppression module 1015 can perform proper overlap-add operations between primary signal P(f) and the substituted waveform. For example, when a wind noise corrupted portion of primary signal P(f) is substituted for a comparatively cleaner portion of an adaptively filtered reference signal R(f) (provided by reference signal adjustment module 1035), primary microphone wind noise suppression module 1015 can smooth the boundaries between the portion of the adaptively filtered reference signal R(f) and primary signal P(f) using proper overlap-add operations.
A general, overlap-add operation of two signals can be defined by:
s(n)=sout(n)·wout(n)+sin(n)·win(n); n=0 . . . N−1
where sout is the signal to be faded out, sin is the signal to be faded in, wout is the fade-out window, win is the fade-in window, and N is the overlap-add window length. The general overlap-add operation, defined by the above equation, can be used to smoothly merge primary signal P(f) with a substituted waveform. In addition, any suitable fade-in window, fade-out window, and overlap-add window length can be used.
Turning now to the description of SD-WND module 505, which was described previously in regard to
In one embodiment, SD-WND module 505 is configured to compare the magnitude or energy of certain frequencies of a frame of primary signal P(f) to corresponding magnitudes or energies of an expected wind noise spectrum. For example, because wind noise is often concentrated in the lower frequency range of speech (e.g., <2250 Hz), SD-WND module 505 can compare the magnitude or energies of only those frequencies of primary signal P(f), within the lower frequency range of speech, to corresponding magnitudes or energies of the expected wind noise spectrum. If a difference in magnitude or energy between the spectrums is greater than a certain threshold, then SD-WND module 505 determines that wind noise is absent in primary signal P(f). Otherwise, SD-WND module 505 determines that wind noise is present in primary signal P(f). Primary microphone SD-WND module 505 provides, as output, primary microphone SD-WND signal that indicates whether wind noise is present or absent in the frame of primary signal P(f).
SD-WND module 510 is configured to operate in a similar manner as SD-WND module 505. However, SD-WND module 510 is configured to detect the presence or absence of wind noise in a frame of reference signal R(f). SD-WND module 510 provides, as output, reference microphone SD-WND signal that indicates whether wind noise is present or absent in the frame of samples of reference signal P(f).
It should be noted that spectral derivation based wind noise detection is a single channel method and is applied on primary signal P(f) and reference signal R(f) separately (i.e., without using the information contained in the other signal). In addition, it should be noted that the thresholds used by SD-WND modules 505 and 510, to determine whether wind noise is present or absent in primary signal P(f) and reference signal R(f), can be different in value.
Turning now to the description of C-WND module 515, which was also described previously in regard to
Using the three facts noted above, C-WND module 515 detects whether wind noise is present or absent in primary signal P(f) and reference signal R(f), on a frame-by-frame basis, by examining the relationship between: (i) the maximum normalized correlation of primary signal P(f) in an estimated pitch period range; (ii) the maximum normalized correlation of reference signal R(f) in an estimated pitch period range; and (iii) the cross-channel normalized correlation between primary signal P(f) and reference signal R(f).
In one embodiment, if all three of these correlation values are above some defined threshold, it is assumed that primary signal P(f) and reference signal R(f) include voiced speech and wind noise is not present in either primary signal P(f) or reference signal R(f).
In another embodiment, if the cross-channel correlation value in (iii) is above the defined threshold and the same-channel correlation values in (i) and (ii) are below the defined threshold, it is assumed that primary signal P(f) and reference signal R(f) include unvoiced speech and/or background noise and wind noise is not present in either primary signal P(f) or reference signal R(f).
In yet another embodiment, if one of the two same-channel correlation values determined in (i) and (ii) is above the defined threshold and the other is below the defined threshold, and the cross-channel correlation value in (iii) is also below the defined threshold, then it is assumed that wind noise is present in the signal with the same-channel correlation value below the defined threshold and that wind noise is not present in the signal with the same-channel correlation value above the defined threshold (or at least to a much less extent).
It should be noted that different defined thresholds can be used for comparison against each correlation value. It should be further noted that the relative differences, between the three correlation values, can be further used to detect whether wind noise is present or absent in primary signal P(f) and reference signal R(f). For example, in addition to requiring all three correlation values be above some defined threshold in order to assume that wind noise is not present in either primary signal P(f) or reference signal R(f), it can be further required that the relative difference in value between one or more of the correlation values be within some defined range. In addition, it should be further noted that the three correlation values can be non-normalized in other embodiments.
C-WND module 515 provides, as output, two intermediate wind noise detection signals 1020 based on the relationship between the correlation values as outlined above. More specifically, C-WND module 515 provides a primary microphone C-WND signal and a reference microphone C-WND signal, as output, to respectively indicate whether wind noise is present or absent in primary signal P(f) and reference signal R(f).
Turning now to the description of ALGD-WND module 1105, ALGD-WND module 1105 is configured to detect the presence or absence of wind noise in primary signal P(f) and adjusted reference signal 1040 based on the average value of the logarithmic gain difference between corresponding frequency components of primary signal P(f) and adjusted reference signal 1040. Adjusted reference signal 1040 has been generated by reference signal adjustment module 1030, illustrated in
In general, wind noise picked up by primary microphone 104 (which provides primary signal P(f)) or reference microphone 106 (which provides reference signal R(f)) often will not be picked up (or at least not to the same extent) by the other microphone because air turbulence caused by wind is usually a fairly local event. Therefore, a difference in energy between corresponding sub-bands of primary signal P(f) and adjusted reference signal 1040 (which was generated based on reference signal R(f)) can provide a good indication as to whether wind noise is present or absent in each signal.
In one embodiment, ALGD-WND module 1105 is configured to receive each frame of primary signal P(f) and adjusted reference signal 1040 already transformed into the frequency domain. In another embodiment, ALGD-WND module 1105 is configured to receive each frame of primary signal P(f) and adjusted reference signal 1040 in the time domain and is configured to calculate the discrete Fourier transform (DFT) of each frame to transform the frames into the frequency domain. ALGD-WND module 1105 can calculate the DFT using, for example, the Fast Fourier Transform (FFT). In general, the resulting frequency domain signal describes the magnitudes and phases of component cosine waves (also referred to as component frequencies) that make up the time domain frame, where each component cosine wave corresponds to a particular frequency between DC and one-half the sampling rate used to obtain the time domain frame.
For example, and in one embodiment, each time domain frame of primary signal P(f) and adjusted reference signal 1040 includes 128 samples and is transformed into the frequency domain using a 128-point DFT by ALGD-WND module 1105 or some other module not shown. The 128-point DFT provides 65 values that represent the magnitudes of the component cosine waves that make up the time domain frame. In another embodiment, each time domain frame of primary signal P(f) and adjusted reference signal 1040 includes N samples and is transformed into the frequency domain using an M-point DFT by ALGD-WND module 1105 or some other module not shown, where N and M are integer numbers and M is greater than or equal to N. When M is larger than N, the N samples of primary signal P(f) and adjusted reference signal 1040 can be padded with M-N zeroes.
Once the magnitudes of the component cosine waves are obtained for a frame of primary signal P(f) and for a frame of adjusted reference signal 1040, ALGD-WND module 1105 is configured to group the component cosine waves into sub-bands, where a sub-band can include one or more component cosine waves. Component cosine waves assigned to a particular sub-band can be grouped by adding their corresponding energies together or the logarithm of their corresponding magnitudes together. The resulting sum represents an estimated energy of the sub-band. For sub-bands that contain single component cosine waves (which can be all of them in one embodiment), the estimated energy of these sub-bands can simply be set equal to the magnitude of their respective component cosine waves or the logarithm of the magnitude of their respective component cosine waves.
In one embodiment, ALGD-WND module 1105 is configured to determine a difference in energy between corresponding sub-bands of primary signal P(f) and adjusted reference signal 1040 by dividing the calculated sub-band energies of primary signal P(f) by corresponding sub-band energies of adjusted reference signal 1040. In another embodiment, ALGD-WND module 1105 is configured to determine a difference in energy between corresponding sub-bands of primary signal P(f) and adjusted reference signal 1040 by subtracting the sub-band energies of adjusted reference signal 1040 from corresponding sub-band energies of primary signal P(f) to determine differences in energy.
In another embodiment, ALGD-WND module 1105 is configured to determine a difference in energy between corresponding sub-bands of primary signal P(f) and adjusted reference signal 1040 for sub-bands corresponding only to a lower frequency range of speech. As discussed above, the expected spectrum of wind noise has an envelope that decays in a roughly linear fashion with frequency and is often concentrated in the lower frequency range of speech (e.g., <2250 Hz). Therefore, upper sub-bands that correspond to higher frequencies of speech (e.g., >2250 Hz) can be ignored because wind noise generally does not corrupt those frequencies.
Once the differences in energy between corresponding sub-bands of primary signal P(f) and adjusted reference signal 1040 are determined, ALGD-WND module 1105 can average the differences in energy together and provide the result as output via ALGD-WND signal. Assuming that adjusted reference signal 1040 matches primary signal P(f) signal well in terms of delay, gain, spectral shape, and background noise level, an ALGD-WND signal around 0 dB indicates that adjusted reference signal 1040 and primary signal P(f) are matching well and there is little or no wind noise in either signal. A large positive ALGD-WND signal indicates that there is wind noise in primary signal P(f) signal and no wind noise in adjusted reference signal 1040, or that primary signal P(f) has more wind noise than adjusted reference signal 1040. Conversely, a large negative ALGD-WND signal indicates that there is wind noise in adjusted reference signal 1040 and not in primary signal P(f), or adjusted reference signal 1040 has more wind noise than primary signal P(f).
Turning now to the description of SMNR-WND module 1110, SMNR-WND module 1110 is configured to divide the energy of primary signal P(f) by the energy of the difference between primary signal P(f) and adjusted reference signal 1040 to obtain a special SNR value referred to as the signal-to-matching-noise ratio (SMNR). Assuming that adjusted reference signal 1040 matches primary signal P(f) well in terms of delay, gain, spectral shape, and background noise level, then the calculated SMNR value should be large when there is little or no wind noise present in either primary signal P(f) and adjusted reference signal 1040. For example, the calculated SMNR value can be on the order of 30 to 50 dB when there is little or no wind noise present in either primary signal P(f) or adjusted reference signal 1040. On the other hand, when there is wind noise present in primary signal P(f) or adjusted reference signal 1040, the calculated SMNR value should be comparatively smaller. Thus, a small SMNR indicates there is likely wind noise in one of the two signals, while a large SMNR indicates there is likely little or no wind noise present in either of the two signals.
Referring now to
In operation, control module 1205 is configured to receive primary microphone wind noise detection signal 1025 and reference microphone wind noise detection signal 1030 that respectively indicate whether wind noise is present or absent in primary signal P(f) and reference signal R(f) (and, thereby, whether wind noise is present or absent in adjusted reference signal 1040). Based on these two signals, control module 1205 controls the operation of waveform substitution module 1210 and, if included, PLC module 1215, weighted sum module 1220, and single channel noise suppression module 1225. More specifically, based on different wind noise scenarios indicated by primary microphone wind noise detection signal 1025 and reference microphone wind noise detection signal 1030, control module 1205 use a different one of waveform substitution module 1210, PLC module 1215 weighted sum module 1220, and single channel noise suppression module 1225 to suppress wind noise in primary signal P(f) or make primary signal P(f) more consistent across time. The resulting signals from these modules are provided as output via wind noise suppressed primary signal {circumflex over (P)}(f).
Different responses of control module 1205 to each wind noise scenario are described further below under the assumption that primary microphone wind noise suppression module 1015 is implemented in noise suppression system 300 illustrated in
In a first scenario, when primary microphone wind noise detection signal 1025 and reference microphone wind noise detection signal 1030 indicate that wind noise is absent or below some threshold in primary signal P(f) and reference signal R(f) (and, thereby, absent or below some threshold in adjusted reference signal 1040), control module 1205 is configured to bypass waveform substitution module 1210 and, if included, PLC module 1215, weighted sum module 1220, and single channel noise suppression module 1225. In this scenario, control module 1205 is configured to set wind noise suppressed primary signal {circumflex over (P)}(f) equal to primary signal P(f). Although not shown in
Acoustic noise suppression module 310, illustrated in
In a second scenario, when primary microphone wind noise detection signal 1025 indicates that wind noise is absent or below some threshold in primary signal P(f) and reference microphone wind noise detection signal 1030 indicates that wind noise is present or above some threshold in reference signal R(f) (and, thereby, present or above some threshold in adjusted reference signal 1040), control module 1205 is configured to bypass waveform substitution module 1210 and, if included, PLC module 1215 and weighted sum module 1220.
In this second scenario, acoustic noise suppression module 310, illustrated in
However, simply restricting acoustic noise suppression module 310 from performing acoustic noise suppression can lead to its own problems. For example, if acoustic noise suppression module 310 provides, on average, X dB of acoustic noise reduction when wind noise is absent or below some threshold in both primary signal P(f) and reference signal R(f), simply turning ANC 320 off when wind noise is present or above some threshold in reference signal R(f) will cause the acoustic background noise level in primary signal P(f) to be X dB higher in the regions where reference signal R(f) is corrupted by wind noise. If this is not dealt with, the acoustic background noise level in primary signal P(f) will modulate with the presence of wind noise in reference signal R(f).
To combat this problem, control module 1205 can use single-channel noise suppression module 1225 to apply single-channel noise suppression with X dB of target noise suppression to primary signal P(f) during this second wind noise scenario. Doing so will help to maintain a roughly constant background noise level. Single-channel noise suppression module 1225 provides this single-channel noise suppressed signal, as output, via wind noise suppressed primary signal {circumflex over (P)}(f).
In the third scenario, when primary microphone wind noise detection signal 1025 indicates that wind noise is present or above some threshold in primary signal P(f) and reference microphone wind noise detection signal 1030 indicates that wind noise is absent or below some threshold in reference signal R(f) (and, thereby, absent or below some threshold in adjusted reference signal 1040), control module 1205 is configured to use waveform substitution module 1210 to replace (at least a portion of) the wind noise corrupted frame of primary signal P(f) with (at least a portion of) the comparatively cleaner frame of adjusted reference signal 1040 as described above in
In the fourth and final scenario, when both primary microphone wind noise detection signal 1025 and reference microphone wind noise detection signal 1030 indicate that wind noise is present or above some threshold in primary signal P(f) and reference signal R(f) (and, thereby, present or above some threshold in adjusted reference signal 1040), control module 1205 can apply Packet Loss Concealment (PLC) using PLC module 1215 and/or can perform a weighted sum method using weighted sum module 1220 to suppress wind noise in primary signal P(f).
For example, if the wind noise does not last too long in both primary signal P(f) and reference signal R(f), control module 1205 can use PLC module 1215 to perform PLC techniques to replace a current frame of primary signal P(f) with an extrapolated version of the current frame from previous frame(s) of primary signal P(f) that were not corrupted by wind noise. Such a PLC-based method, however, often works well only when the time period of wind noise does not last too long. For example, if the burst of wind noise lasts less than about 20 ms, the PLC-based method can be quite effective. If the burst of wind noise lasts about 20 to 40 ms, the output audio quality from the PLC-based method varies depending on whether the speech signal segment of primary signal P(f), corrupted by wind noise, is sufficiently stationary. If the burst of wind noise lasts more than about 40 to 60 ms, the PLC-based method tends to produce unnatural tonal distortion. Hence, if wind noise lasts 40 ms the output signal of PLC module 1215 should be ramped down toward zero or some other method should be used.
It should be noted that if the wind noise is only moderate so that it is still possible to estimate the pitch period and the signal gain from the wind-noise-corrupted portion of primary signal P(f), then PLC module 1215 can perform a modified version of the PLC-based method. In a traditional PLC method, the waveform of a frame is completely lost and the waveform extrapolation can only be based on the waveform in the previous frame(s). In the present invention, if it is determined that the wind noise is only moderate in primary signal P(f) and it is still possible to estimate some speech parameters (such as the pitch period and the signal gain) of the wind-noise-corrupted portion of primary signal P(f) with reasonable reliability, then PLC module 1215 can extrapolate the waveform of previous frame(s) using speech parameters estimated from the current frame of wind-noise-corrupted primary signal P(f). As long as these speech parameters can be estimated with reasonable reliability, this method should work better than the traditional PLC-based method, not only in that the audio quality will be better, but also in that the extrapolation of the waveform can likely be performed for a wind burst with longer duration.
Moreover, if one of the microphone signals (i.e., primary signal P(f) and reference signal R(f)) has a lesser degree of wind noise and can be used to estimate such speech parameters more reliably than using the other microphone signal, then the PLC operation can be performed by PLC module 1215 based on estimated speech parameters of the microphone signal with a lesser degree of wind noise.
If the wind noise in both microphone signals last longer than the PLC-based method can handle, control module 1205 can use weighted sum module 1220 to suppress wind noise in primary signal P(f) during the fourth wind noise scenario. Weighted sum module 1220 is specifically configured to weight primary signal P(f) and adjusted reference signal 1040 and then sum the weighted signals to suppress wind noise in primary signal P(f). The weights are assigned by weighted sum module 1220 such that the higher the relative energy of the microphone signal, the lower the weight. For example, in an ideal situation, let r be the estimated ratio of the wind noise intensity in primary signal P(f) over the wind noise intensity in adjusted reference signal 1040, then the weight for primary signal P(f) can be chosen as:
and the weight for the secondary microphone signal can be chosen as:
Such a weighted sum will tend to have an output signal biased toward the microphone signal that has a lesser degree of wind noise relative to the speech level. Thus, when the relative intensity of wind noise changes dynamically between the two signals (i.e., primary microphone signal P(f) and adjusted reference signal 1040), this weighted sum output signal will always and automatically “steer” toward the signal with a lesser degree of wind noise.
If the wind noise in the two signals is equally strong relative to the speech level in each, we have w1=w2=0.5. Even in this case, the weighted sum method still gets 3 dB of improvement in the signal-to-wind-noise ratio. This is because the wind noise in the two signals are generally uncorrelated, while the speech signals from the two signals are generally in phase and are in fact almost identical. Thus, after scaling the signals by 0.5 and adding them together, the speech component in the summed up signal stays essentially unchanged in the output signal. On the other hand, after scaling the signals by 0.5 and adding them together, the wind noise component is decreased by about 3 dB compared to the unchanged level of the speech component because the wind noise in the two signals are generally uncorrelated. Hence, there is a 3 dB improvement in the signal-to-wind-noise ratio after the weighted sum method is performed by weighted sum module 1220.
If the wind noise intensity ratio r is difficult to estimate reliably, weighted sum module 1220 can alternatively use the ratio of the energy values of primary signal P(f) and adjusted reference signal 1040 averaged over some frequency sub-bands as a rough substitute. However, in this case care should be taken to detect the condition when the noise reference microphone is covered, for example, by a user's hand or finger, which greatly reduces the level of adjusted reference signal 1040. If this situation is detected, the weighted sum method above can be bypassed to prevent the primary microphone signal from being wiped out.
Referring now to
The method of flowchart 1300 begins at step 1305 and transitions to step 1310. At step 1310, wind noise detection is performed on primary signal P(f) and reference signal R(f) using multiple methods. More specifically, at step 1310 wind noise detection is performed to detect the presence or absence of wind noise in primary signal P(f) using two or more wind noise detection methods and to detect the presence or absence of wind noise in reference signal R(f) using two or more wind noise detection methods. Each wind noise detection method produces a wind noise detection signal that indicates whether wind noise is present or absent. For example, one or more of the following methods can be performed to determine if primary signal P(f) or reference signal R(f) contain wind noise: spectral-deviation based wind noise detection, correlation based wind noise detection, average log gain difference based wind noise detection, and signal-to-matching wind noise detection. The resulting wind noise detection signals corresponding to primary signal P(f) are then combined to produce a single wind noise detection signal for primary signal P(f), and the resulting wind noise detection signals corresponding to reference signal P(f) are then combined to produce a single wind noise detection signal for reference signal R(f). Further details regarding wind noise detection using multiple methods were described above in regard to
At step 1315, a determination is made as to whether wind noise is absent or below a threshold in both primary signal P(f) and reference signal R(f), as indicated by the wind noise detection signals produced at step 1310. If wind noise is absent or below a threshold in both primary signal P(f) and reference signal R(f), flowchart 1300 proceeds to step 1320 where no wind noise reduction is performed on primary signal P(f). Otherwise, flowchart 1300 proceeds to step 1325.
At step 1325, a determination is made as to whether wind noise is present or above a threshold in reference signal R(f) and absent or below a threshold in primary signal P(f), as indicated by the wind noise detection signals produced at step 1310. If wind noise is present or above a threshold in reference signal R(f) and absent or below a threshold in primary signal P(f), flowchart 1300 proceeds to step 1330 where single channel noise suppression is performed using single channel noise suppression module 1225 as discussed above in regard to
At step 1335, a determination is made as to whether wind noise is present or above a threshold in primary signal P(f) and absent or below a threshold in reference signal R(f), as indicated by the wind noise detection signals produced at step 1310. If wind noise is present or above a threshold in primary signal P(f) and absent or below a threshold in reference signal R(f), flowchart 1300 proceeds to step 1340 where waveform substitution is performed using waveform substitution module 1210 as discussed above in regard to
At step 1345, it is assumed that wind noise is present or above a threshold in both primary signal P(f) and reference signal R(f). In this instance, when wind noise is present or above a threshold in both primary signal P(f) and reference signal R(f), PLC is performed using PLC module 1215 as discussed above in regard to
It will be apparent to persons skilled in the relevant art(s) that various elements and features of the present invention, as described herein, can be implemented in hardware using analog and/or digital circuits, in software, through the execution of instructions by one or more general purpose or special-purpose processors, or as a combination of hardware and software.
The following description of a general purpose computer system is provided for the sake of completeness. Embodiments of the present invention can be implemented in hardware, or as a combination of software and hardware. Consequently, embodiments of the invention may be implemented in the environment of a computer system or other processing system. An example of such a computer system 1400 is shown in
Computer system 1400 includes one or more processors, such as processor 1404. Processor 1404 can be a special purpose or a general purpose digital signal processor. Processor 1404 is connected to a communication infrastructure 1402 (for example, a bus or network). Various software implementations are described in terms of this exemplary computer system. After reading this description, it will become apparent to a person skilled in the relevant art(s) how to implement the invention using other computer systems and/or computer architectures.
Computer system 1400 also includes a main memory 1406, preferably random access memory (RAM), and may also include a secondary memory 1408. Secondary memory 1408 may include, for example, a hard disk drive 1410 and/or a removable storage drive 1412, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, or the like. Removable storage drive 1412 reads from and/or writes to a removable storage unit 1416 in a well-known manner Removable storage unit 1416 represents a floppy disk, magnetic tape, optical disk, or the like, which is read by and written to by removable storage drive 1412. As will be appreciated by persons skilled in the relevant art(s), removable storage unit 1416 includes a computer usable storage medium having stored therein computer software and/or data.
In alternative implementations, secondary memory 1408 may include other similar means for allowing computer programs or other instructions to be loaded into computer system 1400. Such means may include, for example, a removable storage unit 1418 and an interface 1414. Examples of such means may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM, or PROM) and associated socket, a thumb drive and USB port, and other removable storage units 1418 and interfaces 1414 which allow software and data to be transferred from removable storage unit 1418 to computer system 1400.
Computer system 1400 may also include a communications interface 1420. Communications interface 1420 allows software and data to be transferred between computer system 1400 and external devices. Examples of communications interface 1420 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, etc. Software and data transferred via communications interface 1420 are in the form of signals which may be electronic, electromagnetic, optical, or other signals capable of being received by communications interface 1420. These signals are provided to communications interface 1420 via a communications path 1422. Communications path 1422 carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link and other communications channels.
As used herein, the terms “computer program medium” and “computer readable medium” are used to generally refer to tangible storage media such as removable storage units 1416 and 1418 or a hard disk installed in hard disk drive 1410. These computer program products are means for providing software to computer system 1400.
Computer programs (also called computer control logic) are stored in main memory 1406 and/or secondary memory 1408. Computer programs may also be received via communications interface 1420. Such computer programs, when executed, enable the computer system 1400 to implement the present invention as discussed herein. In particular, the computer programs, when executed, enable processor 1404 to implement the processes of the present invention, such as any of the methods described herein. Accordingly, such computer programs represent controllers of the computer system 1400. Where the invention is implemented using software, the software may be stored in a computer program product and loaded into computer system 1400 using removable storage drive 1412, interface 1414, or communications interface 1420.
In another embodiment, features of the invention are implemented primarily in hardware using, for example, hardware components such as application-specific integrated circuits (ASICs) and gate arrays. Implementation of a hardware state machine so as to perform the functions described herein will also be apparent to persons skilled in the relevant art(s).
The present invention has been described above with the aid of functional building blocks illustrating the implementation of specified functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed.
In addition, while various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be understood by those skilled in the relevant art(s) that various changes in form and details can be made to the embodiments described herein without departing from the spirit and scope of the invention as defined in the appended claims. Accordingly, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.
This application claims the benefit of U.S. Provisional Patent Application No. 61/413,231, filed on Nov. 12, 2010, which is incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 61413231 | Nov 2010 | US |
