This application claims priority under 35 USC §119 or §365 to Great Britain Patent Application No. 1401689.3 entitled “Audio Signal Processing” filed Jan. 31, 2014 by Karsten Vandborg Sorensen the disclosure of which is incorporate in its entirety.
Audio signal processing refers to the intentional altering of an audio signal to achieve a desired effect. It may occur in the analogue domain, digital domain or a combination of both and may be implemented, for instance, by a generic processor running audio processing code, specialized processors such as digital signal processors having architectures tailored to such processing, or dedicated audio signal processing hardware. For example, audio captured by a microphone of a user device may be processed prior to and/or following transmission over a communication network as part of a voice or video call.
An audio signal may be processed by an audio processing chain comprising a plurality of audio signal processing components (hardware and/or software) connected in series; that is whereby each component of the chain applies a particular type of audio signal processing (such as gain, dynamic range compression, echo cancellation etc.) to an input signal and supplies that processed signal to the next component in the chain for further processing, other than the first and last components which receive as an input an initial analogue audio signal (e.g. a substantially unprocessed or ‘raw’ audio signal as captured from a microphone or similar) and supply a final output of the chain (e.g. for supplying to a loudspeaker for play-out or communication network for transmission) respectively. Thus variations in processing by one component in the chain can cause variations in the output of subsequent components in the chain.
One type of audio processing component that may be used in such a chain is a noise suppression component. The audio signal may comprise a desired audio component but also an undesired noise component; the noise suppression component aims to suppress the undesired noise component whilst retaining the desired audio component. For instance, an audio signal captured by a microphone of a user device may capture a user's speech in a room, which constitutes the desired component in this instance. However, it may also capture undesired background noise originating from, say, cooling fans, environmental systems, background music etc.; it may also capture undesired signals originating from a loudspeaker of the user device for example received from another user device via a communication network during a call with another user conducted using a communication client application, or being output by other applications executed on the user device such as media applications—these various undesired signals can all contribute to the undesired noise component of the audio signal.
Disclosed is an audio signal processing device comprising an input for receiving a noisy audio signal, a variable gain component and a noise suppression component. The noisy audio signal has a desired audio component and a noise component. The variable gain component and the noise suppression component are respectively configured to apply a gain and a noise suppression procedure to the audio signal, thereby generating a gain adjusted noise reduced audio signal. The aggressiveness of the noise suppression procedure is rapidly changed responsive to a change in the applied gain. That change is a change from a current value by an amount substantially matching the change in applied gain to a new value. The aggressiveness is then gradually returned to the current value.
An equivalent method and computer program product configured to implement that method are also disclosed.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Nor is the claimed subject matter limited to implementations that solve any or all of the disadvantages noted in the Background section.
For a better understanding of the present subject matter and to show how the same may be carried into effect, reference will now be made by way of example to the accompanying drawings in which:
The present disclosure considers a situation in which a variable gain component and a noise suppression (noise reduction) component are connected in series and are respectively configured to receive and process a noisy audio signal (e.g. a microphone signal) having a desired audio component (e.g. a speech signal) and a noise component (e.g. background noise). The variable gain component is configured to apply a changeable gain to its input. It may, for instance be an automatic gain component configured to automatically adjust the applied gain in order to maintain a desired average signal level (automatic gain control being known in the art) or a manual gain component configured to adjust the applied gain in response to suitable a user input. The noise suppression component is configured to apply a noise suppression procedure to its input in order to suppress the noise component of the audio signal e.g. by applying a spectral subtraction technique whereby the noise component is estimated during periods of speech inactivity, and a noise reduced signal is estimated from the noisy audio signal using the noise component estimate (spectral subtraction being known in the art). The noise suppression component and the variable gain component constitute a signal processing chain configured to generate a gain adjusted estimate of the desired audio component.
In order to improve perceptual quality, the noise suppression procedure may be configured such that the level of the noise component is attenuated relative to the original noisy signal but intentionally not removed in its entirety (even if the estimate of the noise component is near-perfect). That is, such that a noise component is always maintained in the noise reduced signal estimate albeit at a level which is reduced relative to the noisy audio signal such that a ‘fully’ clean signal is intentionally not output.
Whilst this does have the effect of improving perceptual quality, an unintended consequence is that a change in the gain applied by the variable gain component causes a noticeable change in the level of the noise component remaining in the noise reduced signal estimate; this can be annoying for a user.
In accordance with the present subject matter, the noise suppression component is configured to be responsive to such a change in the gain applied by the variable gain component in a way that makes this change more transparent (that is less noticeable) to the user. To an extent, the disclosed subject matter is about “decoupling” the respective changes in level of the desired audio component and the noise component, thereby enabling one gain adaptation speed for changing the desired signal level, and another for changing the noise level. Before describing particular embodiments, a context in which the subject matter can be usefully applied will be described.
The user device 104 executes an instance of a communication client, provided by a software provider associated with the communication system 100. The communication client is a software program executed on a local processor in the user device 104. The client performs the processing required at the user device 104 in order for the user device 104 to transmit and receive data over the communication system 100.
The user device 110 corresponds to the user device 104 and executes, on a local processor, a communication client which corresponds to the communication client executed at the user device 104. The client at the user device 110 performs the processing required to allow the user 108 to communicate over the network 106 in the same way that the client at the user device 104 performs the processing required to allow the user 102 to communicate over the network 106. The user devices 104 and 110 are endpoints in the communication system 100.
The audio signal captured by the microphone of the first user device 104 is transmitted over the network 106 for playing out by the second user device 110 e.g. as part of an audio or video call conducted between the first and second users 102, 108 using the first and second user devices 104, 110 respectively.
With reference to
As shown in
The system 300 comprises a noise suppression component 312 and a variable gain component 302. The variable gain component 302 has a first input which is connected to an output of the noise reduction component 312, a second input connected to receive a gain factor Gvar(k) and an output connected to supply a processed audio signal for further processing, including packetization, at the first user device 104 before transmission to the second user device 108 over the network 106 (e.g. as part of a voice or video call). The noise suppression component 312 has a first input connected to receive the microphone signal y(t)—having a desired audio component s(t) and a noise component n(t)—from the microphone 212, and a second input connected to receive the gain factor Gvar(k). The noise reduction component 312 and variable gain component 302 are thus connected in series and constitute a signal processing chain, the first input of the noise reduction component being an input of the chain and the output of the variable gain component being an output of the chain.
The microphone 212 is shown as supplying the microphone signal to the signal processing chain directly for the sake of convenience. As will be appreciated, the microphone may in fact supply the microphone signal y(t) via other signal processing components such as analogue-to-digital converter components.
The variable gain component 302 component applies an amount of gain defined by the gain factor Gvar(k) to its first input signal to generate a gain adjusted signal. The noise suppression component applies a noise suppression procedure to its first input signal to generate an estimate of the desired audio component thereof. This is described in detail below.
The noise suppression minimum gain factor calculation component 404 has an input connected to receive the gain factor Gvar(k), and an output connected to a first input of the noise suppression gain factor calculation component 406. The noise suppression gain factor calculation component 406 also has a second input connected to receive a noise signal power estimate |Nest(k,f)|2 and a third input connected to the output of the Fourier transform component 408.
Audio signal processing is performed by the system 300 on a per-frame basis, each frame k, k+1, k+2 . . . being e.g. between 5 ms and 20 ms in length. The variable gain component 302 and the noise suppression component 312 each receive respective input audio signals as a plurality of input sequential audio frames and provide respective output signals as a plurality of output sequential audio frames.
The Fourier transform component 408 performs a discrete Fourier transform operation on each audio frame k to calculate a spectrum Y(k,f) for that frame. The spectrum Y(k,f) can be considered a representation of a frame k of the microphone signal y(t) in the frequency domain. The spectrum Y(k,f) is in the form of a set of spectral bins e.g. between 64 and 256 bins per frame, with each bin containing information about a signal component at a certain frequency (that is in a certain frequency band). For dealing with wideband signals, a frequency range from e.g. 0 to 8 kHz may be processed, divided into e.g. 64 or 32 frequency bands. The bands may or may not be of equal width—they could for instance be adjusted in accordance with the Bark scale to better reflect critical bands of human hearing.
The noise suppression minimum gain factor calculation component 404 calculates, on a per-frame k basis, a noise suppression minimum gain factor Gmin(k) which is supplied to the noise reduction gain factor calculation component 406. The noise reduction gain factor calculation component 406 calculates, on a per-frame k basis, a noise suppression gain factor Glimited(k,f) which is supplied to the noise reduced signal calculation component 402. The noise reduced signal calculation component 402 calculates a frequency-domain noise reduced signal estimate Ynr(k,f) which is supplied to the variable gain component 302. The noise reduced signal estimate Ynr (k,f) for a frame k is calculated by adjusting the spectrum Y(k,f) for that frame by an amount specified by the noise suppression gain factor Glimited(k,f); that is, by applying a frequency-dependent gain Glimited(k,f) across the spectrum Y(k,f) to reduce the contribution of the noise component n(t) to the spectrum of the microphone signal y(t) relative to that of the desired audio component s(t).
The inverse Fourier transform component performs an inverse discrete Fourier transform operation on the frequency-domain noise reduced signal estimate Ynr (k,f)—that operation being the inverse of the Fourier transform operation performed by the Fourier transform component 408—to calculate a time-domain noise reduced signal estimate ynr (t). The noise component n(t) is still (intentionally) present in the noise reduced signal ynr (t) but at a lower level than in the noisy microphone signal y(t). The noise reduced signal estimate is provided by the noise suppression component as a plurality of sequential clean-signal-estimate audio frames. The Fourier transform and inverse Fourier transform operations could, in practice, be implemented as fast Fourier transform operations.
The functionality and interaction of these noise suppression components will be described in more detail below.
The variable gain component 302 performs a gain adjustment of the noise reduced signal ynr(t) to generate a gain adjusted audio signal by applying, to each frame k, an amount of gain defined by the variable gain factor Gvar(k) to that frame k of the time-domain noise reduced signal estimate ynr (t). The gain adjusted audio signal is provided by the variable gain component as a plurality of sequential gain-adjusted-signal audio frames. Alternatively, the inverse Fourier transform may be disposed after the variable gain component 302 in the system 300 such that the gain adjustment is performed in the frequency domain rather than the time domain.
The gain factor Gvar(k) may vary between frames and, in embodiments, may also vary inside a frame (from sample-to-sample). For instance, Gvar(k) may be varied inside a frame by smoothing approaching a corrected value.
Alternatively, the positions of the variable gain component 302 and the noise reduction component 312 may be reversed relative to their arrangement as depicted in
The signal processing chain may also comprise other signal processing components (not shown), connected before, after and/or in between the noise reduction component 312 and the variable gain component 302. That is, the signal processing functionality implemented by executing communication client application 206 may include more signal processing functionality than that shown in
The aggregate functionality of the noise reduction component and the variable gain component is to apply, as part of the signal processing method, a combination of a gain and a noise reduction procedure to the noisy audio signal y(t) thereby generating a gain adjusted, noise reduced audio signal having a noise-to-signal power ratio which is reduced relative to the noisy audio signal y(t). This is true irrespective of their order and/or disposition in the signal processing chain (that is, irrespective of the temporal order in which the gain and the noise suppression procedure are applied in series relative to one another and/or relative to any other audio signal processing if performed on the audio signal in series with the application of the gain and noise suppression).
The audio signal processing method will now be described in detail with reference to
The method involves adjusting the aggressiveness of the noise suppression procedure to apply more noise reduction immediately following a gain increase (and the opposite for a decrease) and then slowly returning to ‘regular’ aggressiveness afterwards, ‘regular’ aggressiveness being a level of aggressiveness which is chosen to optimize the perceptual quality of the noise suppression procedure. Here, the “aggressiveness” of the noise suppression procedure is a measure of the extent to which the contribution of the noise component to overall signal level is reduced by the noise suppression procedure and can be quantified, for instance, as an amount by which signal power of the noise component is reduced relative to that of the desired audio component by the noise suppression procedure. Typically, the ‘regular’ aggressiveness will be set so as to ensure that some noise always remains after noise reduction albeit at a level which is reduced relative to that prior to noise reduction, rather than being completely removed—as discussed above, this is for reasons of enhanced perceptual quality.
The aggressiveness of the noise suppression procedure is changed by an amount substantially matching the change in applied gain. Matching the change in the aggressiveness of the noise suppression to the change in applied gain counteracts the effect that the change in applied gain would otherwise have on the level of the noise component remaining in the noise reduced signal estimate (i.e. prevents a ‘jump’ in the level of the remaining noise that would otherwise occur due to the ‘jump’ in applied gain) such that, immediately following the change in applied gain, the level of the noise remaining in the noise reduced signal estimate is substantially unchanged despite the change in the applied gain, with the applied gain thereby acting only to change the level of the desired audio component as intended and not the level of the noise component immediately following the change in applied gain.
It is still desirable to eventually return the aggressiveness to the ‘regular’ level to retain optimal perceptual quality, which will almost certainly cause a change in the level of the noise remaining in the signal estimate; however, making the change in the aggressiveness a gradual change ensures that this noise level change is also a gradual, rather than rapid, change. The level of the audible noise that remains in the gain adjusted noise reduced signal estimate after noise suppression thus varies more slowly than it otherwise would, making the adjustment of the gain less noticeable to the user while preserving the desired adjustment of the desired audio component.
Background noise reduction (BNR)—including, but not limited to, power spectral subtraction and other forms of spectral subtraction such as magnitude spectral subtraction—often applies a noise reduction limit or “target” which limits the extent of the noise reduction that can be applied to the noisy audio signal in order to generate a noise reduced signal estimate (that is, which restricts the amount by which the magnitude or power of the noise component can be reduced by the noise suppression procedure). In this case, the limit sets the aggressiveness of the noise reduction, thus the aggressiveness can be adjusted by adjusting this limit. Often, this limit can be expressed as a minimum gain or maximum attenuation (these being the multiplicative inverse of one another when expressed as a ratio of a signal to a gain adjusted signal and the additive inverse of one another when expressed on a logarithmic scale such as dB) that can be applied to the noisy audio signal at any given time for the purposes of reducing the power or magnitude of the noise component. A lower attenuation (greater gain) limit causes less aggressive noise suppression and a greater attenuation (lower gain) limit causes more aggressive noise suppressions. The limit may take a constant value of e.g. 12 dB of attenuation (−12 dB of gain), 12 dB being the maximum permissible noise suppression attenuation (−12 dB being the minimum permissible noise suppression gain) that can be applied to the noisy audio signal to generate a noise reduced signal estimate. Choosing a non-zero limit ensures that the noise component always remains in the noise reduced signal estimate albeit at a reduced level relative to the original noisy audio signal, rather than being completely removed (discussed above). 12 dB is widely recognized as a good trade-off between noise reduction and speech distortion—for comparison, e.g., 18 dB would be considered to be slightly aggressive, and would in extreme cases lead to audible speech distortion.
In embodiments, it is this noise reduction attenuation limit/target that is rapidly increased (resp. decreased) from a current value (e.g. 12 dB) by substantially the same amount as the gain has been increased (resp. decreased) by, and then gradually returned to that current value (e.g. 12 dB). For example, in response an increase (resp. decrease) in the applied gain of 3 dB, the noise reduction attenuation limit might be immediately changed to 12 db+3 db=15 dB (resp. 12 dB−3 dB=9 dB), and then gradually returned to 12 dB.
At step S502, the client 206 receives the noisy audio signal y(t) having the desired audio component s(t) and the noise component n(t) from the microphone 212. The noisy audio signal y(t) can be considered a sum of the noise component n(t) and the desired component s(t). Here, the desired component s(t) is a speech signal originating with the user 102; the noise signal n(t) may comprise background noise signals and/or undesired audio signals output from the loudspeaker 210 as discussed above.
At step S504, the noise suppression component 312 applies a noise suppression procedure to the audio signal y(t). In this embodiment, the noise suppression component applies a type of power spectral subtraction. Spectral subtraction is known in the art and involves estimating a power of the noise component n(t) during periods of speech inactivity (i.e. when only the noise component n(t) is present in the microphone signal y(t)). A noise signal power estimate |Nest(k,f)|2 for a frame k may, for example, be calculated recursively during periods of speech inactivity (as detected using a known voice activity detection procedure) as
|Nest(k,f)|2=b*|Nest(k−1,f)|2+(1−b)*|Y(k,f)|2
where b is a suitable decay factor between 0 and 1. That is, as the noise signal power estimate |Nest(k−1, f)|2 of the frame k−1 updated by a calculated signal power |Y(k, f)|2 of the next adjacent frame k (calculated as the square of the magnitude of the spectrum Y(k, f) for frame k).
The noise component n(t) is (partially) suppressed in the audio signal y(t) by the noise reduced signal calculation component 402 applying to the audio signal spectrum Y(k,f) an amount of gain as defined by the noise suppression gain factor Glimited(k,f), as follows:
|Ynr(k,f)|2=Glimited(k,f)2*|Y(k,f)|2
That is, a noise reduced signal power estimate |Ynr(k, f)|2 is obtained by multiplying the squared noise suppression gain factor Glimited(k,f) with the signal power |Y(k, f)|2 of the noisy audio signal y(t) (noise suppression gain thus being applied in the magnitude domain). Phase information for the original frame k is retained and can be used to obtain the noise reduced signal estimate Ynr(k, f) (that is, a noise reduced signal spectrum for frame k) from the power estimate |Ynr(k, f)|2. The time-domain noise reduced signal estimate ynr (t) is calculated by the inverse Fourier transform component 410 performing the inverse Fourier transform on the frequency domain noise reduced signal estimates (i.e. noise reduced signal spectra) for each frame in sequence.
An unlimited noise suppression gain factor Gunlimited(k,f) is calculated by the noise suppression gain factor component 406 as:
The noise suppression gain factor Glimited(k,f) is calculated as:
Glimited(k,f)=max[Gunlimited(k,f),Gmin(k)].
That is, as a maximum of the unlimited noise suppression gain factor Gunlimited(k,t) and the noise suppression minimum gain factor Gmin(k). The unlimited noise suppression gain factor thus is applied to a frame k only to the extent that it is above the noise suppression minimum gain factor Gmin(k) for that frame k. Decreasing the lower gain limit Gmin (k) for a frame k increases the aggressiveness of the noise suppression procedure for that frame k as it permits a greater amount of noise signal attenuation; increasing the lower gain limit Gmin (k) decreases the aggressiveness of the noise reduction procedure for that frame k as it permits a lesser amount of noise signal attenuation.
In the absence of other considerations a lower limit of, say, −12 dB may be favoured in order to improve perceptual quality and, in known spectral subtraction techniques, the lower limit is typically fixed at around that value for this reason. In contrast, here, the lower limit Gmin (k) may vary from frame to frame (and, in embodiments, within a given frame—see below)—that is, the aggressiveness of the noise suppression procedure may vary from frame to frame (or within a given frame)—as required in order to track any changes in the gain applied by the variable gain component for reasons discussed above and in a manner that will be described in detail below.
At step S506, an amount of gain defined by the gain factor Gvar(k) is applied to the noise reduced signal estimate s(t) by the variable gain component 302. This applied gain can vary from one frame to the next frame (and as discussed may also vary within a given frame). The gain factor Gvar(k) is varied automatically as part of an automatic gain control (AGC) process such that the average or peak output of the noise reduced signal estimate s(t) is automatically adjusted to a desired level e.g. to maintain a substantially constant peak or average level even in the presence of signal variations. The automatic gain control process may, for instance, be employed throughout a voice or video call with the applied gain thus changing at points in time during the call. Alternatively or additionally, the gain factor Gvar(k) may be varied manually in response to a user input e.g. the user 102 electing to adjust their microphone level.
In this embodiment, the gain factor Gvar(k) varies from an initial value Gvar,initial as a to a new target value Gvar,target. The variation from the initial value to the target value is a smooth variation in that the gain factor Gvar(k) varies from the initial value to the target value as a first (steep) function of time having a first time constant τ1. The time constant τ1 is a time it takes for the applied gain to change from the initial value Gvar,initial by (1−1/e)≈63% of the total amount Δ1 by which the applied gain eventually changes (i.e. Δ1=Gvar,target−Gvar,initial—that is a difference between the target value and the initial value); that is, τ1 is the time it takes for the applied gain to change from Gvar,initial to Gvar,initial+Δ1*(1−1/e). This may be effected, for instance, by first order recursive smoothing of Gvar(k) from the initial value to the target value by updating the applied gain Gvar(k) as per equation 1, below:
Gvar(k)=Gvar,target+d*[Gvar(k−1)−Gvar,target]
where 0<d<1 is a smoothing parameter which determines the first time constant τ1. When the gain factor Gvar(k) is smoothed as per equation 1, the gain factor changes exponentially towards the target Gvar,target as Gvar,target−Δ1*e−(t−t
Whilst smooth, the chance in the applied gain from the initial value to the target value is nonetheless a rapid change in that the first time constant has a value of around 50-250 ms (which can be achieved by setting the smoothing parameter d in equation 1 accordingly). In other words, a variable gain ‘target’ changes instantly (e.g. as a step function) to the new target value of Gvar,target, and the applied gain Gvar(k) follows the gain target, rapidly but nonetheless smoothly moving towards the new target value in a short amount of time (that amount of time being dependent on both the first time constant τ1 and the amount Δ1 by which the applied gain changes). It is undesirable for the noise level to change this fast, particularly if the applied gain change is large (as this would result in a corresponding large, rapid change in the noise level).
Exemplary variations in Gvar (k) are illustrated in graph 600 of
The noise suppression minimum gain factor Gmin (k) as used for a frame k is calculated (updated) in the linear domain as per equation 2, below:
with c being a smoothing factor between 0 and 1. Thus, for example, if the applied gain Gvar(k) is doubled (resp. halved), the noise suppression lower-limit Gmin(k) is halved (resp. doubled) in order to match the effect of doubling (resp. halving) the gain factor Gmin(k).
That is, for as long as the applied gain Gvar(k) is varying, the changes in the applied gain are matched by changing the noise suppression minimum gain from a current value (Gmin)) to a new value Gnew, the new value Gnew being the value the noise suppression lower limit reaches when the applied gain levels off—e.g. at frame “k+3” in
The change in the noise suppression lower limit thus tracks the change in the applied gain such that the change in the applied gain and the change in the noise suppression aggressiveness from the current value to the new value are both rapid and have substantially the same duration.
The term c*[Gmin (k−1)−Gmin] in the above equation 2 is a first order recursive smoothing term effecting first order recursive smoothing. For as long as the applied gain remains constant from frame to frame following a change (i.e. as long as the gain Gvar (k−1) applied to the current frame k−1 remains equal to the gain Gvar (k) applied to the next adjacent frame k), the first order recursive smoothing acts to gradually return the noise suppression minimum gain factor to a constant level of Gmin. Thus, following a change in the applied gain which causes a corresponding and rapid change in the nose suppression minimum gain, the noise suppression minimum gain (and hence the aggressiveness of the noise suppression procedure) is gradually returned to the constant level Gmin. This corresponds to step S510 of
This Gmin value is chosen as a lower limit which would optimise perceptual quality in the absence of any changes in the gain Gvar (k) applied by the variable gain component 302. The constant Gmin may, for instance, take a value of −12 dB or thereabouts (that is, an attenuation of +12 dB or thereabouts).
The smoothing factor c is chosen to effect a gradual return to the constant level Gmin. That is, such that the noise suppression lower limit Gmin (k) varies as a second function of time (substantially shallower than the first function of time) having a second time constant τ2 which is substantially longer than that of the preceding rapid change in the noise suppression lower limit, the second time constant τ2 being around e.g. 10-40 seconds (>>τ1≈50-250 ms) such that it takes around 10-40 seconds for Gmin(k) to change by (1−1/e)≈63% of a difference Δ2=Gmin−Gnew between the constant value Gmin and the new value Gnew (the total change in aggressiveness) i.e. such that it takes τ2≈10-40 seconds for Gmin(k) to change from Gnew to Gnew+Δ2*(1−1/e). When the noise suppression minimum gain Gmin (k) is smoothed as per line 2 of equation 2, the gain factor returns exponentially towards the constant Gmin as Gmin−Δ2*e−(t−t′
During this time, the level of the noise component remaining in the noise reduced signal estimate ynr (t) will vary, but will do so gradually due to the gradual change in Gmin (k) and will thus be less noticeable to the user.
The rapid change in the applied gain (which has substantially the same duration as the rapid change in aggressiveness) is thus faster than the subsequent gradual return by a factor of about τ2/τ1—that is, the applied gain (partially) changes by a fraction 0<p<1 (i.e. a percentage 0%<p %<100%) of the total change in applied gain (i.e. changes from the initial value Gvar,initial to an intermediate gain value Gvar,initial+Δ1*p) over a first time interval T1 and the aggressiveness of the noise suppression procedure (partially) changes by that same fraction p but of the total change in aggressiveness (i.e. changes from the new value Gnew to an intermediate aggressiveness value Gnew+Δ2*p) over a second time interval T2, the second interval T2 being longer than the first time interval T1 by a factor of τ2/τ1 (i.e. T2=(τ2/τ1)*T1≧approx. 40). This is true for different values of p in the range (0,1) (i.e. for different percentages in the range (0%, 100%) e.g. 1%, 5%, 10%, 20%, 50%, 70%, 90% etc.). This is illustrated in
As the gradual return of the noise suppression aggressiveness has a second time constant τ2 of no less than about 10 seconds and the rapid change in the noise suppression aggressiveness has a first time constant τ1 of no greater than about 250 ms=0.25 seconds,
—that is, the second interval is longer than the first interval by at least a factor of about 40.
The time it takes a first-order auto-regressive smoother (with exponential output after changes)—e.g. as effected by equation 1 or line 2 of equation 2—to approach the input value by a certain relative amount (p %) will only depend on the time constant (τ1, τ2) defined by the filter coefficient (smoothing parameter b, c) and not the size of the change (in gain/aggressiveness). The time constant (τ1, τ2) is how the convergence time of a first order smoother is usually described; that is the smoother of equation 1 has a convergence time of the first time constant τ1 and the smoother of equation 2, line 2 has a convergence time of the second time constant τ2 substantially longer than the first (by at least a factor of about 40).
From a strict mathematical point of view, the first and second functions would, if left ‘unchecked’, take an infinite amount of time to converge to the target gain value Gvar,target and the constant noise suppression minimum level Gmin respectively (which are asymptotic values). This will of course not be the case in reality e.g. due to rounding errors. That it strictly speaking takes infinite amount of time to reach the input value, is of negligible importance—this is acceptable (as are rounding errors making the convergence happen earlier), and the output of the smoother is kept ‘on-track’ by the input regardless.
The aggressiveness is changed from the initial value to substantially the current value over a first (finite) duration (Δt1 in
In general, the first duration is sufficiently short to counteract the effect that the change in applied gain would otherwise have on the noise level, and the second duration is sufficiently long to ensure that the eventual change in the nose level is perceptibly slower than it would otherwise be as a result of the change in applied gain.
As an example, if the applied gain is increased by 3 dB, the noise suppression component 312 would be applying 15 dB of noise suppression rapidly afterwards (that is the applied noise suppression gain lower limited by −15 dB), gradually and smoothly returning to a less aggressive suppression of e.g. 12 dB over the next 20 seconds or so. Conversely, if the applied gain is decreased by 3 dB, the noise suppression component 312 would be applying 9 dB of noise suppression (that is the applied noise suppression gain lower limited by −9 dB), gradually and smoothly returning to a more aggressive suppression of e.g. 12 dB over the next 20 seconds or so.
In practice, it may be desirable for frames k, k+1, k+2 . . . to overlap to some extent. This overlap may, for instance, be of order 25% to 50% of the frame length (which may around 5 ms to 20 ms) which means an overlap of order 1.25 ms-10 ms. That is, the audio signal y(t) is segmented into audio frames such that an initial portion of audio in frame k is replicated as a final portion of the next frame k+1 etc.—this is illustrated in
Whilst in the above, the change in applied gain is a ‘smooth’ change, in principle the applied gain could be changed as a step function from one frame to the next adjacent frame. In this case, when the applied gain factor Gvar (k) is changed from one frame to the next as a step function, a consequence of the frame overlap is to nonetheless effectively ‘smooth’ this step function such that the applied gain effectively varies substantially continuously from an initial value to a target value over an interval of time equal to the frame overlap (of order 1 ms-10 ms), as illustrated in
As used herein, the phrase “changing the aggressiveness of the noise suppression procedure by an amount substantially matching the change in the applied gain” (or similar) is used to mean that the change in aggressiveness matches (i.e. counteracts) the effect of the change in applied gain on the noise component (more specifically, when the change in aggressiveness substantially counteracts the effect of the change in applied gain on the level of the noise component such that the level of the noise component in the noise reduced signal is substantially unchanged immediately following the change in applied gain).
This does not necessarily mean that there is any one particular numerical relationship between the magnitudes of the changes and, in particular, does not necessarily mean that the respective magnitudes of the changes are equal (this may or may not be the case). For instance, a change of 1 dB in the applied gain from 1 dB to 2 dB could be matched by changing the noise suppression aggressiveness by −1 dB (e.g. from −12 dB to −13 dB)—in this case the effect of the applied gain change is matched by an aggressiveness change of equal magnitude in dB. However, a change of in the applied gain from 1 to 2 in the linear domain (which is a change of 2−1=1 in the linear domain) could be matched by changing the noise suppression aggressiveness from e.g. 0.25 to ½*0.25=0.125 in the linear domain (which is a change of 0.25−0.125=0.125 in the linear domain)—in this case the effect of the applied gain change is matched by an aggressiveness change which is not equal in magnitude to the change in applied gain. Further, the applied gain could in principle be implemented in one domain (e.g. linear domain or logarithmic domain) and the noise suppression could be implemented in a different domain (e.g. logarithmic domain or linear domain) in which cast the respective changes in in the different domains are unlikely to be equal in magnitude. That is, the change in the aggressiveness substantially matches the change in applied gain when the effect of the former is matched by the latter regardless of the respective domains in which the gain and noise suppression procedure are applied.
Whilst in the above-described method of
Moreover, whilst in the above, the gain and noise suppression component are connected in series, it is envisaged that a similar effect could be achieved by gain/noise suppression components connected in parallel i.e. with at least one gain component and at least one noise suppression component each acting ‘directly’ on the noisy audio signal—rather than one acting on the output of the other—to generate separate respective outputs which are then aggregated e.g. as a (possibly weighted) sum to provide a final output audio signal.
Moreover, whilst in the above the disclosed technique is applied to a near-end signal prior to transmission over a communication network to a far-end user, alternatively or additionally the disclosed techniques may be applied to a far-end signal received over the communication network from the far-end user e.g. before being output from a near-end loudspeaker (e.g. 210). That is, an equivalent signal processing chain may perform equivalent processing on an audio signal received from the network 106 before it is output via speaker 210 as an alternative or addition to a signal processing chain performing audios signal processing on an audio signal received from the microphone 212 of device 300 before it is transmitted via network 106. Thus, a signal processing chain may have an input connected to receive an audio signal received via the network 106 from the second user device 108 and an output connected to supply a processed audio signal to the loudspeaker 210 of device 104.
Further, whilst in the above, the aggressiveness of a noise suppression procedure is rapidly changed from a current value to a new value responsive to a change in applied gain, then gradually returned to the current value by first order recursive smoothing, this gradual return can be effected by any number of alternative means. For instance, the gradual change could be a linear change back to the current value with the current value being reached e.g. 10-40 seconds after the change in applied gain, or higher-order recursive smoothing could be employed to effect the gradual return. Similarly, the rapid change in applied gain could be a linear change from the initial value to the target value over a duration of e.g. about 50-250 ms, or higher order recursive smoothing could be employed to effect the rapid change.
The noisy audio signal may be received as a plurality of (discrete) portions (e.g. audio frames or audio samples) and the aggressiveness and gain may be updated at most per portion (i.e. new values thereof may be calculated at most per portion with one calculated value being used for the entirety of a given portion).
Further, whilst in the above, the subject matter is described in the context of a real-time communication system, it will be appreciated that the disclosed techniques can be employed in many other contexts, both in relation to ‘live’ and pre-recorded noisy audio signals. Further, whilst in the above the subject matter is implemented by an audio signal processing device in the form of user device (such as a personal computer, laptop, tablet, smartphone etc.), in alternative embodiments the subject matter could be implemented by any form of audio signal processing device such as a dedicated audio signal processing device e.g. an audio effects unit, rack-mounted or otherwise.
Generally, any of the functions described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), or a combination of these implementations. The terms “module,” “functionality,” “component” and “logic” as used herein generally represent software, firmware, hardware, or a combination thereof. This includes, for example, the components of
For example, the user devices may also include an entity (e.g. software) that causes hardware of the user devices to perform operations, e.g., processors functional blocks, and so on. For example, the user devices may include a computer-readable medium that may be configured to maintain instructions that cause the user devices, and more particularly the operating system and associated hardware of the user devices to perform operations. Thus, the instructions function to configure the operating system and associated hardware to perform the operations and in this way result in transformation of the operating system and associated hardware to perform functions. The instructions may be provided by the computer-readable medium to the user devices through a variety of different configurations.
One such configuration of a computer-readable medium is signal bearing medium and thus is configured to transmit the instructions (e.g. as a carrier wave) to the computing device, such as via a network. The computer-readable medium may also be configured as a computer-readable storage medium and thus is not a signal bearing medium. Examples of a computer-readable storage medium include a random-access memory (RAM), read-only memory (ROM), an optical disc, flash memory, hard disk memory, and other memory devices that may us magnetic, optical, and other techniques to store instructions and other data.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
Number | Name | Date | Kind |
---|---|---|---|
6563931 | Soli | May 2003 | B1 |
7016507 | Brennan | Mar 2006 | B1 |
7454332 | Koishida et al. | Nov 2008 | B2 |
7555075 | Pessoa et al. | Jun 2009 | B2 |
8185389 | Yu et al. | May 2012 | B2 |
8538763 | Yu | Sep 2013 | B2 |
20070237271 | Pessoa | Oct 2007 | A1 |
20080147387 | Matsubara et al. | Jun 2008 | A1 |
20090010453 | Zurek et al. | Jan 2009 | A1 |
20090304191 | Hoang Co Thuy | Dec 2009 | A1 |
20090313009 | Kovesi | Dec 2009 | A1 |
20110013792 | Iwano | Jan 2011 | A1 |
20130262101 | Srinivasan | Oct 2013 | A1 |
20140211965 | Hetherington | Jul 2014 | A1 |
Number | Date | Country |
---|---|---|
1211671 | Jun 2002 | EP |
Entry |
---|
Rosenstrauch, “Sound Connections with Environmental Optimizer II”, Available at <http://www.resound.com/˜/media/DownloadLibrary/ReSound/White%20papers/environmental-optimizer-II-white-paper.pdf>, Dec. 17, 2013, 5 pages. |
“International Search Report and Written Opinion”, Application No. PCT/US2015/013158, dated Apr. 24, 2015, 9 Pages. |
“International Preliminary Report on Patentability”, Application No. PCT/US2015/013158, dated Feb. 16, 2016, 5 pages. |
Number | Date | Country | |
---|---|---|---|
20150222988 A1 | Aug 2015 | US |