VOICE OPTIMIZATION IN NOISY ENVIRONMENTS

TECHNICAL FIELD

The present invention relates to voice optimization of audio and more precisely to a method and a device for providing voice optimization in noisy environments.

BACKGROUND

Portable electronic equipment is used by virtually everybody, everywhere. For instance, a mobile phone is carried at all times and can be used to make calls or listen to audio. The audio listened to may be music, but podcasts and audiobooks are becoming increasingly common. With the increased use and portability of electronic devices for communication and entertainment, the risk that the audio is consumed in a noisy environment is increased. With music, a noisy environment may be nothing more than a nuisance, but when it comes to listening to speech audio, the noisy environment may make the speech unintelligible over the noise.

Speech intelligibility of speech audio will depend on the signal to noise ratio, in this case, the ratio between the speech audio and the noise. Historically, speech intelligibility is improved by modifying the signal to noise ratio. The brute force approach is to amplify the voice signal such that it is intelligible over the noise, needless to say, this approach may cause damage to the hearing of the person listening to the speech audio. Another approach is, if headphones are used, to decrease the noise by forming the headphones to attenuate outside noise or to utilizing active noise cancellation. The noise attenuation will depend on acoustic design and the fitting of the headphones on the user. The active noise cancellation requires significant processing power with increased material cost and energy consumption as a result.

From the above, it is understood that there is room for improvements.

SUMMARY

An object of the present invention is to provide a new type of voice optimization which is improved over prior art and which eliminates or at least mitigates the drawbacks discussed above. More specifically, an object of the invention is to provide a method and an audio device that is improves the intelligibility of speech or voiced audio in noisy environments. These objects are achieved by the technique set forth in the appended independent claims with preferred embodiments defined in the dependent claims related thereto.

In a first aspect, a method of increasing speech intelligibility of an audio stream comprising speech audio is presented. The method is performed in real-time by an audio device and comprises detecting an ambient noise, estimating an internal noise based on the ambient noise, and determining a voice filter based on the estimated internal noise and the audio stream. The method further comprises applying the voice filter to the audio stream to provide a target audio stream, and outputting the target audio stream to one or more transducers thereby generating an internal sound of the audio device. In addition to this, the method comprises detecting the internal sound of the audio device and the step of determining the voice filter is further based on the detected internal sound and comprises subtracting the estimated internal noise from the detected internal sound to provide a true audio stream. The method further comprises updating the voice filter based on a difference between the target audio stream and the true audio stream.

In one variant, the step of determining the voice filter further comprises comparing the estimated internal noise to one or more masking thresholds, and updating the voice filter based the comparing. This is beneficial as it provides for an energy and computational efficient way of determining if the audio is masked by the noise or not.

In one variant, said one or more masking thresholds are calculated by performing a critical band analysis of the audio stream. The critical band analysis comprises auditory masking by frequency spreading. This is beneficial as it increases the accuracy of the masking thresholds.

In one variant, the method further comprises filtering the audio-stream to compensate for a hearing profile associated with a user of the audio device. This is beneficial as the speech intelligibility is further increased and optimized for the user.

In one variant, the step of determining the voice filter is done after the filtering, such that the determining is based on an audio stream compensated for a hearing profile associated with a user of the audio device. This is beneficial as the same voice filer algorithm may be used regardless of user and the computational effort may be reduces as some compensation is already applied through the hearing profile.

In one variant, the step of determining the voice filter further comprises determining a playback phon based on a playback volume, and the step of updating the voice filter is further based on an equal loudness contour associated with the determined phon. This is beneficial as the speech intelligibility will change over volume, but not evenly across all frequencies and compensating for this increase speech intelligibility regardless of playback volume.

In one variant, the step of determining the playback phon is further based on the internal sound internal sound. This is beneficial as it gives an accurate reading of the actual sound pressure level experienced by the user.

In one variant, the step of determining the voice filter further comprises smoothing a gain of the voice filter in frequency by convolution using a frequency window function. This is beneficial as it removes unwanted differences between adjacent groups of frequencies.

In one variant, the step of determining the voice filter further comprises averaging the gain of the voice filter using an exponentially weighted moving average comprising one or more weighting parameters. This is beneficial as it removes unwanted differences between adjacent groups of frequencies.

In one variant, the step of determining the voice filter further comprises applying configurable mixing setting to select the degree at which the voice filter is to be applied to the audio stream. This is beneficial as it makes the amount of improvement customizable and the user may select the desired amount of compensation.

In one variant, the step of estimating internal noise is implemented by one or more Recurring Neural Networks, RNN. The use of RNN is beneficial as it allows an accurate and efficient way of estimating the internal noise.

In one variant, the ambient noise is detected by an external microphone operatively connected to the audio device. This is beneficial as it provides an accurate measure of the ambient noise.

In one variant, the ambient noise is limited to a maximum audio bandwidth of up to 10 kHz, preferably up to 8 kHz. This is beneficial as it further decreases the computational complexity of the present method.

In one variant, the method further comprises applying Active Noise Cancellation, ANC, to the audio stream after applying the voice filter to the audio stream. This is beneficial as the noise of the internal sound is further reduced.

In a second aspect, an audio device is presented. The audio device comprises one or more transducers, at least one internal microphone arranged to detect an internal sound at an ear cavity of a user, and a processing module operatively connected to the internal microphone, to said one or more transducers and to an external microphone. The processing module is configured to perform the method the present invention.

In one variant, the external microphone is comprised in the audio device. This is beneficial as the data from the microphone is readily available to the processing module.

In a third aspect, and audio system for increasing speech intelligibility in real-time is presented. The system comprising a portable electronic device operatively connected to an audio device and configured to transfer an audio stream comprising speech audio to the audio device, wherein the audio device is the audio device according to the present invention.

In one variant, an ambient noise is sensed by an external microphone comprised in the electronic device, and the electronic device is further configured to transfer the ambient noise sensed by the external microphone to the audio device. This is beneficial as additional noise data may be provided by the external microphone of the audio device. Alternatively or additionally, the audio device may be configured without an external microphone, thus decreasing cost of the audio device.

In a fourth aspect, a computer program product is presented. The computer program product is configured to, when executed by a processing module, causes the processing module to perform the method of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will be described in the following; references being made to the appended diagrammatical drawings which illustrate non-limiting examples of how the inventive concept can be reduced into practice.

FIGS. 1a-b are schematic views of an audio system according to embodiments of the invention.

FIG. 2 is a schematic view of a voice filter and its signals according to embodiments of the present invention.

FIG. 3 is a block view of a method of increasing speech intelligibility of an audio stream according to embodiments of the present invention.

FIG. 4 is an exemplary plot of speech optimization according to embodiments of the present invention.

FIG. 5 is an exemplary plot of equal loudness contours according to embodiments of the present invention.

FIG. 6 is an exemplary plot of speech optimization according to embodiments of the present invention.

FIG. 7 is a block view of determining of voice a voice filter according to embodiments of the present invention.

FIG. 8 is a computer program product according to embodiments of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, certain embodiments will be described more fully with reference to the accompanying drawings. The invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention, such as it is defined in the appended claims, to those skilled in the art.

The term “coupled” is defined as connected, although not necessarily directly, and not necessarily mechanically. Two or more items that are “coupled” may be integral with each other. The terms “a” and “an” are defined as one or more unless this disclosure explicitly requires otherwise. The terms “substantially,” “approximately,” and “about” are defined as largely, but not necessarily wholly what is specified, as understood by a person of ordinary skill in the art. The terms “comprise” (and any form of comprise, such as “comprises” and “comprising”), “have” (and any form of have, such as “has” and “having”), “include” (and any form of include, such as “includes” and “including”) and “contain” (and any form of contain, such as “contains” and “containing”) are open-ended linking verbs. As a result, a method that “comprises,” “has,” “includes” or “contains” one or more steps possesses those one or more steps, but is not limited to possessing only those one or more steps.

FIG. 1a is a simplified view of a real-time audio system 1 for improving speech intelligibility of an audio source signal 20 in the form of an audio stream 20. The corresponding audio system 1 is also depicted in a schematic view in FIG. 1b to give an alternative illustration of component and feature location. The audio stream 20 comprises speech audio and may be any suitable form of streamed audio. The speech audio may be any speech audio in digital form, such as podcast, audio books, voice communication etc. The audio stream 20 is typically transferred from an electronic device 10 to an audio device 30. The transferring is preferably wireless via a suitable wireless interface such as BLE etc., but may also be wired. The electronic device 10 may be any suitable electronic device 10 but is preferably portable electronic device 10 such as a mobile terminal 10. The audio device 30 is preferably a pair of headphones 30 and may be of any suitable design e.g. supra-aural, circum-aural or in-ear, but, as readily understood by the skilled person, any audio device 30 workable with the presented invention is suitable. The audio device 30 is adapted to receive the audio stream 20 from the mobile terminal 10, which means that if the audio stream 20 is transferred via BLE, the audio device comprises one or more BLE chipset. Further to this, the audio device 30 comprises one or more transducers 35, preferably two transducers 35, and necessary hardware and software modules to transform the received audio stream 20 to analogue signals possible to be outputted by the one or more transducers 35. The audio stream 20 is associated with a playback volume 15, set by any device 10, 30 of the audio system 1 indicating a level at which the audio stream 20 is to played at. The hardware of the audio device 30 comprises at least one processing module 32. The processing module 32 may comprise a suitable MCU, or plurality of MCUs, one or more DSPs, one or more memories, and may further comprise the required hardware for communication across the interface, e.g. a BLE transceiver.

The transducer(s) 35 of the audio device 30 are configured to generate sound directed to an ear cavity of a user of the audio device 30. The audio device 30 is provided with one or more internal microphones 36 arranged to measure the sound generated by the transducers 35. The sound is preferably measured at the ear cavity of the user when the audio device 30 is used by the user. Preferably, one internal microphone 36 is provided to measure the sound generated by each of the transducers 35.

The audio system 1 is further provided with one or more external microphones 5. The external microphone 5 is external to the audio device 30 and may be any suitable microphone 5 operatively connected to the processing module 32 of the audio device 30. The external microphone 5 may be comprised in the audio device 20 when, e.g. the audio device 30 is a headset, the external microphone may be arranged to detect a voice of the user of the audio device 30. Alternatively, or additionally, the external microphone 5 may be comprised in the portable electronic device 10 when, e.g. the portable electronic device 10 is a mobile terminal 10.

Turning now to FIG. 2, a conceptual summary of the present invention will be given. The invention is preferably implemented on an audio device 30 as presented herein. The speech intelligibility of the audio stream 20 is increased by means of a voice filter 50. This voice filter 50 applied to the audio stream 20 creating a target audio stream 20′ that is output to the one or more transducers 35. The target audio stream 20′ is, compared to the audio stream 20, filtered to increase the speech intelligibility. Further details regarding the inner workings of the audio filter 50 will be given throughout this disclosure. The voice filter 50 is determined based at least on the ambient noise 40 and an internal sound 37. The ambient noise 40 may be detected by the one or more external microphones 5 operatively connected to the processing module 32 of the audio device 30. The internal sound 37 is, or is an estimate of, the sound at the ear cavity of the user of the audio device 30. The internal sound 37 is measure by the internal microphone 36 associated with the one or more transducers 35. Preferably, the audio device 30 comprises at least one voice filter 50 associated with each of the transducers 35 of the audio device 30.

With reference to FIG. 3, an overview of a method 100 of increasing speech intelligibility of the audio stream 20 comprising speech audio will be described. It should be mentioned that the method 100 may be described as increasing speech intelligibility by processing of an audio stream 20 comprising speech audio. The method 100 may be performed by the audio device 30 as presented herein. Each of the steps of the method 100 will be described in further detail elsewhere in this disclosure. The method 100 comprises outputting 150 the target audio stream 20′ to the user of the audio device 30. That is to say, the audio filter 50 is applied 140 to the audio stream 20 to provide the target audio stream 20′ that is provided to the transducer 35 in order to generate the internal sound 37. The internal sound 37 is detected 160, typically by the internal microphone 36 is detecting 110 the ambient noise 40. This is typically performed by one or more the external microphones 5 measuring the ambient noise and providing this to the processing module 32 of the audio device 30. The ambient noise 40 is used to estimate 120 an internal noise 40′, the internal noise 40′ is noise as perceived by the user's ear drum. The audio stream 20, the estimated internal noise 40′ and the detected internal sound 37 are used to determine the speech filter 50.

In some embodiments of the method 100, it further comprises filtering 105 the audio stream 20 to compensate for a hearing profile HL(f_k) of a user of the system 1. This is beneficial since hearing disabilities and/or impairments of the user are compensated for in addition to the speech intelligibility of the method. Preferably the hearing profile HL(f_k) compensation is applied to the audio source signal 20 prior to applying or determining 130 the voice filter 50, such that the voice filter 50 is determined based on an audio stream 20 compensated for hearing disabilities and/or impairments of the user. This is beneficial since it effectively removes differences between users and the same methodology for determining 130 the voice filter 50 may be used for all users. Additionally, since the compensation with respect to the hearing capabilities only affects the audio stream 20, it will, in most cases, directly improve the speech intelligibility. Further to this, the hearing profile compensation can be accounted for in the determination of the voice filter 50 and no pre-processing is required until the voice filter 50 is applied 140. The relation between the hearing profile HL(f_k) and the ambient noise 40 may be important, as the processing result to increase speech intelligibility may, for some users, have reduced effect with respect to the hearing capabilities of the user if the users hearing profile HL(f_k) is not considered.

It should be mentioned that in audio systems 1 or audio devices 30 utilizing noise cancellation techniques, such as active noise cancellation, ANC, the noise cancellation is preferably applied to the audio stream 20 after the voice filter 50. This is beneficial since doing noise cancellation decreases the noise level, but at the same, may cause the audio signal to become distorted. The degree of distortion depends on the configuration of the noise cancellation and the tuning and/or calibration of the noise cancellation technique.

The target audio stream 20′ may become distorted or otherwise negatively impacted by e.g. digital to analogue converter, the transducer operation and the position of the audio device 30 on the user. Consequently, it is beneficial to detect the internal sound 37 presented at the eardrum of the user after removal of noise by the noise cancellation, compare true audio stream 37′, i.e. the internal sound 37 after removal of internal noise 40′, to the target audio stream 20′ and act to minimize the differences.

As already indicated, the audio system 1 is an audio system 1 with real-time constraints. The audio stream 20 is received as digital samples either per sample basis or per frame basis. The collection of samples in frames can be done elsewhere or as a part of the system 1, e.g. by the electronic device 10. The audio stream 20 comprise a collection of N samples with sample rate F_ssamples per second, these are formed into an audio signal frame with frame (time) index l. The audio stream 20 can be mono or stereo.

The voice filter 50 is preferably based on psychoacoustic masking and comprise a psychoacoustic model derived from a speech intelligibility index or equivalent e.g. Articulation Index, Speech Transmission Index or Short Term Objective Intelligibility, and theory of tonal masking of noise. A frequency gain of the voice filter 50 is calculated such that the internal noise 40′ is masked by the target audio stream 20′, this will be explained in more detailed in other sections of this disclosure.

The estimated internal noise 40′ may be provided in a number of different ways and the input to the estimation 120 of the internal noise 40′ is at least the ambient noise 40 detected 110 by one of the external microphones 5. The ambient noise 40 is provided as a microphone signal by the external microphone and is preferably represented in a frame-wise construction substantially equal to that of the audio stream 20. The microphone signal may also be a stereo signal, such a signal is typically termed dual microphone signal. A dual microphone signal comprises two independent microphone signals formatted as a single stereo signal. As previously explained, there may be several external microphones 5 in the system 1, and the step of estimating 120 the internal noise 40′ may comprise determining to use e.g. only one microphone signal of all microphone signals provided by the external microphones 5. A decision pertaining to which external microphone to use may be based on e.g. a highest signal level, proximity to the transducer etc. All external microphones 5 may be processed separately in order to obtain ambient noise 40 from each of the processed external microphones 5. The external microphones 5 may be processed to obtain a stereo signal, and even processed to obtain a direction of the ambient noise 40 such that each transducer 35 may be associated with a different ambient noise 40.

As the skilled person will understand after digesting the teachings herein, there may, based on resource management and optimization of available processing power in the real-time audio system 1, be a trade-off necessary whether to utilize several external microphones 5 or not for detecting 110 the ambient noise 40. This trade-off may be application dependent. When the audio device 30 is e.g. a pair of headphones 30, and if enough processing capabilities exist, two separate estimations 120 of the internal noise 40′ are feasible, one for a left ear of a user and one for a right ear of the user. However, if e.g. processing capabilities are insufficient or if stringent requirements on current consumption exist, a rational assumption may be the ambient noise 40 is substantially equal in for the left and the right ear and the same internal noise 40′ may be utilized for both ears.

The external microphones 5 may be sampled with a different sample rate compared to the audio signal. Given that the important frequency range for voice communication is up to 8 kHz, the external microphones 5 be bandlimited to a maximum bandwidth of 10 kHz, or preferably 8 kHz. The lower bandwidth reduces processing load, memory load and current consumption. The maximum bandwidth may be reduced even further to reduce the processing load, the memory load and the current consumption additionally, but the maximum bandwidth has to be traded-off for requirements of the ambient noise 40.

It should be noted, as the skilled person will appreciate after reading this disclosure, that the external microphones 5 will produce a signal comprising both the ambient noise 40 and additional sound sources. Only the ambient noise 40 is relevant which means that e.g. echo originating from the transducer and near-end talk from listener engaging in a conversation is beneficial to exclude from the signal produced by the external microphones 5. This is beneficial since it reduces the risk that that the additional sound sources are miss-classified as ambient noise 40.

One solution to avoid miss-classifying the additional sound as ambient noise is to use one or more noise estimation techniques e.g. higher degree statistics, Cepstrum analysis, Auto-regressive modelling or non-parametric methods as Welch spectrum and Minimum variance method. Typically, for minimum effort implementations, the method 100 may stop the detection 110 of ambient noise 40 and/or the estimating 120 of the internal noise 40′ if additional sound sources are detected by the external microphone(s) 5. The distinction between background noise and voice sources may be solved using e.g. a voice activity detector, VAD.

In one embodiment of the present invention, the internal noise 40′ is estimated 120 by a Recurring Neural Network, RNN. This will be explained in more detail in other sections of the disclosure, but one benefit is that the complexity of implementation and configuration of e.g. estimating 120 the internal noise 120, detection of additional sounds, voice detection etc. are exchanged into training and operating the RNN, which is well described in the theory of machine learning.

Regardless of how the internal noise 40′ is estimated 120, a representation of the internal noise 40′ comprise an average energy values E_v(b), b=1 . . . N_Bof the internal noise 40′ only for each auditory filter band, or critical band b. The concept of critical bands b is explained in the following sections.

As previously explained, the frequency gain of the voice filter 50 is calculated such that the internal noise 40′ is masked by the target audio stream 20′. In order to accomplish this, the audio stream 20 is filtered represented either in frequency or as a critical band b representation. This may be accomplished by having the audio stream 20 divided into sub-frames, allowing for up to e.g. 50% overlap with a previous sub-frame. The sub-frames may be windowed using a suitable window function e.g. Hamming, Hanning, Triangular windows etc. A power spectrum P_x(k)=|X(k)|²is calculated using the sub-framed time domain data and a fast Fourier transform implementation, FFT, where k is a frequency-bin index. The resolution of the frequency transform is preferably selected based on the sample rate F_sand a sub-frame size. Typically, a trade-off between resolution and resource demand is required.

The quantities described in the frequency domain are all represented in sound pressure level, SPL, such that P_x(k)/N is the power spectral density in SPL per frequency-bin index k and referenced to a free-field reference point located at the listeners ear cavity. The conversion from digital signals to sound pressure level is done by an appropriate frequency dependent scaling, one scaling frequency function per microphone 5, 36 and one scaling frequency function per transducer 35. The scaling functions are predetermined and/or configurable and preferably stored in a memory operatively coupled to the processing module 32. The scaling functions can be considered as a calibration step conducted once during the design or configuration of the audio device 30. Typically, but not limited to, a scaling function for a microphone 5, 36 consist of one scale value per frequency-bin index k and can be estimated from a microphone frequency response. A scaling frequency function for a transducer 35 would correspond to a transducer frequency response including scaling due to e.g. the distance to a reference point, typically the ear of the listener. As way of exemplifying, for a pair of headphones 30, the scaling frequency function for the audio stream 20 would be based on a frequency response of the transducer 35 referenced to an ear-reference-point, ERP.

A cochlear model divides the audio stream 20 into N_Bfrequency bands, each frequency band representing one critical band b. The number of critical bands N_Bcan be set according to desired resolution in the frequency domain making it possible to directly control the granularity at which the audio stream 20 can be adjusted by the voice filter 50. As the skilled person will understand, there is a trade-off between frequency resolution and resource demand, increasing the resolution require a larger cochlear model and therefore both a higher computational effort and a more complex implementation. The inventors behind this disclosure has found that that 20 frequency bands, N_B=20, is a reasonable choice regarding frequency resolution and computational complexity. Without loss of generality, the division into critical bands b may be made using an equivalent rectangular bandwidth, ERB, scale and a gamma-tone filter bank. Other scales and filter types may be utilized to correctly provide the cochlear model. For a general signal, for each critical band b, an averaged energy e_x(b)=1/N Σ_kP_x(k)|F_b(k)|²is calculated using the power spectrum and where F_b(k) is the frequency response of the filter associated with the critical band b in the gamma-tone filter bank. The energy in each band is represented as a vector by ē_x=[e_x(1), . . . , e_x(N_B)]^T.

In the following, the psychoacoustic property of masking will be explained in some further detail and in particular how a first signal can mask a second signal, resulting in the second signal not being perceived. For explanatory purposes, assume that the audio stream 20 is approximated by a tonal signal while the ambient noise 40 is a wideband noise, the theory of tone masking noise applies.

A masking threshold T(b) associated with the critical band b is calculated based on the critical band analysis of the audio stream 20 which may comprise masking in frequency, simultaneous frequency masking by spreading, and temporal masking by gain smoothing.

The critical band analysis is applied in order to obtain a critical band representation ē_sof the audio stream 20 as ē_s=[e_s(1), . . . , e_s(N_B)]. Note that this means that the masking threshold T(b) as described above, will, in embodiments where compensation 105 according to a hearing profile HL(f_k) is firstly applied, take hearing impairments of the user into account.

The simultaneous frequency masking by spreading mentioned above may be described by spreading function SF that models frequency spreading. The spreading function SF may be given by: SF_dB(x)=15.81+7.5(x+0.474)−17.5√{square root over (1+(x+0.474)²)} dB, where x has unit of Barks¹and −SF_dB(x) is described in dB. It should be noted that frequency spreading is a convolution in the critical band domain and may be represented by a convolutional kernel matrix S. The masking threshold T(b) per critical band b for tone-masking-noise is then given as a masking threshold vector T=γSē_s, where—T=[T(1), . . . , T(N_B)]^T, γ=diag[γ(1), . . . , γ(N_B)] and −γ(x)=1/(10^(X^L^+x)/10) and x equals the Bark number. Both SF_dB(x) and γ(x) may be translated from Bark to the critical band domain as is used herein. X_Lis a configuration parameter that state the constant difference (in dB) between the masking threshold T and the noise energy, i.e. internal noise 40′, per critical band ē_v=[e_v(1), . . . , e_v(N_B)]. A typical value of this configuration parameter is X_L=14.5. A higher value result in higher sensitivity to noise energy and therefore require a higher audio stream 20 energy to mask the ambient noise 40. ¹The Bark scale is a psychoacoustical scale.

The audio system 1 is preferably configured to compute at least one masking threshold T to be used in the voice optimization, i.e. in improving speech intelligibility. Depending on the application, e.g. headphones 30 etc. there can be two or more masking thresholds T calculated by the audio system 1. Each masking threshold T is typically related to one or more transducers 35.

Based on the masking threshold T, the critical band representation ē_sof the audio stream 20, which may comprise the hearing impairment compensation, and the noise critical band representation ē_v, the voice optimization will calculate a frequency dependent gain in the form of the voice filter 50 to improve speech intelligibility for the listener.

Voice optimization in this context refers to the process of calculating a frequency dependent gain, represented by a vector g_opt, that is applied to the audio stream which is then considered voice optimized in the form of the target audio stream 20′. For each critical band b, the noise energy e_v(b), i.e. internal noise 40′, associated with each critical band b is compared with the masking threshold T(b) associated with each critical band b. If the noise energy is below the masking threshold T(b), the audio stream 20 is determined to have masked the noise and no amplification of the audio stream 20 is necessary at the corresponding critical band b. That is to say, for this critical band b, the gain of the voice filter 50 is determined 130 to be unity or lower, but preferably at least as large as is necessary to fulfil the noise energy being below the masking threshold T(b). If the noise energy exceeds the masking threshold T(b), the noise is not masked, and an amplification of the audio source signal is needed. That is to say, for this critical band b, the gain of the voice filter 50 is determined 130 to be greater than unity. The amplification can be represented in many ways, and the skilled person will, after digesting this disclosure, have no problem in industrializing any of the teachings herein.

The amplification of the audio stream 20 in each critical band b by √{square root over (g_opt(b))} result in a voice optimized signal, i.e. the target audio stream 20′.

Note that the disclosure described herein performs simultaneous masking in frequency that is included in the model (matrix S). When calculating the optimal gain g_opt(b) it may be preferable to state a general formulation including all frequency bands rather than calculating the gain individually for each frequency band. A formulation of optimal gain g as per this invention includes all frequency bands and the simultaneous masking in frequency. Consequently, the present invention will result in an optimized usage of the available gain parameters compared to when each gain is calculated individually and not including simultaneous masking in the model.

As previously stated, the audio stream 20, ē_s, is used to calculate the masking threshold T(b) to which the estimated internal noise 40′ is compared and deemed masked or not. The presentation of the voice optimized signal 20′, the target audio stream 20′, at the user's ear cavity via the transducer 35 may be impacted by several frequency dependent components, among which perhaps the most important is the position and fit of the headphones 30, resulting in that the presented version of the target audio stream 20′ does not have the expected frequency content. The perceived optimized voice signal ē_sis part of the microphone signal 37 along with the estimated internal ambient noise 40′.

Opposed to ANC or noise reduction techniques where the noise is measured by a combination of the external microphone 5 and the internal microphone 36, in the present invention, the internal noise 40′, ē_v, is estimated based on the external noise 40. Hence, the presented voice optimized signal ē_scan be estimated by, but not limited to, e.g. subtracting ē_vfrom the internal sound 37. One further benefit of this is that it enables real-time adjustment to the voice optimization processing such that ē_swill converge towards ē_sin some pre-defined measure, e.g. root-mean.-square, and that can then account for e.g., changing position or degree of fitting of the headphones 30 and make the method 100 more robust and resilient.

In one non-limiting example, the amplification may be such that a gain is calculated which, when applied to the audio stream 20, would correspond to a resulting masking threshold T. Assume that when the noise energy is equal to or less than the masking threshold T, no amplification is needed, and amplification is needed for the frequency bands where the noise energy exceeds the threshold. Note that the opposite may very well be true and that a negative gain, i.e. an attenuation may be possible in both cases. The resulting optimal voice filter 50 will only comprise the necessary amplifications, and in some cases redistribute energy from non-masked frequencies to masked frequencies applied. The desired goal function T_d(b) per frequency band b may be defined as:

$T_{d} (b) = {\begin{matrix} T (b), if e_{v} (b) \leq T (b) \\ e_{v} (b) if e_{v} (b) > T (b) \end{matrix}$

Using the desired goal function T_d=[T_d(1), . . . , T_d(N_B)], a weighted least square norm optimization problem may be formulated which may be solved for g_optusing the method of Lagrange multipliers and Karush-Kuhn-Tucker conditions²which implies minimize ∥Tg−T_d∥_W², subjected to

$\begin{matrix} g (i) \geq 0, \forall i \\ {\bar{g}}^{T} {\bar{e}}_{s} = \sum_{i} e_{s} (i) \end{matrix}$

W is a diagonal weighting matrix, the main diagonal in this example is populated by the frequency band weighting as given by a speech intelligibility index³. The optimization utilizes the frequency spreading explicitly in the processes of which frequencies shall be amplified or attenuated with an importance weighting over frequencies. ²Boyd, S., & Vandenberghe, L. (2009). Convex Optimization. Cambridge University Press. ETSI. (2012). Speech and multimedia Transmission Quality (STQ); Speech quality performance in the presence of background noise; Part 1: Background noise simulation technique and background noise database 202 396-1. ETSI³S3.5-1997, A. (1997). Methods for calculation of the Speech Intelligibility Index. ANSI.

In another non-limiting example, the weighting matrix W may be populated based on the error between the target audio stream 20′ provided to the transducer 35 and the corresponding detected internal sound 37 provided by the internal microphone 36 after the internal noise 40′ has been removed i.e., the internal noise 40′ noise estimated 120 is subtracted from the signal provided by the internal microphone 36. A suitable weighting matrix W may in this case be based on an error in the frequency domain, preferably in the auditory band domain, and even more preferably such that appropriate weight-values are in the range of [0-1] and most preferably normalized to e.g. a root-mean-square value of unity, e.g. using the correlation coefficient between ē_sand ē_swhere high correlation, i.e. signals being highly similar, corresponds to a low weight, i.e. no focus on the error at this frequency band, and vice versa.

The optimal gain g_optis preferably converted from critical band representation to amplitude over frequency-bin representation H_g(k). Such conversion preferably comprises an interpolation over the employed critical band division e.g., in this example a gammatone filterbank such that H_g(k)=√{square root over (Σ_b|F_b(k)|²g_opt(b))}.

Observe that H_g(k) is updated once for every new frame l of audio stream 20 and internal noise 40′. Typically, an audio frame is relatively short, a number of samples of the audio frame may be less than an equivalent of 100 ms duration. The human ear, on the other hand, has an integration time of up to 100-300 ms. In addition to this, when applying adaptive frequency adjustment, the listener must experience stability in the tonal balance of the audio stream 20, failure to accomplish this may result in discomfort to the user. Another aspect is the frequency variation in the voice optimized signal, i.e. the target audio stream 20′. Adjacent frequency bands must not differ too much in degree of adjustment or an annoying sensation may arise. All of these properties are subjective and, after reading this disclosure, known to the skilled person.

The inventors behind this disclosure have realized that the gain of the voice filter 50 may be processed to order to mitigate the above subjective effects. In one embodiment, the gain of the voice filter 50 is smoothed in frequency by convolution with a frequency window function e.g. a triangular window or similar to assert that isolated frequency-bins do not have either too high amplification or too high attenuation compared to adjacent frequency-bins, i.e. the variation in gain between frequency-bins is limited. In one embodiment the window can be set to a typical value of [0.15,0.7,0.15], i.e. after convolution the resulting gain in each frequency band consist of a 15/15 percent ratio of the adjacent bands and 70 percent ratio of the current band. Typically, it may be unwise to include more than 3-5 critical bands in such convolution operation since each critical band is more independent of adjacent bands the further apart they are in frequency. In another additional or alternative embodiment, the gain of the voice filter 50 is averaged using an exponentially weighted moving average with weighting parameter T_i. The weighting parameter T_imay e.g. be selectable by the user or set to a fixed value corresponding to e.g. the integration time of the human ear i.e. −Ti=0.3. This will effectively also slow down the update rate and therefore allow the user to adjust the hearing to the frequency coloration

As is evident to the skilled person after learning of the method 100 according to the present disclosure, the method 100 is concerned with processing (filtering, changing) the audio stream 20 to increase speech intelligibility of the audio stream 20. The method 100 does not, in its most general form, comprise adding any inverse of the ambient noise 40 or estimated internal noise 40′. The method 100 rather changes (filters, processes) the audio stream 20 by the voice filter 50. The voice filter 50 is a filter adapted to increase speech intelligibility, it is not concerned with removing the ambient noise 40 or the estimated internal noise 40′, it is rather concerned with adapting the audio stream 20 to make increase intelligibility of speech comprised in the audio stream 20 when listened to in noisy environments. Filtering the speech audio of the audio stream 20 such that its intelligibility is increased will change the frequency content of the speech audio. In other words, a voice of a person uttering the speech audio may appear foreign or distorted after being subjected to the voice filter, but the intelligibility of the speech is increased.

It should be emphasized that the gain adjustment of the voice filter 50 will be distributed over all frequencies of a source signal bandwidth due to the simultaneous masking and the energy constraint. Hence, at some frequencies, where the noise is masked, the audio source signal may be attenuated and vice versa. This phenomena is illustrated in FIG. 4 showing an example of the audio stream 20 as a solid line, the internal noise 40′ as a densely dashed line, the masking threshold T as a dashed line and the target audio stream 20′ as a dotted line. The optimization visualized in FIG. 4 is valid for one frame. The target audio stream 20′ has been energy redistributed to mask the internal noise 40′ at all frequency bands. As can be seen in FIG. 4 when comparing the target audio stream 20′ to the audio stream 20, the target audio stream 20′ has, by the voice filter 50, been amplified at some frequencies and attenuated at other frequencies in accordance with the teachings herein. It should be noted that if the internal noise 40′ is below the masking threshold T for all frequencies, then no voice enhancement is necessary due to background noise.

If applicable, a hearing impairment compensation may be described as, but is not limited to, a filter in either time or frequency domain that counter-act or mitigates the hearing impairments of the user. Hearing impairments may be described by the hearing profile HL(f_k), a frequency function HL(f_k), f_kindicate a set of discrete frequencies, (typically a set of 5-7 frequencies are used) in the unit of Hearing Level dB per frequency, dB HL. The hearing profile HL(f_k) is equal or equivalent to, but is not limited to, an audiogram that is the result of an audiology exam where a tone audiogram was conducted. No impairments corresponds to 0 dB HL and increasing values, i.e. values larger than 0, indicate a hearing impairment or deficiency. The creation of a compensation to mitigate the hearing impairments is described later. In one embodiment, the hearing impairment compensation is defined in the frequency-domain by a frequency function H_HI(k), i.e. a filter which is the compensation due to the users hearing profile HL(f_k), and can be applied 105 to the audio stream 20 before the voice optimization. Alternative, it can be included in the voice optimization 130 without prior processing of the audio stream 20. As previously shown, H_HI(k) can be grouped into critical band representation and applied as a frequency-bin scaling to ē_s, thereby it will be included in the resulting optimal gain H_g(k). A final frequency amplitude adjustment is given by H_HI(k)H_g(k)=H_vo(k) of the voice filter 50. This means that the voice enhancement provided by the voice filter 50 due to the ambient noise 40 in low noise conditions may be a unity gain voice filter 50 over all frequencies since the hearing impairment compensation provides sufficient speech intelligibility improvement.

In one embodiment, the user may, via e.g. a mixing setting select the degree m to which the speech compensation to mask the ambient noise 40 shall be applied. For each frequency-bin and −m=[0, . . . ,], such that:

20*log₁₀(|H_vo(k)|) dB=(1−m)·20 log₁₀(|H(k)|)+m*20 log₁₀(H_g(k)), where m=0 corresponds to no frequency adjustments due to the background noise. Noteworthy is that the phase response is preserved as is, i.e.

$∠ H_{v o} (k) = ∠ H_{HI} (k) .$

In addition to compensating for the hearing profile HL(f_k) of the user, the present invention may optionally be combined with a volume dependent compensation, VDC. As explained, the audio stream 20 comprises a frequency spectrum, and this frequency spectrum of the audio signal will be perceived differently at different playback sound pressure levels. This can be seen when comparing equal loudness contours 300, see FIG. 5, at different SPL. By compensating for the difference between the different equal loudness contours 300, the audio stream 20 is more likely to be perceived as intended, i.e. as it is perceived when mastered at the master in the cause of pre-recorded data such as podcasts or audio books.

A number of equal loudness contours 300 are shown in FIG. 5. An equal loudness contour 300 is a measure of SPL, typically presented in decibel as dB Sound Pressure Level SPL. The sound pressure level is provided over a frequency spectrum, typically the audible spectrum for humans. Each point in the equal loudness contour 300 represents a sound pressure level for which a listener perceives a constant loudness when compared to a pure tone presented at 1 kHz with a certain phon. A phon is defined in DIN 45631 and ISO 532, and will herein be defined to mean that a 1 kHz tone with a 50 dB SPL has a loudness of 50 phons and all other frequencies perceived with the same loudness as this signal also have a loudness of 50 phons.

In FIG. 5, the equal loudness 300 contours are plotted for six different phons, i.e. there are six plots in the graph of FIG. 5. In FIG. 5, equal loudness contours 300 are presented for 0, 20, 40, 60, 80 and 100 phons. The loudness contours 300 of FIG. 5 are loudness contours 300 according to ISO 226:213. These contours 300 are corrected compared to Fletcher-Munson curves provided by Harvey Fletcher and Wilden A. Munson in 1933. The Fletcher-Munson curves are commonly referred to as the first description of the concept of equal loudness. If the sound pressure level had no effect on how the frequency spectrum of an audio signal was perceived by a listener, the offset between the contours 300 in FIG. 5 would be the same across all frequencies.

Additionally, or alternatively to the hearing profile filtering 105, the method 100 may further comprise determining 136 a playback phons based on a playback volume 15 of the audio stream and/or the detected internal sound 37.

As stated above, if the internal noise 40′ is below the masking threshold T for all frequencies, then no voice enhancement is necessary due to background noise. This can be a the result in e.g., low ambient noise environments or if e.g., the hearing impairment compensation have resulting in a signal level where the corresponding threshold T is above the internal ambient noise 40′

In the following embodiment, the hearing impairment is calculated based on the users hearing profile HL(f_k). Utilizing what is known as Speech Intelligibility Index Count-the-dots Audio form⁴, a further optimization problem may be formed. Given a hearing profile HL(f_k), it is desired to provide a filter H_HI(k) to adjust the hearing threshold to maximize a standardized Articulation Index, AI, or Speech Intelligibility Index, SII, as given by the count-the-dot audiogram. The AI is calculated as the number of dots below the hearing profile HL(f_k) when the hearing profile HL(f_k) is plotted on the defined diagram, this is illustrated by the dashed line in FIG. 6. ⁴https://www.audiologyonline.com/articles/20q-aided-speech-intelligibility-index-23707

In one embodiment, that can be described as an optimization formulation where a set of gains h_HI(k) as function of frequency-bins are optimized to amplify/attenuation and to redistribute the energy of the audio stream such that the intelligibility index is maximized,

$\begin{matrix} \max_{h_{HI} (k)} AI (h_{HI} (k)) \\ AI (h_{HI} (k)) = \sum_{k^{'}} D_{k}^{'}, k^{'} \in {k^{'} ❘ 1 \leq k^{'} \leq 100, D_{k}^{'} \leq HL (f_{k}) / h_{HI} (k)} \\ \sum_{k} {❘ h_{HI} (k) ❘}^{2} = γ \end{matrix}$

The filter H_HI(k)=h_HI(k) is consequently created from the necessary gains at frequencies k, such that the total energy change in the target audio stream 20′ by the resulting filter is equal to gamma. In one embodiment γ=1 corresponds to an energy redistribution as exemplified in FIG. 6, another example is γ=2 were a constant gain of 2 over all frequencies are obtained. Gamma may be set depending on the amplification possibilities in the system 1, typically it is set to 1 (unity) as exemplified above if no amplification is allowed or 2 if a doubling is possible. Again, same reasoning applies here as to the energy distribution due to background noise, it is not possible to amplify over all frequencies.

FIG. 6 depicts an exemplary count-the-dot audiogram wherein the dotted line is hearing test result in unit dB HL (hearing level). The solid line is the result after the compensation filter has been applied. As seen, the AI has increased, i.e. there are more dots are below the solid line than the dotted line. As it is desirable to have the overall energy change provided by the filter H_HI(k) to be unity, some dots are lost. The results shown in FIG. 6 are valid for a sound pressure of 60 dB, i.e. normal conversational level or listening level in quite environments. Since an approximate sound pressure level of the target audio stream 20′ when played by the transducer 35 is available e.g. by pre-calibration of volume steps of the system 1 and corresponding sound pressure levels, by measurement of the sound pressure levels using the internal microphone 36, it is possible to adjust the e.g. the hearing profile accordingly.

The VDC is, as mentioned above, in one optional embodiment implemented using a pre-calibrated table making it a stationary method which does not change the filter H_HI(k) based on the dynamics of the audio stream 20. Consequently, the filter H_HI(k) is only updated if the volume setting, i.e. the playback volume 15 of the audio device 30, is changed. In one embodiment the pre-calibrated table contains scaling factors a(k) for each volume step and frequency-bin that are applied to H_HI(k) e.g, a(k)H_HI(k). The size of the pre-calibrated table depend on the number of volume step and the number of frequency-bins used in the calculations.

In one embodiment the VDC compute two hearing compensations in the frequency domain, one for each ear. In one additional embodiment, it may be configured to combine the left ear compensation and the right ear compensation in order to provide a single hearing compensation suitable for both ears. This is important when playback utilize several transducers and the sound from each transducer can physically reach each ear of the listener.

In another embodiment of an implementation of the VDC, that may very well be combined with the embodiment above, the feedback microphone signal is used making it a dynamic method that updates the filter H_HI(k) based on the levels of the target audio stream 20′ as it is played by the transducer 35. This approach calculates the compensation more often. In order to avoid sharp transients in the audio signal, the update rate of the filter H_HI(k) may be kept low, about 0.25-1 Hz.

Although count-the-dot audiogram is provided with the assumption that the signal or speech level is 60 dB SPL, the result and method 100 herein may very well with correct result be scaled with the sound pressure level. As a non-limiting example, a 10 dB increase in volume, corresponding to 10 dB SPL, would correspond to the dotted line of FIG. 6 being offset 10 dB towards 0 dB HL on the hearing level scale, the y-axis. This means that, due to the increased volume, the speech intelligibility increases and the corresponding hearing impairment compensation is adjusted accordingly. A lower volume has an analogue example. As the skilled person would have deduces, if the dotted line is above the area with the dots including any offsets due to increase signal level, no enhancement is necessary.

Other methods of scaling the results and adjusting the hearing impairments based on the count-the-dots audiogram and the teachings here should now be apparent for a person skilled in the art.

As previously mentioned, the inventors behind this disclosure have realised that the internal noise 40′ may be accurately modelled based on the ambient noise 40 by means of Machine Learning. The internal noise 40′ is known from ANC technology to be the external noise 40 filtered by the primary (acoustic) path. Wherein the latter describe the impact on the external noise 40 when propagating from exterior to the headphones 30 into the ear cavity. The primary path is the important (unknown) noise transfer function that must be found in real-time and at high-degree of accuracy for the ANC to operate correctly i.e., for the ANC technology to form the correct anti-noise that cancels (at therefore attenuate) the internal noise 40′. The real-time and accuracy requirements on ANC technology typically dictated that ANC technology is executed in dedicated hardware.

For the present invention, the real-time and accuracy requirements when estimating the internal noise 40′ is much lower compared to ANC technology, as will be described later. Further to this, no dedicated hardware is necessary. Many of the same aspects, known to a person skilled in the art, still exist e.g. the estimation of internal noise 40′ must exclude echo from near-end audio stream when rendered by transducers 35, and instead of complex real-time adaptive filtering and calibrations (as for ANC), a neural network is used to model the primary path including the separation between noise, echo and near-end talker.

The neural network of a preferred embodiment is RNN. RNNs are typically based on Long Short Term Memory, LSTM, or Gated Recurring Units, GRU. In general, a feature vector and a output vector of the RNN may be selected in numerous ways and the selection of the two, the size of the RNN along with the training data quality and the training of RNN will determine the performance of the RNNs ability to output the desired output data given the input, i.e. feature data.

The size of the RNN is set by the training result and the resource constraints posed by the implementation into a real-time audio system. Generally, the size i.e. the number of units and the number of hidden layers of the RNN is a design selection as is the choice of the number of points in an FFT calculation. A typical size of the RNN is e.g. 200-300 units with 3-4 hidden layers. The computational demand of the RNN can be decreased e.g. by selecting a lower order RNN and increasing the error on the output and/or by skipping and structured pruning of RNN units.

It should be noted that the absolute level of the signal provided by the external microphone 5 is important for the application. Hence, an approach for data augmentation may be adopted in which each training example is pre-filtered with a randomized generated second-order filter. Thereby training the RNN to be robust against frequency variations due to an external microphone 5 frequency response tolerance and an external microphone 5 placement variation. Levels of the individual signal in training examples are preferably varied for robustness against level variations.

It is apparent that the above RNN description is but one non-limiting example provided as one working example of the RNN. A person skilled in the art can surely, after digesting the teachings herein, device a different example, varying the feature set, the output set and/or the training e.g. data set, loss function, optimization process etc.

Feature extraction for the RNN may be provided by speech recognition theory. And the following embodiments are to be considered as non-exhaustive examples of feature extractions that are possible to combine with one another in any order or set imaginable.

In one embodiment, the features comprise a discrete cosines transform of the logarithm of the energy per critical band of the signal provided by the microphone 5. In one embodiment, the features comprise the spectrum of the signal provided by the microphone 5, represented in critical bands as described in the present disclosure. In a further embodiment, the features further comprise an average energy in the entire sub-frame. In one embodiment, the features comprise a delta change in amplitude between current sub-frame and previous sub-frame cepstral logarithmic coefficients covering up to at least 600 Hz, preferably up to at least 1000 Hz. Thereby including, with high certainty, the vocal fundamentals of typical voices. In one embodiment, the features comprise a binary signal stating if a non-noise source signal is active or not e.g. a simple level detector to state if user is talking or echo from transducer is present.

The outputs of the RNN typically comprise the average energy values E_v(b), b=1 . . . N_Bof the internal noise 40′ for each auditory filter band. In one embodiment the outputs of the RNN also comprise, in addition to the above, a binary signal indicating noise or not-noise, a second binary signal indicating low or high level, a third binary signal indicating that a near-end signal is active. Additional signals like the binary signals described might not be directly used in the algorithm to calculate the optimal voice filter 50, but for a person skilled in the art it is easily recognized that relevant outputs such as the described can help to obtain a better result when training the RNN.

In training the RNN, one important aspect is the generalization capabilities of the RNN, such that it will operate correctly even in conditions not used during the training. Therefore, raining examples are preferably made up by a combination of background noise e.g. full-size car 100 km/h and Cafeteria⁵, echo signal and near-end talk with varying levels and filtrations as stated above. Echo signal and near-end talk signal are preferably independent i.e., the same utterances do not exist simultaneously. ⁵ETSI. (2012). Speech and multimedia Transmission Quality (STQ); Speech quality performance in the presence of background noise; Part 1: Background noise simulation technique and background noise database 202 396-1. ETSI.

In one embodiment, the ground truth of the training is constituted by the noise (only) power per auditory filter based on the spectrum of the noise signal (only), divided into auditory filters, and referenced (measured) in the headphone-and-ear cavity. This will therefore include the primary path i.e., the outside the headphone to inside the headphone-and-ear cavity. This is important since the headphones have at least a high-frequency attenuation on the noise due to the acoustic seal when worn (depending on the headphone type, in-ear, over-ear and on-ear). If the headphones also have Active Noise Cancellation (typically operative in the frequency range of 150-900 Hz), the noise outside the headphones are much different from the noise inside the cavity.

A system that can facilitate the presentation of noise, near-end speech and echo and at the same time record the internal noise 40′ (ground truth) is industry standard and the process is fully automated. In one non-limiting example, a scenario is started where background noise is rendered from a multi-speaker setup in an measurement chamber, the headphones under test is situated on the head-and-torso simulator that records the internal noise as the signal that reaches the microphone placed in each ear-simulator. Simultaneously, the background noise is recorded by the external microphone 5 on the headphones 30. After the scenario is completed, each signal is conditioned, time adjusted and converted into either a feature set or a ground truth set.

In summary the usage of machine learning and recurrent neural networks in the modelling of the internal noise 40′ result in a noise estimation in an auditory band model, the removal of near end talk and echo without using complex Voice Activity Detection or Echo cancellation and it will model a primary path from outside the headphone to inside the headphone-and-ear cavity.

Several detailed implementation of the different aspects of the voice filter 50 are presented throughout this disclosure. Regardless of how the voice filter 50 is determined, the voice filter 50, described above as H_vo(k), is applied 140 to the digital source signal, the audio stream 20. There may be a multitude of voice filters 50, each voice filter 50 provides a target audio stream 20′ to be rendered on a transducer 35. There are several approaches to process the audio stream 20 and the person skilled in the art will see, after reading this disclosure, several others then the following two examples. The voice filter 50 may, in one embodiment, be applied 140 by e.g. conversion of frequency function to finite impulse response filter if the phase response is less important, it can be a symmetric impulse response filter resulting in linear phase. The voice filter 50 may, in one embodiment, be applied 140 by multiplication in the frequency domain by an overlap-and-add method to avoid circular convolution when multiplying to frequency functions.

In one preferred embodiment, the audio device 30 comprises at least one voice filter 50 per transducer 35.

In one embodiment, the voice filter 50 providing the target audio stream 20′ is energy normalized. This may lead to high peak amplitudes in the time domain signal. In a further embodiment, the target audio stream 20′ is attenuated to ensure that the signal amplitude of the target audio stream 20′ is not too high for the final signal format. The signal amplitude may then be then transformed into the correct format without distortion using e.g. a standard Limiter or Dynamic Range Controller, DRC. It should be noted that no additional processing apart from controlling the signal amplitude is required. The limiter and DRC can be other components of the digital audio system and are preferably included for the sake of hearing safety.

With reference to FIG. 7, some further non-exhaustive steps of determining 130 the voice filter 50 will be explained. The following embodiments are possible to combine with any of the other embodiments presented herein and are fully compatible with the detailed examples and calculations disclosed herein.

In one embodiment of the method 100, the step of determining 130 the voice filter 50 comprises subtracting 132 the estimated internal noise 40′ from the detected internal sound 37. This will provide a true audio stream 37′ that is what the target audio stream 20′ actually sounds like at the ear of the user. Based on a difference between the target audio stream 20′ and the true audio stream 37′, it is consequently possible to update 138 the voice filter 50 based on this difference. This effectively creates a control loop where it is possible to ensure that the target audio stream 37 is actually what the user hears. This is beneficial as it enables the voice filter may be updated based on e.g. how the audio device is worn by a user and how well the audio device fits at the ear of the user.

In one embodiment of the method 100, the step of determining 130 the voice filter 50 comprises comparing 134 the estimated internal noise 40′ and one or more masking thresholds T. This is substantially comparing 134 the densely dashed line, the estimated internal noise 40′, to the dashed line, the masking threshold T, of FIG. 4. The voice filter is then updated based on the comparing 134 to provide the target audio stream 20′. In a further optional embodiment, the method 100 the masking thresholds T are calculated by performing a critical band analysis 137 of the audio stream 20. The critical band analysis 137 comprises auditory masking by frequency spreading.

In one embodiment of the method 100, it is configured to compensate for the playback volume 15 of the audio device 30 such as described in reference to FIG. 5. This comprises, in the step of determining 130 the voice filter 50, determining 136 a playback phon based on the playback volume 15. The playback phon may be determined from the playback phon by means of a transfer function describing a transfer function of the sound through the transducer 35. Further to this, the step of updating 138 the voice filter 50 is further based on an equal loudness contour associated with the determined phon. In a further embodiment, the playback phon is further based on the detected 160 internal sound 37.

The present invention will, in addition to solving the previously presented problem, provide increased speech intelligibility substantially regardless of how a user of the audio device 30 chooses to carry the audio device 30. Typically, the transducers 35 of an audio device are configured to work at a certain load. This load is in the form of the air cavity between the user and the transducer 35. If the audio device 30 is e.g. a pair of closed headphones, the air cavity is formed by the headphones 30 being carried firmly and tightly about the outer ear of the user. However, as not all ears are the same, and not all users carry their audio devices 30 in the same way, the load of the transducer 35 will differ between users and the thereby the sound of the audio device 30. The present invention solves also this problem by detecting 160 the internal sound 37 which will differ depending on how the audio device 30 is worn.

VOICE OPTIMIZATION IN NOISY ENVIRONMENTS

Information

Publication Number

Date Filed

Date Published

Inventors

Original Assignees

CPC

International Classifications

Abstract

Description

Claims

Priority Claims (1)

PCT Information