Patent Grant
8000487
This disclosure relates generally to hearing assistance devices, and more particularly to frequency translation by high-frequency spectral envelope warping in hearing assistance devices.
Hearing assistance devices, such as hearing aids, include, but are not limited to, devices for use in the ear, in the ear canal, completely in the canal, and behind the ear. Such devices have been developed to ameliorate the effects of hearing losses in individuals. Hearing deficiencies can range from deafness to hearing losses where the individual has impairment responding to different frequencies of sound or to being able to differentiate sounds occurring simultaneously. The hearing assistance device in its most elementary form usually provides for auditory correction through the amplification and filtering of sound provided in the environment with the intent that the individual hears better than without the amplification.
In order for the individual to benefit from amplification and filtering, they must have residual hearing in the frequency regions where the amplification will occur. If they have lost all hearing in those regions, then amplification and filtering will not benefit the patient at those frequencies, and they will be unable to receive speech cues that occur in those frequency regions. Frequency translation processing recodes high-frequency sounds at lower frequencies where the individual's hearing loss is less severe, allowing them to receive auditory cues that cannot be made audible by amplification.
One way of enhancing hearing for a hearing impaired person was proposed by Hermansen, Fink, and Hartmann in 1993. “Hearing Aids for Profoundly Deaf People Based on a New Parametric Concept,” Hermansen, K.; Fink, F. K.; Hartmann, U; Hansen, V. M., Applications of Signal Processing to Audio and Acoustics, 1993. “Final Program and Paper Summaries,” 1993 IEEE Workshop on, Vol., Iss, 17-20 October 1993, pp. 89-92. They proposed that a vocal tract (formant) model be constructed by linear predictive analysis of the speech signal and decomposition of the prediction filter coefficients into formant parameters (frequency, magnitude, and bandwidth). A speech signal was synthesized by filtering the linear prediction residual with a vocal tract model that was modified so that any high frequency formants outside of the range of hearing of a hearing impaired person were transposed to lower frequencies at which they can be heard. They also suggested that formants in low-frequency regions may not be transposed. However, this approach is limited in the amount of transposition that can be performed without distorting the low frequency portion of the spectrum (e.g., containing the first two formants). Since the entire signal is represented by a formant model, and resynthesized from the modified (transposed) formant model, the entire signal may be considerably altered in the process, especially when large transposition factors are used for patients having severe hearing loss at mid and high frequencies. In such cases, even the part of the signal that was originally audible to the patient is significantly distorted by the transposition process.
In U.S. Pat. No. 5,571,299, Melanson presented an extension to the work of Hermansen et. al. in which the prediction filter is modified directly to warp the spectral envelope, thereby avoiding the computationally expensive process of converting the filter coefficients into formant parameters. Allpass filters are inserted between stages in a lattice implementation of the prediction filter, and the fractional-sample delays introduced by the allpass filters determine the nature of the warping that is applied to the spectral envelope. One drawback of this approach is that it does not provide direct and complete control over the shape of the warping function, or the relationship between input frequency and transposed output frequency. Only certain input-output frequency relationships are available using this method.
In U.S. Pat. No. 5,014,319, Leibman relates a frequency transposition hearing aid that classifies incoming sound according to frequency content, and selects an appropriate transposition factor on the basis of that classification. The transposition is implemented using a variable-rate playback mechanism (the sound is played back at a slower rate to transpose to lower frequencies) in conjunction with a selective discard algorithm to minimize loss of information while keeping latency low. This scheme was implemented in the AVR TranSonic™ and ImpaCt™ hearing aids. However, in at least one study, this variable-rate playback approach has been shown to lack effectiveness in increasing speech intelligibility. See, for example, “Preliminary results with the AVR ImpaCt Frequency-Transposing Hearing Aid,” McDermott, H. J.; Knight, M. R.; J. Am. Acad. Audiol., 2001 March; 12 (3); 121-7 11316049 (P, S, E, B), and “Improvements in Speech Perception with use of the AVR TranSonic Frequency-Transposing Hearing Aid,” McDermot, H. J.; Dorkos, V. P.; Dean, M. R.; Ching, T. Y.; J. Speech Lang. Hear. Res. 1999 December; 42(6):1323-35. Some disadvantages of this approach are that the entire spectrum of the signal is transposed, and that the pitch of the signal is, therefore, altered. To address this deficiency, this method uses a switching system that enables transposition when the spectrum is dominated by high-frequency energy, as during consonants. This switching system may introduce errors, especially in noisy or complex audio environments, and may disable transposition for some signals which could benefit from it.
In U.S. Patent Application Publication 2004 0264721 (issued as U.S. Pat. No. 7,248,711), Allegro et. al. relate a method for frequency transposition in a hearing aid in which a nonlinear frequency transposition function is applied to the spectrum. In contrast to Leibman, this algorithm does not involve any classification or switching, but instead transposes low frequencies weakly and linearly and high frequencies more strongly. One drawback of this method is that it may introduce distortion when transposing pitched signals having significant energy at high frequencies. Due to the nonlinear nature of the transposition function (the input-output frequency relationship), transposed harmonic structures become inharmonic. This artifact is especially noticeable when the inharmonic transposed signal overlaps the spectrum of the non-transposed harmonic structure at lower frequencies.
The Allegro algorithm is described as a frequency domain algorithm, and resynthesis may be performed using a vocoder-like algorithm, or by inverse Fourier transform. Frequency domain transposition algorithms (in which the transposition processing is applied to the Fourier transform of the input signal) are the most-often cited in the patent and scholarly literature (see for example Simpson et. al., 2005, and Turner and Hurtig, 1999, U.S. Pat. No. 6,577,739, U.S. Patent Application Publication 2004 0264721 (issued as U.S. Pat. No. 7,248,711) and PCT Patent Application WO 0075920). “Improvements in speech perception with an experimental nonlinear frequency compression hearing device,” Simpson, A.; Hersbach, A. A.; McDermott, H. J.; Int J. Audiol. 2005 May; 44(5):281-92; and “Proportional frequency compression of speech for listeners with sensorineural hearing loss,” Turner, C. W.; Hurtig, R. R.; J Acoust Soc Am. 1999 August; 106(2):877-86. Not all of these method render transposed harmonic structure inharmonic, but they all share the drawback that the pitch of transposed harmonic signals are altered.
Kuk et. al. (2006) discuss a frequency transposition algorithm implemented in the Widex Inteo hearing aid, in which energy in the one-octave neighborhood of the highest-energy peak above a threshold frequency is transposed downward by one or two octaves (a factor of two or four) and mixed with the original unprocessed signal. “Linear Frequency Transposition: Extending the Audibility of High-Frequency Information,” Francis Kuk; Petri Korhonen; Heidi Peeters,; Denise Keenan; Anders Jessen; and Henning Andersen; Hearing Review 2006 October. As in other frequency domain methods, one drawback of this approach is that high frequencies are transposed into lower frequencies, resulting in unnatural pitch transpositions of the sound. Additional artifacts are introduced when the harmonic structure of the transposed signal overlaps the spectrum of the non-transposed harmonic structure at lower frequencies.
Therefore, an improved system for improved intelligibility without a degradation in natural sound quality in hearing assistance devices is needed.
Disclosed herein, among other things, is a system for frequency translation by high-frequency spectral envelope warping in a hearing assistance device for a wearer. According to various embodiments, the present subject matter includes a method for processing an audio signal received by a hearing assistance device, including: filtering the audio signal to generate a high frequency filtered signal, the filtering performed at a splitting frequency; transposing at least a portion of an audio spectrum of the filtered signal to a lower frequency range by a transposition process to produce a transposed audio signal; and summing the transposed audio signal with the audio signal to generate an output signal, wherein the transposition process includes: estimating an all-pole spectral envelope of the filtered signal; applying a warping function to the all-pole spectral envelope of the filtered signal to translate the poles above a specified knee frequency to lower frequencies, thereby producing a warped spectral envelope; and exciting the warped spectral envelope with an excitation signal to synthesize the transposed audio signal. It also provides for scaling the transposed audio signal and summing the scaled transposed audio signal with the audio signal. It is contemplated that the filtering includes, but is not limited to high pass filtering or high bandpass filtering. In various embodiments, the estimating includes performing linear prediction. In various embodiments, the estimating is done in the frequency domain. In various embodiments the estimating is done in the time domain.
In various embodiments, the pole frequencies are translated toward the knee frequency and may be done so linearly using a warping factor or non-linearly, such as using a logarithmic or other non-linear function. Such translations may be limited to poles above the knee frequency.
In various embodiments, the excitation signal is a prediction error signal, produced by filtering the high-pass signal with an inverse of the estimated all-pole spectral envelope. The present subject matter in various embodiments includes randomizing a phase of the prediction error signal, including translating the prediction error signal to the frequency domain using a discrete Fourier Transform; randomizing a phase of components below a Nyquist frequency; replacing components above the Nyquist frequency by a complex conjugate of the corresponding components below the Nyquist frequency to produce a valid spectrum of a purely real time domain signal; inverting the DFT to produce a time domain signal; and using the time domain signal as the excitation signal. It is understood that in various embodiments the prediction error signal is processed by using, among other things, a compressor, peak limiter, or other nonlinear distortion to reduce a peak dynamic range of the excitation signal. In various embodiments the excitation signal is a spectrally shaped or filtered noise signal.
In various embodiments the system includes combining the transposed signal with a low-pass filtered version of the audio signal to produce a combined output signal, and in some embodiments the transposed signal is adjusted by a gain factor prior to combining.
The system also provides the ability to modify pole magnitudes and frequencies.
This Summary is an overview of some of the teachings of the present application and not intended to be an exclusive or exhaustive treatment of the present subject matter. Further details about the present subject matter are found in the detailed description and appended claims. The scope of the present invention is defined by the appended claims and their legal equivalents.
The following detailed description of the present subject matter refers to subject matter in the accompanying drawings which show, by way of illustration, specific aspects and embodiments in which the present subject matter may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present subject matter. References to “an”, “one”, or “various” embodiments in this disclosure are not necessarily to the same embodiment, and such references contemplate more than one embodiment. The following detailed description is demonstrative and not to be taken in a limiting sense. The scope of the present subject matter is defined by the appended claims, along with the full scope of legal equivalents to which such claims are entitled.
The present subject matter relates to improved speech intelligibility in a hearing assistance device using frequency translation by high-frequency spectral envelope warping. The system described herein implements an algorithm for performing frequency translation in an audio signal processing device for the purpose of improving perceived sound quality and speech intelligibility in an audio signal when presented using a system having reduced bandwidth relative to the original signal, or when presented to a hearing-impaired listener sensitive to only a reduced range of acoustic frequencies.
One goal of the proposed system is to improve speech intelligibility in the reduced-bandwidth presentation of the processed signal, without compromising the overall sound quality, that is, without introducing undesirable perceptual artifacts in the processed signal. In embodiments implemented in a real-time listening device, such as a hearing aid, the system must conform to the computation, latency, and storage constraints of such real-time signal processing systems.
Hearing Assistance Device Application
In one application, the present frequency translation system is incorporated into a hearing assistance device to provide improved speech intelligibility without undesirable perceptual artifacts in the processed signal.
Frequency Translation System Example
The system of
In one embodiment, the filter 210 in the lower block is omitted. In one embodiment the filter 210 is replaced by a simple delay compensating for the delay incurred by filtering in the upper processing branch.
In one embodiment the frequency translation processor 320 is programmed to perform a piecewise linear frequency warping function. Greater detail of one embodiment is provided in
The three algorithm parameters described above, the splitting frequency, the warping function knee frequency, and the warping ratio, determine which parts of the spectral envelope are processed by frequency translation, and the amount of translation that occurs.
The frequency warping function governs the behavior of the frequency translation processor, whose function is to alter the shape of the spectral envelope of the processed signal. In such embodiments, the pitch of the signal is not changed, because the spectral envelope, and not the fine structure, is affected by the frequency translation process. This process is depicted in
In one embodiment of the present system a whitened excitation signal, derived from linear predictive analysis, is processed using a warped spectral envelope filter to construct a new signal whose spectral envelope is a warped version of the envelope of the input signal, having peaks above the knee frequency translated to lower frequencies. In one embodiment, the peak frequencies are computed directly from the values of the complex poles in the filter derived by linear prediction. In one embodiment the peak frequencies are estimated by examination of the frequency response of the filter. Other approaches for determining the peak frequencies are possible without departing from the scope of the present subject matter.
By translating the peak frequencies according to the frequency warping function described above, a new warped spectral envelope is specified which is used to determine the coefficients of the warped spectral envelope filter. In one embodiment, the filter pole frequencies can be modified directly, so that the spectral envelope described by the filter is warped, and peak frequencies above the knee frequency (such as 2 kHz shown in
The whitened excitation signal, derived from linear predictive analysis, may be subjected to further processing to mitigate artifacts that are introduced when the high-frequency part of the input signal contains very strong tonal or sinusoidal components. For example, the excitation signal may be made maximally noise-like (and less impulsive) by a phase randomization process. This can be achieved in the frequency domain by computing the discrete Fourier transform (DFT) of the excitation signal, and expressing the complex spectrum in polar form (magnitude and phase, or angle). The phase of components at and below the Nyquist frequency (half the sampling frequency) are replaced by random values, and the components above the Nyquist frequency are made equal to the complex conjugate of corresponding (mirrored about the Nyquist component) components below the Nyquist frequency, so that the representation corresponds to a real time domain signal. This frequency domain representation is then inverted to obtain new excitation signal.
In various alternative embodiments, the excitation signal may be replaced by a shaped (filtered) noise signal. The noise may be shaped to behave like a speech-like spectrum, or may be shaped by a highpass filter, and possibly using the same splitting filter used to isolate the high-frequency part of the input signal. In such an implementation, it is generally not necessary to compute the excitation (prediction error) signal in the linear predictive analysis stage.
In other alternative embodiments, the excitation signal may be subjected to dynamics processing, such as dynamic range compression or limiting, or to non-linear waveform distortion to reduce its impulsiveness, and the artifacts associated with frequency transposition of signals with strongly tonal high-frequency components.
The output of the frequency translation processor, consisting of the high-frequency part of the input signal having its spectral envelope warped so that peaks in the envelope are translated to lower frequencies, and optionally scaled by a gain control, is combined with the original, unmodified signal to produce the output of the algorithm.
The present system provides the ability to govern in very specific ways the energy injected at lower frequencies according to the presence of energy at higher frequencies.
Time Domain Spectral Envelope Warping Example
In the time domain process of
It is understood that variations in process order and particular filters may be substituted in systems without departing from the scope of the present subject matter.
Frequency Domain Spectral Envelope Warping Example
In the frequency domain process of
The complex sub-band excitation signal, E(wk), and complex frequency response {H(wk)} are multiplied 1010 to provide a sampled warped spectral envelope signal in the frequency domain {X(wk)}. This sampled warped spectral envelope signal in the frequency domain {X(wk)} can be further processed in the frequency domain by other processes and ultimately converted into the time domain for transmission of processed sound according to one embodiment of present subject matter.
Examples of Combined Whitening and Shaping Filters
In some embodiments, computational savings can be achieved by combining the application of the all-zero FIR filter, to generate the prediction error signal, and the application of the all-pole warped spectral envelope filter to the excitation signal, into a single filtering step.
The all-pole spectral envelope filter is normally implemented as a cascade (or sequence) of second-order filter sections, so-called biquad sections or biquads. Those practiced in the art will recognize that, for reasons of numerical stability and accuracy, as well as efficiency, high-order recursive filters should be implemented as a cascade of low-order filter sections. In the implementation of an all-pole filter, each biquad section has only two poles in its transfer functions, and no (non-trivial) zeros. However, the zeros in the FIR filter can be implemented in the biquad sections along with the spectral envelope poles, and in this case, the FIR filtering step in the original frequency translation algorithm can be eliminated entirely. An example is provided by the system 1100 in
In
In one embodiment, the zeros corresponding to (unwarped) roots of the predictor polynomial should be paired in a single biquad section with their counterpart warped poles in the frequency translation algorithm. Since not all poles in the spectral envelope are transformed in the frequency translation algorithm (only complex poles above a specified knee frequency), some of the biquad sections that result from this pairing will have unity transfer functions (the zeros and unwarped poles will coincide). Since the application of these sections ultimately has no effect on a signal, they can be omitted entirely, resulting in computational savings and improved filter stability.
In the present frequency translation algorithm, the highpass splitting filter makes poles on the positive real axis uncommon, but it frequently happens that poles are found on the negative real axis (poles at the Nyquist frequency, or half the sampling frequency) and these poles should not be warped, but should rather remain real poles (at the Nyquist frequency) in the warped spectral envelope. Moreover, it may happen that a pole is found below the knee frequency in the warping function, and such a pole need not be warped. Poles such as these whose frequencies are not warped can be omitted entirely from the filter design. In the case of a predictor of order 8, for example, if one pole pair is found on the negative real axis, a 25% savings in filtering costs can be achieved by omitting one second order section. If additionally one of the poles is below the knee frequency, the savings increases to 50%.
In addition to achieving some computational savings, this modification may make the biquad filter sections more numerically stable. In some embodiments, for reasons of numerical stability and accuracy, filter sections including both poles and zeros are implemented, rather than only poles.
It is understood that the system of
It is understood that variations in process order and particular conversions may be substituted in systems without departing from the scope of the present subject matter.
The present subject matter includes a method for processing an audio signal received by a hearing assistance device, including: filtering the audio signal to generate a high frequency filtered signal, the filtering performed at a splitting frequency; transposing at least a portion of an audio spectrum of the filtered signal to a lower frequency range by a transposition process to produce a transposed audio signal; and summing the transposed audio signal with the audio signal to generate an output signal, wherein the transposition process includes: estimating an all-pole spectral envelope of the filtered signal; applying a warping function to the all-pole spectral envelope of the filtered signal to translate the poles above a specified knee frequency to lower frequencies, thereby producing a warped spectral envelope; and exciting the warped spectral envelope with an excitation signal to synthesize the transposed audio signal. It also provides for scaling the transposed audio signal and summing the scaled transposed audio signal with the audio signal. It is contemplated that the filtering includes, but is not limited to high pass filtering or high bandpass filtering. In various embodiments, the estimating includes performing linear prediction. In various embodiments, the estimating is done in the frequency domain. In various embodiments the estimating is done in the time domain.
In various embodiments, the pole frequencies are translated toward the knee frequency and may be done so linearly using a warping factor or non-linearly, such as using a logarithmic or other non-linear function. Such translations may be limited to poles above the knee frequency.
In various embodiments, the excitation signal is a prediction error signal, produced by filtering the high-pass signal with an inverse of the estimated all-pole spectral envelope. The present subject matter in various embodiments includes randomizing a phase of the prediction error signal, including translating the prediction error signal to the frequency domain using a discrete Fourier Transform; randomizing a phase of components below a Nyquist frequency; replacing components above the Nyquist frequency by a complex conjugate of the corresponding components below the Nyquist frequency to produce a valid spectrum of a purely real time domain signal; inverting the DFT to produce a time domain signal; and using the time domain signal as the excitation signal. It is understood that in various embodiments the prediction error signal is processed by using, among other things, a compressor, peak limiter, or other nonlinear distortion to reduce a peak dynamic range of the excitation signal. In various embodiments the excitation signal is a spectrally shaped or filtered noise signal.
In various embodiments the system includes combining the transposed signal with a low-pass filtered version of the audio signal to produce a combined output signal, and in some embodiments the transposed signal is adjusted by a gain factor prior to combining.
The system also provides the ability to modify pole magnitudes and frequencies.
The present subject matter includes hearing assistance devices, including, but not limited to, cochlear implant type hearing devices, hearing aids, such as behind-the-ear (BTE), in-the-ear (ITE), in-the-canal (ITC), or completely-in-the-canal (CIC) type hearing aids. It is understood that behind-the-ear type hearing aids may include devices that reside substantially behind the ear or over the ear. Such devices may include hearing aids with receivers associated with the electronics portion of the behind-the-ear device, or hearing aids of the type having a receiver in-the-canal. It is understood that other hearing assistance devices not expressly stated herein may fall within the scope of the present subject matter.
It is understood one of skill in the art, upon reading and understanding the present application will appreciate that variations of order, information or connections are possible without departing from the present teachings. This application is intended to cover adaptations or variations of the present subject matter. It is to be understood that the above description is intended to be illustrative, and not restrictive. The scope of the present subject matter should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5014319 | Leibman | May 1991 | A |
| 5771299 | Melanson | Jun 1998 | A |
| 6169813 | Richardson et al. | Jan 2001 | B1 |
| 6577739 | Hurtig et al. | Jun 2003 | B1 |
| 6980665 | Kates | Dec 2005 | B2 |
| 7580536 | Carlile et al. | Aug 2009 | B2 |
| 20040264721 | Allegro et al. | Dec 2004 | A1 |
| 20060253209 | Hersbach et al. | Nov 2006 | A1 |
| Number | Date | Country |
|---|---|---|
| WO-0076920 | Dec 2000 | WO |
| WO-2007135198 | Nov 2007 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20090226016 A1 | Sep 2009 | US |
