1. Field of the Invention
The present invention relates generally to speech processing, and more particularly to a parametric speech codec for achieving high quality synthetic speech in the presence of background noise.
2. Description of the Prior Art
Parametric speech coders based on a sinusoidal speech production model have been shown to achieve high quality synthetic speech under certain input conditions. In fact, the parametric-based speech codec, as described in U.S. application Ser. No. 09/159,481, titled “Scalable and Embedded Codec For Speech and Audio Signals,” and filed on Sep. 23, 1998 which has a common assignee, has achieved toll quality under a variety of input conditions. However, due to the underlying speech production model and the sensitivity to accurate parameter extraction, speech quality under various background noise conditions may suffer.
Accordingly, a need exists for a system for processing audio signals which addresses these shortcomings by modeling both speech and background noise simultaneously in an efficient and perceptually accurate manner, and by improving the parameter estimation under background noise conditions. The result is a robust parametric sinusoidal speech processing system that provides high quality speech under a large variety of input conditions.
The present invention addresses the problems found in the prior art by providing a system and method for processing audio and speech signals. The system and method use a pitch and voicing dependent spectral estimation algorithm (voicing algorithm) to accurately represent voiced speech, unvoiced speech, and mixed speech in the presence of background noise, and background noise with a single model. The present invention also modifies the synthesis model based on an estimate of the current input signal to improve the perceptual quality of the speech and background noise under a variety of input conditions.
The present invention also improves the voicing dependent spectral estimation algorithm robustness by introducing the use of a Multi-Layer Neural Network in the estimation process. The voicing dependent spectral estimation algorithm provides an accurate and robust estimate of the voicing probability under a variety of background noise conditions. This is essential to providing high quality intelligible speech in the presence of background noise.
Various preferred embodiments are described herein with references to the drawings:
Referring now in detail to the drawings, in which like reference numerals represent similar or identical elements throughout the several views, and with particular reference to
I. Harmonic Codec Overview
A. Encoder Overview
The encoding begins at Pre Processing block 100 where an input signal so(n) is high-pass filtered and buffered into 20 ms frames. The resulting signal s(n) is fed into Pitch Estimation block 110 which analyzes the current speech frame and determines a coarse estimate of the pitch period, PC. Voicing Estimation block 120 uses s(n) and the coarse pitch PC to estimate a voicing probability, PV. The Voicing Estimation block 120 also refines the coarse pitch into a more accurate estimate, PO. The voicing probability is a frequency domain scalar value normalized between 0.0 and 1.0. Below PV, the spectrum is modeled as harmonics of PO. The spectrum above PV is modeled with noise-like frequency components. Pitch Quantization block 125 and Voicing Quantization block 130 quantize the refined pitch PO and the voicing probability PV, respectively. The model and quantized versions of the pitch period (PO, Q(PO)), the quantized voicing probability (Q(PV)), and the pre-processed input signal (so(n)) are input parameters of the Spectral Estimation block 140.
The Spectral Estimation algorithm of the present invention first computes an estimate of the power spectrum of s(n) using a pitch adaptive window. A pitch PO and voicing probability PV dependent envelope is then computed and fit by an all-pole model. This all-pole model is represented by both Line Spectral Frequencies LSF(p) and by the gain, log2Gain, which are quantized by LSF Quantization block 145 and Gain Quantization block 150, respectively. Middle Frame Analysis block 160 uses the parameters s(n), PO, A(PO), and A(PV) to estimate the 10 ms mid-frame pitch PO—mid and voicing probability PV—mid. The mid-frame pitch PO—mid is quantized by Middle Frame Pitch Quantization block 165, while the mid-frame voicing probability PV—mid is quantized by Middle Frame Voicing Quantization block 170.
B. Decoder Overview
The decoding principle of the present invention is shown by the block diagram of
The Parameter Interpolation block 220 interpolates the magnitude Mag(k) and MinPhase(k) envelopes to a 10 ms basis for use in the Subframe Synthesizer. The log2Gain and PV are passed into the SNR Estimation block 230 to estimate the signal-to-noise ratio (SNR) of the input signal s(n). The SNR and PV are used in Input Characterization Classifier block 240. This classifier outputs three parameters used to control the postfilter operation and the generation of the spectral components above PV. The Post Filter Attenuation Factor (PFAF) is a binary switch controlling the postfilter. The Unvoiced Suppression Factor (USF) is used to adjust the relative energy level of the spectrum above PV. The synthesis unvoiced centre-band frequency (FSUV) sets the frequency spacing for spectral synthesis above PV.
Subframe Synthesizer block 250 operates on a 10 ms subframe basis. The 10 ms parameters are either obtained directly from the unquantization process (F0mid, PV—mid), or are interpolated. The FrameLoss flag is used to indicate a lost frame, in which case the previous frame parameters are used in the current frame. The magnitude envelope Mag(k) is filtered using a pitch and voicing dependent Postfilter block 260. The PFAF determines whether the current subframe is postfiltered or left unaltered. The sine-wave amplitudes Amp(h) and frequencies freq(h) are derived in Calculate Frequencies and Amplitudes block 270. The sine-wave frequencies freq(h) below PV are harmonically related based on the fundamental frequency F0. Above PV, the frequency spacing is determined by FSUV. The sine-wave amplitudes Amp(h) are obtained by sampling the spectral magnitude envelope Mag(k). The amplitudes Amp(h) above PV are adjusted according to the suppression factor USF. The parameters F0, PV, MinPhase(k) and freq(h) are fed into Calculate Phase block 280 where the final sine-wave phases Phase(h) are derived. Below PV, the minimum phase envelope MinPhase(k) is sampled at the sine-wave frequencies freq(h) and added to a linear phase component derived from F0. All phases Phase(h) above PV are randomized to model the noise-like characteristic of the spectrum. The amplitudes Amp(h), frequencies freq(h), and phases Phase(h) are fed into the Sum of Sine-Waves block 290 which performs a standard sum of sinusoids to produce the time-domain signal x(n). This signal is input to Overlap Add block 295. Here, x(n) is overlap-added with the previous subframe to produce the final synthetic speech signal shat(n) which corresponds to input signal so(n).
II. Detailed Description of Harmonic Encoder
A. Pre-Processing
As shown in
B. Pitch Estimation
The pitch estimation block 110 implements the Low-Delay Pitch Estimation algorithm (LDPDA) to the input signal s(n). LDPDA is described in detail in section B.6 of U.S. application Ser. No. 09/159,481, filed on Sep. 23, 1998 and having a common assignee; the contents of which are incorporated herein by reference. The only difference from U.S. application Ser. No. 09/159,481 is that the analysis window length is 271 instead of 291, and a factor called β for calculating Kaiser window is 5.1, instead of 6.0.
C. Voicing Estimation
C.1. Adaptive Window Placement
Nw=K*Pc,
where K depends on pitch values of the current frame and the previous frame. An offset D is computed in block 3020 based on Nw. If D is greater than 0, three blocks of signal with the same window size but different locations are extracted from a circular buffer, as indicated in blocks 3030, 3040 and 3050. Around the coarse pitch, three time-domain correlation coefficients are computed from the three blocks of signals in blocks 3035, 3045 and 3055. This time-domain auto-correlation is shown in the following equation:
where Rci is the correlation coefficient, si(n) is the input signal and PC is the coarse pitch. The block of speech with the highest correlation value is fed into Apply Hanning Window block 3070. This windowed signal is finally used for calculating the power spectrum with a FFT of length Nfft in the block 3100 of
C.2. Pitch Refinement
where Nfft is the length of FFT, M is the number of analysis band, E(m) represents the multi-band energy at the m'th band, Pw is the power spectrum and B(m) is the boundary of the m'th band. The multi-band energy is quarter-root compressed in block 3315 as shown below:
Ec(m)=E(m)0.25, 0≦m<M.
The pitch refinement consists of two stages. The blocks 3320, 3330 and 3340 give in detail how to implement the first stage pitch refinement. The blocks 3350, 3360 and 3370 explain how to implement the second stage pitch refinement. In block 3320, Ni pitch candidates are selected around the coarse pitch, PC. The pitch cost function for both stages can be expressed as shown below:
where NRc(m,Pi) is the normalized correlation coefficients of m'th band for pitch Pi, which can be computed in the frequency domain using the following equations:
In block 3330, the cost functions are evaluated from the first Z bands. In block 3360, the cost functions are calculated from the last (M-Z) bands. The pitch candidate who maximizes the cost function of the second stage is chosen as the refined pitch PO of the current frame.
C.3. Compute Multi-Band Coefficients
After the refined pitch PO is found, the normalized correlation coefficients Nrc(m) and the energy E(m) are re-calculated for each band in block 3400 of
A normalization factor No is given below:
where w(n) is the Hanning window and ss(n) is the windowed signal.
By applying the normalization factor No, the multi-band energy E(m) and the normalized correlation coefficient Nrc(m) are calculated by using the following equations:
C.4. Voice Classification
The blocks 3510, 3520 and 3525 show how to generate the feature Rc. After calculating the normalized multi-band correlation coefficients and the multi-band energy in block 3400, the normalized correlation coefficient of certain bands can be estimated by:
where Rt(a,b) is the normalized correlation coefficient from band a to band b. Using the above equation, the low-band correlation coefficient RL is computed in block 3510 and the full-band correlation coefficient Rf is computed in block 3520. In block 3525, the maximum of RL and Rf is chosen as the feature Rc.
The blocks 3530, 3550 and 3560 give in detail how to compute the feature NEL. Energy from the a'th band to b'th band can be estimated by:
The low-band energy, EL, and the full-band energy, Ef, are computed in block 3530 and block 3540 using this equation. The normalized low-band energy NEL is calculated by:
NEL=C*(EL−NS),
where C is a scaling factor to scale down NEL between −1 to 1, and NS is an estimate of the noise floor from block 3550.
As shown in
The Multilayer Neural Network, block 3590, is chosen to classify the current frame to be a voiced frame or an unvoiced frame. There are three layers in this network: the input layer, the middle layer and the output layer. The number of nodes for the input layer is six, the same as the number of input features. The number of hidden nodes is chosen to be three. Since there is only one voicing output Vout, the output node is one, which outputs a scalar value between 0 to 1. The weighing coefficients for connecting the input layer to hidden layer and hidden layer to output layer are pre-trained using back-propagation algorithm described in Zurada, J. M., Introduction to Artificial Neural Systems, St. Paul, Minn., West Publishing Company, pages 186-90, 1992. By non-linearly mapping the input features through the Neural Network Voice Classifier, the output Vout will be used to adjust the voicing decision.
C.5. Voicing Decision
In
TH0=C1−C2*Vm2,
and the variations between two neighbor bands is given by:
Δ=C3−C4*Vmhu 2,
where C1, C2, C3 and C4 are pre-defined constants. Finally, the threshold of m'th band is computed as:
TH(m)=TH0+m*Δ, 0≦m<M.
The next step for the voicing decision is to find a cutoff band, CB, where the corresponding boundary, B(CB), is the voicing probability, PV. The flowchart of this algorithm is shown in
Secondly, a weighted normalized correlation coefficient from the current band to the two past bands must be greater than T2. The coefficient of the i'th band WRC(i) is calculated in block 3725 and is shown in the following equation:
where the weighting factors A0, A1, and A2 are chosen to be 1, 0.5 and 0.08. These weighting factors act as hearing masks. Finally, the distance between two selected voiced bands has to be smaller than another threshold, T3, as shown in 3750. If all three conditions are met, the current band is defined as the voiced cutoff band CB.
After all the analysis bands are tested, CB is smoothed by the previous frame in block 3755. Finally, CB is converted to the voicing probability PV in block 3760.
D. Spectral Estimation
Finally, the complex spectrum F(k) is calculated in FFT block 530 from the windowed speech signal f(n) by an FFT of length N.
Peak(h) contains a peak frequency location for each harmonic bin up to the quantized voicing probability cutoff Q(PV). The number of voiced harmonics is specified by:
and fs is the sampling frequency.
The parameters Peak(h), and P(k) are used in block 630 to calculate the voiced sine-wave amplitudes specified by:
The quantized fundamental frequency Q(F0), Q(PV), and the unvoiced centre-band analysis spacing specified by:
are used as input to block 640 to calculate the unvoiced centre-band frequencies. These frequencies are determined by:
The selection of FAUV has an effect both on the accuracy of the all-pole model and on the perceptual quality of the final synthetic speech output, especially during background noise. The best range was found experimentally to be 60.0-90.0 Hz.
The sine-wave amplitudes at each unvoiced centre-band frequency are calculated in block 650 by the following equation:
A smooth estimate of the spectral envelope PENV(k) is calculated in block 660 from the sine-wave amplitudes. This can be achieved by various methods of interpolation. The frequency axis of this envelope is then warped on a perceptual scale in block 670. An all-pole model is then fit to the smoothed envelope PENV(k) by the process of conversion to autocorrelation coefficients (block 680) and Durbin recursion (block 685) to obtain the linear prediction coefficients (LPC), A(p). An 18th order model is used, but the order model used for processing speech may be selected in the range from 10 to about 22. The A(p) are converted to Line Spectral Frequencies LSF(p) in LPC-To-LSF Conversion block 690.
The gain is computed from PENV(k) in Block 695 by the equation:
E. Middle Frame Analysis
The middle frame analysis block 160 consists of two parts. The first part is middle frame pitch analysis and the second part is middle frame voicing analysis. Both algorithms are described in detail in section B.7 of U.S. application Ser. No. 09/159,481.
F. Quantization
The model parameters comprising the pitch PO (or equivalently, the fundamental frequency F0), the voicing probability PV, the all-pole model spectrum represented by the LSF(p)'s, and the signal gain log2Gain are quantized for transmission through the channel. The bit allocation of the 4.0 kb/s codec is shown in Table 1. All quantization tables are reordered in an attempt to reduce the bit-error sensitivity of the quantization.
F.1. Pitch Quantization
In the Pitch Quantization block 125, the fundamental frequency F0 is scalar quantized linearly in the log domain every 20 ms with 8 bits.
F.2. Middle Frame Pitch Quantization
In Middle Frame Pitch Quantization block 165, the mid-frame pitch is quantized using a single frame-fill bit. If the pitch is determined to be continuous based on previous frame, the pitch is interpolated at the decoder. If the pitch is not continuous, the frame-fill bit is used to indicate whether to use the current frame or the previous frame pitch in the current subframe.
F.3. Voicing Quantization
The voicing probability PV is scalar quantized with four bits by the Voicing Quantization block 130.
F.4. Middle Frame Voicing Quantization
In Middle Frame Quantization, the mid-frame voicing probability Pvmid is quantized using a single bit. The pitch continuity is used in an identical fashion as in block 165 and the bit is used to indicate whether to use the current frame or the previous frame PV in the current subframe for discontinuous pitch frames.
F.5. LSF Quantization
The LSF Quantization block 145 quantizes the Line Spectral Frequencies LSF(p). In order to reduce the complexity and store requirements, the 18th order LSFs are split and quantized by Multi-Stage Vector Quantization (MSVQ). The structure and bit allocation is described in Table 2.
In the MSVQ quantization, a total of eight candidate vectors are stored at each stage of the search.
F.6. Gain Quantization
The Gain Quantization block 150 quantizes the gain in the log domain (log2Gain) by a scalar quantizer using six bits.
III. Detailed Description of Harmonic Decoder
A. Complex Spectrum Computation
The log2Gain, F0, and PV are used to normalize the magnitude envelope to the correct energy in Normalize Envelope block 720. The log2 magnitude envelope Mag(k) is normalized according to the following formula:
where HV, HUV, and uvfreq( ) are calculated in an identical fashion as in block 410 of
The frequency axis of the envelopes MinPhase(k) and Mag(k) are then transformed back to a linear axis in Unwarp block 730. The modified IRS filter response is re-applied to Mag(k) in IRS Filter Decompensation block 740.
B. Parameter Interpolation
The envelopes Mag(k) and MinPhase(k) are interpolated in Parameter Interpolation block 220. The interpolation is based on the previous frame and current frame envelopes to obtain the envelopes for use on a subframe basis.
C. SNR Estimation
The log2Gain and voicing probability PV are used to estimate the signal-to-noise ratio (SNR) in SNR Estimation block 230.
D. Input Characterization Classifier
The SNR and PV are used in the Input Characterization Classifier block 240. The classifier outputs three parameters used to control the postfilter operation and the generation of the spectral components above PV. The Post Filter Attenuation Factor (PFAF) is a binary switch controlling the postfilter. If the SNR is less than a threshold, and PV is less than a threshold, PFAF is set to disable the postfilter for the current frame.
The Unvoiced Suppression Factor (USF) is used to adjust the relative energy level of the spectrum above PV. The USF is perceptually tuned and is currently a constant value. The synthesis unvoiced centre-band frequency (FSUV) sets the frequency spacing for spectral synthesis above PV. The spacing is based on the SNR estimate and is perceptually tuned.
E. Subframe Synthesizer
The Subframe Synthesizer block 250 operates on a 10 ms subframe size. The subframe synthesizer is composed of the following blocks: Postfilter block 260, Calculate Frequencies and Amplitudes block 270, Calculate Phase block 280, Sum of Sine-Wave Synthesis block 290, and OverlapAdd block 295. The parameters of the synthesizer include Mag(k), MinPhase(k), F0, and PV. The synthesizer also requires the control flags FSUV, USF, PFAF, and FrameLoss. During the subframe corresponding to the mid-frame on the encoder, the parameters are either obtained directly (F0mid, Pvmid) or are interpolated (Mag(k), MinPhase(k)). If a lost frame occurs, as indicated by the FrameLoss flag, the parameters from the last frame are used in the current frame. The output of the subframe synthesizer is 10 ms of synthetic speech Shat(n).
F. Postfilter
The Mag(k), F0, PV, and PFAF are passed to the PostFilter block 260. The PFAF is a binary switch either enabling or disabling the postfilter. The postfilter operates in an equivalent manner to the postfilter described in Kleijn, W.B. et al., eds., Speech Coding and Synthesis, Amsterdam, The Netherlands, Elsevier Science B.V., pages 148-150, 1995. The primary enhancement made in this new postfilter is that it is made pitch adaptive. The pitch (F0 expressed in Hz) adaptive compression factor gamma used in the postfilter is expressed in the following equation:
The pitch adaptive postfilter weighting function used is expressed in the following equation:
The following constants are preferred:
Fmin=125 Hz,
Fmax=175 Hz,
γmin=0.3,
γmax=0.45,
llow=1000 Hz
G. Calculate Frequencies and Amplitudes
The sine-wave amplitudes for the voiced harmonics are calculated in Calculate Sine-Wave Amplitudes block 910 by the formula:
AV(h)=2.0(Mag(vfreq(h))+1.0); h=0, 1, 2, . . . , HV−1
In the next step, the unvoiced centre-band frequencies uvfreqAUV(h) are calculated in blocks 920 in the identical fashion done at the encoder in block 410 of
AAUV(h)=2.0(Mag(uvfreqAUV(h))+1.0); h=0, 1, 2, . . . , HUV−1
The amplitudes AAUV(h) at the analysis spacing FAUV are calculated to determine the exact amount of energy in the spectrum above PV in the original signal. This energy will be required later when the synthesis spacing is used and the energy needs to be rescaled.
The unvoiced centre-band frequencies uvfreqSUV(h) are calculated at the synthesis spacing FSUV in block 940. The method used to calculate the frequencies is identical to the encoder in block 410 of
ASUV(h)=2.0(Mag(uvfreqSUV(h))+1.0); h=0, 1, 2, . . . , HSUV−1
where HSUV is the number of unvoiced frequencies calculated with FSUV.
The amplitudes ASUV(h) are scaled in Rescale block 960 such that the total energy is identical to the energy in the amplitudes AAUV(h). The energy in AAUV(h) is also adjusted according to the unvoiced suppression factor USF.
In the final step, the voiced and unvoiced frequency vectors are combined in block 970 to obtain freq(h). An identical procedure is done in block 980 with the amplitude vectors to obtain Amp(h).
H. Calculate Phase
The parameters F0, PV, MinPhase(k) and freq(h) are fed into Calculate Phase block 280 where the final sine-wave phases Phase(h) are derived. Below PV, the minimum phase envelope MinPhase(k) is sampled at the sine-wave frequencies freq(h) and added to a linear phase component derived from F0. This procedure is identical to that of block 756, FIG. 7 in U.S. application Ser. No. 09/159,481.
I. Sum of Sine-Wave Synthesis
The amplitudes Amp(h), frequencies freq(h), and phases Phase(h) are used in Sum of Sine-Wave Synthesis block 290 to produce the signal x(n).
J. Overlap-Add
The signal x(n) is overlap-added with the previous subframe signal in OverlapAdd block 295. This procedure is identical to that of block 758, FIG. 7 in U.S. application Ser. No. 09/159,481.
What has been described herein is merely illustrative of the application of the principles of the present invention. For example, the functions described above and implemented as the best mode for operating the present invention are for illustration purposes only. Other arrangements and methods may be implemented by those skilled in the art without departing from the scope and spirit of this invention.
This application is a divisional patent application of and claims priority to co-pending U.S. patent application Ser. No. 09/625,960, filed Jul. 26, 2000, which claims priority from United States Provisional Application filed on Jul. 26, 1999 by Aguilar et al. having U.S. Provisional Application Ser. No. 60/145,591, the contents of each of which are incorporated herein by reference.
Number | Date | Country | |
---|---|---|---|
60145591 | Jul 1999 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 09625960 | Jul 2000 | US |
Child | 11261969 | Oct 2005 | US |