Embodiment of the present invention described herein generally relate to the field of speech processing.
A source filter model may be used for speech synthesis or other vocal analysis where the speech is modeled using an excitation signal and a synthesis filter. The excitation signal is a sequence of pulses and can be thought of as modeling the air out of the lungs. The synthesis filter can be thought of as modeling the vocal tract, lip radiation and the action of the glottis.
Methods and systems in accordance with embodiments of the present invention will now be described with reference to the following figures:
In an embodiment, a method of extracting speech synthesis parameters from an audio signal is provided, the method comprising:
In a further embodiment, both the pulsed excitation signal and the complex cepstrum are modified to reduce the difference between the reconstructed speech and the input speech.
Modifying the pulsed excitation signal and the complex cepstrum may comprise the process of:
The difference between the reconstructed speech and the input speech may be calculated using the mean squared error.
In an embodiment, the pulse height az is set such that az=0 if az<0 and az=1 if az>0 before recalculation of the complex cepstrum. This forces the gain information into the complex cepstral as opposed to the excitation signal.
In one embodiment, re-calculating the complex cepstrum comprises optimising the complex cepstrum by minimising the difference between the reconstructed speech and the input speech, wherein the optimising is performed using a gradient method.
For use with some synthesizers, it is easier perform synthesis using the complex cepstrum, decomposed into phase parameters and minimum phase cepstral components.
The above method may be used for training parameters for use with a speech synthesizer, but it may also be used for vocal analysis. Since the synthesis parameters model the vocal tract, lip radiation and the action of the glottis extracting these parameters and comparing them with either known “normal” parameters from other speakers or even earlier readings from the same speaker, it is possible to analyse the voice. Such analysis can be performed for medical applications, for example, if the speaker is recovering from a trauma to the vocal tract, lips or glottis. The analysis may also be performed to see a speaker is overusing their voice and damage is starting to occur. Measurement of these parameters can also indicate some moods of the speaker, for example, if the speaker is tired, stressed or speaking under duress. The extraction of these parameters can also be used for voice recognition to identify a speaker.
In further embodiments, the extraction of the parameters is for training a speech synthesiser, the synthesiser comprising a source filter model for modeling speech using an excitation signal and a synthesis filter, the method comprising training the synthesis parameters by extracting speech synthesis parameters from an input signal. After the parameters have been extracted or derived, they can be stored in the memory of a speech synthesiser.
When training a speech synthesizer, the excitation and synthesis parameters may be trained separately to the text or with the text input. Where the synthesiser stores text information, during training, it will receive input text and speech, the method comprising extracting labels from the input text, and relating extracted speech parameters to said labels via probability density functions.
In a further embodiment, a text to speech synthesis method is provided, the method comprising:
As noted above, the complex cepstrum parameters may be stored in said memory as minimum phase cepstrum parameters and phase parameters, the method being configured to produce said excitation signal using said phase parameters and said synthesis filter using said minimum phase cepstrum parameters.
A system for extracting speech synthesis parameters from an audio signal is provided in a further embodiment, the system comprising a processor adapted to:
In a further embodiment, a text to speech system is provided, the system comprising a memory and a processor adapted to:
Since the present invention can be implemented by software, the present invention encompasses computer code provided to a general purpose computer on any suitable carrier medium. The carrier medium can comprise any storage medium such as a floppy disk, a CD ROM, a magnetic device or a programmable memory device, or any transient medium such as any signal e.g. an electrical, optical or microwave signal.
The text signal is then directed into a speech processor 3 which will be described in more detail with reference to
The speech processor 3 takes the text signal and turns it into speech corresponding to the text signal. Many different forms of output are available. For example, the output may be in the form of a direct audio output 5 which outputs to a speaker. This could be implemented on a mobile telephone, satellite navigation system etc. Alternatively, the output could be saved as an audio file and directed to a memory. Also, the output could be in the form of an electronic audio signal which is provided to a further system 9.
Connected to the output module 63 is output for audio 67. The audio output 67 is used for outputting a speech signal converted from text input into text input 63. The audio output 67 may be for example a direct audio output e.g. a speaker or an output for an audio data file which may be sent to a storage medium, networked etc.
In use, the text to speech system 51 receives text through text input 63. The program 55 executed on processor 53 converts the text into speech data using data stored in the storage 57. The speech is output via the output module 65 to audio output 67.
s(n)=h(n)*e(n) (1)
where h(n) is a slowly varying impulse response representing the effects of the glottal flow, vocal tract, and lip radiation. The excitation signal e(n) is composed of delta pulses (amplitude one) or white noise in the voiced and unvoiced regions of the speech signal, respectively. The impulse response h(n) can be derived from the speech signal s(n) through cepstral analysis.
First, the excitation is initialised. In step S103, glottal closure incidents (GCIs) are detected from the input speech signal s(n). There are many possible methods of detecting GCIs for example, based on the autocorrelation sequence of the speech waveform.
These GCIs are then used to produce the first estimate of the positions of the pulses in the excitation signal in step S105.
Next, the signal is segmented in step S107 in time to form segments of speech on the basis of the detected GCIs 301. In an embodiment the windowed portions of the speech signal sw(n) are set to run from the previous GCI to the following GCI as shown by window 303 in
The signal is then subjected to FFT in step S109 so that sw(n) is converted to the Fourier domain sw(ω). A schematic of the phase response after this stage is shown in
Next, in step S113, the complex cepstrum calculation is performed to derive the cepstral representation of h(n).
The cepstral domain representation of s(n) is
Where |S(ejω)| and θ(ω) are respectively the amplitude and phase spectrum of s(n). ŝ(n) is by definition an infinite and non-causal sequence. If pitch synchronous analysis with an appropriate window to select two pitch periods is performed, then samples of ŝ(n) tend to zero as n→∞. If the signal e(n) is a delta pulse or white noise, then a cepstral representation of h(n), here defined as the complex cepstrum of s(n) can be given by ĥ(n)=ŝ(n) so that |n|≦C, where C is the cepstrum order.
At synthesis time, which will be discussed later, the complex cepstrum of s(n), ĥ(n)is converted into the synthesis filter impulse response h(n) in step S115.
The above explained complex cepstrum analysis is very sensitive to the position and shape of the analysis window as well as to the performance of the phase unwrapping algorithm which is used to estimate the continuous phase response θ(ω).
In step S117 h(n) derived from step S115 is excited by e(n) to produce the synthesised speech signal {tilde over (s)}(n). The excitation signal e(n) is composed of pulses located at the glottal closure instants. In this way, only the voiced portions of the speech signal are taken into account.
Therefore, its is assumed that the initial cepstrum fairly represents the unvoiced regions of the input speech signal s(n) in step S101.
In step S119, the synthesised speech signal (n) is compared with the original input speech s(n):
w(n)=s(n)−{tilde over (s)}(n)=s(n)−e(n)*h(n). (6)
In step S121, the positions of the pulses of the excitation signal e(n), representing the pitch period onset times, are optimized given initial complex cepstrum ĥ(n). Next, in step S123, the complex cepstrum ĥ(n) for each pre-specified instant in time is estimated given the excitation signal e(n) with updated pulse positions. Both procedures are conducted in a way that the mean squared error (MSE) between natural, s(n), and reconstructed speech, {tilde over (s)}(n) is minimized. In the following sections these procedures are described.
In step S121, this procedure is conducted by keeping H(z) for each frame t={0, . . . , T−1}, where T is the number of frames in the sentence, constant, and minimizing the mean squared error of the system of
Considering matrix notation, the error signal w(n) can be written as:
with s being a N+M-size vector whose elements are samples of the natural speech signal s(n), e contains samples of the excitation signal e(n), M is the order of h(n), and is N the number of samples of s(n). The (M+N)×N matrix H has the following shape.
where hn contains the impulse response of H(z) at the n-th sample position.
Considering that the vector e has only Z non-zero samples (voiced excitation), then {tilde over (s)} can be written as
where {a0, . . . , aZ−1} are the amplitudes the non-zero samples of e(n).
The mean squared error of the system is the term to be minimized,
The optimal pulse amplitude âz which minimizes (13) can be given by ∂ε/∂az=0, which results in
By substituting (15) into (14), an expression for the error considering the estimated amplitude âz can be achieved
where it can be seen that the only term which depends on the z-th pulse is the last one in the right side of (16). Therefore, the estimated position {circumflex over (p)}z is the one which minimizes εâ
The term Δp is the range of samples in which the search for the best position in the neighbourhood of pz is conducted.
In step S123, the complex cepstrum is re-estimated. In order to calculate the complex cepstrum based on the minimum MSE, a cost function must be defined in step S125. Because the impulse response h(n) is associated with each frame t of the speech signal, the reconstructed speech vector {tilde over (s)} can be written in matrix form as
where T is the number of frames in the sentence, and
are the synthesis filter coefficients vector at the t-th frame of s(n). The (K+M)×(M+1) matrix At is given by
where et is the excitation vector where only samples belonging to the t-th frame are non-zero, and K is the number of samples per frame.
By considering(17), the MSE can be written as
The optimization is performed in the cepstral domain. The relationship between the impulse response vector ht and its corresponding complex cepstrum vector ĥt=[ĥi(−C) . . . ĥi(C)]T, can be written by
where exp (•) means a matrix formed by taking the exponential of each element of the matrix argument, and L is the number of one-sided sampled frequencies in the spectral domain. The elements of the (2L+1)×(2C+1) matrix D1, and the (M+1)×(2L+1) matrix D2 are given by
where {ω−L, . . . , ωL} are the sampled frequencies in the spectrum domain, with ω0=0, ωL=π, and ω−l=−ωl. It should be noted that warping can be used by implemented by appropriately selecting the frequencies {ω−L, . . . , ωL}. By substituting (22) into (21) a cost function relating the MSE with ĥt is obtained
Since the relationship between cepstrum and impulse response, ht=f(ĥt), is nonlinear, a gradient method is utilized to optimize the complex cepstrum. Accordingly, a new re-estimation or the complex cepstrum is given by
where γ is a convergence factor, and ∇ĥ
which results in:
where diag(•) means a diagonal matrix formed with the elements of the argument vector.
In an embodiment, the method may use the following algorithm where the index i indicates iteration number for the complex cepstrum re-estimation procedure described in relation to steps S123 to S125.
1) Initialize {p0, . . . , pZ-1} as the instants used for initial cepstrum calculation
2) Make az=1, 0≦z<Z−1
3) Get an initial estimate of the complex cepstrum for each frame: {ĥ0(0), . . . , ĥT-1(0)}
Recursion
1) For each pulse position {p0, . . . , pZ-1}
2) For each pulse amplitude {a0, . . . , aZ-1}
3) For each frame {t=0, . . . , T −1}
4) If the SNRseg between natural and reconstructed speech is below a desirable threshold, go to Step 1
5) Stop
Initialization for the algorithm in Table 1 can be done by conventional complex cepstrum analysis. The glottal closure instants can be used to represent the positions {p0, . . . , pZ−1}.
Estimates of the initial frame-based complex cepstra {ĥ0, . . . ĥT-1} can be taken in several ways.
The simplest form would be to consider ĥt equal to the complex cepstrum obtained in the GCI immediately before frame t. Other possible ways are interpolation of pitch-synchronous cepstra over the frame, or interpolation of amplitude and phase spectra.
Assuming that the initial GCIs do not need to be accurate, during the pulse optimization process, negative amplitudes az<0 are strong indicators that the corresponding GCIs should not be there, whereas high amplitudes indicate that one or more pulses are missing. To solve the first problem, amplitudes are set to zero az=0 whenever the algorithm finds that the amplitudes are negative (recursive step 2). Such empirical solution assumes that there is not polarity reversal during in the initial complex cepstra.
By forcing the condition az=1 if az>0, the above algorithm forces the gain information into the complex cepstral as opposed to the excitation signal.
Stopping criterion can be based on the segmental signal-to-noise ratio (SNRseg) between natural and reconstructed speech or maximum number of iterations. A SNRseg>15˜dB would mean that the reconstructed speech is fairly close to its natural version. However, sometimes this value can not be reached due to the poor estimates of the initial complex cepstrum and corresponding GCIs. Usually 5 iterations are adequate to reach convergence.
Although the above discussion has referred to optimising both the complex cepstral and the excitation signal, for speech synthesis it is important to include the gain information in these parameters, therefore eliminating the need to store the excitation pulse amplitudes.
A method for complex cepstrum optimization has been proposed. The approach searches for the best pitch onset position given initial estimates of the complex cepstrum, followed by complex cepstrum re-estimation. The mean squared error between natural and synthesized speech is minimized during the optimization process. During complex cepstrum re-estimation, no windowing or phase unwrapping is performed.
Deriving the complex cepstrum means that the speech signal in its full representation is being parameterised. By extracting the complex cepstrum through the minimisation of the mean squared error between natural and synthetic speech means that a more accurate representation of the speech signal can be achieved. This can result in speech synthesizer which can achieve better quality and expressiveness.
The above method produces synthesis filter parameters and excitation signal parameters derived from the complex cepstrum of an input speech signal. In addition to these, when training a system for speech synthesis other parameters will also be derived. In an embodiment, the input to such a system will be speech signals and corresponding input text.
From the input speech signals, the complex cepstrum parameters are derived as described in relation to
Labels are extracted from the input text. From these statistical models are then trained which comprise means and variances of the synthesis filter parameters (derived from the complex cepstrum as described above), the log of the fundamental frequency F0, the aperiodicity components and phoneme durations are then stored. In an embodiment, the parameters will be clustered and stored as decision trees with the leaves of the tree corresponding to the means and variances of a parameters which correspond to a label or a group of labels.
In an embodiment, the system of
x(n)=xm(n)*xa(n). (30)
The minimum-phase cepstrum, {circumflex over (x)}m(n)is a causal sequence and can be obtained from the complex cepstrum, {circumflex over (x)}(n) as follows:
where C is the cepstral order. The all-pass cepstrum {circumflex over (x)}a(n) can then be simply retrieved from the complex and minimum-phase cepstrum as
{circumflex over (x)}
a(n)={circumflex over (x)}(n)−{circumflex over (x)}m(n), n=−C, . . . , C. (32)
By substituting (31) into (32) it can be noticed that the all-pass cepstrum {circumflex over (x)}a(n) is non-causal and anti-symmetric, and only depends on the non-causal part of {circumflex over (x)}(n)
Therefore, {{circumflex over (x)}(−C), . . . , {circumflex over (x)}(−1)} carries the extra phase information which is taken into account when using complex cepstrum analysis. For use in acoustic modeling phase parameters are derived, defined as the non-causal part of {circumflex over (x)}(n).
φ(n)=−{circumflex over (x)}(−n−1)={circumflex over (x)}a(n+1), n=0, . . . , Ca, (34)
where Ca<C is the order of the phase parameters.
When training parameters for use in systems of the above described types, the complex cepstrum based synthesis filter can be realized as the cascade of an all pass filter, derived from the phase parameters, and where only the phase information is modified and all other information is preserved, and a minimum phase filter, derived from the minimum-phase cepstrum. In such systems, the training method will comprise a further step of decomposing the complex cepstrum into phase and minimum phase components. These parameters can be used to from decision trees and pre-stored in a synthesiser product.
1) means and variances of the minimum phase cepstrum parameters;
2) means and variances of the fundamental frequency;
3) means and variances of the aperiodicity components;
4) means and variances of the phoneme durations;
5) means and variances of the phase parameters.
and
1) decision trees for the minimum phase cepstrum parameters
2) decision trees for the fundamental frequency;
3) decision trees for the aperiodicity components;
4) decision trees for the phoneme durations;
5) decision trees for the phase parameters.
Text is input at step S201. Labels are then extracted from this text in step S203. The labels give information about the type of phonemes in the input text, context information etc. Then, the phone durations are extracted in step S205, from the stored decision trees and means and variances for phone duration. Next, by using both the labels and generated durations the other parameters are generated.
In step S207, F0 parameters are extracted using the labels and the phone durations. The F0 parameters are converted into a pulse train t(n) in step S209.
In step S211 which may be performed concurrently, before or after step S207, the phase parameters are extracted from the stored decision trees and means and variances for phase. These phase parameters are then converted to into an all pass impulse response in step S213. This filter is then used to in step S215 to filter the pulse train t(n) produced in step S209.
In step S217, band aperiodicity parameters are extracted from stored decision trees. The band-aperiodicity parameters are interpolated to result in L+1 aperiodicity coefficients {α0, . . . , αL}. The aperiodicity parameters are used to derive the voiced Hv and unvoiced Hu filter impulse in step S219.
The voiced filter impulse is applied to the filtered voice pulse train tm(n) in step S221. A white noise signal, generated by a white noise generator, is input to the system to represent the unvoiced part of the signal and this is filtered by the unvoiced impulse response in step S223.
The voiced excitation signal which has been produced in step S221 and the unvoiced excitation signal which has been produced in step S223 are then mixed to produce mixed excitation signal in step S225.
The minimum phase cepstrum parameters are then extracted in step S227 using the text labels and phone durations. The mixed excitation signal is then filtered in step S229 using minimum phase cepstrum signal to produce the reconstructed voice signal.
Although the above description has been mainly concerned with the extraction of an accurate complex cepstrum for the purposes of training a speech synthesiser, the systems and methods described above have applications outside that of speech synthesis. For example, because h(n) contains information of the glottal flow (glottal effect on the air that passes though the vocal tract), h(n) gives information on the quality/style of the voice of the speaker, such as if he/she is tense, angry, etc, as well as being used for voice disorder detection.
Therefore, the detection of h(n) can be used for voice analysis.
While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed novel methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.
Number | Date | Country | Kind |
---|---|---|---|
1221637.0 | Nov 2012 | GB | national |