Patent Grant
7567896
The present invention relates to generating synthesized speech through concatenation of speech segments that are derived from a large prosodically-rich corpus of speech segments including using an additional dictionary of speech segment identifier sequences.
Machine-generated speech can be produced in many different ways and for many different applications. The most popular and practical approach towards speech synthesis from text is the so-called concatenative speech synthesis technique in which segments of speech extracted from recorded speech messages are concatenated sequentially, generating a continuous speech signal.
Many different concatenative synthesis techniques have been developed, which can be classified by their features:
A common method for generating speech waveforms is by a speech segment composition process that consists of re-sequencing and concatenating digital speech segments that are extracted from recorded speech files stored in a speech corpus, thereby avoiding substantial prosody modifications.
The quality of segment resequencing systems depends among other things on appropriate selection of the speech units and the position where they are concatenated. The synthesis method can range from restricted input domain-specific “canned speech” synthesis where sentences, phrases, or parts of phrases are retrieved from a database, to unrestricted input corpus-based unit selection synthesis where the speech segments are obtained from a constrained optimization problem that is typically solved by means of dynamic programming.
Table 1 establishes a typology of TTS engines depending on several characteristics.
All the technologies mentioned in Table 1 are currently available in the TTS market. The choice of TTS integrators in different platforms and products is determined by a compromise between processing power needs, storage capacity requirements (footprint), system flexibility, and speech output quality.
In contrast to corpus-based unit selection synthesis, canned speech synthesis can only be used for restricted input domain-specific applications where the output message set is finite and completely described by means of a number of indices that refer to the actual speech waveforms.
While canned speech synthesizers use large units such as phrases (described in E. Klabbers, “High-Quality Speech Output Generation Through Advanced Phrase Concatenation,” Proc. of the COST Workshop on Speech Technology in the Public Telephone Network: Where are we today?, Rhodes, Greece, pages 85-88, 1997), words (described in H. Meng, S. Busayapongchai, J. Glass, D. Goddeau, L. Hetherington, E. Hurley, C. Pao, J. Polifroni, S. Sene, and V. Zue, “WHEELS: A Conversational System In The Automobile Classifieds Domain,” in Proc. ICSLP '96, Philadelphia, Pa., October 1996, pp. 542-545), and morphemes, corpus-based speech synthesizers use smaller units such as phones (described in A. W. Black, N. Campbell, “Optimizing Selection Of Units From Speech Databases For Concatenative Synthesis,” Proc. Eurospeech '95, Madrid, pp. 581-584, 1995), diphones (described in P. Rutten, G. Coorman, J. Fackrell & B. Van Coile, “Issues in Corpus-based Speech Synthesis,” Proc. IEE symposium on state-of-the-art in Speech Synthesis, Savoy Place, London, April 2000), and demi-phones (described in M. Balestri, A. Pacchiotti, S. Quazza, P. L. Salza, S. Sandri, “Choose The Best To Modify The Least: A New Generation Concatenative Synthesis System,” Proc. Eurospeech '99, Budapest, pp. 2291-2294, September 1999).
Both types of applications use a different unit size because the size of the database grows exponentially with the size of the unit under the condition of full coverage. Canned speech synthesis is widely used in domain specific areas such as announcement systems, games, speaking clocks, and IVR systems.
Corpus-based speech synthesis systems make use of a large segment database. A large segment database refers to a speech segment database that references speech waveforms. The database may directly contain digitally sampled waveforms, or it may include pointers to such waveforms, or it may include pointers to parameter sets that govern the actions of a waveform synthesizer. The database is considered “large” when, in the course of waveform reference for the purpose of speech synthesis, the database commonly references many waveform candidates, occurring under varying linguistic conditions. In this manner, most of the time in speech synthesis, the database will likely offer many waveform candidates from which a single waveform is selected. The availability of many such waveform candidates can permit prosodic and other linguistic variation in the speech output stream.
Speech resequencing systems access an indexed database composed of natural speech segments. Such a database is commonly referred as the speech segment database. Besides the speech waveform data, the speech segment database contains the locations of the segment boundaries, possibly enriched by symbolic and acoustic features that discriminate the speech segments. The speech segments that are extracted from this database to generate speech are often referred in speech processing literature as “speech units” (SU). These units can be of variable length (e.g. polyphones). The smallest units that are used in the unit selector framework are called basic speech units (BSUs). In corpus-based speech synthesis, these BSUs are phonetic or sub-word units. If part of a synthesized message is constructed from a number of BSUs that are adjacent in the speech corpus (i.e. convex sequence of BSUs), then the concatenation step can be avoided between these units. We will use the term Monolithic Speech Unit (MSU) when it's necessary to emphasize that a given speech unit corresponds to a convex sequence of BSUs.
A corpus-based speech synthesizer includes a large database with speech data and modules for linguistic processing, prosody prediction, unit selection, segment concatenation, and prosody modification. The task of the unit selector is to select from a speech database the ‘best’ sequence of speech segments (i.e. speech units) to synthesize a given target message (supplied to the system as a text).
The target message representation is obtained through analysis and transformation of an input text message by the linguistic modules. The target message is transformed to a chain of target BSU representations. Each target BSU representation is represented by a target feature vector that contains symbolic and possibly numeric values that are used in the unit selection process. The input to the unit selector is a single phonetic transcription supplemented with additional linguistic features of the target message. In a first step, the unit selector converts this input information into a sequence of BSUs with associated feature vectors. Some of the features are numeric, e.g. syllable position in the phrase. Others are symbolic, such as BSU identity and phonetic context. The features associated with the target diphones are used as a way to describe the segmental and prosodic target in a linguistically motivated way. The BSUs in the speech database are also labeled with the same features.
For each BSU in the target description, the unit selector retrieves the feature vectors of a large number of BSU candidates (e.g. diphones as illustrated in
Each of these candidate BSUs is scored by a multi-dimensional cost function that reflects how well its feature vector matches the target feature vector—this is the target cost. A concatenation cost is calculated for each possible sequence of BSU candidates. This too is calculated by a multi-dimensional cost function. In this case the cost reflects the cost of joining together two candidate BSUs. If the prosodic or spectral mismatch at the segment boundaries of two candidates exceeds the hearing threshold, concatenation artifacts occur.
In order to reduce and preferably avoid concatenation artifacts, masking functions (as defined in G. Coorman, J. Fackrell, P. Rutten & B. Van Coile, “Segment selection in the L&H Realspeak laboratory TTS system”, Proceedings of ICSLP 2000, pp. 395-398) that facilitate the rejection of bad segment combinations in the unit selection process are introduced. A dynamic programming algorithm is used to find the lowest cost path through all possible sequences of candidate BSUs, taking into account a well-chosen balance between target costs and concatenation costs. The dynamic programming assesses many different paths, but only the BSU sequence that corresponds with the lowest cost path is retained and converted to a speech signal by concatenating the corresponding monolithic speech units (e.g. polyphones as illustrated in
Although the quality of corpus-based speech synthesis systems is often very good, there is a large variance in the overall speech quality. This is mainly because the segment selection process as described above is only an approximation of a complex perceptual process.
The input phonetic data sequence is converted by the target generator 111 into a multi-layer internal data sequence to be synthesized. This internal data sequence representation, known as extended phonetic transcription (XPT), contains mainly the linguistic feature vectors (including phonetic descriptors, symbolic descriptors, and prosodic descriptors) such as those in the speech segment database 141.
The unit selector 131 retrieves from the speech segment database 141 descriptors of candidate speech units that can be concatenated into the target utterance specified by the XPT transcription. The unit selector 131 creates an ordered list of candidate speech units by comparing the XPTs of the candidate speech units with the target XPT, assigning a target cost to each candidate. Candidate-to-target matching is based on symbolic feature vectors, such as phonetic context and prosodic context, and numeric descriptors, and determines how well each candidate fits the target specification. Poorly matching candidates may be excluded at this point.
The unit selector 131 determines which candidate speech units can be concatenated without causing disturbing quality degradations such as clicks, pitch discontinuities, etc. Successive candidate speech units are evaluated by the unit selector 131 according to a quality degradation cost function. Candidate-to-candidate matching uses frame-based information such as energy, pitch and spectral information to determine how well the candidates can be joined together. Using dynamic programming, the best sequence of candidate speech units is selected for output to the speech waveform concatenator 151.
The speech waveform concatenator 151 requests the output speech units (e.g. diphones and/or polyphones) from the speech unit database 141 for the speech waveform concatenator 151. The speech waveform concatenator 151 concatenates the speech units selected forming the output speech that represents the target input text.
It has been reported that the average quality of unit selection synthesis is increased if the application domain is closer to the domain of the recordings. Canned speech synthesis, which is a good example of domain specific synthesis, results in high quality and extremely natural synthesis beyond the quality of current corpus-based speech synthesis systems. The success of canned speech synthesis lies in the size of the speech segments that are being used. By recording words and phrases in prosodic contexts similar to the ones in which they will be used, a very high naturalness can be achieved. Because the segments used in canned speech applications are large, they embed detailed linguistic and paralinguistic information. It is not straightforward to embed this information in synthesized speech waveforms by concatenating smaller segments such as diphones or demi-phones using automatic algorithms.
The quality of domain-specific unrestricted input TTS can be further increased by combining canned speech synthesis with corpus-based speech synthesis into carrier-slot synthesis. Carrier-slot speech synthesis combines carrier phrases (i.e. canned speech) with open slots to be filled out by means of corpus-based concatenative synthesis. The corpus-based synthesis can take into account the properties of the boundaries of the carriers to select the best unit sequences.
Canned speech synthesis systems work with a fixed set of recorded messages that can be combined to create a finite set of output speech messages. If new speech messages have to be added, new recordings are required. This also means that the size of the database grows almost linearly with the number of messages that can be generated. Similar remarks can be made about corpus-based synthesis. Whatever speech unit is used in the database, it is desirable that the database offers sufficient coverage of the units to make sure that an arbitrary input text can be synthesized with a more or less homogeneous quality. In practical circumstances it is difficult to achieve full coverage. In what follows we will refer to this as the data scarcity problem.
A common approach to increase the number of messages that can be synthesized with high quality is to add more speech data to the speech unit database until the average quality of the system saturates. This approach has several drawbacks such as:
The speech segment database development procedure starts with making high quality recordings in a recording studio followed by auditory and visual inspection. Then an automatically generated phonetic transcription is verified and corrected in order to describe the speech waveform correctly. Automatic segmentation results and prosodic annotation are manually verified and corrected. The acoustic features (spectral envelope, pitch, etc.) are estimated automatically by means of techniques well known in the art of speech processing. All features which are relevant for unit selection and concatenation are extracted and/or calculated from the raw data files.
Single speaker speech compression at bit rates far below the bit rates of traditional coding systems can be accomplished by resequencing speech segments. Such coders are referred to as very low bit rate (VLBR) coders. Initially, VLBR coding was achieved by modeling speech as a sequence of acoustically segmented variable-length speech segments.
Phonetic vocoding techniques can achieve lower bit rates by extracting more detailed linguistic knowledge of the information embedded in the speech signal. The phonetic vocoder distinguishes itself from a vector quantization system in the manner in which spectral information is transmitted. Rather than transmitting individual codebook indices, a phone index is transmitted along with auxiliary information describing the path through the model.
Phonetic vocoders were initially speaker specific coders, resulting in a substantial coding gain because there was no need to transmit speaker specific parameters. The phonetic vocoder was later on extended to a speaker independent coder by introducing multiple-speaker codebooks or speaker adaptation. The voice quality was further improved where the decoding stage produced PCM waveforms corresponding to the nearest templates and not based on their spectral envelope representation. Copy synthesis was then applied to match the prosody of the segment prototype appropriately to the prosody of the target segment. These prosodically modified segments are then concatenated to produce the output speech waveform. It was reported that the resulting synthesized speech had a choppy quality, presumably due to spectral discontinuities at the segment boundaries.
The naturalness of the decoded speech was further increased by using multiple segment candidates for each recognized segment. In order to select the best sounding segment combination, the decoder performs a constrained optimization similar to the unit selection procedure in corpus-based synthesis.
Extremely low bit rates were achieved by combining an ASR system with a TTS system. But these systems are very error prone because they depend on two processes that introduce significant errors.
A representative embodiment of the present invention includes a system and method for producing synthesized speech from message designators. A first large speech segment database references speech segments, where the database is accessed by speech segment designators. Each speech segment designator is associated with a sequence of speech segments having at least one speech segment. A segmental transcription database references segmental transcriptions that can be decoded as a sequence of segment designators, where the segmental transcription database is accessed by the message designators. Each message designator is associated with a fixed message. A first speech segment selector sequentially selects a number of speech segments referenced by the speech segment database using a sequence of speech segment designators that is decoded from a segmental transcription retrieved from the segmental transcription database. A speech segment concatenator in communication with the first speech segment database concatenates the sequence of speech segments designated by a segmental transcription from the segmental transcription database to produce a speech signal output.
A further embodiment includes a digital storage medium in which the speech segments are stored in speech-encoded form, and a decoder that decodes the encoded speech segments when accessed by speech segment selector.
Another embodiment includes a system and method for producing synthesized speech from input text and from input message designators. A first and a second large speech segment database reference speech segments, where the database is accessed by speech segment designators. Each speech segment designator is associated with a sequence of basic speech segments having at least one basic speech segment. A segmental transcription database references segmental transcriptions, where each segmental transcription can be decoded as a sequence of segment designators of the first large speech segment database, and wherein the segmental transcription database is accessed by the message designators, each message designator being associated with a fixed message. A text message database references text messages that correspond to the orthographic representation of the segmental transcriptions of the segmental transcription database. A first speech segment selector sequentially selects a number of speech segments referenced by the first speech segment database using a sequence of speech segment designators that is decoded from the segmental transcription corresponding to the message designator. A text analyzer converts the input text into a sequence of symbolic segment identifiers. A second speech segment selector, in communication with the second speech segment database, selects, based at least in part on prosodic and acoustic features, speech segments referenced by the database using speech segment designators that correspond to a phonetic transcription input. A message decoder activates the first speech segment selector if the input text corresponds to a text message from the text message database or activates the second speech segment selector if the input text does not correspond to a message from the text message database. A speech segment concatenator in communication with the first and second speech segment database concatenates the sequence of speech segments designated by a segmental transcription from the segmental transcription database to produce a speech signal output.
In a further embodiment, the first and second speech segment database may be the same, or the first speech segment database may be a subset of the second speech segment database, or the first and second speech segment database may be disjoint. The first and second database may reside on physically different platforms such that a data stream consisting of segment transcriptions, speech transformation descriptors, and control codes is transmitted from one platform to another enabling distributed synthesis.
In various embodiments, the messages may correspond to words and/or multi-word phrases, such as for a talking dictionary application. The segment designators may be one or more of the following types: (i) diphone designators, (ii) demi-phone designators, (iii) phone designators, (iv) triphone designators, (v) demi-syllable designators, and (vi) syllable designators.
The speech segment concatenator may not alter the prosody of the speech segments. The speech segment concatenator may smooth energy at the concatenation boundaries of the speech segments, and/or smooth the pitch at the concatenation boundaries of the speech segments.
The segment selector may be tunable and alternative segment candidates may be selected by a user to generate a segmental transcription database. The segment selector may be trained on a given segment transcriptor database and alternative segment candidates may be selected by a user or automatically to generate a segmental transcription database or speech.
Embodiments may also include closed loop corpus-based speech synthesis, i.e., speech synthesis consisting of an iteration of synthesis attempts in which one or more parameters for unit selection or synthesis are adapted in small steps in such a way that speech synthesis improves in quality.
The following description is illustrative of the invention and is not to be construed as limiting the invention. Several details are described to obtain a thorough understanding of present invention. However, in certain circumstances, well known, or conventional details are not described in order not to obscure the present invention in detail. Reference throughout this specification to “one embodiment”, “an embodiment”, “preferred embodiment” or “another embodiment” indicates that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearance of the phrase “in one embodiment”, “in an embodiment”, or “in a preferred embodiment” in various places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristic may be combined in any suitable manner in one or more embodiments.
Various embodiments of the present invention are directed to techniques for corpus-based speech synthesis based on concatenation of carefully selected speech units, such as that described in G. Coorman, J. De Moortel, S. Leys, M. De Bock, F. Deprez, J. Fackrell, P. Rutten, A. Schenk & B. Van Coile, “Speech Synthesis Using Concatenation Of Speech Waveforms,” U.S. Pat. 6,665,641, incorporated herein by reference. Such approaches can lead to synthetic speech that is perceptually indistinguishable from speech produced by a human speaker, which we refer to as “transparent synthesis.”
From a perceptual point of view, transparent synthesis results are equivalent to natural speech signals and can thus be added to the segment database. These transparent synthesis results are intrinsically phoneme segmented and annotated because they are derived from segmented and annotated speech data. The transparent synthesis results are not monolithic but are composed of a sequence of monolithic speech units. Therefore we will also refer to them as “compound messages.”
When added to the speech database, the unit selector can extract convex chains of speech units (i.e. chains of consecutive speech units) from the compound messages. We will refer to these convex chains of BSUs as “compound monolithic speech units” (CMSUs) to distinguish them from the traditional monolithic speech units. All elementary units derived from compound messages that are added to the large segment database will be referred to as “compound speech units” (CSUs) to distinguish them from the standard basic speech units. As will be shown further on, the feature vector of a CSU will often differ from the feature vector of the corresponding BSU from which it is drawn from.
The term “compound” as used in compound speech unit has a double meaning. Compound refers to the compound messages that compound speech units are extracted from, and also to the fact that the feature vector is the compound of a modified linguistic feature vector and an acoustic feature vector that belongs to the corresponding BSU.
CMSUs have the same properties for synthesis as monolithic speech units, but are not adjacent in the original recorded speech signal from which they are extracted. The unit selector of the diphone system, depicted in
In one embodiment of the invention, the speech quality of a corpus-based synthesis is enhanced by adding compound speech units to the speech segment database resulting in an increase of the average segment length. This approach offers various advantages which may include that:
The addition of compound speech messages can be done in various different ways. Because the compound speech messages are composed out of segments that are already in the database, no extra acoustic information needs to be added. The compound speech messages can be broken down into a sequence of BSUs. These BSUs can be described by symbolic speech unit feature vectors derived by transplanting the target feature vector description to the compound speech message possibly followed by a hand correction after auditory feedback (done, for example, by a language expert).
The symbolic feature vectors associated with the BSUs are extracted from the hand corrected symbolic feature values. For example, in the phoneme string, primary and secondary stress are automatically obtained through a set of the language modules. Because the language modules are not perfect, and because of pronunciation variation, an extra manual correction step might be required. Therefore this symbolic representation can be quite different from the automatically generated annotation by the grapheme-to-phoneme conversion. However, by transplanting the automatically generated symbolic target feature vectors to the compound messages, the data in the speech segment database and the grapheme-to-phoneme converter will better match. An embodiment of this invention uses automatically annotated compound speech units to achieve a better match between symbolic feature generation in the grapheme-to-phoneme conversion and the symbolic feature vectors used in speech segment database.
Besides expanding the concept of adjacency, the segment database is enriched by new, slightly modified feature vectors through the addition of compound messages to the large segment database. By adding compound messages to the database, only non-acoustic feature values are subjected to a possible modification. For example, the phonetic context, the position of the unit in the sentence or the level of prominence may differ from their original. In this way, variation is added to the segment database without resorting to. new recordings. Non-convex speech unit sequences that are retrieved as convex sequences from the compound utterances have the same advantages as monolithic speech units.
Each speech unit feature vector that belongs to a BSU in the database represents a single point in the multidimensional feature space. By adding speech units from compound utterances to the speech base, one BSU can be represented by an ensemble of points in the multidimensional feature space. Thus adding compound speech units to a speech segment database reduces the data scarcity of that speech segment database. The storage and the use of compound speech units are claimed by the invention.
Database Organization
The addition of many compound speech units to the speech unit database introduces redundancy. The unit feature vector contains linguistic, paralinguistic and acoustic features. The acoustic features remain the same for all unit feature vectors that related to the same BSU waveform. For each CSU, the acoustic features remain the same, and should therefore be stored only once.
A separation of the acoustic features from the other features as shown in
Speech synthesis requires that a speech segment be identified in the linguistic space, the acoustic space and the waveform space. Therefore, the segment identifier might consist out of three parts. In corpus-based synthesis, the segment identifier corresponds typically to a unique index that is used directly or indirectly to address and retrieve the linguistic and acoustic feature vectors and the speech waveform parameters of a given speech segment (BSU). The addressing can for example be done through an intermediate step of consulting address lookup tables.
The use of compound speech units extinguishes the uniqueness concept of the segment identifier because a single acoustic feature vector can be referenced by more than one compound speech unit. To avoid confusion, the segment identifier is now defined as a unique identifier that references directly or indirectly the invariant part of the segment description (i.e. acoustic features if any and waveform parameters). The segment descriptor is defined as the combination of the linguistic feature vector and the segment identifier. The acoustic feature vectors are stored in the acoustic database or in a database that is linked with the acoustic database, while the linguistic feature vectors are stored in the segment descriptor database (that can in some implementation be physically included in the acoustic database).
A segment descriptor contains the linguistic feature vectors and a segment identifier that is or that can be transformed to a pointer to the speech segment representation in the acoustic database. The acoustic feature vector contains among others acoustic features for concatenation cost calculation (such as pitch and mel-cepstrum at the edges) but also features such as average pitch and energy level. The linguistic feature vector includes among other things prominence, boundary strength, stress, phonetic context and position in the phrase. For applications such as dictionary pronunciation systems, linguistic and/or acoustic feature vectors might not be required for the application and can therefore be omitted. Each CSU that corresponds to a given BSU has the same segment identifier.
In one embodiment of the invention, a high quality CPU-intensive unit selector (
Use of compound speech units in corpus-based synthesis is a way of training the unit selector by incorporating higher precision perceptual information through data addition. This is somewhat analogous to automatic speech recognition (ASR), where recognition accuracy is increased by training on large corpora of recorded speech. Recorded speech is applied to the ASR system and evaluation and training is done automatically using the known text transcription of the corpus. In the present context of text-to-speech (TTS), text is applied to the speech synthesis system and perceptual evaluation of the generated output speech is required (e.g. by listening) as a feedback training mechanism.
Speech Unit Database Reduction
Embodiments present interesting issues with regards to speech unit database reduction. Besides reduction in database size (making embodiments more suitable for small footprint platforms), the unit selection process can increase in speed as the number of BSU candidates is reduced. For speech unit database reduction, which speech units can be removed from the database needs to be determined in such a way that the degradation is minimal. One way to solve this problem is by using an auditory-motivated distance measure in the feature vector space. But since the feature vector space is of a high dimension, the relationship between the (linguistic) features and the quality is complex and difficult to understand. Therefore it is difficult to construct auditory-motivated distance measures.
As discussed above, after constructing many compound speech units, each BSU can be described by a set of symbolic feature vectors. The level of overlap between the feature sets may be a good measure for the redundancy of the speech units. Besides the level of overlap, the size of the sets can also be used as a measure to indicate the importance of a speech segment.
Constructing CSUs after an initial stage of database creation can immediately enrich the database without making additional recordings, thereby reducing the amount of additional recordings that are required to create a large speech base. Standard database creation relies heavily on efficient text selection to ensure rich coverage of acoustic and symbolic features in the database. Clustering techniques such as vector quantization (VQ) can be applied afterwards to reduce the size of the database without degrading the resulting synthesis quality, basically by removing redundancy that crept into the database during development.
One proposed framework for database creation (
An extreme and simplified application of using synthesis feedback consists of listening to target words and adding them to the database as CSU when they have transparent quality. This has several advantages:
The use of compound speech units in corpus-based speech synthesis can be seen as an exploration/exploitation of the speech unit feature space. The parameter settings that have an influence on the unit selection process limit the space of unit combinations. Several settings of those parameters can be tried out in order to enlarge the space of speech unit combinations and to make more efficient use of the parameter settings.
Composition Procedure
Besides finding an optimal set of features, cost functions, and weights, it is also important to have the right sort of speech data. It could be that the amount of prosodic variation needed is simply not present within an existing speech database. To increase the prosodic coverage of the speech database it might be necessary to first add prosodically rich data to the speech segment database. The new data should be carefully selected to increase prosodic variation while keeping redundancy to a minimum. To ensure variety and naturalness it is better to add continuously recorded messages to the speech segment database. These recordings are more difficult to process, e.g. the automatic segmentation and labeling of the recordings is more difficult because the speech contains more assimilation and more artifacts like clicks and breathing noises.
Output Validation
Validation can help to find synthesis results of transparent quality. The validation corresponds to a good/bad classification of the synthesis results in two distinct partitions based on perceptual measures.
There are many ways to facilitate the validation process. A semi-automatic validation process where a first machine classification is performed by means of simple segment continuity measures may be followed by a “manual” validation of a smaller set of computer generated utterances. This is the simple validation scheme will be referred to as “simple validation”.
The Use Of Multiple Unit Selectors
The selected path is a function of the parameters of the unit selector. The unit selector assesses many different paths but only the best one needs to be retained. But other paths besides the chosen one can result in good or even better speech quality. Therefore, it is useful to explore the space of the possible “best” unit sequences by varying the parameters of the unit selector, and to select the best one by listening to it or by using objective supra-segmental quality measures.
In a practical situation, the outputs of N (>1) unit selectors with different parameter settings can be compared, and the best synthesis result chosen (if it is acceptable).
During the validation process several statistics of the costs of the different unit selectors are collected and stored in a training database. This training database can be used to train a classifier that can be used as an automatic validation tool.
In one embodiment, a decision tree, well-known by those familiar with speech technology, is trained on the cost vectors of the unit selectors. The cost vectors are of fixed dimension and contain the accumulated cost and some statistics (such as maximum and average) of the sub-costs of the concatenation costs and the target costs. Other well-known techniques such as neural networks can similarly be used for this task.
Stochastic Unit Selector
In each candidate list, many segments may share the same target cost value because the symbolic cost function calculation involves a small set of symbolic features. Most symbolic features produce a small set of cost values. Segments with an identical target cost do not necessarily sound equal. It is very likely that different segments with the same target cost will have a different prosodic realization. In the deterministic approach, the differentiation between the segments with equal target cost is done by examining their ability to join to neighboring segments (i.e. concatenation cost calculation). As discussed above, many transitions can't be differentiated either. This means that in an optimal framework where the cost functions are tuned optimally there might be several paths with the same best cumulative cost.
The use of piecewise constant segments in the masking function encourages less differentiation between the candidate segments. It is very likely that (especially for large databases) certain “equally good” paths are not taken into account because the combination of node- and transition-costs are identical. In order to bring more variation in the unit selection process (in order to discover better and more compound messages) probabilities can be introduced at the level of the unit selector.
All cost functions in combination with their masking functions used in traditional unit selectors are monotone rising functions. However, a small increase in cost between different segments does not necessarily mean that there will be an audible degradation of the signal quality.
By introducing a small noise level superimposed on the piece-wise constant (flat) parts of the masking function, the unit selection process will become non-deterministic and will provide variation without audible quality loss. In a further step, some noise can be added to the non-constant parts of the masking function also. In this way a variety of “quasi-equal quality” segment sequences is obtained. The noise level will finally determine if the differences in quality between the best sequence (noise less) and the quasi-optimal sequence will be audible. By controlling the noise level we can obtain variation and produce “equally good” speech unit sequences.
Besides using an additive noise level, one can substitute the cost and eventually the masking function with a random generator with a distribution depending on the arguments of the cost function (typically the feature distance) in such a way that the probability density function of the noise generator (described by its mean and variance for example) reflects the penalty (corresponding to the cost) that the developer wants to assign to it. An example is shown in
The stochastic unit selector can successfully be used in a multi-unit selector framework as described above. However, the stochastic unit selector can also be used in another multi-unit selector framework in which a large number of successive unit selections are done by means of the same stochastic unit selector and where the statistics of the selected units of the successive unit selections are used in order to select the best segment sequence. One embodiment of the invention selects the segment sequence that corresponds with the most frequent units.
Closed Loop Validation (Automatic)
It is difficult to automatically judge if a synthesized utterance sounds natural or not. However it is doable to estimate the audibility of acoustic concatenation artifacts by using acoustic distance measures.
The unit selection framework is strongly non-linear. Small changes of the parameters can lead to a completely different segment selection. In order to increase the synthesis quality for a given input text, some synthesizer parameters can be tuned to the target message by applying a series of small incremental changes of adaptive magnitude. We will call this the closed loop approach.
For example, audible discontinuities can be iteratively reduced by increasing the weight on the concatenation costs in small steps over successive synthesis trials until all (or most) acoustic discontinuities fall below the hearing threshold. The adaptation of the synthesizer parameters is done automatically. This scheme is presented in
In one embodiment of the invention, the one-shot unit selector of a corpus-based synthesizer is replaced by an adaptive unit selector placed in a closed loop. The process consists of an iteration of synthesis attempts in which one or more parameters in the unit selector are adapted in small steps in such a way that speech synthesis gradually improves in quality at each iteration. One drawback of this adaptive approach is that the overall speed of the speech synthesis system decreases
Another embodiment of the invention iteratively fine-tunes the unit selector parameters based on the average concatenation cost. The average concatenation cost can be the geometric average, the harmonic average, or any other type of average calculation.
Alternatives To Increase Segmental Variability
A typical corpus-based speech synthesizer synthesizes only one utterance for a given input message. This single synthesis result is than accepted or rejected by means of a binary decision strategy (listener or automatic technique). A rejection of a single synthesis result does not always mean that there is no possible basic speech unit combination for a given input text that could lead to transparent quality. This is mainly because the unit selector is not able to model the real perceptual cost.
As an alternative, the N-best synthesis results can be presented to the classifier (i.e. listener/machine). The N-best synthesis results are found based on the N-best paths trough the candidate speech units in the dynamic programming step. Unfortunately the N-best synthesis results will share many speech unit combinations leading to small variations between the synthesis results.
An efficient approach that results in completely different unit combinations is obtained by a series of N different synthesis phases. The first synthesis phase is accomplished through normal synthesis. In the following phases, some units that were selected in a previous synthesis phase are removed from the unit candidate lists. The selection of the units that are withheld from synthesis in the successive phases is based on the target cost of the remaining units. For example: if the target cost of the other candidate units is unacceptably high then the unit is not removed from the unit candidate list, however if there are remaining units with sufficient low cost, than alternative units can be chosen. In other words we look only for new candidates in the node feature space in the neighborhood of the best units.
It is further possible to automate the selection process if reference recordings are available. The N-best synthesis results can be scored automatically by dynamic time warping them with the reference recording (preferably of the same speaker). The synthesis result with the smallest cumulative path cost is the winner and can eventually be further evaluated in a listening experiment.
Creation Of Compound Utterances By Means Of Dynamic Time Warping (DTW)
This approach starts from recorded speech that is not added to the database but that will be used to select segments based on its acoustic realization only.
The composition algorithm looks as follows:
The “Composition Table”: Automatic Unit Composition Based On Concatenation Cost
For a given speech unit database it is possible to construct a speech unit concatenation cost matrix, which we will refer to as a “combination matrix.” The number of combinations grows quadratic with the size of the database, extremely large combination matrices are not affordable for speech synthesis. However, a large number (e.g. 500,000) of the most frequent CSUs can be stored (i.e. compound speech units with negligible internal concatenation costs and similar linguistic features at their internal boundaries). If the composition process is calculated off-line, more precise and complex error measures can be used to calculate the perceptual quality of the CSU. It is possible for instance to incorporate the error resulting from the waveform concatenation process into the concatenation cost. High quality speech unit combinations that are not adjacent in the original recording from which they are extracted can be stored in an automatically generated “composition table”.
Compound Speech Unit Dictionaries (CSU Dict)
The basic flow of a general corpus-based TTS system is shown in
The parameters of a unit-selector of a system are tuned towards a general optimal performance given the content of the speech database and the feature set. This general performance reflects the quality of the system. The general optimal performance is therefore sub-optimal for very specific tasks (due to the generalization error), e.g. pronunciation of proper names, city names, high natural sounding speech generation of sentences from which subunits are lacking form the speech database.
To solve this problem one could infinitely add data to the speech database. But that is a sub-optimal solution since it increases the size of the database and is a labor-intensive task (the data needs to be recorded and processed). Also due to generalization of the unit selector, it may not be able to retrieve all newly added data.
Tagging the newly added data as sub-database might help. When encountering this tag, the unit selector performs a dedicated search in a dedicated sub-database. Again, the outcome of the unit selector is not guaranteed, and tagging and adding data still involves a manual task by the speech database developer. A better solution in terms of quality, effort, memory, and processing power is to introduce the principle of segment descriptor lookup and segment descriptor user dictionaries (i.e., a dictionary containing the compound speech units).
This very same principle can be applied to a full TTS system (see
At run time, the unit-selector consults the segment descriptor dictionary. The segment identifier stream could be pre-loaded into the dynamic programming grid, if the prosodic and join features are available for the segment descriptors from the segmental dictionary. The dynamic programming algorithm (DP) searches for the optimal solution. Non-linear weights on the segment descriptors from the dictionaries will guarantee a seamless integration of the units retrieved from the dictionary into a new segmental stream. This principle takes it one step further than the standard carrier-slot approach where the carriers are described by means of phonetic streams. If the prosodic and join features are not available for the segments then the unit selector is by-passed and lookup and synthesis can start.
For closed datasets the segment descriptor dictionary can be accessed immediately from the orthography thereby replacing both the grapheme-to-phoneme conversion and the unit selector module. Homographs must be tagged correctly then.
Corpus-Based Canned Speech Synthesizer
There are some analogies between the use of compound speech units and canned speech synthesis. In one embodiment of the invention, aspects of canned speech synthesis and corpus-based speech synthesis systems are combined to create a corpus-based canned speech synthesis system that can easily be extended and changed by the user without falling back on extra recordings. Just like carrier-slot applications, it helps to fill the gap between the traditional canned speech synthesis applications and corpus-based synthesis approach. The basic speech unit may be “small” (e.g. diphone) such as in traditional corpus-based synthesis.
A single prototype speech segment may be used as a building block to generate a number of different speech messages. On average, one prototype speech segment may be used in the construction of more than one speech message. In order to generate speech, the corpus-based canned speech synthesizer accesses a large prosodically-rich database of small speech segments. In order to find the right speech segments, the corpus-based canned speech synthesizer utilizes a database of segment identifier sequences that can be interpreted as a compressed representation of the messages to be synthesized.
The selection of the speech segments is done off-line by means of a unit selector that acts on the same segment database, preferably assisted by a listener who fine-tunes and validates output speech messages. However, as mentioned before, the validation process can also be done automatically or can be assisted by an automatic means.
The optimal sequence of segment identifiers is stored in a database that can be consulted by the synthesis application or system in order to reproduce the output speech message. For each target segment, the segment database contains many prototypes (candidates) covering many different prosodic realizations, enabling the listener to synthesize many different realizations of the same utterance by, for example, fine-tuning or iterating through the N-best list of the unit selector. Embodiments can also be used in combination with unrestricted-input corpus-based speech synthesis in order to enhance shortcomings of the system or to improve on a certain application domains (e.g. pronunciation of words for language learning etc.)
Another embodiment of the invention consists of a prosodically-rich speech segment database containing a large number of small speech segments (such as diphones and demi-phones etc.), a lookup device and a number of lookup tables that enable speech segment retrieval, and a synthesizer that is capable of concatenating speech segments producing speech waveform messages. Each message that has to be synthesized is encoded as an entry in one or more databases in the form of a sequence of one or more segment identifiers. This non-empty sequence of segment identifiers is called a segmental transcription (in analogy to a phonetic transcription). The segmental transcription is than used by the lookup engine to sequentially retrieve the segments to be concatenated.
In one specific embodiment, the speech segments are encoded and stored as a sequence of parameters of different types. This means that the speech segment retrieval process includes a speech decoder. The process of encoding and decoding of speech waveforms is well known and understood by those familiar with the art of speech processing.
Once the complete speech database has been created, the incremental bit-rate to represent additional speech messages will be very low, and will be mainly determined by the number of bits required to represent the segment identifiers. The word size of the segment identifier is, among other things, dependent on the size of the database. However by taking into account that not all pairs of speech units can be joined together, the bit rate can be further decreased. For example, in the case of diphones, only segments ending and starting with the same phoneme may be joined. By partitioning the set of all diphone segments into classes corresponding to their first phoneme, the segment identifiers can be represented more efficiently.
Because the average length of the variable size units that are created by selecting adjacent speech segments is significantly larger than the length of a basic speech segment from the large prosodic rich segment database, the residual bit rate can be further reduced by applying a run-length encoding technique by ordering the segment identifiers naturally as they occur in the segment database and encoding the segmental transcription as a sequence of couples of segment identifiers and number of adjacent segments. Because of the low bit-rate representation, applications such as talking dictionary systems in which mainly words, compound words, and short phrases are synthesized on low-end platforms, are particularly suited for this synthesis method.
A tool that creates in response to some user actions (e.g. repeated validation), segmental transcriptions for entries that need a speech prompt may be provided to the customer. With the aid of this tool, the customer can generate speech messages and segmental transcriptions through a corpus-based synthesis technique that selects its units from a database that is identical to the database used on the target application. This guarantees the same speech quality as if the message was generated by the target application by using the same segmental transcription.
In order to generate the highest possible speech quality (higher than the speech that can be derived from a standard corpus-based synthesizer), the unit selection process may be fine tuned or a list of alternative message generations may be considered. The phonetic input string may also be modified (e.g., accentuation, pause, and/or tuning of phonetics for specific names, etc.). The phonetic string can be provided automatically by the grapheme-to-phoneme module, or it can be retrieved from a dictionary. The best speech message can then be selected from a set of relevant candidates and the segment descriptors of this message can be retained in a separate database called a “Customer Certified Database”. The customer certified database can be loaded into a TTS system (see principle compound speech units dictionary, CSUDict.) or the RSW system or into the customer tool itself which is explained in more detail in
The transcription pointer table C02 (
For example a partitioning in groups of four entries would result in a coding gain at the expense of an average of 1.5 additions per access. This must be compared to 1 subtraction that is needed if only positions were stored. The indices stored in customer database C01 (
The segmental transcription database C03 (
The virtual segment identifiers are ordered appropriately and are then appended sequentially to the segment position table C04 of
The segment position table C04 (
Such an encoding scheme allows for flexible speech compression that can deviate from the typical frame-based approach, resulting in a much higher coding gain. This approach also allows for the use of independent prosodic and spectral prototypes, which might further decrease the size of the speech segment database. Efficient coding schemes such as VQ and piece-wise linear compression can be used and may require extra tables that are not shown in
Distributed TTS System
Embodiments of the current invention can also be used for a distributed TTS system in which the segment identifier stream is generated on one platform (server platform) and transmitted to another platform (e.g. client platform) where the units are retrieved from a parametric speech database and converted into a speech waveform (see
The server platform receives a text input [D01]. The text is properly converted to a phonetic string by a text preprocessor and a grapheme-to-phoneme conversion module [D02]. A high quality unit selector searches the optimal sequence of units from either a large database [D04] or a small database [D05]. When the large database is used, the transformation-mapping module maps the segments to the small database [D06]. This provides the flexibility to upgrade the database on the server while maintaining the client (embedded device) as such.
To increase variety (e.g., by voice transformation or prosody transplantation) speech can be input and aligned with the text to the server. The transformation unit generates the transformation parameters [D10] for the sequence of segment identifiers that is closest to the prosody of the donor speech (search for possible minimal manipulation). In the specific case of pure segment mapping, the transformation parameters are also generated where needed.
The transmitted data stream [D09] contains (next to a control protocol) an initialization code containing a database identifier (DBid), the number of segment identifiers and transformation parameters that are in the stream (nSegs), a sequence of segment identifiers Segid(1 . . . nSegs), and a series of transformation parameters TF(1 . . . nSegs) aligned with the segment identifiers. The transformation parameters consist of a time manipulation sequence (Time TF), a fundamental frequency manipulation sequence (F0 TF), and a spectral manipulation sequence (Spectral TF) [D10]. Not all transformation parameters need to be generated for this system; in other words, the transmitted data stream can be as simple as just a sequence of segment identifiers with empty transformation parameters.
The client platform receives the transmitted data stream [D11] and decodes [D12] it. The speech parameters are retrieved from the embedded database [D13] by means of an indexation scheme based on the segment identifiers. If the segment aligned transformation parameters are available, the speech parameters are transformed. This transformation can be rate, pitch, and/or spectral manipulation. Next to that, the user of the client can apply a message-wide transformation of pitch (F0), rate and spectrum (λ), If specified, these transformation parameters are applied to all segments of the message. Finally, the speech parameters are converted into waveforms [D14] and concatenated in order to generate the output speech waveform.
Possible applications include a TTS system to read back data from RDS-receivers, a TTS system to read back traffic messages, a TTS system to read back speech in radio controlled toys etc..
Acoustically Compound Speech Units: Beyond The Acoustic Barrier
Currently, segment resequencing systems convey a more human-sounding synthesized speech than other type of synthesizers because of the intrinsic segmental quality and variability; but they demand more computational resources in terms of processing power and storage capacity and offer less flexibility. The degree of flexibility to modify the default speech output in concatenative systems depends on the availability and scope of signal manipulation techniques. In concatenative speech synthesis, the degradation of the speech quality is typically correlated with the amount of prosody modification applied to the speech signals.
Corpus-based speech synthesis draws on large prosodically-rich speech segment databases. Many of those speech segments sound similar and vary only slightly in some parameters. For example, several BSUs will have a similar spectral trajectory and differ substantially in prosody while other BSUs that have substantially different spectral trajectories will have similar pitch, duration, or energy contours. BSUs that have all acoustic parameters alike are redundant and can be replaced by a CSU where after the original waveform parameters are removed from the speech segment database. Because one or more acoustic parameters often show resemblance, it is possible to enlarge the compound speech unit concept to acoustic parameters also.
Two speech segments (first and second) are acoustically similar if the first segment can be modified with no perceptual quality loss by means of prosody transplantation/modification techniques (well known by those familiar in the art of speech processing), resulting in a new (third) speech segment that sounds like the second segment. Searching acoustically similar speech segments can be done by dynamic time warping, a technique well known in the art of speech processing. The acoustic similarity measure can be used to reduce the size of the database.
The optimization problem of finding the speech segments that create the maximum reduction in the speech waveform database can be done through vector quantization (clustering), also well known in the art of speech processing. The term acoustically compound speech unit (ACSU) will be used to refer to speech unit representations that share an incomplete acoustic representation. In other words, a set of ACSUs refers to a common acoustic representation that does not entirely describe the acoustics of the speech unit.
Each ACSU representation of that set of ACSUs embeds some segment-specific acoustic information (e.g. pitch track, energy contour, rate contour) that is complementary to the common acoustic information. The segment-specific acoustic information differentiates the ACSU from other ACSUs of that set. In order to reconstruct an ACSU, the warping path, the intonation and energy contour, and a reference to the speech waveform parameters need to be stored and consulted at synthesis time. The introduction of ACSUs requires that the speech segment database be organized differently. An embodiment of the invention uses a multi-prosodic representation as shown in Table 2. In this representation, all acoustically similar segments are represented by a common description followed by the differentiating elements.
The warping path, which is typically frame oriented, defines a discrete spectral mapping function from one speech segment to another. In practice, the warping path is a monotonically increasing function of the frame index. Under this condition, the warping path can be represented as a repeat vector indicating how frequently a given frame must be repeated. The spectral repeat vector indicates the frame indices where the spectral vectors are to be updated. The number of spectral vectors in a diphone will always be less than or equal to the number of frames. This is because there is variable frame length coding of the spectrum; i.e., similar spectra are not repeated. Also for all different prosodic realizations the same spectral vectors are used but they can be used at different time positions.
For each redundant speech segment, a pitch track and a time warping contour may be stored in place. The pitch track can be stored efficiently as a sequence of breakpoints that represents a piece-wise linear pitch contour (preferably in the log domain). The time warping contour non-linearly maps the time scale of a basis segment to the time scale of the “redundant” segment. The time warp contour is monotonically increasing and can be stored differentially.
There are at least two options for the encoding of the spectral parameters. The simplest method is to take over the entire spectral trajectory of the corresponding basis segment. In order to avoid altering the perception of the segments, conservative measures should be used. However, a larger coding gain can be expected if the differences between the basis segment and the “redundant” segment are stored. In the latter case, the number of basis segments will be smaller.
The spectral trajectory represents a number of spectral vectors Si (such as LPC or LSP vectors, possibly enriched with some excitation information such as a coded residual signal) that allows reconstruction of the spectral trajectory of the speech segment. The number of spectral vectors Ns used for the spectral vector representation is smaller than or equal to the actual size of the speech segment expressed in vectors. This is because the spectral vectors are determined through a technique called variable frame rate coding where similar consecutive spectral vectors are replaced by a single spectral vector, well known in the art of speech processing. The reconstruction of the real spectral trajectory in the time domain is done by means of the spectral repeat-vector.
The spectral repeat vector represents the frame indices where spectral vector updates are required. The synthesizer can use the spectral vectors as they are or it can interpolate between the updated spectral vectors to smooth the spectral trajectory. The length of the spectral repeat vector is related to the total number of frames of the speech segment. The spectral repeat vector R contains only binary elements. For example a “0”-symbol for ri means no spectral update required at frame index i while a “1 ” -symbol for ri means that a spectral update is required at frame index i. The number of spectral vectors in a diphone will always be less than or equal to the number of frames. This is because variable frame length coding of the spectrum is used; i.e., similar spectra are not repeated. Also for all different prosodic realizations the same spectral vectors are used at possibly different time positions.
So assuming Ns=4 and Nf=8, then the spectral repeat vector [10011010] means spectral vector 1 is used for frame indices 1, 2 and 3; spectral vector 2 is used for frame index 4; spectral vector 3 is used for frame indices 5 and 6; spectral vector 4 is used for frame indices 7 and 8 (the spectral repeat vector is at least of length Ns so Nf>=Ns). This means that in this described implementation we cannot produce speech segments that are shorter than Ns frames. This is a limitation that should be taken into account during the clustering process, however it is straightforward for those familiar with the art of speech or information processing to create other data structures that allow shortening.
The voicing information is coded under the assumption that most BSUs have none or only 1 change in voicing status. So the information can be fit in 1 bit for the initial voicing status, and in 1 bit for the final voicing status. If the two voicing states are different, then another code is needed to indicate the position of the spectral vector where the change takes place. The voicing decision is attached to a spectral vector. In exceptional cases, a code must be provided to encode a double change in voicing status within a segment (e.g. diphone).
The pitch block is a piecewise linear approximation of the intonation contour of the segment. It consists of a (binary) breakpoint vector P (e.g., P=[p1, p2, . . . , pn]=[1100101100]) indicating the frame positions in the voiced regions of the breakpoints followed by the pitch data at the breakpoints. The pitch data is a sequence of pitch values and pitch slope values represented at a certain precision and preferably defined in the log-domain (e.g. semi-tones). The pitch slope values represent pitch increments that have a precision that is typically higher than the precision of the pitch values themselves (because of the cumulative calculations).
A “0”-symbol for pj means that there is no update at frame index j while a “1”-symbol for pj indicates an update of the pitch data. An isolated breakpoint at position j ([. . . 010. . . ], i.e. a “1”-symbol surrounded at each side by at least one “0”-symbol) indicates an update of the slope value for the pitch for the j-th voiced frame. Two or more (say N) subsequent breakpoints (e.g. [. . . 01110. . . ] indicate that the pitch value will be updated at N−1 consecutive frames, followed by a slope value corresponding to the N-th “1”-symbol. The energy block is similarly represented as the pitch block.
If “read-all” philosophy is used, Np−1 bytes can be stored to find the correct offset for each realization. If “read-selective” philosophy is used, then one could argue to store Np bytes, as not only the offset but also the length must be known. On the other hand storing Np−1 bytes can be enough in a “read-selective” philosophy too, provided that a maximum size of a prosodic realization is known so that enough information can be read to decode the last prosodic realization in cases this is requested. This saves 1 byte for every spectral realization. The trade-off depends on the ratio of the average versus the maximal size of a prosodic realization as well as the frequency of use, i.e., how often will the system need access to a last prosodic realization (or the number of prosodic realizations per spectral realization).
Prosody Modification
To go beyond the prosodic variety that the speech database can provide, prosody modification can be used. Other components such as the unit selector can benefit from the introduction of prosody modification (even for small levels). Prosody modification in the form of segment boundary smoothing allows relaxing the continuity constraints used in the unit selector. Prosody modification can also be used to imply a prosody contour on the synthesized speech. Prosody transplantation techniques, well known in the art of speech processing, can be used to create new ACSUs that can be added to the segment database in a similar way as CSUs are added to the database.
Spectral Transformation
To enable speaker transformation (e.g. copy synthesis, cartoon voices, voice rejuvenation or voice ageing transformation, etc.) frequency warping of the spectral parameters can be applied. To enable this, one can send in addition to a segment identifier, a spectral warping factor. At the retrieval and interpolation moment of the spectral vectors, the warping into frequency domain is applied. The warping effect can be performed in a general way (same warping for all segments), or a segment-by-segment varying warping factor (see also distributed TTS system).
CSU-Based Unit Selector Bootstrap Training Algorithm
The validation of CSUs through iterative listening is a labor-intensive task. If reference data is available, this task could be automated by computing an objective perceptual distance measure. If there is no reference data available (e.g., very specific domains), an iterative verification by listening to all possible paths is probably needed. When a listening result is satisfactory, the dynamic programming path of the unit selector is stored as a sequence of segment descriptors into a dedicated database. After having done the listening verification on a dataset, it is advantageous to perform a bootstrap training on the feature weights (wƒi) and feature functions (F(ƒi))of the unit selector(s) so that the probability that the unit selection automatically generates the correct paths increases.
The learning algorithm shown in
Ep=(wovertap(100−overlap(t, o))+wdtwCostpath(t, o))2
The training method uses the steepest descent algorithmic approach adapted for this specific purpose and tries to minimize the error (Ep) by adapting the feature weights (wƒi) and feature functions (F(ƒi)) such as duration and pitch probability density functions and also the masking functions. This training method is very similar to the training method of a multi-layer feed-forward neural net. As an alternative training method a dataset can be generated that is composed out of the feature weights (wƒi) and feature functions (F(ƒi)) the features (ƒi) and the error (Ep) by keeping the input of the unit selector constant and letting the feature weights vary. The optimal feature weights and feature functions can be obtained by applying statistical and clustering learning-based methods on the dataset.
Glossary
The definitions below are pertinent to both the present description and the claims following this description.
“Diphone” is a fundamental speech unit composed of two adjacent half-phones. Thus the left and right boundaries of a diphone are in-between phone boundaries. The center of the diphone contains the phone-transition region. The motivation for using diphones rather than phones is that the edges of diphones are relatively steady-state and so it is easier to join two diphones together with no audible degradation, than it is to join two phones together.
“High level” linguistic features of a polyphone or other phonetic unit include with respect to such unit (without limitation), accentuation, phonetic context, and position in the applicable sentence, phrase, word, and syllable.
“Large speech database” refers to a speech database that references speech waveforms. The database may directly contain digitally sampled waveforms, or it may include pointers to such waveforms, or it may include pointers to parameter sets that govern the actions of a waveform synthesizer. The database is considered “large” when, in the course of waveform reference for the purpose of speech synthesis, the database commonly references many waveform candidates, occurring under varying linguistic conditions. In this manner, most of the time in speech synthesis, the database will likely offer many waveform candidates from which a single waveform is selected. The availability of many such waveform candidates can permit prosodic and other linguistic variation in the speech output stream.
“Low level linguistic features” of a polyphone or other phonetic unit includes, with respect to such unit, pitch contour and duration.
“Polyphone” is more than one diphone joined together. A triphone is a polyphone made of 2 diphones.
“SPT (Simple Phonetic Transcription)” describes the phonemes. This transcription is optionally annotated with symbols for lexical stress, sentence accent, etc . . . Example (for the word ‘worthwhile’): #‘werT-’wYl#
“Triphone” has two diphones joined together. It thus contains three components—a half phone at its left border, a complete phone, and a half phone at its right border.
Embodiments of the invention may be implemented in any conventional computer programming language. For example, preferred embodiments may be implemented in a procedural programming language (e.g., “C”) or an object oriented programming language (e.g., “C++”). Alternative embodiments of the invention may be implemented as pre-programmed hardware elements, other related components, or as a combination of hardware and software components.
Embodiments can be implemented as a computer program product for use with a computer system. Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium. The medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., microwave, infrared or other transmission techniques). The series of computer instructions embodies all or part of the functionality previously described herein with respect to the system. Those skilled in the art should appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies. It is expected that such a computer program product may be distributed as a removable medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web). Of course, some embodiments of the invention may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments of the invention are implemented as entirely hardware, or entirely software (e.g., a computer program product).
Although various exemplary embodiments of the invention have been disclosed, it should be apparent to those skilled in the art that various changes and modifications can be made which will achieve some of the advantages of the invention without departing from the true scope of the invention.
This application claims priority from provisional application 60/537,125, filed Jan. 16, 2004, the contents of which are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5153913 | Kandefer et al. | Oct 1992 | A |
| 5384893 | Hutchins | Jan 1995 | A |
| 5479564 | Vogten et al. | Dec 1995 | A |
| 5490234 | Narayan | Feb 1996 | A |
| 5611002 | Vogten et al. | Mar 1997 | A |
| 5630013 | Suzuki et al. | May 1997 | A |
| 5652828 | Silverman | Jul 1997 | A |
| 5749064 | Pawate et al. | May 1998 | A |
| 5774854 | Sharman | Jun 1998 | A |
| 5913193 | Juang et al. | Jun 1999 | A |
| 5920840 | Satyamurti et al. | Jul 1999 | A |
| 5970453 | Sharman | Oct 1999 | A |
| 5978764 | Lowry et al. | Nov 1999 | A |
| 6665641 | Coorman et al. | Dec 2003 | B1 |
| 6980955 | Okutani et al. | Dec 2005 | B2 |
| 7069216 | DeMoortel et al. | Jun 2006 | B2 |
| 7136818 | Cosatto et al. | Nov 2006 | B1 |
| 7219060 | Coorman et al. | May 2007 | B2 |
| Number | Date | Country | |
|---|---|---|---|
| 20050182629 A1 | Aug 2005 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60537125 | Jan 2004 | US |
