Patent Application
20230410790
This invention relates to synthesis of speech with foreign fragments.
Text-to-speech (TTS) systems synthesize speech waveforms from textual input. Some conventional TTS systems process textual input to determine phonetic transcriptions of the words or subsets of the words (i.e., letters or groups of letters termed “graphemes”) in the textual input. The phonetic transcriptions are provided to a synthesizer that converts the phonetic transcriptions into speech waveforms that can be output using, for example, a loudspeaker.
Certain textual inputs to TTS systems include fragments of one or more words in in a language that is in a second language. An example of such a textual input that is “J'ai lu Harry Potter hier” where the TTS system is configured to synthesize native French-language text as would be spoken by a French speaker, but Harry Potter is foreign-language text because it is the proper name of a fictional British wizard. One approach may be to ignore the change of language and use the same synthesis rules for the whole input. Another approach may be to switch to between French and English synthesis rules (i.e., as if switching between a native French and a native English speaker mid-sentence). However, such approaches may not produce natural sounding synthesized speech.
In a general aspect, an approach to speech synthesis accepts text that has a foreign language fragment and produces a synthesized waveform that pronounces the native language text as spoken by a native speaker and pronounces the foreign language fragment in a manner that would be pronounced by the native speaker, which may not correspond to a “correct” pronunciation of the foreign language fragment by a speaker of that foreign language.
Some TTS systems may, for example, identify that a text input includes both French and English words and then use French phonemes to synthesize the French words and English (e.g., British English) phonemes to synthesize the English words using pronunciation rules that are native to each of the languages. One drawback of this technique is that the synthesized speech may sound unnatural because midway through the sentence it seems that a French speaker suddenly switches over to speaking perfect English and then switches back to speaking perfect French. Other techniques may replace English phonemes for the textual input with their closest French equivalents, or may use French grapheme-to-phoneme rules to synthesize the English Fragment. The resulting synthesized speech may sound unnatural because aspects of the text may be unnaturally omitted from the English part of the synthesized speech.
Aspects described herein address the above-described drawbacks of conventional techniques by synthesizing foreign language text embedded in a native language textual input in a way that imitates the way a native language speaker would pronounce the foreign language text. For example, many native French speakers are at least somewhat proficient English speakers. Even though the French language doesn't include the h phoneme, those French speakers recognize that the “H” in the name “Harry” is pronounced in the English language and make a (likely imperfect) attempt to pronounce the “H” when saying “Harry.” In effect, those native French speakers are using an augmented French phoneme set to pronounce English words. Aspects described herein would imitate the French speaker's attempt at pronouncing the “H” in “Harry” by using an augmented French phoneme set that includes extra phonemes that French speakers use when pronouncing English words, resulting in more natural and realistic sounding synthesized speech.
In a general aspect, a method for synthesizing speech from a textual input includes receiving the textual input, the textual input including native words in a native language and foreign words in a foreign language, and processing the textual input to determine a phonetic representation of the textual input. The processing includes determining a native phonetic representation of the of the native words, and determining a nativized phonetic representation of the foreign words. Determining the nativized phonetic representation includes forming a foreign phonetic representation of the foreign words using a foreign phoneme set, and mapping the foreign phonetic representation to the nativized phonetic representation according to a model of a native speaker's pronunciation of foreign words.
Aspects may include one or more of the following features.
The native phonetic representation may include phonemes from a native phoneme set. The nativized phonetic representation may include phonemes from a native phoneme set. The nativized phonetic representation may include phonemes from a second set of phonemes different from the native phoneme set. The second set of phonemes may include a phoneme set for the foreign language.
The mapping of the foreign phonetic representation to the nativized phonetic representation may use contextual information associated with the foreign phonetic representation. The contextual information may include a grapheme representation of the foreign text associated with the foreign phonetic representation. The contextual information may include an alignment of graphemes in the grapheme representation of the foreign text to phonemes in the foreign phonetic representation. The contextual information may include location information of phonemes in the foreign phonetic representation.
The model of the native speaker's pronunciation of foreign words may be based on training data comprising foreign textual phrases and phonetic transcriptions of a native speaker's pronunciation of the foreign textual phrases. At least some of the native speaker's pronunciations of foreign textual phrases may be mispronunciations. The method may include providing a combination of the native phonetic representation of the of the native words and the nativized phonetic representation of the foreign words to a waveform synthesizer for synthesis of a speech waveform. The method may include synthesizing the speech waveform based on the combination of the native phonetic representation of the of the native words and the nativized phonetic representation of the foreign words using the waveform synthesizer.
The method may include configuring a neural network according to the model of a native speaker's pronunciation of foreign words. Mapping the foreign phonetic representation to the nativized phonetic representation according to the model of a native speaker's pronunciation of foreign words may include applying one or more mapping rules. The method may include identifying the foreign words in the textual input.
In another general aspect, a system for synthesizing speech from a textual input, the system includes an input for receiving the textual input, the textual input including native words in a native language and foreign words in a foreign language and one or more processors configured to process the textual input to determine a phonetic representation of the textual input. The processing includes determining a native phonetic representation of the of the native words, and determining a nativized phonetic representation of the foreign words. Determining the nativized phonetic representation includes forming a foreign phonetic representation of the foreign words using a foreign phoneme set, and mapping the foreign phonetic representation to the nativized phonetic representation according to a model of a native speaker's pronunciation of foreign words.
In another general aspect, software stored in a non-transitory form on a computer-readable medium includes instructions for causing a computing system to synthesize speech from a textual input including to receive the textual input, the textual input including native words in a native language and foreign words in a foreign language and process the textual input to determine a phonetic representation of the textual input. The processing includes determining a native phonetic representation of the of the native words, and determining a nativized phonetic representation of the foreign words. Determining the nativized phonetic representation includes forming a foreign phonetic representation of the foreign words using a foreign phoneme set and mapping the foreign phonetic representation to the nativized phonetic representation according to a model of a native speaker's pronunciation of foreign words.
In another general aspect, a method for determining configuration data for configuring a module for mapping a foreign phonetic representation to a nativized phonetic representation includes receiving training data comprising a plurality of textual representations of foreign language words or phrases and a corresponding plurality of reference phonetic representations of the foreign language words or phrases and processing the training data to form the configuration data.
Aspects may include one or more of the following features.
Processing the training data may include for each textual representation of a foreign language word or phrase forming a foreign phonetic representation of the foreign word or phrase using a foreign phoneme set, mapping the foreign phonetic representation to the nativized phonetic representation according to using a model of a native speaker's pronunciation of foreign words, determining a difference between the nativized phonetic representation and a reference phonetic representation corresponding to the foreign language word or phrase, and updating the model of the native speaker's pronunciation of foreign words based at least in part on the determined difference.
An advantage of training a mapping from a correct foreign pronunciation to a pronunciation in the target language, as compared for example to training a full grapheme-to-phoneme mapping that maps foreign text to the target phoneme set, may be that relatively less training data is needed to represent the manner in which native speakers “nativize” the foreign fragments. Another advantage arises when using graphemes as part of the method of nativizing the foreign fragments because attributes of the foreign language pronunciation such as vowel coloring (e.g., very short vowel phonemes such as schwa $) may be lost in the English phonetic transcription. Such attributes can be restored by using the graphemes as part of nativizing the foreign fragments.
Another advantage may be that different mappings can be trained for different target users. For example, speakers that live in a generally bilingual area may pronounce the foreign fragments more closely to their correct foreign form as compared to speakers from an area where that foreign language is not spoken. To accommodate “natural” synthesis of foreign fragments to the users from such different areas, different mappings may be used.
Other features and advantages of the invention are apparent from the following description, and from the claims.
Referring to
In some examples, the system 100 includes a language parser 108, a French language processing pipeline 110, an English language processing pipeline 112, and a waveform generator 114. The language parser identifies French words or phrases 114 in the textual input 102 and identifies English words or phrases 116 in the textual input 102. The French words or phrases 114 are provided to and processed by the French language processing pipeline 110 and the English words or phrases 116 are provided to and processed by the English language processing pipeline 116. The outputs of the French language processing pipeline 112 and the English language processing pipeline 112 are combined and provided to the waveform generator 114, which processes the combined output to generate the synthesized speech waveform 104.
The French language processing pipeline 110 includes a French grapheme-to phoneme (G2P) module 118, which identifies graphemes (i.e., letters or groups of letters) in the French words or phrases 114 and determines French language phonemes associated with the identified graphemes. The output of the graphemes to French phonemes module 118 is a sequence of French phonemes 121 corresponding to the French words or phrases 114.
The English language processing pipeline 112 processes the English words or phrases 116 to ultimately generate a sequence of phonemes in a “French+” phoneme set. In general, this phoneme set is denoted “French+” to account for phoneme set including French phonemes and possibly additional phonemes that a native French speaker would use when attempting to pronounce English words but that may not be required to pronounce French words. In some examples, the additional phonemes are existing English phonemes and/or new phonemes unique to French speakers attempting to pronounce English words. The English language processing pipeline 112 includes a graphemes to English phonemes module 120, an English to French+ phoneme mapping module 123, and an optional override mapping module 124.
The graphemes to English phonemes module 120 identifies graphemes in the English words or phrases 116 and determines English language phonemes associated with the identified graphemes. The output of the graphemes to English phonemes module 120 is a sequence of English phonemes 122 corresponding to the English words or phrases 116.
The sequence of English phonemes 122 (and optionally the graphemes associated with the English words or phrases 116 and aligned to the English phonemes) are provided to the English to French+ phoneme mapping module 123, which maps the sequence of English phonemes 122 into a sequence of French+ phonemes 126 (as is described in greater detail below). In some examples, the English to French+ phoneme module 123 is implemented as a neural network that is parameterized by mapping parameters, θ 125. Optionally, the sequence of French+ phonemes are processed by the override mapping module 124 to override certain phoneme mappings using override rules (as is described in greater detail below).
The sequence of French phonemes 121 and the sequence of French+ phonemes 126 are provided to a combiner 128, which combines the two sequences of phonemes according to the order of the words and phrases in the textual input 102 to generate the combined French and French+ phonemes 130. The combined French and French+ phonemes 130 are proved to the waveform generator 114, which processes the phoneme sequence to generate the synthesized speech waveform 104.
Referring to
The French words or phrases 110, “J'ai lu” and “hier” are provided to the graphemes to French phonemes module 118 (shown twice in
The English words or phrases 116, “Harry Potter” are provided to the graphemes to English phonemes module 120. The graphemes to English phonemes module 120 processes “Harry Potter” to generate the sequence of English phonemes 122 'h @ R+ I _'p A+. t $ R+. The sequence of English phonemes 122 (and optionally the associated English graphemes) are provided to the English to French+ phoneme mapping module 123. The English to French+ phoneme mapping module 123 maps the sequence of English phonemes 122 to a sequence of French+ phonemes 126 h a. 'R+ I _p O. 't E+ R+. Note that the sequence of French+ phonemes 126 includes phonemes h and R+, which are not present in native French phoneme set, but instead are borrowed from the English phoneme set.
When the English graphemes are provided to the English to French+ phoneme mapping module 123, the module recognizes a mapping between certain English graphemes and French+ phonemes and inserts French+ phonemes when corresponding English graphemes are present in the textual input.
The combiner 128 combines the sequence of French phonemes 121 and the sequence of French+ phonemes 126 to form the combined French and French+ phonemes 130, which is provided to the waveform generator 114 to generate the synthesized speech waveform 104.
Referring to
In other examples, the English to French+ phoneme mapping module 123 implements a sequence-to-sequence mapping where a sequence of English phonemes is mapped to a sequence of French+ phonemes in a way that accounts for a context of each phoneme (and possibly grapheme) in the greater sequence of phonemes.
As is noted above, when the English graphemes are provided to the English to French+ phoneme mapping module 123, the phoneme mapper 232 is configured to identify certain English graphemes that should be replaced by French+ phonemes. For example, the phoneme mapper 232 may use the “e” grapheme 239 in its determination that the $ English phoneme 234 is mapped to the E+ French+ phoneme 235.
Referring to
In
Referring to
In operation, the training system 500 processes the set of textual English phrases 542 one at a time by reading a textual English phrase 546 from the set of textual English phrases 542. The textual English phrase 546 is processed by a graphemes to English phonemes module 520 to generate a sequence of English phonemes 522 corresponding to the textual English phrase 546.
The sequence of English phonemes 522 (and optionally the associated English graphemes) are provided to an English to French+ phoneme mapping module 523. The English to French+ phoneme mapping module 523 maps the sequence of English phonemes 522 to a mapped sequence of French+ phonemes 526 based on the current mapping parameters, θ 125.
The mapped sequence of French+ phonemes 526 and the reference sequence of French phonemes 548 associated with the textual English phrase 546 are provided to a loss computation module 550, which computes a loss value, L characterizing a difference between the mapped sequence of French+ phonemes 526 and the reference sequence of French+ phonemes 548. The loss value, L is provided to an optimization module 552, which updates the mapping parameters, θ 125 to minimize the loss value over the set of training data.
The training system 500 repeats this parameter update procedure for all of the English phrases 542, causing the mapping parameters, θ to converge.
In some examples, the reference sequences of French+ phonemes are aligned (automatically or manually) to the textual English phrases (or the corresponding sequences of English phonemes) to improve training performance.
In some examples, each reference sequence of French+ phonemes is hand transcribed. In other examples, each reference sequence of French+ phonemes is generated by performing phoneme recognition using a French+ phoneme set on speech from typical French speakers. In such examples, the mapping parameters, θ will configure the mapper to mimic the population of French speakers used to generate the training data (i.e., a population of French speakers who are fluent in English would result in different mapping behavior than a population of French speakers who are only familiar with the English language).
In other examples, each reference sequence of French+ phonemes is generated by performing phoneme recognition using a French+ phoneme set on speech spoken by a French speaker reading the English phrases.
The examples described above use a neural network and optionally rules to map English phonemes into an augmented French phoneme set (French+). However, in some examples, no neural network is used, and a rule-based approach is sufficient. Alternatively, other machine learning approaches, such as use of a decision tree, may be used instead of a neural network.
The examples described above map English phonemes into phonemes in an augment French phoneme set. However, it isn't required that the French phoneme set is augmented—the native French phoneme set may include sufficient phonemes for training a mapper to generate speech that imitates a native French speaker who is attempting to pronounce English words.
While the system described above produces a waveform for presentation as an acoustic signal, the output may comprise a representation from which the waveform may be generated. For example, the phoneme sequence may be processed to form a spectrogram (energy vs. frequency vs. time) from which the waveform may be determined in a language-independent manner.
Alternative mappings approaches may be used to process the foreign phoneme sequence to form the target phoneme sequence. For example, a sequence-to-sequence neural network (e.g., a Transformer Neural Network or a Recurrent Neural Network, such as an LSTM) may be used. Alternative training approaches may be used, with the target phoneme sequence being determined by phoneme recognition by one or a combination of native speakers uttering the foreign fragments, hand transcription of such native speaker's utterances. Furthermore, the mapping may be trained to directly match the audio of the native speakers.
In alternative approach, different foreign fragments may be processed differently, for example, according to their length. For example, it may be that short fragments are mapped closed to the native language while longer fragments retain more of their proper foreign pronunciation. Similarly, named entities may retain their proper pronunciation more than common expressions. Therefore, alternatively may have a mapping process that can vary the degree to which the foreign fragment is mapped to the native pronunciation. Furthermore, users may have a preference for the degree of “nativization” of a pronunciations, which may be settable to by the user.
In some examples, lexica are used to override at least some part of the mapping of the English phonemes into the augmented French phoneme set (French+) in the case of exceptions that are not properly captured by the neural network or rule-based mapping.
As is noted above, in some examples, graphemes and phonemes are aligned prior to processing using a classification model or rule-based mapper. Different types of grapheme-phoneme alignment techniques can be used such as Expectation Maximization-based alignment, phonetic similarity-based alignment, constraint-based alignment, using an Integer Programming framework, alignment by aggregation (i.e., a combination of integer programming and expectation maximization) or any other suitable alignment technique.
The approaches described above can be implemented, for example, using a programmable computing system executing suitable software instructions or it can be implemented in suitable hardware such as a field-programmable gate array (FPGA) or in some hybrid form. For example, in a programmed approach the software may include procedures in one or more computer programs that execute on one or more programmed or programmable computing system (which may be of various architectures such as distributed, client/server, or grid) each including at least one processor, at least one data storage system (including volatile and/or non-volatile memory and/or storage elements), at least one user interface (for receiving input using at least one input device or port, and for providing output using at least one output device or port). The software may include one or more modules of a larger program, for example, that provides services related to the design, configuration, and execution of dataflow graphs. The modules of the program (e.g., elements of a dataflow graph) can be implemented as data structures or other organized data conforming to a data model stored in a data repository.
The software may be stored in non-transitory form, such as being embodied in a volatile or non-volatile storage medium, or any other non-transitory medium, using a physical property of the medium (e.g., surface pits and lands, magnetic domains, or electrical charge) for a period of time (e.g., the time between refresh periods of a dynamic memory device such as a dynamic RAM). In preparation for loading the instructions, the software may be provided on a tangible, non-transitory medium, such as a CD-ROM or other computer-readable medium (e.g., readable by a general or special purpose computing system or device), or may be delivered (e.g., encoded in a propagated signal) over a communication medium of a network to a tangible, non-transitory medium of a computing system where it is executed. Some or all of the processing may be performed on a special purpose computer, or using special-purpose hardware, such as coprocessors or field-programmable gate arrays (FPGAs) or dedicated, application-specific integrated circuits (ASICs). The processing may be implemented in a distributed manner in which different parts of the computation specified by the software are performed by different computing elements. Each such computer program is preferably stored on or downloaded to a computer-readable storage medium (e.g., solid state memory or media, or magnetic or optical media) of a storage device accessible by a general or special purpose programmable computer, for configuring and operating the computer when the storage device medium is read by the computer to perform the processing described herein. The inventive system may also be considered to be implemented as a tangible, non-transitory medium, configured with a computer program, where the medium so configured causes a computer to operate in a specific and predefined manner to perform one or more of the processing steps described herein.
A number of embodiments of the invention have been described. Nevertheless, it is to be understood that the foregoing description is intended to illustrate and not to limit the scope of the invention, which is defined by the scope of the following claims. Accordingly, other embodiments are also within the scope of the following claims. For example, various modifications may be made without departing from the scope of the invention. Additionally, some of the steps described above may be order independent, and thus can be performed in an order different from that described.
