Patent Grant
7263488
The present invention relates to speech synthesis. In particular, the present invention relates to setting prosody in synthesized speech.
Text-to-speech systems have been developed to allow computerized systems to communicate with users through synthesized speech. To produce natural sounding speech, prosodic contours such as fundamental frequency, duration, amplitude and pauses must be generated for the synthesized speech to provide the proper cadence. In many languages, lexical word boundaries provide cues for generating prosodic contours.
For Asian languages, such as Chinese, Japanese and Korean, generating prosodic contours in an utterance is complicated by the fact that the lexical word boundaries in these languages are not apparent from the text. Unlike Western languages such as English, where characters are grouped into words separated by spaces, Asian languages are written in strings of unsegmented single characters. Thus, even multi-character words appear as unsegmented single characters.
In the prior art, efforts were made to improve the cadence or prosody of Asian text-to-speech systems by improving the segmentation of the characters into individual lexical words. However, the resulting speech has not been as natural as desired.
A method and computer-readable medium are provided that identify prosodic word boundaries for an unrestricted text. If the text is unsegmented, it is segmented into lexical words. The lexical words are then converted into prosodic words using an annotated lexicon to divide large lexical words into smaller words and a model to combine the lexical words and/or the smaller words into larger prosodic words. The boundaries of the resulting prosodic words are used to set prosodic contours for the synthesized speech.
The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
With reference to
Computer 110 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer 110 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CDROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer 100.
Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, FR, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.
The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during startup, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation,
The computer 110 may also include other removable/non-removable volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media discussed above and illustrated in
A user may enter commands and information into the computer 110 through input devices such as a keyboard 162, a microphone 163, and a pointing device 161, such as a mouse, trackball or touch pad. Other input devices (not shown) may include a joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 120 through a user input interface 160 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190. In addition to the monitor, computers may also include other peripheral output devices such as speakers 197 and printer 196, which may be connected through an output peripheral interface 190.
The computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be a personal computer, a hand-held device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 110. The logical connections depicted in
When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user input interface 160, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
Memory 204 is implemented as non-volatile electronic memory such as random access memory (RAM) with a battery back-up module (not shown) such that information stored in memory 204 is not lost when the general power to mobile device 200 is shut down. A portion of memory 204 is preferably allocated as addressable memory for program execution, while another portion of memory 204 is preferably used for storage, such as to simulate storage on a disk drive.
Memory 204 includes an operating system 212, application programs 214 as well as an object store 216. During operation, operating system 212 is preferably executed by processor 202 from memory 204. Operating system 212, in one preferred embodiment, is a WINDOWS® CE brand operating system commercially available from Microsoft Corporation. Operating system 212 is preferably designed for mobile devices, and implements database features that can be utilized by applications 214 through a set of exposed application programming interfaces and methods. The objects in object store 216 are maintained by applications 214 and operating system 212, at least partially in response to calls to the exposed application programming interfaces and methods.
Communication interface 208 represents numerous devices and technologies that allow mobile device 200 to send and receive information. The devices include wired and wireless modems, satellite receivers and broadcast tuners to name a few. Mobile device 200 can also be directly connected to a computer to exchange data therewith. In such cases, communication interface 208 can be an infrared transceiver or a serial or parallel communication connection, all of which are capable of transmitting streaming information.
Input/output components 206 include a variety of input devices such as a touch-sensitive screen, buttons, rollers, and a microphone as well as a variety of output devices including an audio generator, a vibrating device, and a display. The devices listed above are by way of example and need not all be present on mobile device 200. In addition, other input/output devices may be attached to or found with mobile device 200 within the scope of the present invention.
A sample and store circuit 310 breaks training speech 308 into individual speech units such as phonemes, diphones, triphones or syllables based on training text 306. Sample and store circuit 310 also samples each of the speech units and stores the samples as stored speech components 312 in a memory location associated with speech synthesizer 300.
In many embodiments, training text 306 includes over 10,000 words. As such, not every variation of a phoneme, diphone, triphone or syllable found in training text 306 can be stored in stored speech components 312. Instead, in most embodiments, sample and store 310 selects and stores only a subset of the variations of the speech units found in training text 306. The variations stored can be actual variations from training speech 308 or can be composites based on combinations of those variations.
Once training samples have been stored, input text 304 can be parsed into its component speech units by parser 314. The speech units produced by parser 314 are provided to a component locator 316 that accesses stored speech units 312 to retrieve the stored samples for each of the speech units produced by parser 314. In particular, component locator 316 examines the neighboring speech units around a current speech unit of interest and based on these neighboring units, selects a particular variation of the speech unit stored in stored speech components 312. Based on this retrieval process, component locator 316 provides a set of stored samples for each speech unit provided by parser 314.
Text 304 is also provided to a semantic identifier 318 that identifies the basic linguistic structure of text 304. In particular, semantic identifier 318 is able to distinguish questions from declarative sentences, as well as the location of commas and natural breaks or pauses in text 304.
Based on the semantics identified by semantic identifier 318, a prosody calculator 320 calculates the desired pitch and duration needed to ensure that the synthesized speech does not sound mechanical or artificial. In many embodiments, the prosody calculator uses a set of prosody rules developed by a linguistics expert. In other embodiments, statistical prosody rules are used.
Prosody calculator 320 provides its prosody information to a speech constructor 322, which also receives retrieved samples from component locator 316. When speech constructor 322 receives the speech components from component locator 316, the components have their original prosody as taken from training speech 308. Since this prosody may not match the output prosody calculated by prosody calculator 320, speech constructor 322 must modify the speech components so that their prosody matches the output prosody produced by prosody calculator 320. Speech constructor 322 then combines the individual components to produce synthesized speech 302. Typically, this combination is accomplished using a technique known as overlap-and-add where the individual components are time shifted relative to each other such that only a small portion of the individual components overlap. The components are then added together.
As discussed in the background, prior art semantic identifiers identify groupings of characters that form lexical words in the text. These lexical words are then used by a prosodic calculator to calculate prosodic contours such as fundamental frequency, duration, amplitude and pauses.
The present inventors have discovered that this technique is not effective in many Asian languages because lexical word boundaries do not match well with the cadence of speech. Instead, the basic rhythm units sometimes form only part of a lexical word and at other times they span more than one lexical word. Such basic rhythm units are called prosodic words.
Unfortunately, such prosodic words are formed dynamically during speech and it is impossible to list all of them into a lexicon. The present invention provides a method and system for identifying the prosodic word boundaries in a text.
Under one embodiment of the present invention, a conversion model and an annotated lexicon are formed to identify lexical words that should be combined into a larger prosodic word and to identify lexical words that should be divided into smaller prosodic words.
The segmented training text is then provided to a prosodic word identifier 408 together with a training speech signal 410. In many embodiments, prosodic word identifier 408 is a panel of human listeners who listen to training speech signal 410 while reading the training text. Each member of the panel marks prosodic word boundaries that he perceived as a single rhythm unit. If a majority of the panel agrees on a prosodic word, a boundary mark is placed.
Once the training text has been annotated with the prosodic word boundaries, the annotated text is provided to a category look-up 414, which identifies a set of categories for each word in the training text. Under embodiments of the present invention, these categories include things such as the lexical word's part of speech in the text, the length of the lexical word, whether the lexical word is a proper name and other similar features of the lexical word. Under some embodiments, some or all of these features are stored in the entry for the lexical word in lexicon 404.
The words and their categories are passed to model trainer 412, which groups neighboring lexical words in the training text into word pairs and groups their corresponding categories into category pairs. The category pairs and the annotations indicating whether a pair of lexical words constitute a prosodic word are then used to train a conversion model 416.
Under one embodiment, conversion model 416 is a statistical model. To train this statistical model, model trainer 412 generates a count of the number of word pairs associated with each unique category pair in the training text. Thus, if four different word pairs formed the same category pair, that category pair would have a count of four. Model trainer 412 also generates a count of the number of lexical word pairs associated with a category pair that was marked as forming a prosodic word by prosodic word identifier 408. These counts are then used to produce a conditional probability described as:
where count(P1) is the number of lexical word pairs with category pair condition Pi, count(T0|P1) is the number of lexical word pairs that form a single prosodic word and have category pair condition Pi, and {tilde over (P)}(T0|P1) is the probability of a lexical word pair forming a prosodic word if the word pair has the category pair condition Pi.
When count(P1) is a small number, the estimated probability is not reliable. Under one embodiment, a weighted probability is used to reduce the contribution of unreliable probabilities. This weighted probability is defined as:
W{tilde over (P)}(T0|P1)={tilde over (P)}(T0|P1)※W(P1) EQ.2
where W{tilde over (P)}(T0|P1) is the weighted probability and W(P1) is a weighting function. Under one embodiment, the weighting function is a sigmoid function of the form:
W(P1)=sigmoid(1+log(count(P1))) EQ.3
which has values between zero and one.
Under one embodiment, the weighted probabilities determined above are compared to a threshold to determine whether lexical words with a particular category pair condition will be designated as forming a prosodic word. If the probability is greater than the threshold for a category pair, lexical words with that category pair will be combined into a prosodic word by conversion model 416 when encountered during speech production. If the probability is less than the threshold, conversion model 416 will not combine the lexical word pair that forms that category pair into a prosodic word.
In other embodiments, conversion model 416 is a classification and regression tree (CART). Under this embodiment, a question list is defined for the conversion model. The classification and regression tree then applies the questions to the category pairs to group the category pairs and their associated lexical word pairs into nodes. The lexical word pairs in each node are then examined to determine how many of the lexical word pairs were designated by prosodic word identifier 408 as forming a prosodic word. Nodes with relatively large numbers of word pairs that form prosodic words are then designated as prosodic nodes while nodes with relatively few word pairs that form prosodic words are designated as non-prosodic nodes.
When the CART model receives text during speech synthesis, it applies the category pairs to the questions in the model and identifies the node for the category pair. If the node is a prosodic node, the lexical words associated with the category pair are combined into a prosodic word. If the node is a non-prosodic node, the lexical words are kept separate.
Each word in large-word file 506 is applied to lexical word segmentation unit 508. Lexical word segmentation unit 508 is similar to segmentation unit 402 of
The smaller lexical words identified by segmentation unit 508 are applied to a category look-up 509, which is similar to category look-up 414 of
Thus, a four-character word may be divided into a two-character word followed by two one-character words by segmentation unit 508. The two one-character words may then be combined into a single prosodic word by conversion model 510.
Lexicon 502 is then annotated to form annotated lexicon 500 by indicating how the larger lexical words should be divided into smaller prosodic words. In particular, the output of conversion model 510 indicates how each larger word should be divided. Thus, in the example above, the four-character word's entry would be annotated to indicate that it should be divided into two two-character prosodic words.
Once the annotated lexicon and the conversion model have been formed, they can be used to identify prosodic words during speech synthesis.
At step 700 of
The first lexical word identified by segmentation unit 602 is selected at step 702 and is provided to splitting unit 606. At step 704, splitting unit 606 segments the lexical word into smaller prosodic words as indicated by annotated lexicon 604. If annotated lexicon 604 indicates that the lexical word is not to be divided, the word is left intact by splitting unit 606.
At step 706, splitting unit 606 determines if this is the last lexical word in the string. If it is not the last lexical word, it stores the present lexical word or the prosodic words formed from the lexical word and selects the next word in the string at step 708. The process of
Steps 704, 706, and 708 are repeated until the last lexical word in the string has been processed by prosodic segmentation unit 606. When the last word has been processed, all of the stored words are passed to category look-up 607 as a modified or intermediate string of words.
Category look-up 607 is similar to category look-up 414 of
At step 710, conversion model 608 selects the first word pair in the modified string of words. This word pair may be formed of two lexical words from text 600, a lexical word and a smaller prosodic word, or two smaller prosodic words. Based on the model parameters and the category pair formed from the set of categories for the two words in the word pair, conversion model 608 determines whether to merge the two words together to form a prosodic word at step 712. If the model indicates that the two words would be pronounced as a single rhythm unit, the words are combined into a single prosodic word. If the model indicates that the words would be pronounced as two rhythm units, the words are left separated.
At step 714, conversion model 608 determines if this is the last word pair in the string. If this is not the last word pair, the next word pair is selected at step 716. Under most embodiments, the next word pair consists of the last word in the current word pair and the next word in the string. If a single prosodic word was formed at step 712, the next word pair consists of the prosodic word and the next word in the string. The process of
Steps 712, 714, and 716 are repeated until the end of the string is reached. The process then ends at step 718 and the modified string is provided to further components 610 that perform the remainder of the semantic identification. This includes such things as determining the sentence construction and using the sentence construction and the prosodic word boundaries to identify pitch contour, duration and pauses or other high level description features such as word initial, word middle or word end. Note that by using prosodic word boundaries to identify these prosodic features, the present invention is thought to provide more natural sounding speech for text, especially Asian text.
Although the prosodic word identification system of the present invention was described above in the context of speech synthesis, the system can also be used to label a training corpus with prosodic word boundaries. Thus, instead of being used directly to identify prosody for a text to be synthesized, the prosodic word identification process can be used to identify prosodic words in a large corpus.
Although the present invention has been described with reference to particular embodiments, workers skilled in the art will recognize that changes may be made in form and detail without departing from the spirit and scope of the invention.
The present application claims priority to a U.S. Provisional application having Ser. No. 60/251,167, filed on Dec. 4, 2000 and entitled “PROSODIC WORD SEGMENTATION AND MULTI-TIER NON-UNIFORM UNIT SELECTION”.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5146405 | Church | Sep 1992 | A |
| 5384893 | Hutchins | Jan 1995 | A |
| 5592585 | Van Coile et al. | Jan 1997 | A |
| 5727120 | Van Coile et al. | Mar 1998 | A |
| 5732395 | Silverman et al. | Mar 1998 | A |
| 5839105 | Ostendorf et al. | Nov 1998 | A |
| 5890117 | Silverman | Mar 1999 | A |
| 5905972 | Huang et al. | May 1999 | A |
| 6064960 | Bellegarda et al. | May 2000 | A |
| 6076060 | Lin et al. | Jun 2000 | A |
| 6101470 | Eide et al. | Aug 2000 | A |
| 6185533 | Holm et al. | Feb 2001 | B1 |
| 6230131 | Kuhn et al. | May 2001 | B1 |
| 6401060 | Critchlow et al. | Jun 2002 | B1 |
| 6499014 | Chihara | Dec 2002 | B1 |
| 6665641 | Coorman et al. | Dec 2003 | B1 |
| 6708152 | Kivimaki | Mar 2004 | B2 |
| 6751592 | Shiga | Jun 2004 | B1 |
| 6829578 | Huang et al. | Dec 2004 | B1 |
| 7010489 | Lewis et al. | Mar 2006 | B1 |
| 20020072908 | Case et al. | Jun 2002 | A1 |
| 20020103648 | Case et al. | Aug 2002 | A1 |
| 20020152073 | DeMoortel et al. | Oct 2002 | A1 |
| Number | Date | Country |
|---|---|---|
| 0 984 426 | Mar 2000 | EP |
| Number | Date | Country | |
|---|---|---|---|
| 20020095289 A1 | Jul 2002 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60251167 | Dec 2000 | US |
