Patent Grant
9286885
The invention is based on a priority application EP 03360052.9 which is hereby incorporated by reference.
The invention relates to a method of generating speech from text and a distributed speech synthesis system for performing the method.
Interactive voice response systems generally comprise a speech recognition system and means for generating a prompt in form of a speech signal. For generating prompts, speech synthesis systems are often used (text-to-speech synthesis TTS). These systems transform text into a speech signal. To this end, the text is phonetized, suitable segments are chosen from a speech database (p.ex. diphones) and the speech signal is concatenated from the segments. If this is to be performed in an environment which allows data transmission, in particular, if one or more distant end terminals such as mobile phones are to be used, special requirements with respect to the end terminal and the transmission capacity exist.
Typically, a TTS is realized centrally on a server in a network, which server performs the task of translating text into acoustic signals. In telecommunications networks the acoustic signals are coded and then transmitted to the end terminal. Disadvantageously, the data volume to be transmitted using this approach is relatively high (p.ex.>4.8 kbit/s).
In another approach the TTS may be implemented in the end terminal. In this case only a text string needs to be transmitted. However, this approach requires a large memory in the end terminal in order to ensure a high quality of the speech signal. Furthermore, the TTS needs to be implemented in each terminal, requiring high computation power in each terminal.
It is the object of the invention to provide a method for generating speech from text which requires only a small memory in an end terminal and which avoids having to transfer large data volumes and a system for performing the method.
This object is achieved by a method of generating speech from text comprising the steps of determining the speech segments necessary to put together the text to be output as speech by a terminal; checking which speech segments are already present in the terminal and which ones need to be transmitted from a server to the terminal; indexing the segments to be transmitted to the terminal; transmitting the speech segments and the indices of segments to be output at the terminal; transmitting an index sequence of speech segments to be put together to form the speech to be output; concatenating the segments according to the index sequence.
This method only requires a relatively small memory in the terminal and low computational power in each terminal. A relatively small number of speech segments is kept in a cache memory in the terminal. Speech segments used in a previous speech message are kept in the cache and may be re-used for subsequent messages. If a new text is to be output as speech by the terminal, only the speech segments which are not yet present in the terminal need to be transmitted to the terminal. Each speech segment is associated with an index allowing access to the speech segment. Even though transmission of an index sequence is sufficient for the inventive method to work, advantageously an index list is kept in the terminal and is updated every time new speech segments are sent to the terminal. The index list may be maintained by the server. Whenever a speech segment is sent to the terminal and stored in the cache, the index list at the terminal may be updated. A copy of the updated list may be kept in the server. The server may update both index lists or it may update the index list in the terminal, which then sends a copy back to the server. If a speech segment stored in the cache is not used for a certain number of speech messages it may be deleted from the cache and replaced by another segment used more often. Hence, only a small number of speech segments is stored in the terminal as compared to a whole database of speech segments. Since only the missing segments for composing a new speech message need to be transmitted from the server, the amount of data transferred from the server to the terminal is reduced. If all the speech segments for a particular output are already present in the terminal, only the index sequence for composing the speech message needs to be transmitted. Speech segments may, p. ex., be single phonemes, groups of phonemes, words or groups of words or phrases.
In a variant of the inventive method the segments to be transmitted to the terminal are chosen from a database of speech segments. The database may comprise a large number of phonemes and/or phoneme groups. Furthermore, whole phonetized words or groups of words may be stored in the database.
Alternatively, diphones may be stored in the database. If a database is used, the contents of the database are also indexed and a second index list allowing access to the database is stored in the server. In the server new speech segments may also be generated from the data available in the database, such that segments are regrouped and new groups of p.ex. phonemes are generated, which may be sent to the terminal and provided with one single index.
Alternatively, the speech segments to be transmitted to the terminal may be generated in the server each time a text is to be output by the terminal. Either the whole text is phonetized and divided into suitable segments or only the missing parts of the text, which have not been phonetized and stored in the terminal cache previously, are phonetized. This approach does not require a database in the server containing speech segments. However, a combination is also possible. If, p.ex., a phoneme needed to output text as speech is not to be found in the database, the missing part may be generated in the server by phonetizing and transmitted to the terminal.
Preferably, the speech generated from the concatenated segments is post-processed. This operation may be performed in the terminal. Post-processing improves the quality of the speech signal.
In a particularly preferred variant of the inventive method the speech segments are associated with a time-to-live value and the index lists at the terminal and the server are maintained according to these values. The time-to-live-value may be chosen by the server according to the application course. Thus, if in a certain application a speech segment is expected to be needed in a subsequent speech message of the application or if a certain speech segment is known to be used often in a particular language, a longer time-to-live value may be associated. The time-to-live-value may be a time or a number of speech messages, dialog steps or interactions. If a particular speech segment has not been used for a given time or a given number of speech messages or dialog steps it may be deleted from the cache. The time-to-live value may be updated, i.e., a new time-to-live value may be associated with a speech segment if it is used while being stored in the cache.
A quick response and output of speech messages can be achieved if subsequent speech to be output is anticipated and necessary segments for the anticipated speech signal are transmitted to the terminal. Thus, missing segments of an anticipated subsequent speech signal can already be transmitted while the previous speech message is still being output or while a command by the user is still being processed, p.ex. by a speech recognition unit, or even while the previous message is still being processed, either in the server or the terminal. Furthermore, upon certain events standardized speech messages need to be output. For example, the request to enter a command needs to be output if a command is expected but not received after a preset time. A user may also have to be prompted to repeat a command if, p. ex., speech is not recognized by the speech recognition system. Such messages can be anticipated and the missing segments for the complete speech messages can be transmitted before the event occurs. Alternatively, such messages can be permanently stored in the cache because they occur very often.
In order to avoid outputting an incomplete speech signal or to output a speech signal at the wrong time, p.ex. while a user is still thinking about the command to enter, an enabling signal may be sent to the terminal, allowing the terminal to start with the speech output. Such a signal may be a separate signal, allowing the output after a certain pause in the interaction. Alternatively, the signal may be the end of the index sequence transmitted from the server to the terminal. The concatenation of the speech signal could already begin while the index sequence is still being transmitted. The end of the sequence may be transmitted with a delay so that upon reception of the last index of the index sequence only the speech segment corresponding to the last index needs to be attached to the speech message concatenated from the previously transmitted indices. The output can thus start immediately after the end of the index sequence is received.
Within the scope of the invention also falls a terminal suitable for outputting speech messages comprising a cache memory for storing speech segments, an index list of the indices associated with the speech segments and means for concatenating the speech segments according to an index sequence. The means for concatenating may be implemented as software and/or hardware. Such a terminal requires only a small memory and a relatively small computational power. The terminal may be a stationary or a mobile terminal. With such a terminal a distributed speech synthesis system can be realized.
A distributed speech synthesis system advantageously further comprises a server for text to speech synthesis comprising means for indexing speech segments and means for selecting missing speech segments to be transmitted to a terminal which are necessary to compose a speech message in the terminal together with speech segments already present in the terminal. The means may be implemented as software and/or hardware. Such a server allows to just transmit missing speech segments for outputting a given text as speech. The terminal is enabled to put together segments already stored in the terminal and the segments transmitted by the server to form a speech signal. The terminal and the server form a distributed speech synthesis system able to perform the inventive method. The server may communicate with several terminals, keeping a copy of the index list of the speech segments stored in the cache memory of each terminal.
Advantageously, the terminal and the server are connected by a communication connection. This may be any connection allowing the transfer of speech segments and index lists, p.ex., a data link or a speech channel. Further advantages can be extracted from the description and the enclosed drawing. The features mentioned above and below can be used in accordance with the invention either individually or collectively in any combination. The embodiments mentioned are not to be understood as exhaustive enumeration but rather have exemplary character for the description of the invention.
An exemplary embodiment of the present invention is shown schematically in the drawing.
In a method of generating speech from text the speech segments necessary to put together the text to be output as speech by a terminal 2 is determined; it is checked, which speech segments are already present in the terminal 2 and which ones need to be transmitted from a server 5 to the terminal 2; the segments to be transmitted to the terminal 2 are indexed; the speech segments and the indices of segments to be output at the terminal 2 are transmitted; an index sequence of speech segments to be put together to form the speech to be output is transmitted; and the segments are concatenated according to the index sequence. This method allows to realize a distributed speech synthesis system 1 requiring only a low transmission capacity, a small memory and low computational power in the terminal 2.
| Number | Date | Country | Kind |
|---|---|---|---|
| 03360052 | Apr 2003 | EP | regional |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5802100 | Pine | Sep 1998 | A |
| 5864812 | Kamai et al. | Jan 1999 | A |
| 5978765 | Nagata | Nov 1999 | A |
| 6188754 | Kikuchi et al. | Feb 2001 | B1 |
| 6275793 | Tavernese | Aug 2001 | B1 |
| 6366883 | Campbell et al. | Apr 2002 | B1 |
| 6496801 | Veprek et al. | Dec 2002 | B1 |
| 6510413 | Walker | Jan 2003 | B1 |
| 6516207 | Boucher et al. | Feb 2003 | B1 |
| 6600814 | Carter et al. | Jul 2003 | B1 |
| 6625576 | Kochanski et al. | Sep 2003 | B2 |
| 6718339 | Eden | Apr 2004 | B2 |
| 6741963 | Badt et al. | May 2004 | B1 |
| 6810379 | Vermeulen et al. | Oct 2004 | B1 |
| 6963838 | Christfort | Nov 2005 | B1 |
| 7013278 | Conkie | Mar 2006 | B1 |
| 7043432 | Bakis et al. | May 2006 | B2 |
| 7308080 | Moriuchi et al. | Dec 2007 | B1 |
| 7440899 | Otsuka et al. | Oct 2008 | B2 |
| 20010047260 | Walker et al. | Nov 2001 | A1 |
| 20020184031 | Brittan | Dec 2002 | A1 |
| 20030028380 | Freeland et al. | Feb 2003 | A1 |
| 20030061051 | Hattori et al. | Mar 2003 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20040215462 A1 | Oct 2004 | US |
