Patent Grant
9720909
Participants in a teleconference have found it helpful to have a transcript of the teleconference conversation. In the event the audio connection of the teleconference is poor, closed-captioning provides users with the ability to read what is being spoken. In the teleconference context, these closed-captioning services are typically performed by a third party stenography service. The third party stenographer sits in on the teleconference to provide the close captioning and a text transcript of the conversation. However, this may prohibit impromptu calls between teleconference participants, who want a text transcript of the conversation. Additionally, if the participants speak and read different languages from one another, a translation of the transcript and closed-captioning may be required. If the participants are to have a meaningful conversation, it may be necessary to generate the closed-captioning in real time. Adding a third-party translator to a teleconference can increase the complexity and cost of a teleconference to such a point that it would be prohibitive for a casual user to conduct such a teleconference.
According to an embodiment of the disclosed subject matter, the implementation may include a conferencing services server, a translation services server, a data storage, a speech-to-text processor, a translation processor, and a text-to-speech processor. The conferencing services may exchange text and audio data signals between communicatively coupled devices. The audio data signals may represent communication in a first spoken language, while the text signals may represent communication in the same or another spoken language. The data storage may store data including text and audio data signals. The translation services server may receive audio signals from devices via a separate communication channel. The speech-to-text processor may be configured to convert the first spoken language audio data signals into text corresponding to the communication in the first spoken language. The translation processor may be configured to translate the first language text into text in a second language. The text-to-speech processor may be configured to convert the second language text into audio signals representing a spoken version of the second language text. The second language text and audio signals may be stored in the data storage. The translation server may deliver the second language text and audio signals to the respective device requesting translation services.
According to an embodiment of the disclosed subject matter, the implementation may include a method for providing an audio and textual transcript of a communication. The method may include receiving audio signals representing speech in a first spoken language at a conferencing services device. The received audio signal may be delivered to an end user device. The delivered audio signals may be received at a translation services server over a separate communication channel with the end user device at which the audio signals may be converted into text of the first spoken language by a processor in the speech-to-text server. The converted text of the first spoken language may be stored. The first spoken language text may be translated into text of a second language. The translated text of a second language may also be stored. The second language text may be converted into audio signals representing speech in the second spoken language, and at least one of the received audio data, the stored text, or the translated audio signals may be delivered back to the end user device over the separate communication channel.
Additional features, advantages, and embodiments of the disclosed subject matter may be set forth or apparent from consideration of the following detailed description, drawings, and claims. Moreover, it is to be understood that both the foregoing summary and the following detailed description are exemplary and are intended to provide further explanation without limiting the scope of the claims.
The accompanying drawings, which are included to provide a further understanding of the disclosed subject matter, are incorporated in and constitute a part of this specification. The drawings also illustrate embodiments of the disclosed subject matter and together with the detailed description serve to explain the principles of embodiments of the disclosed subject matter. No attempt is made to show structural details in more detail than may be necessary for a fundamental understanding of the disclosed subject matter and various ways in which it may be practiced.
There is a need to automate the conversion of a teleconference transcript into text, and providing the text to the users either in the form of closed-captioning or a post-transcript, searchable computer-readable file. In addition, the translation of the text into different languages also needs to be automated so persons with different language capabilities can easily schedule and participate in real-time teleconferences. It would also be beneficial if the text conversion and translation capabilities were applicable to mobile device-based chats and messaging.
A system 100 as illustrated in
The translation services server 145 may also include a processor 145A for coordinating translation functions of the audio or text signals. The translation services server 145 may be communicatively coupled to the end user devices 110, 113, 120 and 123 via a communication network, such as the Internet or a mobile communication network. The end user devices 110, 113, 120 and 123 may include a processor that executes a communication application. The communication application may include functionality to intercept incoming text or audio signals during a teleconference communication, recognize the language of incoming audio signals, and cause the respective device to establish a connection with the translation services server 145 based on the recognition result. For example, if the recognized language does not correspond to a language preference setting of the end user device, the end user device may establish a separate connection with the translation services server 145 and forward the incoming text or audio signals to the translation services server 145 for translation to one of the end user device's set language preferences. Although the system 100 is illustrated with four devices 110, 113,120 and 123 connected to conferencing services server 140, it should be understood that more or fewer end user devices may participate in the communication between devices 110, 113, 120 and 123 through the conferencing services server 140. The conferencing services server 140 may manage a plurality of different teleconference sessions between a plurality of different end user devices. Any of the devices 110, 113,120 and 123 may also be capable of connecting to the translation services server 145. For ease of explanation, only a teleconference session between devices 110 and 120 will be described in detail. Devices 113 and 123 may participate in their own teleconference separate from devices 110 and 120. Any other connected devices, such as devices 113 and 123 may operate in a similar manner as devices 110 and 120.
The translation services server 145 may be connected to the conferencing services server 140. The speech-to-text converter (STC) 150, the text-to-speech converter (TSC) 160 and the text translator 170 may be connected to and managed by the translation services server 145 via a network connection, such as the Internet, LAN or WAN, to allow for the transfer of data and control signals. The speech-to-text converter (STC) 150, the text-to-speech converter (TSC) 160 and the text translator 170 may also be connected directly to one another, which may allow the translation services server 145 to be omitted during some processes. The data storage 130 can be any form of data storage device such as a hard disk, non-volatile memory, FLASH memory or the like.
The STC 150 may be a server with a processor(s) and/or memory. The STC 150 may include inputs for receiving audio and audio video signals from the translation services server 145. The STC 150 may be configured to identify a language of the received input audio signals, and access a plurality of different processes to convert the audio signals of the identified language into text of the identified language. The identity of the language in the input audio signals may be recognized by the STC 150 or indicated by a user preferences signal incorporated in the input audio signals. The TSC 160 may be a server with a processor(s) and/or memory. The TSC 160 may be configured to receive text signals from the STC 150 and the text translator 170, and convert the text into audio signals representing speech. In addition to receiving signals from the STC 150 and the text translator 170, the TSC 160 may receive text signals from other sources such as the translation services server 145.
The text translator 170 may be a processor or may be a processor hosted on a server. The text translator 170 may be configured to translate text from a first language into text of a second language. The text translator 170 may have access to a plurality of translation processes for translating text from one language to another. These different processes may be maintained in data storage 130, or stored within the text translator 170.
The translation services server 145 and the conferencing services server 140 may also exchange data related to timing of the delivery of text or audio signals, communication channel status, and other data useful for managing the communication between the end user devices participating in a communication session. The translation services server 145 may be capable of participating in a plurality of conferences and responding to a plurality of different translation requests from end user devices.
As the end user devices communicate in the teleconference, the participants' speech may be converted by the respective end user device 110 or 120 into encoded audio signals that may be incorporated into an output data stream from the end user devices 110 or 120. At step 210 of
Continuing with the above example, the receiving, or intended, end user device 120 may compare its language preference settings with the language preference identifiers embedded in the encoded audio signals sent by sending end user device 110, or perform language recognition to determine the compatibility of the languages. Upon making the determination that the language of the received audio signals are not compatible with the language preferences of the receiving end user device 120, the receiving, or intended, end user device 120 may send the received audio signals to the translation services server 145 for translation into a second language. (Step 240). As another example, the intended end user device 120 may send the received audio signals directly to the translation services server 145 without performing any type of language recognition or language preference comparison. In this case, the intended end user device 120 may further embed indicators of its language preference setting into the data stream that includes the encoded audio signals. The translation services server 145 may perform language recognition processes to confirm that the language of the audio signals corresponds to the primary language preference indicator, or a secondary language preference indicator provided by the sending end user device 110. If the language preference indicator does not correspond to the language recognized by the language recognition process, the translation services server 145 may perform language translation corresponding to the language recognized by its own processes.
The operation of the translation services server 145 will be described with respect to
At step 235, the STC 150 may return the converted text to the translation services server 145. A decision on the language the converted text is to be translated into may be made at step 245. The translation services server 145 may, for example, review the language preferences indicator included with the audio signals to determine whether the converted text needs to be translated into another language. The translation services server 145 may forward the converted text to a text translator server 170 at step 255. The text translator 170 may call the appropriate translation engine or engines based on the indicated language preference. At step 265, the text translator 170 may perform the translation of the text from the first language into text in the second language, as well as any additional languages, based on the language preference indicators. For example, the text translator 170 may translate the English text to French text (i.e., first language to the second language) using a translation table containing text data corresponding to English words and corresponding to French words or a similar type of translation mechanism. (Step 265). The text translation performed by the text translator 170 may be performed using, for example, statistical machine translation and/or rules-based machine translation. Statistical translation may be performed by searching for sub-phrases (down to a single word) and up to the complete phrase or sentence, in a phrase table of translations. Potential hypotheses may be computed for the many different combinations of matching phrases from the phrase table can be combined to produce a potential translation. Those hypotheses may be scored according to a number of different metrics. An exemplary metric may be a language model that may determine the likelihood of the produced translation being a reasonable sentence in the target language. The produced translation with the highest likelihood score may be chosen. The rules-based machine translation may use linguistic rules and vocabulary tables to produce a translation. The text translator 170 may store the translated text in a text file. At step 270, the text translator 170 may return the translated text to the translation services server 145.
At step 275, the conferencing services server 140 may determine whether the translated text file or converted text files are to be converted into audio files for output as speech. This may, for example, be indicated during the initial set-up of the communication between the participant's devices or based on the indicators in the data stream. If the determination is NO, the text is not to be converted into audio files; the translation services server 145 may deliver the converted text, the translated text or both to the end user device 120. (Step 280).
In response to a determination at step 275 that the text is to be converted into audio files (“YES”), the translation services server 145 at step 285, may deliver the translated text to a TSC 160 server. At step 290, the translated text may be converted by a processor in the TSC 160 server into audio data of speech in a second spoken language. Continuing with the earlier example, the translated English-to-French (first language-to-second language) text may be delivered by the translation services server 145 to the TSC 160. The French (second) language text may be converted by the TSC 160 into audio signals representing speech in French, or the second language. At step 295, the TSC 160 may return the audio signals to the translation services server 145 for delivery to the device 120.
Returning back to step 245, if the decision is NO, the converted text does not need to be translated, the process may proceed to step 275. At step 275, the translation services server 145 may determine whether the converted text files are to be converted into audio files for output as speech. This may, for example, be indicated during the initial set-up of the communication (i.e., teleconference or videoconference) between the end user devices or based on the indicators in the audio data stream. If the determination is NO, the text is not to be converted into audio files; the translation services server 145 may deliver the converted text, the translated text or both back to the end user device 120 for output at step 280.
If the decision at step 275 is YES, the converted text is to be converted to audio signals. The translation services server 145 at step 285 may deliver the translated text to a TSC 160 server. At step 290, the translated text may be converted by a processor in the TSC 160 server into audio data of speech in a second spoken language.
The translation services server 145 may deliver the translated French, or second language, audio signals to the end user device 120 (Step 295). In alternative embodiments, the translation services server 145 may deliver the translated second language text and/or the first language text in addition to the second language audio signals to the device 120. The TSC server 160 may store the audio data in an audio data file. The translation services server 145 may forward an audio and/or text transcript of the translated text or audio to the conferencing services server 140 for incorporation into a transcript of the respective teleconference. For example, each teleconference managed by the conferencing services server 140 may have an identifier. The identifier may be provided to the translation services server 145 by the end user device 120 when connecting to the translation services server 145.
Communication from the end user device 120 to the end user device 110 may be processed in a similar manner as described above with respect to the communication between end user device 110 and 120. Similarly, communication between the end user devices 113 and 123 may communicate with one another, conference services server 140, translation services 145, and end user devices 113 and 123. The processing of the received audio, subsequent conversion to text, and translation may be performed in substantially real time to provide the participants with an experience similar to having a conversation in the same room.
The STC 150, TSC 160 and text translator 170 may also be connected to one another without the translation services server 145 acting as an intermediary. In which case, the STC 150, TSC 160 and text translator 170 may deliver the respective output signals to one another based on control signals from the translation services server 145. The output audio signals to be output to the end device, such as device 120 in the above example, may be delivered by translation services server 145.
As mentioned above, the text generated by the STC 150 may be buffered. The buffered text may be stored in a data file for subsequent use by participants in the chat or teleconference. The data file may be a searchable transcript that can be archived for post conversation retrieval. In an example of a chat, the conversion of the text to speech by the TSC 160 may not be necessary. In this case, step 240 may be optional, and only the translated text may be delivered to the device 120 by the conferencing services server 140. As a result, two translation modes, one mode with audio and another mode without audio may be provided. Participants that speak the same language may indicate that only the converted text should be displayed on the respective devices 110 and 120 to allow the conversation to be followed in the event the audio signal is less than optimal.
During the chat or teleconference, the conferencing services server 140 and the translation services server 145 may respond to real-time control inputs from the respective devices 110 or 120. An end user device 110 may change a language preference indication during the conversation. For example, a user may be more proficient in Chinese, and change the English language preference to a Chinese language preference. In response to the changed language preference, the translation services server 145 may output an updated language preference control signal to the STC 150. In response to the updated language preference control signal, the STC 150 may begin converting the input audio signals to Chinese text instead of English text.
As mentioned above, the translation services server 145 may periodically (e.g., every ten seconds) recognize the language from each device 110, 120 during the teleconference or chat, and may note any change in language from the first or second language to a third language, by generating an updated language preference indicator. For example, a participant may stop speaking English during part of the teleconference, and begin speaking Chinese. In which case, the translation services server 145 upon recognizing the change in languages from the particular device may automatically provide an updated language preference indicator to the STC 150 indicating the new language. This allows the system 100 to accommodate different language capabilities of teleconference or chat participants that may be sharing a device. For example, one of the co-located participants may not be as fluent in the preferred, primary language as another co-located participant, and when a complex discussion needs to occur, it would be advantageous if the participant could change to the language with which they are more fluent.
The conferencing services server 140 may be configured to output the stored first language text, buffered second language text, or output the buffered second language audio signals. The translation services server 145 may be further configured to output the stored second language audio signals and the stored second language text. A transcript of the audio signals and text signal in each of the first, second and third language may be maintained in the data storage 130 or in the respective server 150, 160 or 170 memory. The transcripts may be updated during the exchange of text and audio data signals between the communicatively coupled devices 110 and 120. The audio and text transcripts may be updated during the exchange of text and audio data signals between the communicatively coupled end user devices 110, 120 by the conferencing services server 140.
The stored transcript may also include alternate translation options, so end user devices can deliver other possible translation candidates exist, and the user may be allowed to select the appropriate word. For example, the English words “bare” and “bear” have similar pronunciations, but have different meanings.
Embodiments of the presently disclosed subject matter may be implemented in and used with a variety of component and network architectures.
The bus 310 allows data communication between the central processor 340 and the memory 370, which may include read-only memory (ROM) or flash memory (neither shown), and random access memory (RAM) (not shown), as previously noted. The RAM is generally the main memory into which the operating system and application programs are loaded. The ROM or flash memory can contain, among other code, the Basic Input-Output system (BIOS) which controls basic hardware operation such as the interaction with peripheral components. Applications resident with the computer 300 are generally stored on and accessed via a computer readable medium, such as a hard disk drive (e.g., fixed storage 330), an optical drive, floppy disk, or other storage medium 350.
The fixed storage 330 may be integral with the computer 300 or may be separate and accessed through other interfaces. A network interface 390 may provide a direct connection to a remote server via a telephone link, to the Internet via an internet service provider (ISP), or a direct connection to a remote server via a direct network link to the Internet via a POP (point of presence) or other technique. The network interface 390 may provide such connection using wireless techniques, including digital cellular telephone connection, Cellular Digital Packet Data (CDPD) connection, digital satellite data connection or the like. For example, the network interface 390 may allow the computer to communicate with other computers via one or more local, wide-area, or other networks, as shown in
Many other devices or components (not shown) may be connected in a similar manner (e.g., document scanners, digital cameras and so on). Conversely, all of the components shown in
More generally, various embodiments of the presently disclosed subject matter may include or be embodied in the form of computer-implemented processes and apparatuses for practicing those processes. Embodiments also may be embodied in the form of a computer program product having computer program code containing instructions embodied in non-transitory and/or tangible media, such as floppy diskettes, CD-ROMs, hard drives, USB (universal serial bus) drives, or any other machine readable storage medium, wherein, when the computer program code is loaded into and executed by a computer processor, the computer becomes an apparatus for practicing embodiments of the disclosed subject matter. Embodiments also may be embodied in the form of computer program code, for example, whether stored in a storage medium, loaded into and/or executed by a computer, or transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, wherein when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing embodiments of the disclosed subject matter. When implemented on a general-purpose microprocessor, the computer program code segments configure the microprocessor to create specific logic circuits. In some configurations, a set of computer-readable instructions stored on a computer-readable storage medium may be implemented by a general-purpose processor, which may transform the general-purpose processor or a device containing the general-purpose processor into a special-purpose device configured to implement or carry out the instructions. Embodiments may be implemented using hardware that may include a processor, such as a general purpose microprocessor and/or an Application Specific Integrated Circuit (ASIC) that embodies all or part of the techniques according to embodiments of the disclosed subject matter in hardware and/or firmware. The processor may be coupled to memory, such as RAM, ROM, flash memory, a hard disk or any other device capable of storing electronic information. The memory may store instructions adapted to be executed by the processor to perform the techniques according to embodiments of the disclosed subject matter.
The foregoing description and following appendices, for purpose of explanation, have been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit embodiments of the disclosed subject matter to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to explain the principles of embodiments of the disclosed subject matter and their practical applications, to thereby enable others skilled in the art to utilize those embodiments as well as various embodiments with various modifications as may be suited to the particular use contemplated.
This application is a continuation of U.S. application Ser. No. 13/316,689, filed Dec. 12, 2011. The disclosure of the above application is incorporated herein by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5815196 | Alshawi | Sep 1998 | A |
| 6292769 | Flanagan | Sep 2001 | B1 |
| 6339754 | Flanagan | Jan 2002 | B1 |
| 6859820 | Hauduc | Feb 2005 | B1 |
| 7054804 | Gonzales et al. | May 2006 | B2 |
| 7194687 | Sezan | Mar 2007 | B2 |
| 7734467 | Gao et al. | Jun 2010 | B2 |
| 7830408 | Asthana et al. | Nov 2010 | B2 |
| 7970598 | Flanagan | Jun 2011 | B1 |
| 8051452 | Arseneau | Nov 2011 | B2 |
| 8259910 | Afifi | Sep 2012 | B2 |
| 8327270 | Jones et al. | Dec 2012 | B2 |
| 8498871 | Miglietta et al. | Jul 2013 | B2 |
| 8699994 | Kim | Apr 2014 | B2 |
| 20030009329 | Stahl | Jan 2003 | A1 |
| 20030018726 | Low | Jan 2003 | A1 |
| 20040034521 | Kawakura | Feb 2004 | A1 |
| 20040102957 | Levin | May 2004 | A1 |
| 20040267527 | Creamer | Dec 2004 | A1 |
| 20080022211 | Jones | Jan 2008 | A1 |
| 20080195482 | Sailer | Aug 2008 | A1 |
| 20090099836 | Jacobsen et al. | Apr 2009 | A1 |
| 20090234643 | Afifi | Sep 2009 | A1 |
| 20100185445 | Comerford | Jul 2010 | A1 |
| 20100250231 | Almagro | Sep 2010 | A1 |
| 20100257234 | Caughey | Oct 2010 | A1 |
| 20110134910 | Chao-Suren | Jun 2011 | A1 |
| 20110224981 | Miglietta | Sep 2011 | A1 |
| 20110282645 | Khuda | Nov 2011 | A1 |
| 20110287748 | Angel | Nov 2011 | A1 |
| 20120109631 | Gopal | May 2012 | A1 |
| 20120330644 | Giraudy | Dec 2012 | A1 |
| 20130058471 | Garcia | Mar 2013 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20150356077 A1 | Dec 2015 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 13316689 | Dec 2011 | US |
| Child | 14827826 | US |
