Patent Grant
11501091
This application claims the benefit of U.S. provisional patent application Ser. No. 17/645,973 titled “Real-time speech-to-speech generation (RSSG) apparatus, method and a system therefore”, filed by Sandeep Dhawan et al on Dec. 24, 2021. The provisional patent application is incorporated herein by reference.
The present invention relates generally to the field of Speech-to-speech translation, more particularly related to direct speech-to-speech translation without text conversion to thereby automatically translate from a speech to a sign in real time.
Speech-to-speech translation is the process by which conversational spoken phrases are translated and spoken aloud in a second language. This differs from phrase translation, which is where the system only translates a fixed and finite set of phrases that have been manually entered into the system.
Speech-to-speech translation systems have been developed over the past several decades with the goal of helping people who speak different languages to communicate with each other. Such systems have usually been broken into three separate components: automatic speech recognition to transcribe the source speech as text, machine translation to translate the transcribed text into the target language, and text-to-speech synthesis (TTS) to generate speech in the target language from the translated text. Dividing the task into such a cascade of systems has been very successful, powering many commercial speech-to-speech translation products, including Google Translate.
Conventional speech to speech translation systems involves three different stages Speech to Text (Source Language), Text Translation, and Text to Speech (Target Language). However, physical models need to be included for each stage separately, and most often, model needs to be changed based on the complexity of the sub-task. The available techniques using old technologies such as hidden Markov mode (HMM) and/or Gaussian Markov mode (GMM) is not scalable to larger datasets as they have a relatively shorter ceiling of learning.
US patent application US20070208569 is one such patent which uses emotion database similar to speaker verification and is limited by emotional range in terms of words. It misses out on the phonetic nuances. The more recent deep learning models that have a higher learning ceiling also suffer from information loss due to being separately trained and cascaded, Similarly, three services to be used in each one of the available techniques Speech to Text: ASR, Translation: NMT, Text to Speech: TTS. Further, services (with Paid API Access) that are widely used include Google Speech to Text, IBM Watson, SpeakAI, and Microsoft Cognitive Services.
Thus, most of the conventionally available techniques have critical drawbacks of information loss as the essential information is lost during speech to text conversion. Information can include but is not limited to gender of speaker, age of speaker dialect, accent emotions intonation intent, statement command statement, confirmation-seeking, educational background, cooperativeness of speaker, background conditions and channel information such as but not limited to echo, reverb, noises of different kinds, which are usually present in the surrounding environment where speech is uttered, artificially induced background such as in movies or pre-recorded conversations, music, etc.
Also, the direct speech to speech translation system has the potential of becoming more accurate than a conventional system because errors do not trickle down three separate systems. The direct speech to speech translation system doesn't employ any voice cloning mechanism but related papers do mention use of a cloning system that works on speaker-centric data. The direct speech to speech translation system does Spanish to English as a prototype but has the potential of translating multiple languages into multiple languages.
Further, according to global estimates released by the World Health Organization (WHO), as of February, 2013, more than 360 million people in the world, which is greater than the population of the United States, have hard-hearing. Further, according to statistics of e-Nation Index, as of December, 2014, about a quarter of a million people have hard-hearing in Korea. Impairment by loss of physiological function, including hearing impairment, causes physiological and functional issues and serious issues in financial, social, and emotional aspects as well. Accordingly, media regulation organizations of all over the world start to deal with hearing impairment as an issue of fundamental human rights to improve such issues caused by hearing impairment, and provide three major types of services. A subtitle service includes closed captioning (CC), subtitles for hard of hearing (HoH), and subtitles for deaf and hard of hearing (SDF) which all help a hearing-impaired person not to experience alienation in everyday life. However, each of these types of services has their own limitations.
Further, a dire need exists to generate a sign language in real-time for hard-hearing people once the speech to speech translation is completed.
To summarize the above, information loss in speech to text conversion and inability to hear the translated language with preserved liable vocal emotion information, background information, voice channel conditions and artificially induced audio in a conventional speech to speech translation system are essential drawbacks of the conventional techniques.
Embodiments of the invention provide direct speech to speech translation system is disclosed that solves the above recited issues. The presented direct speech to speech translation system provides direct speech to speech translation system and specific training methodology to preserve essential information of the audio signal in the task of speech to speech translation. It further generates a sign language in real-time for hard-hearing/deaf or hearing impaired people once the speech to speech translation is completed.
All objects, features and advantages of the present invention will become apparent in the following detailed written description.
The Summary is neither intended nor should it be construed as being representative of the full extent and scope of the present invention, which these and additional aspects will become more readily apparent from the detailed description, particularly when taken together with the appended drawings.
Direct speech to speech translation system uses a one-tier approach, creating a unified-model for whole application. The single-model ecosystem takes in audio waveform—a time domain signal, preprocesses it into a frequency domain signal such as a mel spectrogram and gives out a frequency domain signal such as a mel spectrogram to be converted to an audio waveform. The model solves the bottleneck problem by not converting speech directly to text but having text as a byproduct of speech to speech translation, preserving phonetic information along the way. This model also uses pre-processing and post-processing scripts but only for the whole model.
In one embodiment, the Speech to Speech model needs parallel audio samples in two languages. For instance, in the case of Spanish to English Speech to Speech translation, the model would need training samples of audio spoken in Spanish and corresponding English audio. These are called “parallel samples”. The parallel samples can optionally include Spanish and English text as well. The text helps the model train faster.
The training methodology involves augmenting or changing both sides of the audio equally (audio in the first language and the audio in the second language). The Spanish and English specification is an example and does not represent the model's limitation in understanding languages. The Speech to Speech model generalises well on any language pairs.
To allow the model to learn background and other information mentioned in the aforementioned text, the parallel samples could be augmented by one or all of the following data augmentation techniques. In the proposed model, augmentations need to be applied equally on both sides of the parallel samples. In case of Spanish to English conversion, the augmentation needs to be applied to audio samples of Spanish and English equally, and to the same extent.
To allow the model to generate a sign language in real-time for hard-hearing people once the speech to speech translation is completed, it performs below preferably steps:
Following are preferable set of augmentation techniques that can be directly applied to the parallel samples.
Accordingly, in contrast to the conventional systems, the direct speech to speech translation system of the present invention prevents information loss and helps avoid information bottlenecks.
Speech to Speech translation can be used with existing voice chat applications and could potentially bridge the gap between multi-lingual e-commerce store owners and their customers. The Speech to Speech translation can also be used with live phone-calls and voice messages where a speaker speaks from one end in the first language while the listener listens to the voice in another language from the other end. With the advent of new ways in human interaction, audio recognition is increasingly becoming more common in the industry. As WhatsApp, Signal and other chat applications improve their audio chatting services and, as businesses turn towards these chat applications for customer interaction and services, there seems to be a potential market for including multi-lingual interactions here. These conversations can also sound a lot more interactive if they're translated into the voice of the same speaker who spoke the original sentence, preserving the phonetics of the person speaking the sentence. Also, the benefits of the conversion/translation into second language from first language can be transferred to the hearing impaired peoples who are also the part of our society and the user of such communication device such as but not limited to mobile phones or smart phones and laptops.
The method can be configured to be implemented in the form of a software or a plugin or as mobile application.
The accompanying drawings are included to provide a further understanding of the present disclosure, and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
The diagrams are for illustration only, which thus is not a limitation of the present disclosure, and wherein:
The following is a detailed description of embodiments of the disclosure depicted in the accompanying drawings. The embodiments are in such detail as to clearly communicate the disclosure. However, the amount of detail offered is not intended to limit the anticipated variations of embodiments; on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present invention. It may be apparent to one skilled in the art that embodiments of the present invention may be practiced without some of these specific details.
Various terms as used herein are shown below. To the extent a term used, it should be given the broadest definition persons in the pertinent art have given that term as reflected in printed publications and issued patents at the time of filing.
Embodiments of the invention provide direct speech to speech translation system is disclosed that solves the above recited issues. The presented direct speech to speech translation system Provides direct speech to speech translation system and specific training methodology to preserve essential information of the audio signal in the task of speech to speech translation.
In an embodiment, direct speech to speech translation system uses a one-tier approach, creating a unified-model for whole application. The single-model ecosystem takes in audio (mel spectrogram) as an input in time domain signal and gives out audio (mel spectrogram) as an output in time domain signal. The model solves the bottleneck problem by not converting speech directly to text but having text as a byproduct of speech to speech translation, preserving phonetic information along the way. This model also uses preprocessing and post-processing scripts but only for the whole model.
In this embodiment, the Speech to Speech model needs parallel audio samples in two languages. For instance, in the case of Spanish to English Speech to Speech translation, the model would need training samples of audio spoken in Spanish and corresponding English audio. These are called “parallel samples”. The parallel samples can optionally include Spanish and English text as well. The text, in some cases, helps the model train faster.
The training methodology involves augmenting or changing both sides of the audio equally (audio in the first language and the audio in the second language). The Spanish and English specification is an example and does not represent the model's limitation in understanding languages. The Speech to Speech model generalises well on any language pairs.
To allow the model to learn background and other information mentioned in the aforementioned text, the parallel samples could be augmented by one or all of the following data augmentation techniques. In the proposed model, augmentations need to be applied equally on both sides of the parallel samples. In case of Spanish to English conversion, the augmentation needs to be applied to audio samples of Spanish and English equally.
Following are preferable set of augmentation techniques that can be directly applied to the parallel samples.
Artificially create one of the above or multiple augmentations and place them within the room to create more diverse room effects.
In an embodiment,
In an exemplary embodiment, the input audio signal comprises one or more paragraphs spoken by a user in the first language, and wherein the output audio signal comprises audio of one or more translated paragraphs of the one or more paragraphs spoken in the second language.
In an exemplary embodiment, the one or more paragraphs include textual data that has context and content.
In an exemplary embodiment, the input audio signal is split into multiple signals for converting into the MEL spectrogram.
In an exemplary embodiment, the method further comprising: pre-processing the first time domain signal before converting into the MEL spectrogram by using at least one of the techniques such as machine-learning, signal processing, stats-based pre-processing, Digital Signal Processing, or data normalization for data augmentation on the first time domain signal.
In an exemplary embodiment, the method further comprising: converting the input audio signal in the first language directly into the output MEL spectrogram of the second language by preserving phonetic information, which is then converted to an audio signal using the post-processing module.
In an exemplary embodiment, the first language and the second language are different languages.
In an exemplary embodiment, the input audio signal comprises one or more paragraphs spoken by a user in the first language, and wherein the output audio signal comprises one or more translated paragraphs of the one or more paragraphs spoken in the second language.
In an exemplary embodiment, the one or more paragraphs include textual data that has context and content.
In an exemplary embodiment, the input audio signal is split into multiple signals for converting into the MEL spectrogram.
In an exemplary embodiment, the processor pre-processes the first time domain signal before converting into the MEL spectrogram by using techniques for data time domain data augmentation on the first time domain signal.
In an exemplary embodiment, the processor pre-processes the first time domain signal before converting into the MEL spectrogram by using at least one of machine-learning, signal processing techniques, statistic-based pre-processing technique, Digital Signal Processing techniques, data normalization techniques for data augmentation on the first time domain signal.
In an exemplary embodiment, the real-time speech-to-speech generator and sign gestures converter apparatus is configured to convert the input audio signal in the first language directly into the output audio signal in the second language without requirement of any speech to text conversion as an intermediate step.
In an exemplary embodiment, the real-time speech-to-speech generator and sign gestures converter apparatus is configured to convert the input audio signal in the first language directly into the output audio signal in the second language by preserving phonetic information and further convert the second language output into sign language to be read by the user in second language on a display device.
This architectural setting is prone to a bottleneck that is created after ASR, which converts speech to text, discarding information pertaining to phonetics of the speaker. Hence, the model has this inherent limitation of not being able to take in emotional context and phonetic information of the speaker as it moves along the pipeline. Due to this, errors coming in after ASR are also trickled all the way down to outputs of text to speech. The
In an embodiment, as shown in
Mechanism 1: Reading a paragraph of information
Upon performing above, the system creates a batch of data by selecting a subset of data from within the entire dataset. For each sentence from the dataset, gather: Audio in Language 1, Text in Language 1, Audio in Language 2, and Text in Language 2.
To allow the model to generate a sign language in real-time for hard-hearing people once the speech to speech translation is completed, it performs below preferably steps:
In an example, a universal sign language notation database may be used for conversion of speech to sign conversion. In another example, any of the existing systems/solutions (for example, app, called GnoSys) that uses neural networks and computer vision to recognize the speech of language speaker, and then smart algorithms translate it into sign can be used to the implementation of the above feature.
In yet another example, an animation corresponding to the outputted/generated sign (gesture) can be generated. The animation is generated using Signing Gesture Markup Language (SigML).
In another example, the speech-sign converter may translate a speech into a sign, and output a result of translation, for example, the sign.
Although the present invention herein has been described with reference to particular preferred embodiments thereof, it is to be understood that these embodiments are merely illustrative of the principles and applications of the invention. Therefore, modifications may be made to these embodiments and other arrangements may be devised without departing from the spirit and scope of the invention, which is defined by the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6377925 | Greene, Jr. | Apr 2002 | B1 |
| 8330864 | Blumenschein | Dec 2012 | B2 |
| 20040143430 | Said | Jul 2004 | A1 |
| 20090234636 | Rylander | Sep 2009 | A1 |
| 20150227510 | Shin | Aug 2015 | A1 |
| 20170236450 | Jung | Aug 2017 | A1 |
| 20180047387 | Nir | Feb 2018 | A1 |
| 20190138603 | Daley | May 2019 | A1 |
| 20190315227 | Kim | Oct 2019 | A1 |
| 20200075011 | Yao | Mar 2020 | A1 |
| 20200226327 | Matusov | Jul 2020 | A1 |
| 20220139417 | Maxwell | May 2022 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20220327294 A1 | Oct 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17645973 | Dec 2021 | US |
| Child | 17838181 | US |
