Patent Application
20230325612
The present application claims priority under 35 U.S.C. 119(a)-(d) to the Indian Provisional Patent Application Serial No. 202211021330, having a filing date of Apr. 9, 2022, the disclosure of which is hereby incorporated by reference in its entirety.
A voice translator can translate voice input to provide translated voice output. The voice translation uses different Application Programming Interfaces (APIs) and speech recognition libraries of different programming platforms. The voice translation can convert text from one language to another language and saves the recorded file. Artificial Intelligence (AI) techniques have enabled intelligence speech recognition technology that makes cross-language communications easier. However, challenges exist in providing high-quality, accurate voice translations through automatic translators.
Features of the present disclosure are illustrated by way of examples shown in the following figures. In the following figures, like numerals indicate like elements, in which:
For simplicity and illustrative purposes, the present disclosure Is described by referring to examples thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be readily apparent however that the present disclosure may be practiced without limitation to these specific details. In other instances, some methods and structures have not been described in detail so as not to unnecessarily obscure the present disclosure. Throughout the present disclosure, the terms “a” and “an” are intended to denote at least one of a particular element. As used herein, the term “includes” means includes but not limited to, the term “including” means including but not limited to. The term “based on” means based at least in part on.
An AI-driven multi-platform, multilingual translation system is disclosed. The system provides translation services to a plurality of communication platforms having corresponding communication apps or front ends. To provide translation services, the system is coupled to a plurality of translation engines and a plurality of transcription engines. The system receives an audio input and associated metadata of a communication session from one of the communication apps corresponding to one of the plurality of communication platforms holding the communication session. The communication session can include a real-time communication session such as a teleconference with multiple participants, a recorded session wherein a recording is provided to one or more listeners or a combination of real-time and recording sessions, e.g., wherein a recording is played during a teleconference. The communication apps can include not only mobile applications but any front end employed by one or more of the plurality of communication platforms for facilitating user communications. Accordingly, a desktop application enabling user communications can also be considered a ‘communication app’. The audio input and the associated metadata are analyzed to determine the speech context of the audio input. One of the plurality of translation engines is selected based on the speech context for translating the audio input from the source language to a target language. An audio output including the translation of the audio input is provided to one or more listeners of the communication session via the communication app from which the audio input was received. Various other services such as text and grammar correction, summarizing the communication session, etc. are also enabled by the system.
The system includes a voice analyzer and translator coupled to the plurality of communication platforms and the corresponding communication apps via two integration layers. The first integration layer includes platform-specific pipelines connecting the voice analyzer and translator to the corresponding communication apps. The second integration layer includes platform-specific pipelines connecting the voice analyzer and translator to the plurality of communication platforms. Participants of the communication session use one of the plurality of communication platforms and therefore provide the audio input and the associated metadata via the corresponding communication app. The voice analyzer and translator generates the audio output and provides the audio output to the communication platform for transmission to the listener requesting the translation through the communication app.
The audio input and associated metadata are analyzed to determine if multiple speakers are participating in the communication session. If multiple speakers are speaking, the audio input is further analyzed to determine if multiple audio streams of respective multiple speakers are included in corresponding multiple channels or a single channel. If the audio streams are being transmitted in multiple channels then the channels are split. If the multiple audio streams are transmitted in a single channel, then the plurality of speakers or the multiple audio streams are separated out on the corresponding channels via AI-based diarization. Furthermore, the audio input can be transcoded if it is not transmitted in audio formats compatible with the plurality of translation engines.
The audio input is transcribed and a speech context is determined. In an example, the speech context can include among other attributes, the communication platform being used, the source language, the target language, the particular speaker providing the voice input, the domain pertaining to the speech, and other metadata such as the date/time of receipt of the audio input, the length of the voice input of the particular speaker, speaker-specific stream details, etc. The communication session is monitored to detect any change in the speech context. In an example, single-shot language detection is implemented so that the audio input is briefly sampled periodically as part of the speech context detection. When a change in the speech context i.e., any change in the aforementioned attributes is detected, it is further determined if another translation engine needs to be selected in response to the detected change. If yes, another translation engine is selected and the audio output generated from the newly selected translation engine is provided to the listener. The system enables the selection of translation engines via a Q-table. The user feedback on a translation is collected and the Q-table is updated. The next translation request that is received is serviced by referring to the updated Q-table. Thus, if different translation engines produce greater translation accuracies for different voice attributes, then the most accurate translation engines can be selected and used even for a single communication session. Other functions such as a summary of the communication session, sentiment associated with the communications session, actionable items arising from the communication session, etc. can be identified.
The system provides for an end-to-end AI-driven natural language processing (NLP) platform that enables translation of voice communications for a plurality of platforms in a plurality of languages by employing the plurality of translation engines. By separating each audio stream of each speaker into a different channel through diarization, the system improves the audio quality provided to the translation engines thereby increasing the accuracy of the translation output. Furthermore, the translation accuracy is also improved through unique language-based gender models as voice tones for different genders are different. The voice input is converted to text in real-time to further qualify the translation output with sentiment data. The system, therefore, provides near real-time translation with support for both textual and audio formats. The continuous monitoring of the communication sessions enables the system to accommodate on-the-fly changes of source and target languages in addition to efficiently handling translation service outages. The system also provides for realistic audio translation through the continuous sampling of the sentiment of the speaker and transforming the audio output to reflect the speakers' sentiments. The system is customizable as ML models used for language, gender, or sentiment detection can be chosen based on efficiency. The translation is provided in different formats so that users can either choose to listen to the translation or read the translated transcription of the communication session. Furthermore, the system can use cloud platforms to scale thereby supporting numerous users across multiple communication sessions and on multiple communication platforms simultaneously.
For example, the audio input 150 may include one or more audio streams from a voice, video, or other communication, wherein a plurality of speakers 112 are speaking in a source language and listeners/audience may be logged in to the communication session to listen to the plurality of speakers. In an example, the audio input 150 can be provided by the communication platform 1 along with the metadata related to the audio input 150. Further, the listeners/audience may not necessarily understand the source language and may request translation of the audio input 150 into one or more target languages which may be provided by the voice analysis and translation system 100. The voice analysis and translation system 100 can be installed or configured as an intermediary between a communication app and the corresponding one of the plurality of communication platforms 130. In an example, the voice analysis and translation system 100 can be installed on an exclusive server providing the intermediary translation services implementing models such as software as a service (Sans) for the plurality of communication platforms 130. When operated from a cloud server, the voice analysis and translation system 100 can be highly scalable and support thousands of users across different platforms concurrently translating their voice inputs into different languages simultaneously. One or more of the plurality of communication apps 132 installed on a user device can have access to the voice analysis and translation system 100 so that the user/audience/listener 114 may obtain translated versions of any audio streams received via the corresponding communication platforms.
The voice analysis and translation system 100 may include a first integration layer 102, a voice analyzer and translator 104, and a second integration layer 106. The first integration layer 102 includes a plurality of pipelines e.g., pipeline 11, pipeline 12, and pipeline 13 communicatively coupling the plurality of communication applications 132 to the voice analyzer and translator 104. The second integration layer 106 also includes a plurality of pipelines e.g., pipeline 21, pipeline 22, and pipeline 23, that are communicatively coupled to the plurality of communication platforms 130. It may be appreciated that since a pipeline is a communication channel that is programmatically created and operated, the pipelines in the first integration layer 102 can be different from the pipelines of the second integration layer 106 as the former connect the voice analysis and translation system 100 with the plurality of communication apps 132 while the latter connects the voice analysis and translation system 100 with the plurality of communication platforms 130. In an example, the pipelines 11 and 21 can be peculiar to the communication platform 1. Similarly, other pipelines used for other platforms can also be platform-specific. The first integration layer 102 and the second integration layer 106 are both bidirectional layers integrating the voice analyzer and translator 104 with each of the plurality of communication platforms 130. The first integration layer 102 can be configured to capture voice communication metadata and the audio streams from the audio input 150 delivering these to the voice analyzer and translator 104. The second integration layer 106 can be configured to receive the output of the voice analyzer and translator 104 and transmit it as the audio output 152 on the communication platform executing the voice communication session to be provided to the user via the corresponding communication app or front end. In an example, the voice analysis and translation system 100 can include Application Programming Interfaces (APIs) that can be called from each of the plurality of communication platforms 130. In an example, the APIs of the voice analysis and translation system 100 may be commonly implemented across the plurality of communication platforms 130, but the APIs can be accessed by different platforms in different ways via the platform-specific pipelines. Processing the metadata allows the voice analysis and translation system 100 to store key aspects of the communication session or call such as the communication platform, the participants, the start time, the end time, joining time for each participant, the audio channel, and the audio format of the communication platform, etc.
When a listener 114 logs into a communication session of the communication platform 1, the listener 114 may select the voice analysis and translation system 100 to translate the audio of the communication session via a selection of the target language. When the listener activates the voice analysis and translation system 100 and selects the target language, the audio input 150 from the communication session may be received by the voice analyzer and translator 104 through the pipeline 21, coupling the communication platform 1 to the voice analyzer and translator 104. In an example, the audio input 150 may also include metadata related to the communication session. By way of illustration and not limitation, the metadata extracted from the audio input 150 can be used to select from third-party cognitive services, one of a plurality of transcription engines 180 and one of a plurality of translation engines 160 for translating the audio input 150 into the target language requested by the listener. The audio output 152 including the translated version is provided to the listener via the corresponding communication application e.g., the communication app 1.
The voice analysis and translation system 100 can thus, receive the audio input 150 in one language and translate it into a different language selected by a listener 114 thereby facilitating multilingual communications between different users of a communication platform. By way of illustration and not limitation, a communication session can include a teleconference on a telephonic meeting between two or more individuals, speaking different languages. The communication session may also include an agent speaking with the client/customer via the client's telephone via one of the plurality of communication platforms 130 and speaking with the client/customer one language, which may be translated for the client/customer into another language while the audio input provided by the client/customer translated for the agent into the agent's preferred language. The voice analysis and translation system 100 can include or can be communicatively coupled to a data storage 170 that stores data generated or required during the audio translation and transmission process.
In an example, the audio stream(s) may be received from one of the plurality of communication apps 132 e.g., communication app 1, via a single channel. If the audio stream has multiple speakers and is received on a single channel, then the output of the single channel is provided to the diarizer 224 that splits the audio stream into multiple channels, wherein each speaker's voice output i.e., each audio stream is provided on one channel. On the other hand, if it is determined that the audio streams of different speakers are received in separate channels, then the audio input 150 can be provided by the audio stream analyzer 222 directly to the transcoder 226 without being processed by the diarizer 224.
Furthermore, if the metadata of the audio input 150 indicates that the audio streams are received in formats incompatible with one or more of the Al-based audio processor 204, the transcoder 226 converts the audio streams into acceptable audio formats that can be handled by the plurality of translation engines 160 and the plurality of transcription engines 180. Again, processing of the audio input 150 by the transcoder is optional in that if the audio input 150 is received in formats compatible with the plurality of translation engines 160, then no format transformation is needed. For example, the audio input 150 can be received in Wave audio (.wav), .ogg, oflac, .mp3, etc., formats. The output from the transcoder 206 can include .way or .mp3 formats. Compact audio formats such as but not limited to .mp3 formats are preferred over lossless formats such as .way or Free Lossless Audio Codec (.flac) for optimizing processing resources and reducing network latency. Well-tuned transcoding libraries can be used with options provided for optimizing output quality and noise filtration. In the instances where the audio input 150 is received in formats that are incompatible with the voice analyzer and translator 104, the audio input 150 may be processed by the transcoder 226 before being diarized. The output 228 from the homogenizer 202 may include metadata in standard formats which can include information about the speakers, speaker-specific stream details, and the audio streams in compatible formats.
The AI-based audio processor 204 independently processes the audio input 150 which may have been transformed or converted into a compatible format by the transcoder 226. The AI-based audio processor 204 can be configured for detecting the speech context and based at least on the speech context, selecting one of the plurality of translation engines 160 for translating the audio input 150. The plurality of translation engines 160 may include but are not limited to Google® Translate, Azure®, Amazon Web Services (AWS®), SDL Language Cloud®, etc. Similarly, the plurality of transcription engines 180 may include but are not limited to AWS®, Speechmatics®, Azure®, etc. Additionally, the AI-based audio processor 204 can be configured for detecting and correcting sentiments. Therefore, the translated audio output 152 can be configured to convey a ‘sentiment’.
An as-is transcript generated by the selected translation engine from the plurality of translation engines 160 may contain noise, sentences split across multiple lines, etc. While it may represent the communication session as it happened, in many cases, this transcript cannot be used as-is. The summarization process implemented by the summary generator 206 identifies and removes noise, merges multi-line dialogues, and attaches sentiments. The final summary can include the summary of the conversation, sentiment, and the details of the speakers. In an example, the final summary can be provided in a textual format as a call transcript. In an example, the final summary can be provided to a text-to-speech engine so that an audio summary of the communication session with the sentiment data is automatically produced.
A model trainer 208 is also included in the voice analyzer and translator 104 to provide feedback. Upon providing the audio output 152 to the participants/listeners, the voice analysis and translation system 100 can collect their feedback. The feedback can be sought in different aspects including but not limited to the accuracy of the translation, the quality of sound, the domain terminology used in the translation, the sentiments expressed in the audio output 152, etc. Based on the received feedback, the model trainer 208 can further train one or more of the ML classification models used for speech context determination.
In an example, trained machine learning (ML) classification models 320 can be employed by each of the language detectors 322-1, . . . , 322-x, the domain detector 326, the gender detectors 324-1-1, . . . , 324-x-2, and the sentiment detector 328-1-1, . . . 328-x-2, included in the context detector 302. A subset of the ML classification models 320 e.g., language detector 322-1, . . . language detector 322-x, can be trained via supervised or unsupervised techniques for identifying x different languages. Each time the voice analysis and translation system 100 is configured to identify/process a new source language the corresponding language detector CNN can be trained. Therefore, multiple CNNs are used for language detection. The voice analysis and translation system 100 can thus be expanded to identify and translate any number of languages provided the corresponding translation engines are available. The source language of the audio input 150 can be identified from the output of one of the language detectors 322-1, . . . , 322-x, with maximum confidence. The language detectors 322-1, . . . , 322-x, can employ language-specific features for language detection. Each language has certain peculiarities in representing text while it is been spoken or written. For example, in Japanese, when addressing a person respectfully/formally by a name then the title “San” is added at the end. The language context helps in refining the translated text after completing the translation.
The detected language is reused so that two gender detectors e.g., 324-1-1 and 324-1-2, . . . , 324-x-1, 324-x-2 are trained for each language. Based on gender and language, the appropriate CNN model is chosen for sentiment analysis. Therefore, the CNN used for sentiment detection is trained to be gender-specific and language-specific. Accordingly, each of the two gender detectors 324-1-1 and 324-1-2 . . . 324-x-1, 324-x-2, trained for each language detector 322-1, . . . , 322-x, also have corresponding pair of gender-specific sentiment detectors 328-1-1, 328-1-2 and 328-x-1, 328-x-2. The output of the sentiment detectors 328-1-1, 328-1-2, and 328-x-1, 328-x-2 is further provided to the sentiment analyzer 306 after translation of the input audio 150 for sentiment correction of the output transcript generated by the selected translation engine. One or more ML domain classification models can be used in the domain detector 326 for the identification of different domains. The ML domain classification models may be trained via supervised and unsupervised techniques as described above. However, in addition to the trained ML models, the domain detector 326 can also employ a plurality of domain glossaries 350 stored for example, in the data storage 170, to identify particular domains based on the meaning of specific terminology. For example, ‘fine’ in the healthcare domain would mean that everything is ok whereas ‘fine’ in the taxation or finance domain would mean something different. In an example, the plurality of domain glossaries 350 for various domains in different languages may be used so that when the language detector 322 identifies the source language, a subset of the plurality of domain glossaries 350 of the source language is selected for domain identification by the ML domain classification model.
The speech context 334 output by the context detector 302 is provided to the engine selector 304 that selects one of the plurality of translation engines 160 and one of the plurality of transcription engines 180 for translating/transcribing the audio from the audio input 150. The particular transcription engine selected by the engine selector provides a textual output of the audio input 150 in the source language. The textual output in the source language can be provided to the particular translation engine selected by the engine selector 304. The selected translation engine provides a textual output in the form of an output transcript including a translated version of the transcript generated for the audio input 150. The output transcript is provided to the sentiment analyzer 306 for sentiment correction which produces a modified output transcript. In an example, the sentiment output by the gender-specific sentiment detectors 328-1-1, 326-8-1-2, and 328-x-1, 328-x-2 from the audio input 150 in the source language can be applied to the output transcript in a sentiment correction process. The modified output transcript including the correct sentiment(s) can be provided to the post-translation processor 308 for producing the audio output 152 in the target language using text-to-speech conversion.
The translated textual output is provided to the post-translation processor 308 for post-translation operations such as grammar and spelling correction, sentiment correction, etc. In an example, the post-translation processor 308 can also include a text-to-speech engine (not shown) that is used to produce the audio output 152 in the target language from the translated textual output received from the sentiment analyzer 306. In an example, the functions of the auto summarizer 206 to generate communication session call summary can also be implemented by the post-translation processor 308. The textual summary generated by the auto-summarizer 206 can also be provided to a text-to-speech engine to generate an audio summary of the communication session.
In an example, the audio input sampler 330 can include a timer 332 that may be programmatically manipulated. So long as the voice analysis and translation system 100 operates to translate the streams from the audio input 150, the timer 332 is run for a predetermined time and is reset on the expiration of the predetermined time. When the timer 332 expires, the audio input sampler 330 collects the audio streams from the audio input 150 for analysis by the various detectors in the context detector 302. Thus, any changes in speech context 334 including changes of speakers, changes in the domain, sentiment, or even language can be detected. Changes in the speech context 334 can also include an outage of the selected translation engine necessitating the selection of another translation engine. Each time a change in the speech context 334 is detected, it can trigger the engine selector 304 to execute the engine selection process to that the plurality of translation engines 160, the plurality of transcription engines 180 are re-evaluated to determine the optimal engine for the altered speech context. When the optimal translation engine is located for the new speech context, the audio output 152 can be obtained from the newly-selected optimal engine(s). Therefore, the translated output produced by the voice analysis and translation system 100 is superior to the translated output produced by a translation process wherein a translation service is selected once at the commencement of a voice session and used until the conclusion of the session. This is because reevaluation and reselection of the translation engines with changes in the speech context 334 enables the voice analysis and translation system 100 to capture nuances that are otherwise missed by the translation process where the same translation engine is used throughout the session.
When a new engine is introduced or during the initial setup, the Q-table analyzer 404 is expected to be in its training stage. At this point, input from language experts and Subject Matter Experts (SMEs) can be utilized to evaluate each of the plurality of translation engines at random and provide accurate feedback before handing over the new engines to the users. Accordingly, the engine activator 406 can be initially made to pick a translation engine from the plurality of translation engines 160 randomly based on the language pair. The values for the selected engine can be updated by the Q-table analyzer 404. When the feedback is collected in response to the provided audio output 152, the Q-table analyzer 404 can further update the Q-table 402 based on the received feedback. As the number of iterations increases, the Q-table analyzer 404 will be able to compute a more accurate Q-value based on the feedback from the users and hence improve in its predictions of the best engine for each speech context and source/target language pair.
Based on the extracted metadata, e.g., the speaker-specific stream details it can be determined at 508 if the audio stream(s) received in the audio input 150 includes multiple speakers hi a single channel. For example, some communication platforms may include multiple speakers in a single channel, especially for recordings of voice communications, while some communication platforms transmit multiple speakers on multiple channels. If it is determined at 508 that multiple speakers or multiple audio streams are transmitted in a single channel, the method moves to 510 wherein the AI-based diarization process is executed to separate the multiple streams into separate channels so that one speaker is transmitted on one channel. Thus, the quality of the audio provided to the translation engine is improved thereby translated output of higher quality is obtained. If it is determined at 508 that multiple speakers are on multiple channels the method moves to 512 to process the audio input 150 and to generate a transcription of the audio input 150 in the source language. The transcription can be generated by one of the plurality of transcription engines 180 which may be selected based on the metadata 250. The processing of the audio input 150 can include transcoding the audio Input 150 into audio formats that are compatible with one or more of the plurality of translation engines 160.
The text output from the transcription is used for identifying the speech context 334 at 514. Different ML classification models 320 are used for identification of the speech context 334. In an example, the context detector 302 implements stream-wise processing of the audio input 150 for the detection of speech context 334. Based on the speech context 334, one of the plurality of translation engines 160 is selected at 516 for translating the audio input 150. The audio output 152 translated into the target language is provided at 518. Throughout the communication session, the audio streams being input to the voice analysis and translation system 100 are monitored at 520 and it is determined if there is a change in the speech context 334 at 522. If it is detected at 522 that there is a change in the speech context 334, the method moves to 524 to determine if the change in speech context includes a conclusion of the communication session. If yes, the method terminates on the end block. If it is determined at 524 that the communication session has not concluded then the method moves to 514 to identify the new speech context. If no change in the speech context is detected at 522, the monitoring of the communication session continues at 520 and the process can be repeated.
The translated output text may accurately represent the meaning of the speech, but may not reflect the sentiment or emotion of the speaker. The addition of emotions makes the translated audio output more realistic. Pre-trained advanced natural language processing (NLP) models can be used by the sentiment analyzer 306 for text transformation. These models provide greater accuracy and speed if trained for specific languages. Hence based on the output translated language, the NLP model can be chosen. The NLP model can be derived from the text2text generation models from the transformers library. The NLP model can be tuned using custom datasets and a tokenizer. Since this is a translation operation, the NLP model will need to be fine-tuned for each language using language-specific data sets. Based on the sentiment of the speaker, the NLP model can generate sentences that are snore suitable for the speaker's sentiment or emotion. The transformed text will be fed to the Text-to-Speech engine of the post-translation processor 308 for the audio output.
The computer system 1100 includes processor(s) 1102, such as a central processing unit, ASIC or another type of processing circuit, input/output devices 1110. such as a display, mouse keyboard, etc., a network interface 1104, such as a Local Area Network (LAN), a wireless 802.11x LAN, a 3G, 4G or 5G mobile WAN or a WiMax WAN, and a processor-readable medium 1106. Each of these components may be operatively coupled to a bus 1108. The processor-readable or computer-readable medium 1106 may be any suitable medium that participates in providing instructions to the processor(s) 1102 for execution. For example, the processor-readable medium 1106 may be a non-transitory or non-volatile medium, such as a magnetic disk or solid-state non-volatile memory, or a volatile medium such as RAM. The instructions or modules stored on the processor-readable medium 1106 may include machine-readable instructions 1164 executed by the processor(s) 1102 that cause the processor(s) 1102 to perform the methods and functions of the voice analysis and translation system 100.
The voice analysis and translation system 100 may be implemented as software or machine-readable instructions stored on a non-transitory processor-readable medium and executed by one or more processors 1102. For example, the processor-readable medium 1106 may store an operating system 1162, such as MAC OS, MS WINDOWS, UNIX, or LINUX, and code 1164 for the voice analysis and translation system 100. The operating system 1162 may be multi-user, multiprocessing, multitasking, multithreading, real-time, and the like. For example, during runtime, the operating system 1162 is running and the code for the voice analysis and translation system 100 is executed by the processor(s) 1102.
The computer system 1100 may include a data storage 1110, which may include non-volatile data storage. The data storage 1110 stores any data used by the voice analysis and translation system 100. The data storage 1110 may be used as the data storage 170 to store the audio input 150, the speech context 334 including various attributes of the audio input 150, and other data elements which are generated and/or used during the operation of the voice analysis and translation system 100.
The network interface 1104 connects the computer system 1100 to internal systems for example, via a LAN. Also, the network interface 1104 may connect the computer system 1100 to the Internet. For example, the computer system 1100 may connect to web browsers and other external applications and systems via the network interface 1104.
What has been described and illustrated herein is an example along with some of its variations. The terms, descriptions, and figures used herein are set forth by way of illustration only and are not meant as limitations. Many variations are possible within the spirit and scope of the subject matter, which is intended to be defined by the following claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202211021330 | Apr 2022 | IN | national |
