Patent Application
20240371369
Applying standard Natural Language Processing (NLP) techniques that were developed for written language to transcripts of spoken conversations yields worse than expected results. Unlike written text, which is composed of complete and (for the most part) grammatically correct sentences, conversational transcripts are composed of utterances (the smallest unit of speech; continuous, beginning, and ending with a break). Such utterances are often incomplete statements and may not be grammatically correct on their own.
Aspects of the present disclosure inject the context of what an utterance is in response to in a conversation for improving NLP applied to transcripts of spoken conversations. Applying Machine Learning (ML) methods directly to these smaller units of text is the source of the problem that the subject matter of the present disclosure ameliorates.
In an aspect, a computer-implemented method for training a machine learning model comprises receiving an audio signal representing a spoken sequence of utterances, identifying one of the utterances as a target utterance to be transcribed, and defining an utterance pair for processing. The utterance pair, which comprises the target utterance paired with another one of the utterances, is processed with the machine-learned model to generate a prediction of the target utterance. The method also includes evaluating a loss function that compares the prediction to a ground truth value, and modifying one or more values of at least one parameter of the machine-learned model based at least in part on the evaluated loss function.
In another aspect, a computing system classifies a target utterance from an audio signal representing a spoken sequence of utterances. The computing system comprises one or more processors and one or more non-transitory computer-readable media storing a machine-learned model and computer-executable instructions. When executed by the one or more processors, the instructions cause the computing system to perform operations, which include receiving the audio signal representing the spoken sequence of utterances, identifying one of the utterances as the target utterance to be classified, and defining an utterance pair for processing. The utterance pair, which comprises the target utterance paired with another one of the utterances, is processed with the machine-learned model to generate a prediction of the target utterance that is then provided as an output.
In yet another aspect, a machine-learned transcription system comprises a telephony services receiving a spoken conversation between at least two persons and configured for recording the conversation. The recorded conversation comprises a spoken sequence of utterances. A speech to text service of the system is configured for transcribing the spoken sequence of utterances. The system further comprises a data builder service configured for identifying one of the transcribed utterances as a target utterance to be classified and defining an utterance pair for processing. The utterance pair comprises the target utterance paired with another one of the transcribed utterances. A machine learning engine of the system is configured for processing the utterance pair with a machine-learned model to generate a prediction of the target utterance and providing the prediction as an output.
Other objects and features of the present invention will be in part apparent and in part pointed out herein.
Corresponding reference characters indicate corresponding parts throughout the drawings.
The features and other details of the concepts, systems, and techniques sought to be protected herein will now be more particularly described. It will be understood that any specific embodiments described herein are shown by way of illustration and not as limitations of the disclosure and the concepts described herein. Features of the subject matter described herein can be employed in various embodiments without departing from the scope of the concepts sought to be protected.
Aspects of the present disclosure inject the context of what an utterance is in response to in a conversation. For example, without context, a single utterance such as “okay” can have any number of meanings and interpretations, which degrades the accuracy of predictions from a machine learning model. By pairing it with the preceding utterance, the context of what it is in response to is added. By pairing it again with the following utterance, the context of what the statement prompts is also added. As a result, each utterance appears twice in the data sequence thus effectively doubling the amount of training data, which improves model training performance.
The process begins with a recording of a spoken conversation which is then transcribed into text using speech to text (STT) algorithms. The transcription process identifies the various speakers taking part in the conversation and segments the text into utterances labeled by the individual speaking. Training data for ML models is then constructed by creating alternating pairs of prompt and response utterances.
Referring now to
The process 200 reads a leading utterance at 208 and uses it as the Utterance in a new Prompt Response pair. The process 200 then reads a trailing utterance from the alternate speaker and adds it as the response to the current Prompt-Response pair at 210. Pre-processing of utterances may include: Discarding utterances having low confidence scores indicating potential inaccuracies in transcription, as well as discarding utterances where the individual speaker could not be identified.
At 212, the completed Prompt-Response pair is emitted to the output data sequence. In an embodiment, complete prompt-response pairs in the output data sequence are transformed into vector representations using sentence-embedding models such as Bidirectional Encoder Representations from Transformers (BERT). Sentence embedding models are trained to learn contextual representations of words and sentences by predicting masked words and the relationship of sentence sequences.
At 214, the text of the transcript is checked to see if there are any remaining utterances to process. If so, a new Prompt-Response pair is created at 216 and process 200 returns to 210 for loading the next utterance. In this instance, the previous response utterance is used as the prompt in the next data pair. The process proceeds to 212 using the new data pair.
As shown, process 200 ends at 218 when the input transcript text does not have any utterances left to process, as indicated at 214.
Each utterance is used once as the prompt in a pair and as a response in another, creating the following data sequence:
As an example, the process 200 produces a sequence of prompt-response pairs from following transcript:
The sequence of prompt-response pairs includes:
Embodiments of the present disclosure may comprise a special purpose computer including a variety of computer hardware, as described in greater detail herein.
For purposes of illustration, programs and other executable program components may be shown as discrete blocks. It is recognized, however, that such programs and components reside at various times in different storage components of a computing device, and are executed by a data processor(s) of the device.
Although described in connection with an example computing system environment, embodiments of the aspects of the invention are operational with other special purpose computing system environments or configurations. The computing system environment is not intended to suggest any limitation as to the scope of use or functionality of any aspect of the invention. Moreover, the computing system environment should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the example operating environment. Examples of computing systems, environments, and/or configurations that may be suitable for use with aspects of the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
Embodiments of the aspects of the present disclosure may be described in the general context of data and/or processor-executable instructions, such as program modules, stored one or more tangible, non-transitory storage media and executed by one or more processors or other devices. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types. Aspects of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote storage media including memory storage devices.
In operation, processors, computers and/or servers may execute the processor-executable instructions (e.g., software, firmware, and/or hardware) such as those illustrated herein to implement aspects of the invention.
Embodiments may be implemented with processor-executable instructions. The processor-executable instructions may be organized into one or more processor-executable components or modules on a tangible processor readable storage medium. Also, embodiments may be implemented with any number and organization of such components or modules. For example, aspects of the present disclosure are not limited to the specific processor-executable instructions or the specific components or modules illustrated in the figures and described herein. Other embodiments may include different processor-executable instructions or components having more or less functionality than illustrated and described herein.
The order of execution or performance of the operations in accordance with aspects of the present disclosure illustrated and described herein is not essential, unless otherwise specified. That is, the operations may be performed in any order, unless otherwise specified, and embodiments may include additional or fewer operations than those disclosed herein. For example, it is contemplated that executing or performing a particular operation before, contemporaneously with, or after another operation is within the scope of the invention.
When introducing elements of the invention or embodiments thereof, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements.
Not all of the depicted components illustrated or described may be required. In addition, some implementations and embodiments may include additional components. Variations in the arrangement and type of the components may be made without departing from the spirit or scope of the claims as set forth herein. Additional, different or fewer components may be provided and components may be combined. Alternatively, or in addition, a component may be implemented by several components.
The above description illustrates embodiments by way of example and not by way of limitation. This description enables one skilled in the art to make and use aspects of the invention, and describes several embodiments, adaptations, variations, alternatives and uses of the aspects of the invention, including what is presently believed to be the best mode of carrying out the aspects of the invention. Additionally, it is to be understood that the aspects of the invention are not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The aspects of the invention are capable of other embodiments and of being practiced or carried out in various ways. Also, it will be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting.
It will be apparent that modifications and variations are possible without departing from the scope of the invention defined in the appended claims. As various changes could be made in the above constructions and methods without departing from the scope of the invention, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.
In view of the above, it will be seen that several advantages of the aspects of the invention are achieved and other advantageous results attained.
The Abstract and Summary are provided to help the reader quickly ascertain the nature of the technical disclosure. They are submitted with the understanding that they will not be used to interpret or limit the scope or meaning of the claims. The Summary is provided to introduce a selection of concepts in simplified form that are further described in the Detailed Description. The Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the claimed subject matter.
This application claims the benefit of U.S. Provisional Patent Application No. 63/463,718, filed May 3, 2023, the entire disclosure of which is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 63463718 | May 2023 | US |
