Patent Application
20230326450
In speech recognition, uncommon words are less likely to be recognized correctly. It may be more important to customers that these uncommon words, especially first and last names, company names, domain-specific terminology, are recognized correctly. Therefore, it may be desirable to include the ability to include custom vocabulary in the recognition of speech.
In one embodiment, a method of adding a custom vocabulary to a transcription system includes receiving a custom vocabulary at an ASIRW module. The method further includes tokenizing the custom vocabulary with the ASIRW module. The method further includes creating a new WFST (weighted finite-state transducer) with the ASIRW module. The method further includes transcribing audio using the new WFST with the ASIRW module. In one alternative, the tokenizing includes performing a translation model on each word of the custom vocabulary. In another alternative, the custom vocabulary includes phrases. Alternatively, the tokenizing includes creating predicted tokenizations for alternate spellings of the custom vocabulary. In another alternative, the new WFST includes tokenizations for every word in a lexicon, plus tokenizations for the custom vocabulary, plus the predicted tokenizations for the alternate spellings of the custom vocabulary. Alternatively, the creating the new WFST includes running CTC (connectionist temporal classification) decoding.
In one embodiment, a system for adding a custom vocabulary to a transcription system includes a ASIRW module executing code and configured to: receive a custom vocabulary at the ASIRW module. The ASIRW module is further configured to tokenize the custom vocabulary with the ASIRW module. The ASIRW module is further configured to create a new WFST (weighted finite-state transducer) with the ASIRW module. The ASIRW module is further configured to transcribe audio using the new WFST with the ASIRW module. In one alternative, the tokenizing includes performing a translation model on each word of the custom vocabulary. In another alternative, the custom vocabulary includes phrases. Alternatively, the tokenizing includes creating predicted tokenizations for alternate spellings of the custom vocabulary. In another alternative, the new WFST includes tokenizations for every word in a lexicon, plus tokenizations for the custom vocabulary, plus the predicted tokenizations for the alternate spellings of the custom vocabulary. Alternatively, the creating the new WFST includes running CTC (connectionist temporal classification) decoding.
In one embodiment, a system for adding a custom vocabulary to a transcription system includes a ASIRW module executing code and configured to: receive a custom vocabulary at the ASIRW module. The ASIRW module is further configured to tokenize the custom vocabulary with the ASIRW module including performing a translation model on each word of the custom vocabulary. The ASIRW module is further configured to create a new WFST (weighted finite-state transducer) with the ASIRW module. The new WFST includes tokenizations for every word in a lexicon, plus tokenizations for the custom vocabulary, plus the predicted tokenizations for the alternate spellings of the custom vocabulary. The ASIRW module is further configured to transcribe audio using the new WFST with the ASIRW module.
In one embodiment, a non-transitory digital storage medium having a computer program stored thereon to perform a method of adding a custom vocabulary to a transcription system, includes receiving a custom vocabulary at an ASIRW module. The method further includes tokenizing the custom vocabulary with the ASIRW module. The method further includes creating a new WFST (weighted finite-state transducer) with the ASIRW module. The method further includes transcribing audio using the new WFST with the ASIRW module.
In one embodiment, a method of alternative spelling training includes providing a ground-truth text and providing ASR-generated text. The method further includes aligning the ground-truth text and the ASR-generated text with a text alignment tool. The method further includes generating error pairs from the aligning. The method further includes filtering the error pairs according to uncommon words to create filtered error pairs. The method further includes adding the filtered error pairs to the translation model training along with the previously existing training configuration files, to create a trained translation model. The method further includes using the trained translation model to generate a transcript from an audio signal. Alternatively, the ground-truth text results resulting from a live user creating a transcription for the audio signal including words. In another alternative, the ASR-generated text results from automatic speech recognition.
In another embodiment, a system of alternative spelling training includes an alternative spelling training module executing code and configured to: receive a ground-truth text and receive ASR-generated text. The alternative spelling training module further aligns the ground-truth text and the ASR-generated text with a text alignment tool. The alternative spelling training module further generates error pairs from the aligning. The alternative spelling training module further filters the error pairs according to uncommon words to create filtered error pairs. The alternative spelling training module further adds the filtered error pairs to the translation model training along with the previously existing training configuration files, to create a trained translation model. The alternative spelling training module further generates a transcript from an audio signal using the trained translation model.
Certain terminology is used herein for convenience only and is not to be taken as a limitation on the embodiments of the systems and methods for employing alternate spellings for improving the recognition of rare words (herein after “ASIRW”). In one embodiment of an ASIRW, a speech recognition portion may include a “Custom Vocabulary” (CV), which allows a customer/user to submit a list of words which are likely to appear in an audio. The ASIRW may perform a translation model on each CV term to obtain a new tokenization. The system may build a new WFST, which includes the tokenizations for every word in the lexicon, plus the tokenizations for all the CV terms, plus the predicted tokenizations for the alternate spellings of the CV terms. Subsequently, the system runs with CTC (connectionist temporal classification) decoding and the new WFST (weighted finite-state transducer). In another embodiment, based on a custom vocabulary including user submitted words/phrases, the ASIRW builds identifying features for all CV and alternative spellings for all CV. Then the ASIRW recognizes speech using the identifying features of the CV and determines the greatest likelihood of recognized speech to corresponding transcribed text. In some alternatives, the ASIRW performs a tokenization of the CV. In another alternative, alternate spellings of the CV is created. In another alternative, the ASIRW uses a weighted finite-state transducer (WFST). The WFST includes language modeling information, including mappings from word pieces to full words. One objective of such systems is to create transcripts of spoken words from an audio file or a stream of media that are accurate and highly reflective of the actual speech.
In many embodiments, when referring to “Custom Vocabulary” (CV), any method of adding custom vocabulary words is included. The most common way may be user added words. These words may be out of the vocabulary of the automatic speech recognition (ASR) model and/or may sound like a common word but have a unique spelling. In many embodiments, users may use CV if the audio contains truly unique or rare terms that they believe not to be in the standard dictionary, which in many embodiments is a massive dictionary of 500k+ words. Examples of good custom vocabulary terms include made-up words or words with unique spelling such as: sparkletini, timi, Ginnifer. In some configurations, examples of bad custom vocabulary terms that should not include are: Maybelline, e pluribus unum, orthostatic hypotension. In some alternatives, CV may be automatically loaded according to the expected discipline of use. For instance, the system may automatically load medical terminology in some scenarios, where medical personnel are likely to be using the system. It may be advantageous not to include words that may be confused with other words unless those words are likely to be used.
In one embodiment, an ASR model predicts pieces of text in units one to eight characters long called “word pieces”. The ASR model has multiple modes of putting together the pieces of text predicted from an audio: connectionist temporal classification (CTC) decoding, CTC decoding guided with a weighted finite-state transducer (WFST), or WFSTs alone. IN many scenarios, WFST decoding is preferred. The WFST includes language modeling information, including mappings from word pieces to full words. This method of improving recognition of customer-specified words involves adding new mappings from word pieces to the relevant customer-specified words to the WFST.
In many embodiments, training the alternative spellings model is an important aspect of the ASIRW. In many configurations, a first step is generating training data. In many embodiments of this aspect, segments of ASR training data that likely to contain errors by taking utterances from the training set which contain at least one word appearing fewer than n times in our ASR training data are identified. In many configurations, WFST decoding is used. In some alternatives, ASR model is run using connectionist temporal classification (CTC) decoding on the audio segments from this subset of the training data.
In many embodiments, the data is additionally “cleaned.” As part of this process, FSTAlign, an open-source tool for aligning sequences of text, the system aligns the ASR output text from step 1 with the corresponding human-transcribed reference text. FSTAlign identifies groups of contiguous errors (includes insertions, deletions, substitutions). An example of such a system may be found in U. S. Patent Application No. 17/069462, entitled Systems and Methods for Aligning a Reference Sequence of Symbols With Hypothesis Requiring Reduced Processing and Memory, which is hereby incorporated by reference in its entirety. Next, the system extracts the error groups which involve substitution errors. The system identifies the most common X words in our data. The system removes all error pairs which involve any of the most common X words on the reference side.
In many embodiments, the ASIRW additionally includes a training step. The ASIRW trains a transformer-based encoder-decoder translation model on error pairs, using the reference text tokenized into characters as the input, and the ASR output text tokenized with the ASR model’s tokenizer as output.
Once the setup phase is complete, the ASIRW may run speech recognition using the alternate spellings. As part of this process, when the user or customer submits a CV list, the ASIRW runs the translation model on each CV term to obtain a new tokenization. Then the system adds alternative spellings to the WFST. Then the ASIRW builds a new WFST, which includes the tokenizations for every word in the lexicon, plus the tokenizations for all the CV terms, plus the predicted tokenizations for the alternate spellings of the CV terms. The ASIRW runs ASR with this new WFST and in alternatives CTC.
In many embodiments, parts of the system are provided in devices including microprocessors. Various embodiments of the systems and methods described herein may be implemented fully or partially in software and/or firmware. This software and/or firmware may take the form of instructions contained in or on a non-transitory computer-readable storage medium. Those instructions then may be read and executed by one or more processors to enable performance of the operations described herein. The instructions may be in any suitable form such as, but not limited to, source code, compiled code, interpreted code, executable code, static code, dynamic code, and the like. Such a computer-readable medium may include any tangible non-transitory medium for storing information in a form readable by one or more computers such as, but not limited to, read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; a flash memory, etc.
Embodiments of the systems and methods described herein may be implemented in a variety of systems including, but not limited to, smartphones, tablets, laptops, and combinations of computing devices and cloud computing resources. For instance, portions of the operations may occur in one device, and other operations may occur at a remote location, such as a remote server or servers. For instance, the collection of the data may occur at a smartphone, and the data analysis may occur at a server or in a cloud computing resource. Any single computing device or combination of computing devices may execute the methods described.
In various instances, parts of the method may be implemented in modules, subroutines, or other computing structures. In many embodiments, the method and software embodying the method may be recorded on a fixed tangible medium.
While specific embodiments have been described in detail in the foregoing detailed description, it will be appreciated by those skilled in the art that various modifications and alternatives to those details could be developed in light of the overall teachings of the disclosure and the broad inventive concepts thereof. It is understood, therefore, that the scope of this disclosure is not limited to the particular examples and implementations disclosed herein but is intended to cover modifications within the spirit and scope thereof as defined by the appended claims and any and all equivalents thereof.
