Patent Application
20230252994
Most modern automatic speech recognition (ASR) systems are capable of high quality performance under ideal conditions, such as low background noise and clearly enunciated, unaccented human speech. However, interpretive errors can occur when ASR systems are used in noisy environments, or when speech is indistinctly uttered due to age or disability, is heavily accented, is of a different vernacular, or includes linguistic mistakes. Moreover, the reliability of ASR systems can also rely on the corpus of words used to train the system. Consequently, an ASR system trained on a corpus of common language terms may be unreliable when used to transcribe speech that is unique to or characteristic of a specialized language domain, such as medical or other scientific terminology, for example. Consequently, there remains a need in the art for a natural language understanding (NLU) solution capable of performing domain and user intent specific disambiguation of transcribed speech.
The following description contains specific information pertaining to implementations in the present disclosure. One skilled in the art will recognize that the present disclosure be implemented in a manner different from that specifically discussed herein. The drawings in the present application and their accompanying detailed description are directed to merely exemplary implementations. Unless noted otherwise, like or corresponding elements among the figures may be indicated by like or corresponding reference numerals.
As stated above, although most modern automatic speech recognition (ASR) systems are capable of high quality performance under ideal conditions, such as low background noise and clearly enunciated, unaccented human speech, interpretive errors can occur when ASR systems are used in noisy environments, or when speech is indistinctly uttered due to age or disability, is heavily accented, is of a different vernacular, or includes linguistic mistakes. Moreover and as further stated above, the reliability of ASR systems can also rely on the corpus of words used to train the system. Consequently, an ASR system trained on a corpus of common language terms may be unreliable when used to transcribe speech that is unique to or characteristic of a specialized language domain, such as medical or other scientific terminology, by way of example. Consequently, there remains a need in the art for a natural language understanding (NLU) solution capable of performing domain and user intent specific disambiguation of transcribed speech.
The present application is directed to automated systems and methods for performing domain and user intent specific disambiguation of transcribed speech. The disambiguation solution disclosed in the present application reverses the language that is output in transcript form by a conventional ASR system in real-time, while retaining the phonetic hints and context from the transcript, and then superimposes domain and application specific language with the context to disambiguate and correct the transcribed text for a specific domain. The solution disclosed in the present application results in significant improvements in disambiguating domain specific language from common language, thereby advantageously providing improved understanding in situations where audio quality is poor, speech is uttered by non-native speakers, children, the aged, or those with speech disorders, or where confusion can arise due to the unique or distinctive requirements of a particular domain specific vocabulary.
It is noted that the disambiguation solution disclosed herein has a wide variety of practical applications (hereinafter also “use cases”) in which a user issues voice commands. Such use cases can include browsing for or selecting media content in the form of movies, television (TV) programming content, music, video games, and the like, for example. Other examples of use cases for the present novel and inventive concepts include retail shopping, food ordering, obtaining travel directions, and communicating with a doctor or first responder, to name merely a few.
As defined in the present application, the term “intent” refers to a goal oriented psychological state of a human user. Examples of the types of goals determining intent include the acquisition of information, acquisition of a product or service, arrival at a particular destination, or viewing or participating in an event, to name a few. Furthermore, because it is not possible to have definite knowledge of a human user's inner mental state, as used in the present application the term “intent” is to be interpreted as “inferred intent.” Thus, as used herein, the “intent of the user” refers to the “inferred intent of the user.”
In addition, as defined in the present application, the term “domain” refers to an area of specialization such as medicine, the sciences, specific types of entertainment, sports, and specialized pastimes, for example. Thus, the expression “domain-specific language” refers to language that includes words that are unique or characteristic to a particular domain. By contrast “common language” refers to the words and their meanings included in a generic or general purpose dictionary.
In addition, as defined in the present application, the terms “automation,” “automated,” and “automating” refer to systems and processes that do not require human intervention. Although in some implementations a human editor or system administrator may to review the “best transcription” of speech by a user identified by the systems and using the methods described herein, that human involvement is optional. Thus, the methods described in the present application may be performed under the control of hardware processing components of the disclosed automated systems.
Moreover, as defined in the present application, the expression “machine learning model” or “ML model” may refer to a mathematical model for raking future predictions based on patterns learned from samples of data or “training data.” Various learning algorithms can be used to map correlations between input data and output data. These correlations form the mathematical model that can be used to make future predictions on new input data. Such a predictive model may include one or more logistic regression models, Bayesian models, or neural networks (NNs). Moreover, a “deep neural network,” in the context of deep learning, may refer to an NN that utilizes multiple hidden layers between input and output layers, which may allow for learning based on features not explicitly defined in raw data. As used in the present application, a feature identified as an NN refers to a deep neural network. In various implementations, ML models may be trained as classifiers and may be utilized to perform image processing, audio processing, or natural-language processing.
It is noted that, as shown by
As further shown in
It is noted that, in some implementations, disambiguation resource(s) 140 including trained ML model(s) 150 may be incorporated into software code 110 as features of software code 110. That is to say, in some implementations, any or all of the functionality attributed to disambiguation resource(s) 140 by the present disclosure may be attributed to software code 110 alone. It is further noted that although in some implementations, ASR system 134 may be a third party resource and not a feature of system 100 in other implementations, ASR system 134 may be part of system 100, and may be stored in system memory 106.
It is also noted that, in some implementations, as shown in
Although the present application refers to software code 110, user history database 120, and disambiguation resource(s) 140 as being stored in system memory 106 for conceptual clarity, more generally, system memory 106 may take the form of any computer-readable non-transitory storage medium. The expression “computer-readable non-transitory storage medium,” as used in the present application, refers to any medium, excluding a carrier wave or other transitory signal that provides instructions to processing hardware 104 of computing platform 102. Thus, a computer-readable non-transitory storage medium may correspond to various types of media, such as volatile media and non-volatile media, for example. Volatile media may include dynamic memory, such as dynamic random access memory (dynamic RAM), while non-volatile memory may include optical, magnetic, or electrostatic storage devices. Common forms of computer-readable non-transitory storage media include, for example, optical discs, RAM, programmable read-only memory (PROM), erasable PROM (EPROM), and FLASH memory.
Moreover, although
Processing hardware 104 may include multiple hardware processing units, such as one or more central processing units, one or more graphics processing units, and one or more tensor processing units, one or more field-programmable gate arrays (FPGAs), custom hardware for machine-learning training or inferencing, and an application programming interface (API) server, for example. By way of definition, as used in the present application, the terms “central processing unit” (CPU), “graphics processing unit” (GPU), and “tensor processing unit” (TPU) have their customary meaning in the art. That is to say, a CPU includes an Arithmetic Logic Unit (ALU) for carrying out the arithmetic and logical operations of computing platform 102, as well as a Control Unit (CU) for retrieving programs, such as software code 110, from system memory 106, while a GPU may be implemented to reduce the processing overhead of the CPU by performing computationally intensive graphics or other processing tasks. A TPU is an application-specific integrated circuit (ASIC) configured specifically for artificial intelligence (AI) processes such as machine learning.
In some implementations, computing platform 102 may correspond to one or more web servers, accessible over a packet-switched network such as the Internet, for example. Alternatively, computing platform 102 may correspond to one or more computer servers supporting a private wide area network (WAN), local area network (LAN), or included in another type of limited distribution or private network. In addition, or alternatively, in some implementations, system 100 may be implemented virtually, such as in a data center. For example, in some implementations, system 100 may be implemented in software, or as virtual machines.
Although user system 130 is shown as a smartphone or tablet computer in
The specific actions performed by phonetic conversion module 212, domain language transfer module 216 and sentence ranking and selection module 218 will be discussed in detail below by reference to
The operation of system 100 including software code 110/210 will be further described by reference to
Referring to
Flowchart 370 further includes generating a phoneme stream corresponding to transcript 136/236 (action 372). Generation of the phoneme stream corresponding to transcript 136/236 in action 372 may be performed by software code 110/210, executed by processing hardware 104 of computing platform 102, and using phonetic conversion module 212, as described below.
Referring to
As shown in
It is further noted that phonetic conversion module 412 and candidate word pool 417 correspond respectively in general to phonetic conversion module 212 and candidate word pool 217, in
Regarding lexicon 442, it is noted that lexicon 442 may include multiple different lexicons, including one or more common language lexicons and one or more domain-specific lexicons specific to the application or use case that system 100 is configured to support.
Because transcript 136/236/436 includes transcribed text but no acoustic information of what user 128 uttered as speech 132, action 372 converts the transcribed text back into a stream of phonemes. A phoneme, as known in the art, is a distinct unit of sound. A “stream of phonemes” may refer to a sequence of individual phonemes, i.e., a sequence of distinct sounds. To obtain a better understanding of the possible domain-specific word confusions, phoneme stream 413 is analyzed with word boundaries obtained from transcript 136/236/436, as well as without such boundaries to identify unigram and n-gram level confusions using DTW, which, as noted above is a technique that compares temporal sequences of signals that do not perfectly match in order to find the optimal match between the sequences, as described below by reference to action 373 and
Referring once again to
As stated above, the partitioning of phoneme sequences from phoneme stream 136/236/436 into words may be performed in one or both of two different ways: 1) with word boundaries, and 2) without word boundaries.
Comparisons of bounded phoneme sequences with words can result in multiple different unigram variations that could represent words included in speech 132 by user 128. For example, processing hardware 104 may execute software code 110/210 to partition phoneme stream 413 into words using word boundaries obtained from transcript 136/236/436. Similarity metric such as cosine similarity or Euclidean distance can be used to find other confusing words.
Example: “show me all series”
“Show” may be “show” or “sow”
“Me” may be “me” or “mean
“All” may be “all” or “oil”
“Series” may be “series” or “serious”
This type of comparison snakes the search of possible spoken words more elastic by removing the word boundaries and allowing phoneme stream 413 to be partitioned into multiple words, or to merge multiple words into a single word to find potential candidates words.
Example: the two words “all of” may be merged to form the single proper noun “Olaf.”
DTW can be used to find a predetermined integer number “n” best or most plausible word sequences through the application of pruning to avoid the computational burden of calculating similarity metrics for phoneme sequences that appear implausible.
As shown in
According to the example shown in
Flowchart 370 further includes aggregating subsets of the words resulting from the partitioning performed in action 373 to form multiple candidate sentences (action 374). Aggregation of subsets of words included in candidate word pool 417, in
Referring to
It is noted that domain language transfer module 516 and candidate sentence pool 519 correspond respectively in general to domain language transfer module 216 and candidate sentence pool 219, in
Transcript 536 corresponds in general to transcript 136/236/436 in
Action 374 includes reconstructing sentences from words included in candidate word pool 217/417/517. Initially, all, or substantially all permutations of words included in candidate word pool 217/417/517 may be evaluated. For example, a weighted combination of trained common language prediction ML model 552, trained domain-specific language prediction ML model 553 may be used to disambiguate and correct the transcribed text included in transcript 136/236/436/536. Trained common language prediction ML model 552 may be used in order to compensate for what might by a limited domain-specific vocabulary and insufficient domain-specific training data. In some implementations, trained common language prediction ML model 552 may take the form of any readily available language models, such as Bidirectional Encoder Representations from Transformers (BERT), a generalized autoregressive model trained on a large corpus using the permutation language modeling objective, such as XLNet, or any bidirectional language model for effectively using bidirectional content information to model or analyze content based on corrupted input.
Trained domain-specific language prediction ML model 553 may be trained by fine-turning an available pre-trained model, such as XLNet. The objective is to maximize the expected log likelihood of a sequence of words by responding to all possible permutations of the factorization order. In use cases in which system 100 is configured to support searches or queries from user 128 regarding the availability of media content such as movies, TV programming, or video games, for example, domain-specific language data may include titles, characters, actors, synopses, subtitles, reviews, content metadata, and the like.
In addition to using the weighted language models, i.e., trained common language prediction ML model 552, trained domain-specific language prediction ML model 553, word associations may also be considered during the disambiguating and correcting process. Trained word association ML model 554 may be built on the domain-specific language database that contains the associations and dependencies between words as well as implicit associations or co-references such as co-occurrence events. Let R(xi, xj) be the association between word xi and word xj:
Trained word association ML model 554 mainly provides an indication of the existence of association between two words and the group of words that have associations with a given word. A word association score Ra of a length-T word sequence X=[x1, . . . , xT] may be defined as:
In addition to using trained common language prediction ML model 552, trained domain-specific language prediction ML model 553, and trained word association ML model 554, in some implementations it, may also be advantageous to increase the number of domain-specific key word appearances in candidate sentence pool 219/519 by adding key word weights (wk) during the domain language transfer process. The key word score K of a length-T word sequence X=[x1, . . . , xT] may be defined as:
where N (x∈Kt) is the number of words that are key words.
A hypothesized sentence for inclusion in candidate sentence pool 519/519 may be generated using candidate word pool 217/417/517. That is to say, given a hypothesized sequence X=[x1, . . . , xT] with length T, ZT denotes the set of all possible permutations of the index sequence [1, . . . , T]. Let Pclm(xz
where wc, wd, wtam, and wk are the weights of trained common language prediction ML model 552, trained domain-specific language prediction ML model 553, trained word association ML model 553, and the number of key words present in the hypothesized sentence, respectively. It is noted that weights wc, wd, wtam, and wk can be trained. Based on the LLs, a predetermined integer number “n” of the highest scoring sentences may be selected for inclusion in candidate sentence pool 219/519, under the constraint that every hypothesized sentence included in candidate sentence pool must have an LLs higher than the LLs computed for transcript 136/236/436/536. In other words, in use cases in which transcript 136/236/436/536 contains a single sentence, and less than n hypothesized sentences have a respective LLs higher than the LLs of transcript 136/236/436/536, transcript 136/236/436/536 may be included in candidate sentence pool 219/519.
Thus, aggregating the subsets of words included in candidate word pool 217/417/517 to form the candidate sentences of candidate sentence pool 217/519 in action 374 may include determining a confidence score for each of those candidate sentences. In addition, in some implementations, action 374 may include determining a confidence score for transcript 136/236/436/536. Moreover, the confidence score for each candidate sentence and transcript 136/236/436/536 may be determined based on a weighted sum of a word association score and a keyword score of each of the candidate sentences and 136/236/436/536.
According to some implementations, disambiguation resource(s) 140/540 of system 100 may include trained word association machine ML model 554, in those implementations, the word association score may be determined using trained word association ML model 554. Furthermore, according to some implementations, disambiguation resource(s) 140/540 of system 100 may include one or both of trained common language prediction ML model 552 and trained domain-specific language prediction ML model 553. In those implementations, to aggregating the subsets of words to form the candidate sentences in action 374 may be performed using trained common language prediction ML model trained domain-specific language prediction ML model 553, or both, as described above.
Referring once again to
Referring to
It is noted that sentence ranking and selection module 618 corresponds in general to sentence ranking and selection module 218, in
Furthermore, disambiguation resources 640, in
It is noted that, in action 375, a contribution to the final transcription results based on a particular application or use case is sought. Given sentence candidate pool 219/519/619, the ultimate objective is to rank the candidate sentences based on their likely relevance to the intent of user 128.
For the purposes of the present disambiguation solution, an application or use case can correspond to a set of intents and entities where the recognition of each along with given context can inform the transcription process. In action 375, natural language processing techniques can be used to detect intents and entities in history 122a/622a of user 128, which may include a dialogue history of user 128. This can be done using trained goal prediction ML model 655 in the form of a multi-task entity and intent recognition based transformer network or similar model trained on textual examples containing the intents and entities of the given use case. Similarly, the intents and entities are detected for the candidate sentences included in candidate sentence pool 219/519/619. Pre-trained dense feature embeddings from BERT, ConveRT, or similar NNs may be used to encode history 122a/622a of user 128 along with the candidate sentences included in candidate sentence pool 219/519/619 and their associated intents and entities.
Intent classification can be trained using sentence exemplars associated with their ground truth intents, where the examples are encoded by a particular pre-trained embedding. Dot-product loss or another suitable loss can be used to maximize similarity between transformer output and target intent. Standard entity-extraction techniques can include the implementation of a conditional random field (CRF), for example. Such a process can be trained in a multi-task fashion with the intent classification. Thus, action 375 may be perforated by software code 110/210, executed by processing hardware 104 of computing platform 102, and using trained goal prediction ML model 655.
Flowchart 370 further includes performing a ranking of candidate sentences from candidate sentence pool 219/519/619 and transcript 136/236/436/536 based on the intent of user 128, the context of speech 132 by user 128, or both (action 376). In action 376, trained ranking ML model 656 may be implemented as a transformer-based NN optimized for ranking the candidate sentences and transcript 136/236/436/536, and thereby outputting what it predicts to be the most probable, i.e., “best” transcription. This process can be thought of as a constrained variant of the common “text ranking” problem as discussed in the deep learning literature and ongoing research. Typically, text ranking assumes the existence of a corpus of texts C={ai} which can be structured or unstructured. Given a query q, the task of text ranking is to return a ranked list of n texts {X1, . . . , Xn} from a finite collection of texts (corpus), that optimizes a metric of interest, such as average precision for example. Unlike use cases such as “search” where the corpus can consist of billions of web pages, here the corpus is limited to the sentences included in candidate sentence pool 219/519/619.
Trained ranking ML model 556 receives as input one or both of the determined intent of user 128 and the encoded context of speech 132 by user 128, as well as the encoded candidate sentences and their associated intents and entities. The output of trained ranking ML model 556 represents a ranked order of how well each candidate sentence ‘fits’ as the correct transcription. It is noted that, in addition to identifying which sentences are to be included in candidate sentence pool 219/519/619, the confidence scores determined as part of action 374 may also be used in action 376 as an additional basis for ranking the candidate sentences. Action 376 may be performed by software code 110/210, executed by processing hardware 104 of computing platform 102, and using trained ranking ML model 556.
Flowchart 370 further includes identifying, based on the ranking performed in action 376, one of transcript 136/236/436/536 or one of the other candidate sentences included in candidate sentence pool 219/519/619 as best transcription 138/638 of speech 132 by user 128 (action 377). In practical terms, action 377 includes determining which candidate sentence most reasonably follows, given history 122a/622a of user 128 and what is known about the particular use-case. In this sense, action 377 is an attempt to determine which of the sentences included in candidate sentence pool 219/519/619 continues the flow of conversation best, i.e., which makes the most sense. Action 377 may be performed by software code 110/210 executed by processing hardware 104 of computing platform 102.
Regarding best transcription 138/638, it is noted that although in some use cases one of the candidate sentences aggregated in action 374 may be identified as best transcription 138/638 in action 377, in other use cases, original transcript 136/236/436/536 received in action 371 may be identified as best transcription 138/638. Thus, in addition to making domain-specific corrections to transcript 136/236/436/536 when necessary, system 100 is also configured to identify transcript 136/236/436/536 as best transcription 138/638 of speech 132 by user 128 when that is appropriate.
According to the exemplary implementation shown in
With respect to the method outlined by flowchart 370, it is emphasized that actions 372, 373, 374, 375, 376, and 377 (hereinafter “actions 372-377”) may be performed in real-time relative to receiving transcript 136/236/436/536 in action 371, such as within less than ten seconds of receiving transcript 136/236/436/536, for example. Moreover, actions 371 and 372-377 may advantageously be performed in an automated process from which human involvement may be omitted.
According to the exemplary implementation shown in
Thus, the present application discloses automated systems and methods for performing domain and user intent specific disambiguation of transcribed speech. As discussed above, the disambiguation solution disclosed in the present application reverses the language that s output in transcript form by a conventional ASR system in real-time, while retaining the phonetic hints and context from the transcript, and then superimposing domain and application specific language along with the context to disambiguate and correct the transcribed text for a specific domain, The solution disclosed in the present application advances the state-of-the-art by providing significant improvements in disambiguating domain specific language from common language, as well as significant improvements in language understanding in situations where audio quality is poor, linguistic errors are present, speech is uttered by non-native speakers, children, the aged, or those with speech disorders, or where confusion can arise due to the unique or distinctive requirements of a particular domain specific vocabulary.
From the above description it is manifest that various techniques can be used to implement the concepts described in the present application without departing from the scope of those concepts. Moreover, while the concepts have been described with specific reference to certain implementations, a person of ordinary skill in the art would recognize that changes can be made in form and detail without departing from the scope of those concepts. As such, the described implementations are to be considered in all respects as illustrative and not restrictive. It should also be understood that the present application is not limited to the particular implementations described herein, but many rearrangements, modifications, and substitutions are possible without departing from the scope of the present disclosure.
