This invention relates to use of a monophone background model for wakeword detection.
One approach to providing a speech-based user interface for a speech-enabled system is to use a device that monitors an acoustic environment waiting for a user to speak a command that can be interpreted by the system. The system may determine when the speaker intends to interact via the interface by determining when the speaker utters a particular word or phrase designed to “wake” the system. Such a word or phrase may be referred to as a “wakeword” or a “trigger word.”
Speech recognition used to determine the words spoken and further understand the intent of the speaker may be computationally expensive, and may be beyond the computational capacity of devices that may be located in the acoustic environment being monitored. One approach to addressing the limited computational capacity of such devices is to perform some of the computation on a server coupled to the devices over a data network, for instance over the public Internet. In some such approaches, the devices send audio data to the server only after a speaker utters the wakeword, and the server performs much of the computation required to interpret the speaker's input.
Referring to
One technical problem addressed by the user device 110 is limiting the amount of audio data sent to the server 190. It should be recognized that sending the audio data in a continuous manner may consume excessive communication resources as well as computation resources at the server required to process that data. By limiting the sending of audio data to time intervals associated with detection of a trigger, fewer communication and computation resources are consumed. It should also be recognized that in practice, the user device may make errors in detection of the trigger, with both missed actual triggers and false alarms of triggers that are not present. The quality of the user experience is improved when missed triggers are avoided. The amount of avoidable communication and server computation is reduced when false alarms are avoided. There may be a need to achieve low error rates within the computation resources available at the device.
The system 100 described below is an example of a system that provides an improved trigger detection error rate within a limited computation capacity (e.g., limits on the number of operations per second and/or the size of memory available to continuously process monitored audio) of the audio user interface device 110. Before addressing the technical features that yield this improvement, which is not merely an improvement achieved by virtue of applying computing resources to the problem, a description of the overall system is provided below.
Continuing to refer to the embodiment shown in
An acoustic front end (AFE) 130 receives the digitized audio input signals and the digitized audio output signal, and outputs an enhanced digitized audio input signal (i.e., a time sampled waveform). Referring to
In alternative embodiments, the acoustic front end 130 may include various features not described above, including one or more of: a microphone calibration section, which may reduce variability between microphones of different units; fixed beamformers, each with a fixed beam pattern from which a best beam is selected for processing; separate acoustic echo cancellers, each associated with a different beamformer; an analysis filterbank for separating the input into separate frequency bands, each of which may be processed, for example, with a band-specific echo canceller and beamformer, prior to resynthesis into a time domain signal; a dereverberation filter; an automatic gain control; and a double-talk detector.
Referring again to
Referring to
Referring again to
Various forms of feature analyzer 150 may be used. One approach uses probability models with estimated parameters, for instance, Gaussian mixture models (GMMs) to perform the transformation from feature vectors to the representations of linguistic content. Another approach is to use an Artificial Neural Network (ANN) to perform this transformation. Within the general use of ANNs, particular types may be used including Recurrent Neural Networks (RNNs), Deep Neural Networks (DNNs), Time Delay Neural Networks (TDNNs), and so forth. Yet other parametric or non-parametric approaches may be used to implement this feature analysis.
Referring to
In the discussion below, the output of the feature analyzer, for the tth input (i.e., the tth observation vector xt), dt, is treated as representing “scaled likelihood” of each of the linguistic elements q (described in more detail below as states of a Hidden Markov Model) conditioned on that input. That is, the output of the neural network may provide a set of outputs that sum to 1.0 representing conditional probabilities, such that the ith output of the neural network represents the conditional probability p(qt=i|xt), (using conventional notation the items to the right of the vertical bar “|” denotes the conditioning event and the notation p( ) is used without distinction for both probabilities of discrete events and probability density for continuous variable values). Prior to output, the feature analyzer scales each of these conditional probabilities by the unconditional probability of each element as
dt,i=p(qt=i|xt)/p(qt=i),
which is referred to as a “scaled likelihood” because by Bayes Rule, this quantity is also equal to
dt,i=p(xt|qt=i)/p(xt).
Referring to
Before continuing with a description of components of the system 100 that make use of the output of the trigger detector 160 (see
Referring to
a=p(qt+1=j|qt=i),
where the lack of a link in the illustrated state transition diagram denotes that the corresponding state transition probability is zero.
In other embodiments, different type of networks for the subword units may be used, for example, with a different number of states, or with optional skipping of certain of the states. Furthermore, it is not required that all subword units in the network 300 have the same structure. The last state 315 of the last subword unit 312 of the wakeword 310, which is identified with an “*” in the Figure, represents the end of the wakeword. During processing, an end of an utterance of the wakeword corresponds to the MEW being that state. In the network 305 shown in
The background model 305 of the network includes subword units 332 that together generally cover the sounds that occur in speech being processed by the system. In this embodiment, these units 332 generally correspond to English language phonemes, however other sets of linguistically motivated or non-linguistic data-derived units may be used. For linguistically motivated units, phonemes from other languages, or a set of language-independent phonemes covering many languages may be used. Other units such as diphones, syllables, demi-syllables, phoneme classes, etc. may also be used without altering the approach described below. Furthermore, the units themselves may be identified in an unsupervised manner by processing data, for example, using “fenones” or clustering of similar segments of speech. In
The background model 305 optionally includes confusable words and phrases 320. If
During operation, as presented above, the feature analyzer 150 provides successive scaled likelihoods dt to the HMM. The HMM implements “Viterbi” update procedure in which, for each successive update time t, and for all the states j, the quantities vt,i are updated as
(suitably initializing v1,j=πjd1,j for initial probabilities πj).
As introduced above, the scaled likelihood is treated as being equivalent to
dt,i=P(xt|qt=i)/P(xt)
and therefore following the properties of the Viterbi procedure (applying the conventional conditional independence assumption of the observation vectors),
where X1t=(x1, . . . xt), and Q1t=(q1, . . . , qt−1, qt=j) is a state sequence ending in state j at time t. Treating
p(X1t)=Πτp(xτ)
(i.e., treating the observation vectors as independent, which is recognized to be incorrect), it is useful to consider vt,i, to be
As introduced above, a particular state, denoted q* is the last state of the wakeword, and an output of the HMM that is used for detecting the presence of the wakeword ending at a particular time is the time evolution of the quantity
s(t)=vt,q*
as defined above, or a scaled version and/or log version of this quantity. Local peaks in this score are used to locate potential ending times of the wakeword in a likelihood peak detector 272 shown in
During the Viterbi update procedure, states that maximize each of the updates for the vt,j terms are retained in a “traceback” data structure, and therefore having declared that the last state is q* (also referred to herein as qend) at an end time tend, the corresponding state sequence Q1t
For the selected wakeword ending time tend, the state subsequence from tstart to tend is referred to as a time segment, denoted “seg”, for the wakeword and is denoted QKW. For the same ending time, the state q in the background model with the highest Viterbi value vt
A total duration for the wakeword segment is one element (i.e., a feature) of the wakeword feature vector and is computed as
Durseg=tend−tstart+1.
Another wakeword feature is a total score of the segment, computed based on the Viterbi procedure as
for the state sequence QKW, which is treated as
Scoreseg≈p(X|QKW)/p(X)
where X=(xt
Another wakeword feature is an average score computed as
AveScoreseg=Scoreseg/Durseg.
Another wakeword feature is a confidence of the segment, which compares the scaled likehoods along the path in the wakeword part of the HMM and the background part of the HMM. In particular, this feature is computed as
In addition to these four features for the wakeword segment as a whole, a set of features are computed for the subword unit segments of the wakeword. Quantities for a particular subword unit (i.e., phoneme) segment p are denoted by the subscript p.
A total duration Durp is computed corresponding to the number sequential states in QKW are in that segment, computed as
Durp=Tp+1−Tp
where Tp is the starting time of the pth segment (and TP+1 is treated as being equal to tend+1).
A relative duration is computed as the fractional time in the segment as compared to the duration of the wakeword as a whole, computed as
RelDurp=Durp/Durseg.
A confidence of the subword segment is defined as the average ratio between the wakeword state likelihood and the sum of the likelihoods from all the background states, defined as
where qt ∈QKW is the state in the sequence QKW at time t, and where
A speech confidence for this time segment is computed as
where “background\NSP” represents all states of the background model with the omission of the non-speech unit, which is labelled “NSP”.
An number of phone confidence scores are computed making use of an maximum score phone score defined as
where qlp is defined as the first (left) of the three states for the pth subword unit in the wakeword, qcp and qrp are similarly defined for the second (center) and third (right) states. Using this definition, the left, center, and right confidence scores are defined as
Finally, a set of entropies derived from the confidence scores are computed using three-bin distributions, as
where A=al+ac+ar. This quantity provides a measure of variation over the distribution.
The two features are
WithinSlotNormEntropyp=entropy(LPhConfp,CPhConfp,RPhConfp),
for p=1, . . . , P and
AcrossSlotNormEntropyp=entropy(RPhConfp−1,CPhConfp,LPhConfp+1),
for p=1, . . . , P, padding the missing values at p=1 and p=P with zeroes. This latter quantity provides a measure of variation of a match of the input data for the pth unit over multiple of the subword units (i.e., p−1 through p+1) of the event.
Finally, for each combination of a wakeword subword segment p, and a background subword unit b, which has left, center, and right states {qbl,qbc,qbr}, a match score
is computed where maxscoreBG is defined analogously to the maxscore for within wakeword evaluation used above for each of the 44 background units.
Therefore, with P=7 wakeword subword units, and B=44 units it the background model (excluding the non-speech unit), the total number of features computed for a particular wakeword ending time are:
4Durseg,Scoreseg,AveScoreseg,Confseg
P*6Durp,RelDurp,SpeechConfp,CPhConfp,AcrossSlotNormEntropy WithinSlotNormEntropy
(P−1)*2LPhConfp,RPhConfp(P*B)MatchScore
for a total of 373 values. Referring to
Generally, the secondary classifier 278 receives these various computed quantities corresponding to the trigger word ending time and performs a scoring or detection decision based on the inputs. The operation of the classifier depends on values of parameters stored in the detector configuration data 267. In some embodiments, the classifier is implemented using a Support Vector Machine (SVM). The detector logic receives the wakeword feature vector v, and computes a quantity v·a+b, where a and b are parameters that have set values, and compares it with zero, such that if the quantity is greater than zero, the wakeword is treated as having occurred, and if the quantity is not greater than zero, the detected occurrence of the wakeword is treated as a “false alarm” and discarded. In some examples, the SVM makes use of radial basis functions to augment the feature vector.
Referring back to
Referring to
Referring back to
Following processing by the runtime speech recognizer 281, the text-based results may be sent to other processing components, which may be local to the device performing speech recognition and/or distributed across data networks. For example, speech recognition results in the form of a single textual representation of the speech, an N-best list including multiple hypotheses and respective scores, lattice, etc. may be sent to a natural language understanding (NLU) component 291 may include a named entity recognition (NER) module 292, which is used to identify portions of text that correspond to a named entity that may be recognizable by the system. An intent classifier (IC) module 294 may be used to determine the intent represented in the recognized text. Processing by the NLU component may be configured according to linguistic grammars 293 and/or skill and intent models 295. After natural language interpretation, a command processor 296, which may access a knowledge base 297, acts on the recognized text. For example, the result of the processing causes an appropriate output to be sent back to the user interface device for presentation to the user.
The description above focuses on the runtime operation of the audio interface device 110 shown in
Generally, neural network weights and the HMM parameters (e.g., the transition probabilities) are determined together in a “training” procedure which uses data representing a set of audio recording which have been annotate (e.g., transcribed) to indicate occurrences of the wakeword as well as other words in the general vocabulary. Generally, the procedure alternates between associating each input feature vector obtained from the audio recordings with a state (or a state distribution) based on current weights of the neural network and the HMM parameters, and updating the weights of the neural network using an incremental gradient-based optimization procedure generally referred to as “backpropagation,” as well as updating the transition probabilities. In this embodiment, the neural network weights are determined to optimize a cross-entropy loss function, thereby essentially achieving Maximum Likelihood training of the HMM. Other approaches to training the neural network may be used, including approaches that use different loss functions, and approaches that attempt to discriminate accurately between true wakeword occurrences and false alarms.
Generally, the detection logic is addressed in training by determining the values of configuration parameters for the SVM (i.e., a and b) based on detected true and false alarms on a training data set of recorded utterances that are labelled as to having a wakeword or not.
It should be understood that the specific arrangement of the background HMM and the elements of the wakeword feature vector described for the embodiment above may be varied without departing from the general approach. For example, different features of subword units of the wakeword may be used, for example, various average, variation, or maximum scores in each segment, may be used, and difference comparisons of the wakeword subword units and the background subword units may be used, for example, comparing each wakeword subunit against particular subsets of background units. In some examples, the background model may include explicit inter-phoneme transition probabilities (e.g., a phoneme “bigram” model).
In some embodiments, the parameters of the classifier (e.g., the SVM classifier, or an alternative parametric classifier) are training jointly with the parameters of the HMM. For example, a joint gradient-based approach is used such that the training labels for the classifier are propagated through the classifier structure according to the derivatives of the classifier features and to the neural network that provides the state scores for the HMM. In some examples of this approach, alternating stages of computation of a state alignment and training of classifier and HMM parameters are performed.
In yet another embodiment, the structure of the input features (e.g., their functional forms) for the classifier are not necessarily provide and are rather learned from data. For example, a rich set of features based on the alignment of the located instance of the wakeword to subword units of the wakeword and to subword units of the background are provide to a secondary scoring neural network, which is trained independently or jointly with the HMM and its neural network.
In the description above, the wakeword is predefined in the sense that that structure of the HMM is known when the particular utterance is processed by the system. However, it should be understood that such predefinition of the wakeword is not limited to an initial configuration of the interface device. For example, the wakeword may be set by the user, or even determined implicitly from a prior input from the user.
In alternative versions, the predefined event that is being detected is not necessarily word based. For example, a clap, door slam, or other non-linguistic event may be detected using the same approach. Similarly, the sub-units are not necessarily subword units. For example, the sub-units may represent clustered acoustic-based patterns that are found in collected input data.
The use of an HMM, or more particularly, use of a Viterbi scoring procedure, is not essential. For example, other forms of processing based on models that impose at least some deterministic or probabilistic sequence constraint on sub-units located in the input data may be used. Similarly, it is not essential to use an SVM, as other parametric or non-parametric classifiers may be using, including neural networks, or nearest neighbor approaches.
The approaches described above may be implemented in software, in hardware, or using a combination of software and hardware. For example, the software may include instructions stored on a non-transitory machine readable medium that when executed by a processor, for example in the user interface device, perform some or all of the procedures described above. Hardware may include special purpose circuitry (e.g., Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) and the like) for performing some of the functions. For example, some of the computations for the neural network transformations may be implemented using such special purpose circuitry.
It is to be understood that the foregoing description is intended to illustrate and not to limit the scope of the invention, which is defined by the scope of the appended claims. Other embodiments are within the scope of the following claims.
Number | Name | Date | Kind |
---|---|---|---|
9159319 | Hoffmeister | Oct 2015 | B1 |
9542939 | Hoffmeister | Jan 2017 | B1 |
9865253 | De Leon | Jan 2018 | B1 |
20060074664 | Lam | Apr 2006 | A1 |
20070112568 | Fingscheidt | May 2007 | A1 |
20080010065 | Bratt | Jan 2008 | A1 |
20140012586 | Rubin | Jan 2014 | A1 |
20150073795 | Tan | Mar 2015 | A1 |
20160092766 | Sainath | Mar 2016 | A1 |
20160260429 | Jin | Sep 2016 | A1 |
20160358619 | Ramprashad | Dec 2016 | A1 |
20170256255 | Bocklet | Sep 2017 | A1 |
20180068653 | Trawick | Mar 2018 | A1 |
Entry |
---|
Y. Benayed et al., “Confidence Measures for Keyword Spotting Using Suport Vector Machines”, 2003, IEEE, pp. 588-591 (Year: 2003). |
Wilpon, J. G., Miller, L. G., & Modi, P. (Apr. 1991). Improvements and applications for key word recognition using hidden Markov modeling techniques. In Acoustics, Speech, and Signal Processing, 1991. ICASSP-91., 1991 International Conference on (pp. 309-312). IEEE. |
Wilpon, J. G., Rabiner, L. R., Lee, C. H., & Goldman, E. R. (1990). Automatic recognition of keywords in unconstrained speech using hidden Markov models. IEEE Transactions on Acoustics, Speech, and Signal Processing, 38(11), 1870-1878. |
Rose, R. C., & Paul, D. B. (Apr. 1990). A hidden Markov model based keyword recognition system. In Acoustics, Speech, and Signal Processing, 1990. ICASSP-90., 1990 International Conference on (pp. 129-132). IEEE. |
Gish, H., & Ng, K. (Apr. 1993). A segmental speech model with applications to word spotting. In Acoustics, Speech, and Signal Processing, 1993. ICASSP-93., 1993 IEEE International Conference on (vol. 2, pp. 447-450). IEEE. |
Junqua, J. C., & Haton, J. P. (2012). Robustness in automatic speech recognition: fundamentals and applications (vol. 341). Springer Science & Business Media. Ch. 10, “Word-Spotting and Rejection”, 21 pages. |
Rohlicek, J. R., Russell, W., Roukos, S., & Gish, H. (May 1989). Continuous hidden Markov modeling for speaker-independent word spotting. In Acoustics, Speech, and Signal Processing, 1989. ICASSP-89., 1989 International Conference on (pp. 627-630). IEEE. |
Rohlicek, J. R., Jeanrenaud, P., Ng, K., Gish, H., Musicus, B., & Siu, M. (Apr. 1993). Phonetic training and language modeling for word spotting. In Acoustics, Speech, and Signal Processing, 1993. ICASSP-93., 1993 IEEE International Conference on (vol. 2, pp. 459-462). IEEE. |
Rohlicek, J. R. (1995). Word spotting, in R. P. Ramachandran et al. (ed .), Modern Methods of Speech Processing, Kluwer Academic Publishers 1995. pp. 123-157. |
Chen, G., Parada, C., & Heigold, G. (May 2014). Small-footprint keyword spotting using deep neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on (pp. 4087-4091). IEEE. |