Patent Application
20080004876
Embodiments of the present invention are directed to large vocabulary continuous speech recognition (LVCSR) that does not require an initial enrollment procedure. An LVCSR application creates a user profile which includes speech recognition information associated with a specific user. After the user profile is created, the user may commence using the LVCSR application for speech recognition of unknown speech inputs from the user utilizing the information from the user profile.
Embodiments are based on use of a speaker-specific transform based on unsupervised adaptation which uses recognition results as feedback to update the speaker transform. In some specific embodiments, the adaptation is referred to as Online Unsupervised Feature space Adaptation (OUFA) and the adaptation transform is a feature space transform based on Constrained Maximum Likelihood Linear Regression (CMLLR) adaptation, first described in M. J. F. Gales, “Maximum Likelihood Linear Transformations For HMM-Based Speech Recognition”, Technical Report TR. 291, Cambridge University, 1997, the contents of which are incorporated herein by reference. In other embodiments, the adaptation is a model space adaptation which, for example, may use a CMLLR transform or other type of MLLR transform.
The user profile stores information specific to that user and may reflect information from one or more initial audio setup procedures such as an initial Audio Setup Wizard procedure for the microphone. For example, after performing a cepstral mean subtraction (CMS), recognition may be performed on the ASW input (without biasing to the ASW text) and the recognized text of that used to compute a spectral warp factor (vocal tract normalization). The warp factor is used to scale the frequency axis of incoming speech so that it is as if the vocal tract producing the input speech was the same (hypothetical) vocal tract used to produce the acoustic models. For example, spectral warping may be based on a piecewise linear transformation of the frequency axis, further details of which are well-known in the art, and may be found, for example, in S. Wegmann, D. McAllaster, J. Orloff, and B. Peskin, Speaker Normalization On Conversational Telephone Speech, Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing ICASSP'96, Volume 1, pages 339-343, Atlanta (Ga.), USA, May 1996, the contents of which are incorporated herein by reference. Thus, initially, the user profile reflects CMS and spectral warping for the new user.
After a new user profile is first created, step 102, embodiments next initialize an adaptive speaker transform, step 103. In the specific embodiment shown, the speaker transform is based on a Constrained Maximum Likelihood Linear Regression (CMLLR) approach using online unsupervised adaptation statistics from the recognition results. The resulting dynamic speaker transform is relatively responsive to the immediate acoustic environment, for example, spectral variations reflecting specific user speech characteristics and specific characteristics of ambient noise. In some embodiments, the dynamic speaker transform may be complemented by a separate stable speaker transform which is relatively unresponsive to the immediate acoustic environment and may reflect speaker specific characteristics as determined by supervised adaptation such as from a traditional enrollment procedure and/or a post-enrollment acoustic optimization process. The speaker transform may be initialized, step 103, in a variety of specific ways. One approach is to initialize the speaker transform with an identity matrix. Another approach is to initialize the speaker transform from an inverse MLLR transform.
Once a user profile has been created and a speaker transform initialized, the system begins processing input speech for recognition. When an input utterance is present, step 104, the speaker transform is applied, step 105. That is, the input speech vectors for the current utterance are multiplied by the transform matrix that reflects the existing adaptive feature space transformation. Normal speech recognition of the transformed input speech is then performed, step 106, and output to the user's application. In addition, from the speech recognition results of each utterance recognition adaptation statistics are accumulated for the speaker transform, step 107. Every Mth utterance, step 108, for example, every third utterance, the adaptation statistics are used to adapt the speaker transformation, step 109, for example, by updating the CMLLR transform. In some embodiments, this updating may be conditioned on some number T seconds worth of recognition statistics having been collected, and/or whether processor load is relatively low. As when the speaker transform was first initialized, updating of the transform may start from applying the adaptation statistics to an identity matrix or the inverse of an MLLR transform, or from the existing CMLLR transform.
The cycle of input utterance recognition and online unsupervised adaptation repeats from step 104 so long as input speech is present. Once enough speech has been dictated into the system, the user may be encouraged to run or the system may automatically invoke unsupervised model space adaptation to further optimize acoustic models for the user. This acoustic model optimization process is typically an offline process because it requires a great deal of computational resources which are not available when the computing system is busy.
One typical environment for such an application would be a dictation application in a Windows, Macintosh, or Linux computer operating system; for example, Dragon Naturally Speaking by Nuance Communications, Inc. of Burlington, Mass. In such an application, a user sits before a computer video monitor and uses a microphone to enter input speech, which is recognized by the dictation application and displayed on the video monitor as representative text. LVCSR applications have also recently been implemented on portable devices such as cell phones and personal digital assistants (PDAs). In such applications, the use of LVCSR would be similar, with the user providing a speech input via a device microphone and seeing the recognition output as representative text on a device visual display.
Embodiments of the present invention which allow LVCSR without the usual enrollment procedure are based on an Online Unsupervised Feature space Adaptation (OUFA) Module 208 which uses an adaptive Constrained Maximum Likelihood Linear Regression (CMLLR) transform to best fit the feature vectors of a user in the current recognition environment to the model. OUFA uses adaptation data to determine a CMLLR linear transformation that consistently modifies both means and (diagonal) covariances of the Acoustic Models 205. Starting from the Gaussian mixture component distribution:
As discussed above, the OUFA Module 208 can use the OUFA technique as a substitute for normal supervised enrollment. It is also useful even when the user has completed supervised enrollment or after the system completes acoustic model optimization with sufficient amount of input speech, for example, when the immediate acoustic environment during recognition differs from the acoustic environment that was present during enrollment.
In some embodiments, the OUFA Module 208 may accumulate CMLLR statistics with a “forgetting factor.” That is, after an accumulated statistic is used to update the speaker transform some number N times, it is multiplied by a configurable factor between 0 and 1 and new data is then added to the statistic without scaling.
In some embodiments, the OUFA Module 208 may further use-one or more additional optimizations to code for the speaker transform to make it run faster. For example, the OUFA Module 208 may accumulate the CMLLR statistics for some configurable fraction of the highest probability Gaussian components of the aligned acoustic model states. The algorithm that estimates the CMLLR transform also may be initialized from a pre-existing transform when a new transform is computed. The OUFA Module 208 also may postpone accumulation of statistics, and/or the computation and application of an updated CMLLR transform in coordination with processor load, for example, until the start of the next utterance recognition, to minimize recognition latency effects. In other words, adaptation can be delayed if the processor is busy with other tasks. Adaptation may also be run in a separate processor in a multi-core or multi-processor computer system.
Various other software engineering speedups may be usefully applied by the OUFA Module 208 including, without limitation, exploiting the symmetry of the accumulated statistics matrices to perform calculations on only half of each matrix for the CMLLR transform, using scaled integer arithmetic, converting divisions to multiplications where possible, precomputing reusable parts (e.g. denominators in the accumulation expressions), stopping accumulation of statistics early on very long utterances, coordinating the timing of the adaptation statistics accumulation and CMLLR transform update with processor load (e.g., temporarily suspend updating the CMLLR transform when processor load is high), and not accumulating statistics for initialization of the transform if initializing from the existing transform.
Specific embodiments may also employ other useful techniques. For example, after running for a while (i.e., after the system has processed a specific number of utterances or frames), the system may encourage users to run or automatically invoke an acoustic optimization process in which an Adaptation Module 209 re-adapts the users' Acoustic Models 205 using data collected from previously dictated documents. In the specific application of Dragon NaturallySpeaking, this optimization process is known as ACO (ACoustic Optimization). After running for awhile (i.e., after the system has processed a specific number of utterances or frames), unsupervised adaptation can be invoked using either or all of adaptation of the CMLLR transform, and/or MLLR transform, and/or MAP adaptation by the Adaptation Module 209 of the means and variances of the Acoustic Models 205. The CMLLR statistics may also be accumulated directly from the best acoustically scoring model state prior to final decoding. This would allow statistics accumulation in real time as opposed to in latency time, although it is possible that this might lead to a decrease in accuracy. The adaptation may be a feature space adaptation as described above, or similarly, model space adaptation may be used.
Embodiments of the invention may be implemented in any conventional computer programming language. For example, preferred embodiments may be implemented in a procedural programming language (e.g., “C”) or an object oriented programming language (e.g., “C++”). Alternative embodiments of the invention may be implemented as pre-programmed hardware elements, other related components, or as a combination of hardware and software components.
Embodiments can be implemented as a computer program product for use with a computer system. Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, DVD, flash memory devices, or hard disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium. The medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., microwave, infrared or other transmission techniques). The series of computer instructions embodies all or part of the functionality previously described herein with respect to the system. Those skilled in the art should appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies. It is expected that such a computer program product may be distributed as a removable medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or hard disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web). Of course, some embodiments of the invention may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments of the invention are implemented as entirely hardware, or entirely software (e.g., a computer program product).
Although various exemplary embodiments of the invention have been disclosed, it should be apparent to those skilled in the art that various changes and modifications can be made which will achieve some of the advantages of the invention without departing from the true scope of the invention.
