Patent Application
20200312315
An aspect of the disclosure here relates to a system for selecting a high quality audio stream for multichannel speech recognition. Other aspects are also described.
Many consumer electronics devices have voice-driven capabilities, such as the ability to act as or interface with a “personal virtual assistant,” by utilizing multiple sound pick up channels in the form of two or more microphones. A key technique to the success of this type of human-machine interaction is far-field speech recognition because the user is typically at some distance from the device that the user is interacting with. However, the microphones may produce mixed audio signals, which contain sounds from various or diverse sources in the acoustic environment (e.g., two or more talkers in the room and ambient background noise). Also, when a talker in a room is sufficiently far away from the microphones, other types of interference may exist, such as reverberation, directional noise, ambient noise, competing speech, etc. These forms of interference may have a significant impact on the performance of a personal virtual assistant, such as producing false positives or missing commands. A user may expect a device to both accurately and rapidly detect an initial voice trigger phrase so that it can respond with reduced latency.
An aspect of the disclosure is directed toward a system for achieving high performance speech recognition for a multi-stream speech recognition system. A speech enabled device may include several microphones, e.g., forming a microphone array, that can pick up speech of a talker. A speech recognition capability may be activated by a voice trigger, such as a predetermined trigger phrase. The system may include a multichannel digital signal processor that processes the microphone array signals into a beamforming audio stream and a blind source separation audio stream. A speech processor performs tests for the presence of and the quality of a voice trigger on the audio streams, in order to determine whether there is a preferred stream for speech analysis. This may improve the performance of general speech analysis that is performed downstream upon the preferred stream, including trigger phrase detection, automatic speech recognition, and speaker (talker) identification or verification, by integrating information relating to the acoustic environment of each audio stream with a voice trigger score for each audio stream, and the selecting the stream with a highest combined score.
In one aspect, the audio stream input is from the microphones of not just one but several speech devices (at least one stream that contains sound pickup from each of the several speech devices.) The audio streams from the several devices are tested for presence and quality of a voice trigger in order to determine whether there is a preferred stream for speech analysis in order to optimize the performance of subsequent general speech analysis. The stream that achieves the highest combined score may be output to for example an automatic speech recognizer.
In an aspect, an acoustic environment characteristic that is measured may be signal to noise ratio (SNR.) The system may use the detected trigger phrase as an anchor when calculating the signal to noise ratio. Due to the large variability in time and frequency exhibited by different speech sounds, estimation of SNR and other acoustic properties usually takes a long integration time for a statistical model of speech and noise to converge. This anchor-based method greatly reduces the speech variability by focusing on the trigger phrase, which makes it possible to reliably estimate the acoustic property within a short duration. The signal to noise ratio may be calculated using, for example, a root mean square method.
A preferred audio stream (“preferred” because it is expected to show greater performance than other streams when processed by subsequent or “downstream” speech recognition) may be output through the following process. A plurality of audio streams are received or generated. Testing for the presence of a voice trigger is performed, and if the voice trigger is present (in any one of the audio streams) then a quality of the voice trigger in each audio stream is tested. The quality of the voice trigger in each audio stream may be determined using a combined score that is calculated based on i) a voice trigger score and ii) an acoustic environment measurement (for that audio stream.) If the combined score is sufficiently high, e.g., highest across all of the audio streams, then that stream is selected as the “preferred” stream (such that its payload, and not those contained in the other audio streams, undergoes speech recognition analysis.)
The above summary does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the disclosure includes all systems and methods that can be practiced from all suitable combinations of the various aspects summarized above, as well as those disclosed in the Detailed Description below and particularly pointed out in the Claims section. Such combinations may have particular advantages not specifically recited in the above summary.
Several aspects of the disclosure here are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to “an” or “one” aspect in this disclosure are not necessarily to the same aspect, and they mean at least one. Also, in the interest of conciseness and reducing the total number of figures, a given figure may be used to illustrate the features of more than one aspect of the disclosure, and not all elements in the figure may be required for a given aspect.
Several aspects of the disclosure with reference to the appended drawings are now explained. Whenever the shapes, relative positions and other aspects of the parts described are not explicitly defined, the scope of the invention is not limited only to the parts shown, which are meant merely for the purpose of illustration. Also, while numerous details are set forth, it is understood that some aspects of the disclosure may be practiced without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.
The present disclosure is related to a digital speech signal processing system for achieving high performance far-field speech recognition. The system may include a speech enabled device which may be a device with digital signal processing-based speech signal processing capabilities that may use several microphones that form a microphone array to pick up speech of a user that is in an ambient environment of the device. The speech signal processing is to process the microphone signals in order to improve the fidelity of (e.g., reduce word error rate by) an automatic speech recognition (ASR) process or a downstream voice trigger detection process (that is subsequently performed upon a selected one of several audio streams derived from the microphone signals.) The downstream voice trigger detection and ASR may be elements of a “virtual assistant,” through which a user may instruct the device to undertake certain actions by speaking to the device. The speech enabled device could be, but is not limited to, a smartphone, smart speaker, computer, home entertainment device, tablet, a car audio or infotainment system, or any electronics device with interactive speech features.
The multichannel DSP 110 and SAS 115 are generically used here to refer to any suitable combination of programmable data processing components and data storage that conduct the operations needed to implement the various functions and operations of the speech system 107. The multichannel DSP and SAS may be implemented as a system on a chip typically found in a smart phone. The memory unit may be microelectronic, non-volatile random access memory. While the multichannel DSP 110 and the SAS are shown as separate blocks in
The multichannel DSP 110 may receive a plurality of microphone signals from the microphone array 103. The multichannel DSP 110 may process the plurality of microphone array signals into a plurality of audio streams using a beamforming output mode and a blind source separation (BSS) output mode to generate a single beamformed audio stream and a BSS audio stream, respectively. The BSS audio stream could include a plurality of BSS audio streams, wherein one of the BSS audio streams from the plurality of BSS audio streams may contain target speech from a user. In an aspect, the multichannel DSP 110 may use beamforming techniques applied to the plurality of microphone array signals to generate a plurality of beamformed audio streams that are extracted using respective beams steered in different “look directions.”
The audio streams may be transmitted to the SAS 115. The SAS 115 may conduct fitness tests on the audio streams to determine if there is a preferred audio stream available for speech recognition, including a first pass voice trigger detector 118 and an acoustic environment characteristics analyzer 121. The first pass voice trigger detector 118 determines through signal analysis whether any of the audio streams contain a voice trigger (e.g., likely contain a voice trigger), such as a desired trigger phrase that indicates that a user is addressing the device. In an aspect, the first pass voice trigger detector 118 may utilize a deep neural network to convert the acoustic patterns present in the audio streams into a probability distribution over speech sounds, which then outputs as a voice trigger score by using a temporal integration process, wherein the voice trigger score represents the confidence level of a match between an acoustic pattern present in the audio stream and the trigger phrase.
The acoustic environment characteristics analyzer 121 may calculate estimates of acoustic environment measurements for each of the audio streams. Acoustic environment measurements may include, for example, target signal to background noise ratio (signal to noise ratio, SNR), direct to reverberant ratio (DRR), audio signal level, and direction of arrival of the voice trigger. The estimates of the acoustic environment measurements may be adapted to the environment and calculated “live,” or during run time, as the user is talking.
The SAS115 uses a combination of the voice trigger score and the acoustic environment measurements to determine a preferred audio stream (select one of the input audio streams.) For example, the SAS may calculate a stream score (SC) for each input audio stream by adding a voice trigger score (TS) that is multiplied by a voice trigger score weight (W1) and an acoustic environment measurement (AEM), which may include a measurement for a single acoustic environment characteristic or a composite value based on several acoustic environment characteristics, that is multiplied with an acoustic environment measurement weight (W2), such that SC=W1*TS+W2*AEM. The audio stream that has the highest stream score SC is then selected as the preferred audio stream.
In an aspect, the preferred audio stream may be selected by separately evaluating the voice trigger score and the acoustic environment measurements. For example, the SAS 115 may determine whether a beamformed audio stream is preferred, by determining based on the acoustic environment characteristics whether the environment is quiet. In contrast, blind source separation audio stream is preferred, by determining based on the acoustic environment characteristics whether the environment is noisy. If the environment is determined to be noisy, then this indicates the desirability of a blind source separation audio stream (rather than a beamformed audio stream). In that case, the SAS may determine which blind source separation audio stream out of a plurality of blind source separation audio streams is preferred by selecting the blind source separation audio stream with the highest voice trigger score to be the preferred audio stream.
The first pass voice trigger detector 118 may also determine “runtime” information of the voice trigger, as shown in
In an aspect where the acoustic environment measurement is the SNR, a scoring rule may be determined that produces a combined score from the voice trigger score and SNR for each audio stream, for selecting the preferred audio stream. For a beamformed audio stream, the combined score may be calculated as follows: combined score=voice trigger score+alpha*SNR+beta, wherein alpha and beta are tuning parameters that may be tuned based on data for the device. For each BSS audio stream, the combined score may be calculated as follows: combined score=voice trigger score. The audio stream with the highest combined score may then be selected as the preferred audio stream. Other scoring rules may be considered for determining the preferred audio stream. For example, deep learning may be utilized to devise a data-driven combination rule which automatically selects the preferred audio stream given an SNR and voice trigger score. In another example where the acoustic environment measure is the direction of arrival, the direction information may be utilized to determine a preferred audio stream by comparing the direction of arrival of sound before the start time of the voice trigger to the direction of arrival of sound computed during the runtime interval (the interval from start time to end time.) Audio streams for which the direction arrival of sound prior to the voice trigger is similar to the direction of arrival of sound during the runtime are more likely to contain interference.
In an aspect, the SAS 115 may also include a speaker (talker) identifier. The speaker identifier may use pattern recognition techniques to determine through audio signal analysis whether the voice biometric characteristics of the trigger phrase matches the voice biometric characteristics of a desired talker. If the first pass voice trigger detector 118 has determined with sufficient confidence that an audio stream contains a trigger phrase, the speaker identifier may analyze the audio stream to determine whether the speaker of the trigger phrase is a desired speaker. If the speaker identifier determines that the voice biometric characteristics of the speaker of the trigger phrase matches the voice biometric characteristics of the desired speaker, the SAS may output the audio stream as the preferred audio stream. If the speaker identifier determines that the voice biometric characteristics of the speaker of the trigger phrase does not match the voice biometric characteristics of the desired speaker, the SAS may continue monitoring the audio streams until there is a match.
Optionally, at block 508, it may be desirable to determine if the detected voice trigger is spoken by a specific user. If the speaker (talker) of the voice trigger does not match any known user's voice, or it does not match that of a specific user, then no further action is taken (and the monitoring mentioned above continues) until an audio stream containing a voice trigger that matches that spoken by a desired user is detected. If there is a match, then the process continues with block 512.
If the presence of a voice trigger has been determined (regardless of any user match in block 510), then at block 512 a voice trigger score and runtime information (e.g., a start time of the voice trigger and an end time of the voice trigger within a given audio stream) may be calculated for each audio stream of the plurality of audio streams.
Acoustic environment measurements may be calculated for each audio stream of the plurality of audio streams at block 515. The acoustic environment measurements calculations may involve data from the runtime information. At block 518, an audio stream is selected as an output, preferred audio stream. The selection may be based on a combined score calculated for each audio stream of the plurality of audio streams using a combination of the acoustic environment measurements and voice trigger score. The audio stream with the highest combined score may be selected as the preferred audio stream.
Optionally, at block 522, a downstream speech processor may test the preferred audio stream in order to confirm the detected voice trigger. If the voice trigger is not confirmed with a sufficient confidence level, then no further action is taken until an audio stream containing a voice trigger is detected. If the voice trigger is confirmed with a sufficient confidence level, then the preferred stream may be provided to the input of an automatic speech recognizer at block 532.
As described above, one aspect of the present technology is the gathering and use of data available from various sources to improve the accuracy of automatic speech recognition. The present disclosure contemplates that in some instances, this gathered data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data can include demographic data, location-based data, telephone numbers, email addresses, TWITTER ID's, home addresses, data or records relating to a user's health or level of fitness (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other identifying or personal information.
The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used to increase the ability to recognize a specific user. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure. For instance, health and fitness data may be used to provide insights into a user's general wellness, or may be used as positive feedback to individuals using technology to pursue wellness goals.
The present disclosure contemplates that the entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with well-established privacy policies and/or privacy practices. In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining personal information data private and secure. Such policies should be easily accessible by users, and should be updated as the collection and/or use of data changes. Personal information from users should be collected for legitimate and reasonable uses of the entity and not shared or sold outside of those legitimate uses. Further, such collection/sharing should occur after receiving the informed consent of the users. Additionally, such entities should consider taking any needed steps for safeguarding and securing access to such personal information data and ensuring that others with access to the personal information data adhere to their privacy policies and procedures. Further, such entities can subject themselves to evaluation by third parties to certify their adherence to widely accepted privacy policies and practices. In addition, policies and practices should be adapted for the particular types of personal information data being collected and/or accessed and adapted to applicable laws and standards, including jurisdiction-specific considerations. For instance, in the US, collection of or access to certain health data may be governed by federal and/or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); whereas health data in other countries may be subject to other regulations and policies and should be handled accordingly. Hence different privacy practices should be maintained for different personal data types in each country.
Despite the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and/or software elements can be provided to prevent or block access to such personal information data. For example, in the case of user recognition, the present technology can be configured to allow users to select to “opt in” or “opt out” of participation in the collection of personal information data during registration for services or anytime thereafter. In another example, users can select for voice triggers to be deactivated in certain situations, such as when a sensitive conversation is occurring. In addition to providing “opt in” and “opt out” options, the present disclosure contemplates providing notifications relating to the access or use of personal information. For instance, a user may be notified upon downloading an app that their personal information data will be accessed and then reminded again just before personal information data is accessed by the app.
Moreover, it is the intent of the present disclosure that personal information data should be managed and handled in a way to minimize risks of unintentional or unauthorized access or use. Risk can be minimized by limiting the collection of data and deleting data once it is no longer needed. In addition, and when applicable, including in certain health related applications, data de-identification can be used to protect a user's privacy. De-identification may be facilitated, when appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of data stored (e.g., collecting location data a city level rather than at an address level), controlling how data is stored (e.g., aggregating data across users), and/or other methods.
Therefore, although the present disclosure broadly covers use of personal information data to implement one or more various disclosed embodiments, the present disclosure also contemplates that the various embodiments can also be implemented without the need for accessing such personal information data. That is, the various embodiments of the present technology are not rendered inoperable due to the lack of all or a portion of such personal information data. For example, speech enabled actions may be undertaken without advanced speech content analysis based on non-personal information data or a bare minimum amount of personal information, such as the content being requested by the device associated with a user, other non-personal information available to the speech processor, or publicly available information.
While certain aspects have been described and shown in the accompanying drawings, it is to be understood that such are merely illustrative of and not restrictive on the broad invention, and that the invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those of ordinary skill in the art. For example, while
