The present disclosure relates to processing audio signals and, more particularly, to methods and devices for pre-processing audio signals.
Although speech recognition has been around for decades, the quality of speech recognition software and hardware has only recently reached a high enough level to appeal to a large number of consumers. One area in which speech recognition has become very popular in recent years is the smartphone and tablet computer industry. Using a speech recognition-enabled device, a consumer can perform such tasks as making phone calls, writing emails, and navigating with GPS, strictly by voice.
Speech recognition in such devices is far from perfect, however. When using a speech recognition-enabled device for the first time, the user may need to “train” the speech recognition software to recognize his or her voice. Even after training, however, the speech recognition functions may not work well in all sound environments. For example, the presence of background noise can decrease speech recognition accuracy.
While the appended claims set forth the features of the present techniques with particularity, these techniques may be best understood from the following detailed description taken in conjunction with the accompanying drawings of which:
In accordance with the foregoing, a method and apparatus for pre-processing audio signals will now be described.
According to an embodiment, an electronic device is able to select a pre-processing technique that is suited to the environment under which the device is operating. In doing so, the device enhances speech recognition accuracy. In one implementation, the device uses information obtained from the audio signal itself, and information obtained from one or more auxiliary devices.
The device is able to select from any of a number of pre-processing techniques (e.g., single microphone noise suppression, two microphone noise suppression, adaptive noise cancellation) and apply the selected technique to the audio input signal of the device. The selection of the appropriate pre-processing technique may depend on the level of background noise as well as the characteristics of the background noise (e.g., variability, spectral shape, etc.)
One or more auxiliary devices, according to an embodiment, provide additional information on which the pre-processing procedure selection may be made. For example, a Global Positioning Signal (GPS) module can provide information about the location of the device, whether the device is in motion, and its velocity. From the location and velocity of the device, clues about the level of background noise and characteristics of the background noise can be garnered. For example, the device may be located in a quiet home environment, a busy restaurant, a city street, or a highway. It may be stationary or moving at 60 mph. Based on the location and velocity of the device, information about the noise level and noise characteristics can be inferred using prior knowledge (e.g., lookup tables of stored noise levels and characteristics) under similar conditions. Such information can then be used to select the appropriate pre-processing technique for the input signal and thereby enhance the speech recognition performance.
In an embodiment, an electronic device receives an audio signal that has audio information, obtains auxiliary information (such as location, velocity, direction, light, and temperature), and determines, based on the audio information and the auxiliary information, a type of audio environment in which the electronic device is operating. The device selects an audio pre-processing procedure based on the determined audio environment type and pre-processes the audio signal according to the selected pre-processing procedure. The device may then perform speech recognition on the pre-processed audio signal.
Possible implementations for the pre-processing procedure include straight-through signal transmission, single microphone noise suppression, two microphone noise suppression, and adaptive noise cancellation.
In an embodiment, determining the type of audio environment involves determining whether the device is being operated in a vehicle, in a home, in a restaurant, in an office, or on a street.
As used herein, the “audio environment” of a device means the characteristics of the sounds audible to the device other than the sound of the user's speech. Background noise is part of the audio environment.
A “module” as used herein is software executing on hardware. A module may execute on multiple hardware elements or on a single one. Furthermore, when multiple modules are depicted in the figures, it is to be understood that the modules may, in fact, all be executing on the same device and in the same overall unit of software.
When the current disclosure refers to modules and other elements “providing” information (data) to one another, it is to be understood that there are a variety of possible ways such action may be carried out, including electrical signals being transmitted along conductive paths (e.g., wires) and inter-object method calls.
Some of the embodiments described herein are usable in the context of always-on audio (AOA). When using AOA, the device 102 (
Referring to
Although
The device 102 is intended to be representative of a variety of devices including, for example, cellular telephones, personal digital assistants (PDAs), smart phones, or other handheld or portable electronic devices. In alternate embodiments, the device can also be a headset (e.g., a Bluetooth headset), MP3 player, battery-powered device, a watch device (e.g., a wristwatch) or other wearable device, radio, navigation device, laptop or notebook computer, netbook, pager, PMP (personal media player), DVR (digital video recorders), gaming device, camera, e-reader, e-book, tablet device, navigation device with video capable screen, multimedia docking station, or other device.
Embodiments of the present disclosure are intended to be applicable to any of a variety of electronic devices that are capable of or configured to receive voice input or other sound inputs that are indicative or representative of vocalized information.
Further, in the embodiment of
By contrast, the Wi-Fi transceiver 205 is a wireless local area network (WLAN) transceiver 205 configured to conduct Wi-Fi communications in accordance with the IEEE 802.11 (a, b, g, or n) standard with access points. In other embodiments, the Wi-Fi transceiver 205 can instead (or in addition) conduct other types of communications commonly understood as being encompassed within Wi-Fi communications such as some types of peer-to-peer (e.g., Wi-Fi Peer-to-Peer) communications. Further, in other embodiments, the Wi-Fi transceiver 205 can be replaced or supplemented with one or more other wireless transceivers configured for non-cellular wireless communications including, for example, wireless transceivers employing ad hoc communication technologies such as HomeRF (radio frequency), Home Node B (3G femtocell), Bluetooth and/or other wireless communication technologies such as infrared technology.
Although in the present embodiment the device 102 has two of the wireless transceivers 202 (that is, the transceivers 203 and 205), the present disclosure is intended to encompass numerous embodiments in which any arbitrary number of wireless transceivers employing any arbitrary number of communication technologies are present. By virtue of the use of the wireless transceivers 202, the device 102 is capable of communicating with any of a variety of other devices or systems (not shown) including, for example, other mobile devices, web servers, cell towers, access points, other remote devices, etc. Depending upon the embodiment or circumstance, wireless communication between the device 102 and any arbitrary number of other devices or systems can be achieved.
Operation of the wireless transceivers 202 in conjunction with others of the internal components 200 of the device 102 can take a variety of forms. For example, operation of the wireless transceivers 202 can proceed in a manner in which, upon reception of wireless signals, the internal components 200 detect communication signals and the transceivers 202 demodulate the communication signals to recover incoming information, such as voice and/or data, transmitted by the wireless signals. After receiving the incoming information from the transceivers 202, the processor 204 formats the incoming information for the one or more output devices 208. Likewise, for transmission of wireless signals, the processor 204 formats outgoing information, which can but need not be activated by the input devices 210, and conveys the outgoing information to one or more of the wireless transceivers 202 for modulation so as to provide modulated communication signals to be transmitted.
Depending upon the embodiment, the input and output devices 208, 210 of the internal components 200 can include a variety of visual, audio and/or mechanical outputs. For example, the output device(s) 208 can include one or more visual output devices 216 such as a liquid crystal display and/or light emitting diode indicator, one or more audio output devices 218 such as a speaker, alarm, and/or buzzer, and/or one or more mechanical output devices 220 such as a vibrating mechanism. The visual output devices 216 among other things can also include a video screen. Likewise, by example, the input device(s) 210 can include one or more visual input devices 222 such as an optical sensor (for example, a camera lens and photosensor), one or more audio input devices 224 such as the microphone 108 of
As mentioned above, the internal components 200 also can include one or more of various types of sensors 228 as well as a sensor hub to manage one or more functions of the sensors. The sensors 228 may include, for example, proximity sensors (e.g., a light detecting sensor, an ultrasound transceiver or an infrared transceiver), touch sensors, altitude sensors, and one or more location circuits/components that can include, for example, a Global Positioning System (GPS) receiver, a triangulation receiver, an accelerometer, a tilt sensor, a gyroscope, or any other information collecting device that can identify a current location or user-device interface (carry mode) of the device 102. Although the sensors 228 for the purposes of
The memory portion 206 of the internal components 200 can encompass one or more memory devices of any of a variety of forms (e.g., read-only memory, random access memory, static random access memory, dynamic random access memory, etc.), and can be used by the processor 204 to store and retrieve data. In some embodiments, the memory portion 206 can be integrated with the processor 204 in a single device (e.g., a processing device including memory or processor-in-memory (PIM)), albeit such a single device will still typically have distinct portions/sections that perform the different processing and memory functions and that can be considered separate devices. In some alternate embodiments, the memory portion 206 of the device 102 can be supplemented or replaced by other memory portion(s) located elsewhere apart from the mobile device and, in such embodiments, the mobile device can be in communication with or access such other memory device(s) by way of any of various communications techniques, for example, wireless communications afforded by the wireless transceivers 202, or connections via the component interface 212.
The data that is stored by the memory portion 206 can include, but need not be limited to, operating systems, programs (applications), modules, and informational data. Each operating system includes executable code that controls basic functions of the device 102, such as interaction among the various components included among the internal components 200, communication with external devices via the wireless transceivers 202 and/or the component interface 212, and storage and retrieval of programs and data, to and from the memory portion 206. As for programs, each program includes executable code that utilizes an operating system to provide more specific functionality, such as file system service and handling of protected and unprotected data stored in the memory portion 206. Such programs can include, among other things, programming for enabling the device 102 to perform a process such as the process for speech recognition shown in
Referring to
Referring still to
The database 312 has one or more data structures that associate different sets of sensory and audio data with different types of audio environments. These data structures may include, for example, one or more lookup tables that contain locations and audio environments that correspond to the locations. Such a lookup table may be created through testing under similar audio environments.
The GPS module 316 receives a GPS signal and determines the location of the device 300 based on the received signal. The GPS module 316 provides information regarding the determined location (“location data”) to the audio environment determination module 308.
The motion sensor 318 senses the motion of the device 300, such as the device 300's acceleration, velocity, and direction. The motion sensor 318 provides the data regarding the sensed motion (“motion data”) to the audio environment determination module 308. In some embodiments, the motion sensor 318 determines the motion of the device 300 and provides the motion data in the form of the appropriate units of distance, speed, etc. In other embodiments, the motion data is raw, in which case the audio environment determination module determines the motion of the device 300 based on the raw data.
The optical sensor 320 senses the light in the vicinity of the device 300 and provides the information regarding the sensed light (“light data”) such as level, color, and images, to the audio environment determination module 308. The optical sensor 320 may include a photo sensor, photo detector, image sensor, or other suitable device.
The temperature sensor 323 may include a thermistor or other similar device. The temperature sensor senses the temperature in the vicinity of the device 300 and provides information regarding the temperature (“temperature data”) to the audio environment determination module 308.
The proximity sensor 327 senses the presence of objects (including people and materials) in the vicinity of the device 300 and provides information regarding this presence (“proximity data”) to the audio environment determination module 308.
The other auxiliary devices 324 gather other auxiliary information and provide this information to the audio environment determination module 308.
The device 300 also includes a set 325 of pre-processors, including first pre-processor 326, a second pre-processor 328, and a third pre-processor 330. The device 300 may also include other pre-processors, represented by a fourth pre-processor 334.
Each of the pre-processors of the set 325 carries out a pre-processing procedure. Possible pre-processor procedures include a one-mic noise suppression procedure, a two-mic noise suppression procedure, and an adaptive noise cancellation procedure. For example, the first pre-processor 326 could carry out a one-mic noise suppression procedure, the second pre-processor 328 could carry out a two-mic noise suppression procedure, and the third pre-processor 330 could carry out an adaptive noise cancellation procedure. The fourth preprocessor 334 could carry out some combination of the first, second, and third preprocessors 326, 328, and 330. As will be discussed, it is possible that the audio signal does not undergo pre-processing at all.
The device 300 further includes a speech recognition module 336 that converts recognized speech signals to text, or carries out the appropriate action in response to the recognized speech or text.
The audio environment determination module 308 receives the audio information from the signal processing and analysis module 304, and receives the auxiliary information from the set 314 of auxiliary devices. The audio environment determination module 308 processes the audio information and the auxiliary information. Using the processed auxiliary information, the audio environment determination module 308 queries the database 312 and receives a response. The audio environment determination module 308 combines the query response with the audio information (received from the signal processing and analysis module 304) to obtain an audio environment type. The audio environment determination module 308 provides data regarding the audio environment type to the pre-processor selection module 310.
Using audio environment type data, the pre-processor selection module 310 determines which pre-processing method will most enhance the ability of the speech recognition module 336 to recognize speech. From the set 325, the pre-processor selection module 310 selects the pre-processor associated with the determined pre-processing method.
The pre-processor selected by the pre-processor selection module 310 pre-processes the input signal and provides the pre-processed signal to the signal recognition module 336. Based on the pre-processed signal, the speech recognition module 336 determines whether the sound constitutes one or more spoken words. If the sound does, the speech recognition module 336 provides the spoken word or words to one or more applications, represented by the application 338 of
In one embodiment, the device 300 is capable of carrying out a trigger procedure, in which the device 300 is in a dormant, low-power mode, but is continuously monitoring for trigger words, such as “wake up.” In such an embodiment, the speech recognition module 336 operates in a minimal mode in which it does not react to audio signals until a trigger command is detected. When the speech recognition module 336 detects a trigger command, the speech recognition module 336 sends a message to one or more applications 338. The application 338 in this example may be a method that the operating system calls in order to take the device 300 out of sleep mode.
The ways in which the audio environment determination module 310 uses the auxiliary information to determine the audio environment of the device 300 according to various embodiments of the invention will now be described. It is to be understood that audio environment determination module 310 may not necessarily receive, nor need to receive, data from all of the auxiliary devices of the device 300. Also, the device 300 may only have a subset of the set 314 of auxiliary devices.
The GPS module 316 provides location data to the audio environment determination module 308. The audio environment determination module 308 may determine the audio environment of the device 300 at least in part on the location data. In one embodiment, the audio environment determination module 308 has access to map software/service (such as Google Maps, ©2013 Google) and is able to query the map software/service to determine the address at which the device 300 is located and the type of business at that address. For example, if the audio environment determination module 308 queries the map service with the GPS coordinates and receives the address of a restaurant, the audio environment determination module 308 is likely to conclude that the audio environment is “restaurant.”
The audio environment determination module 308 may also use the location information to determine the velocity of the device 300. In particular, the audio environment determination module 308 receives location data updates from the GPS module 316 at regular intervals, and determines the change in location of the device 300 over time. The audio environment determination module 308 determines, based on the location change determination, the velocity of the device 300. The audio environment determination module 308 may make this velocity determination to determine the audio environment of the device 300. For example, if the audio environment determination module 308 determines that the device 300 is moving more than 20 miles per hour, the audio environment determination module 308 may determine that the device 300 is in a moving vehicle.
The motion sensor 318 provides motion data to the audio environment determination module 308. The audio environment determination module 308 may determine the audio environment of the device 300 based at least in part on the motion data. In one embodiment, the audio environment determination module uses the motion data as a supplement to the location data. In an embodiment, the audio environment determination module 308 uses the location data to determine a starting point for the device 300, and determines, based on the motion data and the starting location, the current location at each time interval. The audio environment determination module 308 then determines an audio environment type based at least in part on the current location of the device 300. This may be done in the same manner as location data received solely from the GPS module 316, which has been previously discussed.
The light sensor 320 provides data regarding the level of illumination (“light data”) to the audio environment determination module 308. The audio environment determination module 308 may determine the audio environment of the device 300 based at least in part on the light data. In one embodiment, the audio environment determination module 308 uses the light data to determine whether the device 300 is indoors, outdoors, or stored away. For example, if the light level is very low, then the audio environment determination module may determine that device 300 is stored away. If the light level is high, then the audio environment determination module may determine that device 300 is outdoors. If is the light level is moderate, then the audio environment determination module may determine that device 300 is indoors.
The temperature sensor 323 provides temperature data to the audio environment determination module 308. The audio environment determination module 308 may determine the audio environment of the device 300 based at least in part on the temperature data. In one embodiment, the audio environment determination module 308 uses the temperature data to determine whether the device 300 is indoors or outdoors. For example, if the temperature is moderate, then the audio environment determination module may determine that device 300 is indoors. If the temperature is high or low, then the audio environment determination module 308 may determine that device 300 is outdoors.
The proximity sensor 327 provides proximity data to the audio environment determination module 308. The audio environment determination module 308 may determine the audio environment of the device 300 based at least in part on the proximity data. In one embodiment, the audio environment determination module 308 uses the proximity data to determine whether the device 300 is stowed (e.g., in a purse) or not. For example, if the proximity data indicates that there are objects all around the device 300, then the audio environment determination module 308 may determine that device 300 is stowed.
Referring to
It can be seen from the foregoing that a method and apparatus for pre-processing audio signals has been provided. In view of the many possible embodiments to which the principles of the present discussion may be applied, it should be recognized that the embodiments described herein with respect to the drawing figures are meant to be illustrative only and should not be taken as limiting the scope of the claims. Therefore, the techniques as described herein contemplate all such embodiments as may come within the scope of the following claims and equivalents thereof
The present application claims the benefit of the filing date of U.S. Provisional Application No. 61/776,793, filed Mar. 12, 2013, the entire contents of which are incorporated by reference; U.S. Provisional Application No. 61/798,097, filed Mar. 15, 2013, the entire contents of which are incorporated by reference; and U.S. Provisional Application No. 61/819,960, filed May 6, 2013, the entire contents of which are incorporated by reference.
Number | Date | Country | |
---|---|---|---|
61776793 | Mar 2013 | US | |
61798097 | Mar 2013 | US | |
61819960 | May 2013 | US |