The present invention is related to audio signal processing and more specifically to system and method of adaptive multi-microphone beamforming to enhance speech/audio far-field pickup.
It is quite natural for human beings to use their own voices as an effective means of communication. Indeed children start to use their voices long before they develop other communication skills, such as reading or writing. The broad adoption of mobile devices is another example that demonstrates the proliferation and importance of voice enabled communications throughout the modern world.
Telephony applications have progressed through a long evolution from wired devices to wireless mobile units, and from operator assisted calls, to fully automated end-to-end user calls across the globe. Increasingly, users appreciate the flexibility and freedom afforded by modern telecommunication devices and services. Another step to further this evolution is to completely liberate users' hands from the operation of their mobile communication devices. The use of hands-free modes for phone calls is not only convenient in many situations, but is often required and frequently enforced by the law, for example, as is the case when using mobile phones while driving.
Another rapidly growing technological area that is currently gaining enormous momentum is the vast array of smart or connected devices (also referred to as the Internet of Things or “IoT”, that can be installed almost anywhere including residential homes, office buildings, public spaces, transportation vehicles, and even implanted in human beings. These devices generally include sensors, actuators and the like, and are connected to the Web or other cloud-based services and/or to each other in some fashion. Some residential examples include audio/video equipment, thermostats, appliances, and lighting. IoT devices can be designed and manufactured to respond to voice commands in order to provide increased flexibility and freedom to users.
Major problems that must be overcome when implementing hands-free communications or voice controlled devices are inefficiencies due to the inherent nature of sound waves that degrade when propagating through the air. Specifically, because the strength or intensity of sound waves is inversely proportional to the square of the distance from the source, it becomes increasingly difficult to achieve acceptable results the further away a user is from the input device or microphone.
When a user holds a phone close to his or her mouth, it is not difficult to achieve a sufficiently high signal to noise ratio (SNR), and thus produce acceptable results for voice recognition or noise reduction applications, even in a noisy environment. For example, the volume level of normal speech (as measured close to the human mouth) is approximately 85 dB(A). A background noise level of 70 dB(A) is generally considered a noisy environment, such as a crowded restaurant or bar. This example leads to a SNR of 15 dB, which is large enough to achieve acceptable results for most applications. Examples of such applications include voice recognition accuracy for a voice-controlled device, or a typical noise suppression module for a high quality telephony call.
However, if the user moves only three meters away from the microphone, and still speaks at the same volume, the strength of his or her voice (as measured at the microphone) would now be reduced to around 55 dB(A). Thus, even with a much lower noise level of 50 dB(A), (a level in which most users would describe as quiet), the resulting SNR is only 5 dB, which makes it extremely difficult for applications to produce acceptable results.
In order to mitigate this issue, it is a common industry practice to use multiple microphones, or a microphone array, combined with advanced techniques such as beamforming, to enhance the SNR to produce better results. Traditional beamforming techniques use a “Delay-Sum” approach, which analyze a talker's voice arrival time at each microphone, delays early-arrived speech signals, aligns each of the signals with the latest arrival speech signal, and finally sums up all of the speech signals to create a maximum correlated output speech signal. While this approach is simple and effective, it requires accurate tracking of the user's location relative to the microphones or microphone array to determine the angle of arrival of the speech signals. Errors in determining the user's location relative to the microphones will quickly diminish the beamforming gains, resulting in rapid speech level variations.
Persons skilled in the art would appreciate that, while techniques exist for determining a user's location fairly accurately using multiple microphone inputs, it is nonetheless a very challenging task when ambient noises are present, especially at low SNR conditions. Also, when a user moves around rapidly, such as when walking back and forth inside a home for example, timely and accurate detection of the user's location represents another challenge.
Another difficulty with traditional approaches is that due to design constraints and the like, multiple microphones are not necessarily aligned in a straight line. This makes the estimation of the talker's location even more difficult to calculate and therefore further limits the applicability of traditional methods.
Thus, in order to resolve the limitations of conventional methods and systems and to improve user experience, the present invention provides an adaptive multi-microphone beamforming technique that does not require calculations for the user's location or the direction of arrival of audio signals. In addition, the present invention provides an additional benefit of allowing arbitrary placement of microphones in products without impacting the beamforming performance.
The present invention may be described herein in terms of functional block components and various processing steps. It should be appreciated that such functional blocks may be realized by any number of hardware components or software elements configured to perform the specified functions. For example, the present invention may employ various integrated circuit components, e.g., memory elements, digital signal processing elements, logic elements, look-up tables, and the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices. In addition, those skilled in the art will appreciate that the present invention may be practiced in conjunction with any number of data and voice transmission protocols, and that the system described herein is merely one exemplary application for the invention.
It should be appreciated that the particular implementations shown and described herein are illustrative of the invention and its best mode and are not intended to otherwise limit the scope of the present invention in any way. Indeed, for the sake of brevity, conventional techniques for signal processing, data transmission, signaling, packet-based transmission, network control, and other functional aspects of the systems (and components of the individual operating components of the systems) may not be described in detail herein, but are readily known by skilled practitioners in the relevant arts. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent exemplary functional relationships and/or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may be present in a practical communication system.
Referring now to
Referring now to
Thus, because the two noise signals 321 and 322 remain uncorrelated, their sum does not create a 2× sample value effect in the output signal 330, as does the voice signal from the talker 301. Therefore, the two uncorrelated noise signals added together is simply a noise energy increase of 2, and a noise level increase of 3 d B.
In an ideal case, with a speech signal energy level increase of 6 dB, and a noise level increase of 3 dB, the maximum gain of a two-microphone based delay-sum beamforming approach is 3 dB SNR. However, as previously mentioned, this traditional method requires extremely accurate knowledge regarding the location of the talker in order to calculate the exact time delay required to create a perfectly correlated speech signal. As would be appreciated by persons skilled in the art, it is often very difficult to accurately and precisely detect a talker's location. When such location information is not accurate or unavailable, the performance of such traditional beamforming systems and methods are dramatically reduced as is often the case when a talker is not stationary.
Another difficulty with traditional delay-sum beamforming is that, due to design constraints, such as required product size, and other form factor considerations, multiple microphones are not necessarily aligned in a straight line. This makes the estimation of the talker's location even more difficult to calculate and therefore further limits the applicability of traditional methods. These types of problems are illustrated in
As shown by the examples depicted in
The present invention alleviates the problems found in traditional microphone beamforming methods and systems by not requiring any determination of the direction of arrival of the audio sources. Further, because the orientation of the device and the placement of the microphones are irrelevant, the present invention works equally well under all conditions and may be implemented with less complexity than traditional methods.
In general, as stated by the above-referenced article, an adaptive filter is a filter that self adjusts its transfer function according to an optimizing algorithm. It adapts the performance based on the input signal. Such filters incorporate algorithms that allow the filter coefficients to adapt to the signal statics. Adaptive techniques use algorithms, which enable the adaptive filter to adjust its parameters to produce an output that matches the output of an unknown system. This algorithm employs an individual convergence factor that is updated for each adaptive filter coefficient at each iteration.
As shown in
Referring back now to
The audio signal from the second microphone input 604 is digitized by the A/D converter 608 to become the second input speech signal 682, and is the input to the adaptive prediction module 670. The prediction result 692 is subtracted from the reference signal 691 to obtain the prediction error 693. This prediction error 693 is then used to drive the adaptive prediction module 670, which acts to minimize the prediction error as an objective for the adaptation. The sum of the first input speech signal 681 and the prediction result signal 692 forms the desired output signal 680, which is output to an output device such as a speaker, headphones or the like. Adding such highly correlated signals together results in an output signal 692 with an approximate amplification of 2×.
Please note that in the examples used herein, speech signals are used as examples (such as input speech signals 681 and 682) of the desired type of signals that are enhanced by an embodiment of the present invention. However, in other embodiments, any type of audio signal can be enhanced by the improved techniques described herein, such as music signals and the like, without departing from the scope and breadth of the present invention.
In
Similarly, the digitized second input speech signal 722 is used as reference for the first adaptive prediction module 771, which takes the digitized first input speech signal 721 as input to produce an optimized prediction result signal 731 that minimizes the prediction error between the reference signal 722 and the prediction result signal 731. The sum of 731 and 722 forms the second enhanced signal 741.
The second enhanced signal 741 is used as the reference signal for a second level of prediction according to an example embodiment of the present invention. The first enhanced signal 742 is input to the third adaptive prediction module 773 that produces an optimized prediction result 733 by minimizing the prediction error between second enhanced signal 741 and the prediction result 733. Finally, the sum of 741 and 733 is the desired output signal 798, with is subsequently output to an output audio device.
It should be noted that in this example embodiment, it is assumed that there is a high level of consistency between the first input signal 722 and the second input signal 721. As such, in this example, the second enhanced signal 741 is selected to act as the reference signal to the third adaptive prediction module 773. Indeed, in most cases, were the microphones that comprise the microphone array are closely spaced relative to each other, this consistency is expected. However, in order to minimize any negative effects from inconsistent inputs and to maximize the performance of the present invention, another stage may be added to the embodiment shown in
As shown in
In this example, the better or stronger single is detected in the first step 702, for example, the signal with the highest energy, or other criteria as discussed above is identified in the first step 702. Once this determination is made, the better signal is used as the reference signal and the other signal or weaker signal, is used as the input signal to the third adaptive prediction module 773. In particular, in step 702, if it is determined that signal 742 is better than 741, then as shown in step 704, the signal 742 is used as the reference signal and the signal 741 is used as the input signal to the adaptive prediction module 773. Similarly, if the Signal 741 is better than (or equal to) 742, then as shown in step 703, the signal 741 is the reference signal and the signal 742 is the input signal to the adaptive prediction module 773. In practice, if the signals are equivalent and neither one is better or stronger than the other, than it makes no difference which signal is used as the reference signal and which signal is used as the input signal.
In yet another embodiment of the present invention, this technique of
The digitized second microphone input is the input speech signal 872 that is the input to the second adaptive prediction module 878. Adaptive prediction module 878 functions to minimize the prediction error signal 894 between the reference signal 851 and the prediction result 882. As shown and indicated by the ellipses in
Finally, the sum of the first input speech signal 831 (also the reference signal), and each of the prediction result signals associated with each of the N−1 adaptive prediction filter modules, (such as those shown in 882 and 883), form the desired output signal 898, which is output to an output device.
In yet another embodiment of the present invention, the technique of
The present invention may be implemented using hardware, software or a combination thereof and may be implemented in a computer system or other processing system. Computers and other processing systems come in many forms, including wireless handsets, portable music players, infotainment devices, tablets, laptop computers, desktop computers and the like. In fact, in one embodiment, the invention is directed toward a computer system capable of carrying out the functionality described herein. An example computer system 901 is shown in
Computer system 901 also includes a main memory 906, preferably random access memory (RAM), and can also include a secondary memory 908. The secondary memory 908 can include, for example, a hard disk drive 910 and/or a removable storage drive 912, representing a magnetic disc or tape drive, an optical disk drive, etc. The removable storage drive 912 reads from and/or writes to a removable storage unit 914 in a well-known manner. Removable storage unit 914, represent magnetic or optical media, such as disks or tapes, etc., which is read by and written to by removable storage drive 912. As will be appreciated, the removable storage unit 914 includes a computer usable storage medium having stored therein computer software and/or data.
In alternative embodiments, secondary memory 908 may include other similar means for allowing computer programs or other instructions to be loaded into computer system 901. Such means can include, for example, a removable storage unit 922 and an interface 920. Examples of such can include a USB flash disc and interface, a program cartridge and cartridge interface (such as that found in video game devices), other types of removable memory chips and associated socket, such as SD memory and the like, and other removable storage units 922 and interfaces 920 which allow software and data to be transferred from the removable storage unit 922 to computer system 901.
Computer system 901 can also include a communications interface 924. Communications interface 924 allows software and data to be transferred between computer system 901 and external devices. Examples of communications interface 924 can include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, etc. Software and data transferred via communications interface 924 are in the form of signals which can be electronic, electromagnetic, optical or other signals capable of being received by communications interface 924. These signals 926 are provided to communications interface via a channel 928. This channel 928 carries signals 926 and can be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, such as WiFi or cellular, and other communications channels.
In this document, the terms “computer program medium” and “computer usable medium” are used to generally refer to media such as removable storage device 912, a hard disk installed in hard disk drive 910, and signals 926. These computer program products are means for providing software or code to computer system 901.
Computer programs (also called computer control logic or code) are stored in main memory and/or secondary memory 908. Computer programs can also be received via communications interface 924. Such computer programs, when executed, enable the computer system 901 to perform the features of the present invention as discussed herein. In particular, the computer programs, when executed, enable the processor 904 to perform the features of the present invention. Accordingly, such computer programs represent controllers of the computer system 901.
In an embodiment where the invention is implemented using software, the software may be stored in a computer program product and loaded into computer system 901 using removable storage drive 912, hard drive 910 or communications interface 924. The control logic (software), when executed by the processor 904, causes the processor 904 to perform the functions of the invention as described herein.
In another embodiment, the invention is implemented primarily in hardware using, for example, hardware components such as application specific integrated circuits (ASICs). Implementation of the hardware state machine so as to perform the functions described herein will be apparent to persons skilled in the relevant art(s).
In yet another embodiment, the invention is implemented using a combination of both hardware and software.
While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.
The present application for patent claims priority to Provisional Application No. 62/380,372 entitled “Adaptive Multi-Microphone Beamforming” filed on Aug. 27, 2016 by Dr. Huan-yu Su. The above-referenced Provisional application is incorporated herein by reference as if set forth in full.
Number | Name | Date | Kind |
---|---|---|---|
6738482 | Jaber | May 2004 | B1 |
6781521 | Gardner | Aug 2004 | B1 |
6983055 | Luo | Jan 2006 | B2 |
6999541 | Hui | Feb 2006 | B1 |
7289586 | Hui | Oct 2007 | B2 |
7346175 | Hui | Mar 2008 | B2 |
7426464 | Hui | Sep 2008 | B2 |
7706549 | Zhang | Apr 2010 | B2 |
7720233 | Sato | May 2010 | B2 |
8195246 | Vitte | Jun 2012 | B2 |
8374358 | Buck | Feb 2013 | B2 |
9313573 | Schuldt | Apr 2016 | B2 |
20030053639 | Beaucoup | Mar 2003 | A1 |
20030139851 | Nakadai | Jul 2003 | A1 |
20060015331 | Hui | Jan 2006 | A1 |
20090214054 | Fujii | Aug 2009 | A1 |
20110096941 | Marzetta | Apr 2011 | A1 |
20140126745 | Dickins | May 2014 | A1 |
20150099500 | Chalmers | Apr 2015 | A1 |
Entry |
---|
Takuto Yoshioka et al., “Speech Separation Microphone Array Based on Law of causality and Frequency Domain Processing”, ISCIT 2009, pp. 697-702. |
Number | Date | Country | |
---|---|---|---|
62380372 | Aug 2016 | US |