Patent Application
20100008518
1. Field of the Invention
This invention relates generally to audio processing and more particularly to a microphone array system capable of tracking an audio signal from a particular source while filtering out signals from other competing or interfering sources.
2. Description of the Related Art
Voice input systems are typically designed as a microphone worn near the mouth of the speaker where the microphone is tethered to a headset. Since this imposes a physical restraint on the user, i.e., having to wear the headset, users will typically use the headset for only a substantial dictation and rely on keyboard typing for relatively brief input and computer commands in order to avoid wearing the headset.
Video game consoles have become a commonplace item in the home. The video game manufacturers are constantly striving to provide a more realistic experience for the user and to expand the limitations of gaming, e.g., on line applications. For example, the ability to communicate with additional players in a room having a number of noises being generated, or even for users to send and receive audio signals when playing on-line games against each other where background noises and noise from the game itself interferes with this communication, has so far prevented the ability for clear and effective player to player communication in real time. These same obstacles have prevented the ability of the player to provide voice commands that are delivered to the video game console. Here again, the background noise, game noise and room reverberations all interfere with the audio signal from the player.
As users are not so inclined to wear a headset, one alternative to the headset is the use of microphone arrays in order to capture the sound. However, shortcomings with the microphone arrays currently on the market today is the inability to track a sound from a moving source and/or the inability to separate the source sound from the reverberation and environmental sounds from the general area being monitored. Additionally, with respect to a video game application, a user will move around relative to the fixed positions of the game console and the display monitor. Where a user is stationary, the microphone array may be able to be “factory set” to focus on audio signals emanating from a particular location or region. For example, inside an automobile, the microphone array may be configured to focus around the driver's seat region for a cellular phone application. However, this type of microphone array is not suitable for a video game application. That is, a microphone array on the monitor or game console would not be able to track a moving user, since the user may be mobile, i.e., not stationary, during a video game. Furthermore, a video game application, a microphone array on the game controller is also moving relative to the user. Consequently, for a portable microphone array, e.g., affixed to the game controller, the source positioning poses a major challenge to higher fidelity sound capturing in selective spatial volumes.
Another issue with the microphone arrays and associated systems is the inability to adapt to high noise environments. For example, where multiple sources are contributing to an audio signal, the current systems available for consumer devices are unable to efficiently filter the signal from a selected source. It should be appreciated that the inability to efficiently filter the signal in a high noise environment only exacerbates the source positioning issues mentioned above. Yet another shortcoming of the microphone array systems is the lack of bandwidth for a processor to handle the input signals from each microphone of the array and track a moving user.
As a result, there is a need to solve the problems of the prior art to provide a microphone array that is capable of capturing an audio signal from a user when the user and/or the device to which the array is affixed are capable of changing position. There is also a need to design the system for robustness in a high noise environment where the system is configured to provide the bandwidth for multiple microphones sending input signals to be processed.
Broadly speaking, the present invention fills these needs by providing a method and apparatus that defines a microphone array framework capable of identifying a source signal irrespective of the movement of microphone array or the origination of the source signal. It should be appreciated that the present invention can be implemented in numerous ways, including as a method, a system, computer readable medium or a device. Several inventive embodiments of the present invention are described below.
In one embodiment, a method for processing an audio signal received through a microphone array is provided. The method initiates with receiving a signal. Then, adaptive beam-forming is applied to the signal to yield an enhanced source component of the signal. Inverse beam-forming is also applied to the signal to yield an enhanced noise component of the signal. Then, the enhanced source component and the enhanced noise component are combined to produce a noise reduced signal.
In another embodiment, a method for processing an audio signal received through a microphone array coupled to an interfacing device is provided. The method is processing at least in part by a computing device that communicates with the interfacing device. The method includes receiving a signal at the microphone array and applying adaptive beam-forming to the signal to yield an enhanced source component of the signal. Also, an inverse beam-forming is applied to the signal to yield an enhanced noise component of the signal. The method combines the enhanced source component and the enhanced noise component to produce a noise reduced signal, where the noise reduced signal is a target voice signal. Then, monitoring an acoustic set-up associated with the audio signal as a background process using the adaptive beam-forming inverse beam-forming to track the target signal component, and periodically setting a calibration of the monitored acoustic set-up. The calibration implements blind source separation that uses second order statistics to separate the enhanced source component from the enhanced noise component, and the calibration remains fixed between the periodic setting. By executing this method, the target signal is able to freely move around relative to the microphone array of the interface device.
In yet another embodiment, a computer readable medium having program instructions for processing an audio signal received through a microphone array is provided. The computer readable medium includes program instructions for receiving a signal and program instructions for applying adaptive beam-forming to the signal to yield an enhanced source component of the signal. Program instructions for applying inverse beam-forming to the signal to yield an enhanced noise component of the signal are included. Program instructions for combining the enhanced source component and the enhanced noise component to produce a noise reduced signal are provided.
In still yet another embodiment, a computer readable medium having program instructions for reducing noise associated with an audio signal is provided. The computer readable medium includes program instructions for enhancing a target signal associated with a listening direction through a first filter and program instructions for blocking the target signal through a second filter. Program instructions for combining an output of the first filter and an output of the second filter in a manner to reduce noise without distorting the target signal are provided. Program instructions for periodically monitoring an acoustic set up associated with the audio signal are included. Program instructions for calibrating both the first filter and the second filter based upon the acoustic setup are provided.
In another embodiment, a system capable of isolating a target audio signal from multiple noise sources is provided. The system includes a portable consumer device configured to move independently from a user. A computing device is included. The computing device includes logic configured enhance the target audio signal without constraining movement of the portable consumer device. A microphone array affixed to the portable consumer device is provided. The microphone array is configured to capture audio signals, wherein a listening direction associated with the microphone array is controlled through the logic configured to enhance the target audio signal.
In yet another embodiment, a video game controller is provided. The video game controller includes a microphone array affixed to the video game controller. The microphone array is configured to detect an audio signal that includes a target audio signal and noise. The video game controller includes circuitry configured to process the audio signal. Filtering and enhancing logic configured to filter the noise and enhance the target audio signal as a position of the video game controller and a position of a source of the target audio signal change is provided. Here, the filtering of the noise is achieved through a plurality of filter-and-sum operations.
An integrated circuit is provided. The integrated circuit includes circuitry configured to receive an audio signal from a microphone array in a multiple noise source environment. Circuitry configured to enhance a listening direction signal is included. Circuitry configured to block the listening direction signal, i.e., enhance a non listening direction signal, and circuitry configured to combine the enhanced listening direction signal and the enhanced non-listening direction signal to yield a noise reduced signal. Circuitry configured to adjust a listening direction according to filters computed through an adaptive array calibration scheme is included.
Other aspects and advantages of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.
The present invention will be readily understood by the following detailed description in conjunction with the accompanying drawings, and like reference numerals designate like structural elements.
An invention is described for a system, apparatus and method for an audio input system configured to isolate a source audio signal from a noisy environment in real time through an economic and efficient scheme. It will be obvious, however, to one skilled in the art, that the present invention may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.
The embodiments of the present invention provide a system and method for an audio input system associated with a portable consumer device through a microphone array. The voice input system is capable of isolating a target audio signal from multiple noise signals. Additionally, there are no constraints on the movement of the portable consumer device, which has the microphone array affixed thereto. The microphone array framework includes four main modules in one embodiment of the invention. The first module is an acoustic echo cancellation (AEC) module. The AEC module is configured to cancel portable consumer device generated noises. For example, where the portable consumer device is a video game controller, the noises, associated with video game play, i.e., music, explosions, voices, etc., are all known. Thus, a filter applied to the signal from each of the microphone sensors of the microphone array may remove these known device generated noises. In another embodiment, the AEC module is optional and may not be included with the modules described below. Further details on acoustic echo cancellation may be found in “Frequency-Domain and Multirate Adaptive Filtering” by John J. Shynk, IEEE Signal Processing Magazine, pp. 14-37, January 1992. This article is incorporated by reference for all purposes.
A second module includes a separation filter. In one embodiment, the separation filter includes a signal passing filter and a signal blocking filter. In this module, array beam-forming is performed to suppress a signal not coming from an identified listening direction. Both, the signal passing filter and the blocking filter are finite impulse response (FIR) filters that are generated through an adaptive array calibration module. The adaptive array calibration module, the third module, is configured to run in the background. The adaptive array calibration module is further configured to separate interference or noise from a source signal, where the noise and the source signal are captured by the microphone sensors of the sensor array. Through the adaptive array calibration module, as will be explained in more detail below, a user may freely move around in 3-dimensional space with six degrees of freedom during audio recording. Additionally, with reference to a video game application, the microphone array framework discussed herein, may be used in a loud gaming environment with background noises which may include, television audio signals, high fidelity music, voices of other players, ambient noise, etc. As discussed below, the signal passing filter is used by a filter-and-sum beam-former to enhance the source signal. The signal blocking filter effectively blocks the source signal and generates interferences or noise, which is later used to generate a noise reduced signal in combination with the output of the signal passing filter.
A fourth module, the adaptive noise cancellation module, takes the interferences from the signal blocking filter for subtraction from the beam-forming output, i.e., the signal passing filter output. It should be appreciated that adaptive noise cancellation (ANC) may be analogized to AEC with the exception that the noise templates for ANC are generated from the signal blocking filter of the microphone sensor array, instead of a video game console's output. In one embodiment, in order to maximize noise cancellation while minimizing target signal distorting, the interferences used as noise templates should prevent the source signal leakage that is covered by the signal blocking filter. Additionally, the use of ANC as described herein, enables the attainment of high interference-reduction performance with a relatively small number of microphones arranged in a compact region.
In one embodiment, an exemplary four-sensor based microphone array may be configured to have the following characteristics:
It should be appreciated that the microphone sensor array affixed to a video game controller may move freely in 3-D space with six degrees of freedom during audio recording. Furthermore, as mentioned above, the microphone sensor array may be used in extremely loud gaming environments which include multiple background noises, e.g., television audio signals, high-fidelity music signals, voices of other players, ambient noises, etc. Thus, the memory bandwidth and computational power available through a video game console in communication with the video game controller makes it possible for the console to be used as a general purpose processor to serve even the most sophisticated real-time signal processing applications. It should be further appreciated that the above configuration is exemplary and not meant to be limiting as any suitable geometry, sampling rate, number of microphones, type of sensor, etc., may be used.
The output of the microphone sensors 112-1 through 112-4 is processed through module 124 in order to isolate the source signal and provide output source signal 128b, which may be used as a voice command for a computing device or as communication between users. Module 124 includes acoustic echo cancellation module, adaptive beam-forming module, and adaptive noise cancellation module. Additionally, an array calibration module is running in the background as described below. As illustrated, module 124 is included in video game console 130. As will be explained in more detail below, the components of module 124 are tailored for a portable consumer device to enhance a voice signal in a noisy environment without posing any constraints on a controller's position, orientation, or movement. As mentioned above, acoustic echo cancellation reduces noise generated from the console's sound output, while adaptive beam-forming suppresses signals not coming from a listening direction, where the listening direction is updated through an adaptive array calibration scheme. The adaptive noise cancellation module is configured to subtract interferences from the beam-forming output through templates generated by a signal filter and a blocking filter associated with the microphone sensor array.
The fundamental idea behind beam-forming is that the sound signals from a desired source reaches the array of microphone sensors with different time delays. The geometry placement of the array being pre-calibrated, thus, the path-length-difference between the sound source and sensor array is a known parameter. Therefore, a process referred to as cross-correlation is used to time-align signals from different sensors. The time-align signals from various sensors are weighted according to the beam-forming direction. The weighted signals are then filtered in terms of sensor-specific noise-cancellation setup, i.e., each sensor is associated with a filter, referred to as a matched filter F1 FM, 142-1 through 142-M, which are included in signal-passing-filter 160. The filtered signals from each sensor are then summed together through module 172 to generate output Z(φ,θ). It should be appreciated that the above-described process may be referred to as auto-correlation. Furthermore, as the signals that do not lie along the beam-forming direction remain misaligned along the time axes, these signals become attenuated by the averaging. As is common with an array-based capturing system, the overall performance of the microphone array to capture sound from a desired spatial direction (using straight line geometry placement) or spatial volumes (using convex geometry array placement) depends on the ability to locate and track the sound source. However, in an environment with complicated reverberation noise, e.g., a videogame environment, it is practically infeasible to build a general sound location tracking system without integrating the environmental specific parameters.
Still referring to
One skilled in the art will appreciate that one method for performing the data mining is through independent component analysis (ICA) which analyzes the data and finds independent components through second order statistics in accordance with one embodiment of the invention. Thus, a second order statistic is calculated to describe or define the characteristics of the data in order to capture a sound fingerprint which distinguishes the various sounds. The separation filter is then enabled to separate the source signal from the noise signal. It should be appreciated that the computation of the sound fingerprint is periodically performed, as illustrated with reference to
Blocking filter 164 is configured to perform reverse beam-forming where the target signal is viewed as noise. Thus, blocking filter 164 attenuates the source signal and enhances noise. That is, blocking filter 164 is configured to determine a calibration coefficient F3 which may be considered the inverse of calibration coefficient F2 determined by the adaptive beam-forming process. One skilled in the art will appreciate that the adaptive array calibration referred to with reference to
In one embodiment, the microphone sensor array output signal is passed through a post-processing module to further refine the voice quality based on person-dependent voice spectrum filtering by Bayesian statistic modeling. Further information on voice spectrum filtering may be found in the article entitled “Speech Enhancement Using a Mixture-Maximum Model” by David Burshtein, IEEE Transactions on Speech and Audio Processing vol. 10, No. 6, September 2002. This article in incorporated by reference for all purposes. It should be appreciated that the signal processing algorithms mentioned herein are carried out in the frequency domain. In addition, a fast and efficient Fast Fourier transform (FFT) is applied to reach real time signal response. In one embodiment, the implemented software requires 25 FFT operations with window length of 1024 for every signal input chunk (512 signal samples in a 16 kHz sampling rate). In the exemplary case of a four-sensor microphone array with equally spaced straight line geometry, without applying acoustic echo cancellation and Bayesian model base voice spectrum filtering, the total computation involved is about 250 mega floating point operations (250M Flops).
Continuing with
In one embodiment, at a sampling rate of 16 kHz, approximately 30 blocks are used at the initialization in order to determine the calibration coefficients. Thus, in approximately two seconds from the start of the operation, the calibration coefficients will be available. Prior to the time that the calibration coefficients are available, a default value will be used for F2 and F3. In one embodiment, the default filter vector for F2 is a Linear-Phase All-Pass FIR, while the default value for F3 is −F2.
The method then proceeds to operation 214 where the output of the first filter and the output of the second filter are combined in a manner to reduce noise without distorting the target signal. As discussed above, the combination of the first filter and the second filter is achieved through adaptive noise cancellation. In one embodiment, the output of the second filter is aligned prior to combination with the output of the first filter. The method then moves to operation 216 where an acoustic set-up associated with the audio signal is periodically monitored. Here, the adaptive array calibration discussed above may be executed. The acoustic set-up refers to the position change of a portable consumer device having a microphone sensor array and the relative position to a user as mentioned above. The method then advances to operation 218 where the first filter and the second filter are calibrated based upon the acoustic setup. Here, filters F2 and F3, discussed above, are determined and applied to the signals for the corresponding filtering operations in order to achieve the desired result. That is, F2 is configured to enhance a signal associated with the listening direction, while F3 is configured to enhance signals emanating from other than the listening direction.
In summary, the above described invention describes a method and a system for providing audio input in a high noise environment. The audio input system includes a microphone array that may be affixed to a video game controller, e.g., a SONY PLAYSTATION 2® video game controller or any other suitable video game controller. The microphone array is configured so as to not place any constraints on the movement of the video game controller. The signals received by the microphone sensors of the microphone array are assumed to include a foreground speaker or audio signal and various background noises including room reverberation. Since the time-delay between background and foreground from various sensors is different, their second-order statistics in frequency spectrum domain are independent of each other, therefore, the signals may be separated on a frequency component basis. Then, the separated signal frequency components are recombined to reconstruct the foreground desired audio signal. It should be further appreciated that the embodiments described herein define a real time voice input system for issuing commands for a video game, or communicating with other players within a noisy environment.
It should be appreciated that the embodiments described herein may also apply to on-line gaming applications. That is, the embodiments described above may occur at a server that sends a video signal to multiple users over a distributed network, such as the Internet, to enable players at remote noisy locations to communicate with each other. It should be further appreciated that the embodiments described herein may be implemented through either a hardware or a software implementation. That is, the functional descriptions discussed above may be synthesized to define a microchip configured to perform the functional tasks for each of the modules associated with the microphone array framework.
With the above embodiments in mind, it should be understood that the invention may employ various computer-implemented operations involving data stored in computer systems. These operations include operations requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Further, the manipulations performed are often referred to in terms, such as producing, identifying, determining, or comparing.
The above described invention may be practiced with other computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The invention may also be practiced in distributing computing environments where tasks are performed by remote processing devices that are linked through a communications network.
The invention can also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data which can be thereafter read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium can also be distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion.
Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims. In the claims, elements and/or steps do not imply any particular order of operation, unless explicitly stated in the claims.
This application is a continuation of U.S. application Ser. No. 10/650,409, filed on Aug. 27, 2003, from which priority is claimed under 35 USC § 120, and is herein incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10650409 | Aug 2003 | US |
| Child | 12563089 | US |
