1. Field of the Invention
The present invention is directed to audio conferencing systems, and more particularly to a method of reducing acoustic coupling and howling effects in a full duplex audio conferencing system using beamforming.
2. Description of the Related Art
Beamforming technology (also known as microphone array technology) can be used for spatial directivity of sound pickup in a conferencing system to enhance the quality of near-end speech. For such systems that perform full-duplex communications, acoustic echo cancellation presents numerous challenges.
One problem that is well known in the art relates to combining acoustic echo cancellation with beamforming (see M. Brandstein and D. Ward, “Microphone Arrays. Signal Processing Techniques and Applications”. Springer Verlag, 2001, and H. Buchner, W. Herbordt, W. Kellermann, “An Efficient Combination of Multi-Channel Acoustic Echo Cancellation With a Beamforming Microphone Array”, Proc. Int. Workshop on Hands-Free Speech Communication (HSC), pp. 55-58, Kyoto, Japan, April, 2001). One approach is to perform acoustic echo cancellation on all the microphone signals in parallel, which is computationally intensive. A second approach is to perform acoustic echo cancellation on the spatially filtered signal at the output of the beamformer.
In the approach where acoustic echo cancellation is performed on the spatially filtered signal at the output of the beamformer, it is clearly desirable that the beamformer presents good loudspeaker-to-beam coupling characteristics in order to minimize the amount of echo that has to be cancelled. At the same time, it is also desirable that the beamformer has good directivity in its look direction to provide enhanced speech quality based on spatial directivity. In practice, however, it is impossible to meet both of these requirements with the same beamformer design as there typically is a trade off between the beamformer's directivity and response to the loudspeaker signal, as discussed below.
A number of design requirements must be met to ensure good coupling characteristics. Firstly, the beamformer must be robust to uncorrelated perturbations of the microphone signals. The reason for this is that in a conference system, the physical coupling between the loudspeaker and the microphones is subject to variations caused by the loudspeaker and other structural vibrations or leakage. Therefore, a high sensitivity of the beamformer to such variations may result in excessive acoustic coupling between the loudspeaker and the output of the beam. This, in turn, affects the performance of the acoustic echo cancellation, resulting in echo bursts as well as howling effects due to positive loop gain. It should be noted that the sensitivity problem applies to both structures described above for combining acoustic echo cancellation with beamforming. Indeed, even if acoustic echo cancellation is performed on the microphone signals in parallel, the residual echo signals are not free of amplitude and phase variations due to structural coupling (vibrations and acoustical leaks). Many of these variations will remain on the microphone signals after the individual echo cancellation. When combined in the beamformer they may result in a large error signal if the beamformer is sensitive to the variations described above. The second design requirement that must be met to ensure good coupling characteristics is that the beamformer should also provide strong attenuation of the loudspeaker signal (which can be seen as an interference signal) in order to achieve better echo cancellation performance in the conferencing system.
A further description of the prior art and of a preferred embodiment of the invention is set forth in detail below with reference to the following drawings, in which:
It is well known in the art (see Branstein et al, supra, chapter 2) that there is a trade-off between the directivity towards a talker and the robustness of the beamformer to amplitude and phase variations of the signals received by the microphones. In other words, very directive beamformers (designed with such methods as Minimum Variance Distortionless Response or other superdirective variants) are characterized by a high sensitivity to uncorrelated noise in the microphone signals.
The general problem of achieving directivity and robustness is well-known in the art and several solutions have been proposed. All of them achieve a compromise between directivity and robustness, by introducing a regularization factor in the design of the beamformer (see for instance E. N. Gilbert and S. P. Morgan, “Optimum design of directive antenna arrays subject to random variations”, Bell Syst. Tech. J., pp 637-663, May 1955). This regularization factor can vary between 0 and +∝. When given the value 0, the design results in a pure, superdirective beamformer (with high sensitivity). When given a value that tends to +∝, the design then yields the so-called conventional beamformer (also referred to as a delay-and-sum beamformer, or the Barlett beamformer (see Brandstein et al, supra, chapter 2), which is very robust but not very directive. Any value of the regularization factor between these two extremes represents a compromise between directivity and robustness. The process to find the best compromise for a particular device and a particular application is sometimes a trial-and-error process as in M. Doerbecker, “Mehrkanalige Signalverarbeitung zur verbesserung akustisch gestorter sprachsignale am beispiel elektronischer hörhilfen”, PhD thesis, Dept of Telecommunications, University of TH Aachen, Verlag der Augustinus Buchhandlung, Aachen, Germany, August 1998. It is sometimes an iterative process as in J. M. Kates and M. R. Weiss, “A comparison of hearing-aid array-processing techniques”, J. Acoustic. Soc. Amer., vol. 99, pp 3138-3148, May 1996. All these solutions achieve a trade off between directivity and coupling of the beamformer thus compromising the overall performance of the conferencing system.
The second design requirement mentioned above may be referred to as interference nulling. It can be achieved with standard linear-constraint techniques as described in Van Veen and Buckley, “Beamforming: a versatile approach to spatial filtering”, IEEE ASSP Magazine, April 1998. However, it is known in the art that placing a null (or strong attenuation) in the direction of an interference signal may also significantly affect the beamformer directivity in other directions because it reduces the number of degrees of freedom available to form a beam in its look direction.
The inventors have realized that for the particular application of audio conferencing, the beamformer does not require good coupling characteristics and good near-end directivity at the same time. Moreover, the conferencing device has internal knowledge of which beamformer feature is more important at any given point in time (depending on the dynamics of the conversation: far-end speech, near-end speech, etc). More specifically, the inventors have recognized that an audio conferencing system only requires spatial directivity in the presence of near-end speech, and it only requires good coupling characteristics in the presence of far-end speech. Therefore, the key technical aspect of the present invention is to use two different beamformers: one with good coupling characteristics for far-end-speech (including double-talk) and one with good directivity towards the near-end speaker, for near-end speech. Double-silence can be handled with one or the other scheme the choice between which is an implementation decision, based on the specific characteristics of the particular device.
According to the foregoing, there is no need to compromise between directivity and coupling characteristics in the same beamformer design. The resulting system provides enhanced near-end speech quality (reverberations and local interference are suppressed by the “near-end” directional beamformer) as well as enhanced full-duplex operation (loudspeaker feedback is reduced by the “far-end” beamformer that is used in the presence of far-end signal or double-talk).
The present invention is to be distinguished from half-duplex echo cancellation, wherein the transmit channel is closed during far-end speech activity and partially closed during double-silence. In the present invention, both sound pickup schemes can be designed to provide a similar gain and frequency response towards the near-end talker. The resulting audio conferencing system therefore provides true full-duplex communication, and switching between the two sound-pickup schemes is much less noticeable for the far-end listener than it is with a half-duplex system. Furthermore, the full-duplex conferencing system benefits from maximal directivity when required (i.e. during near-end speech and possibly double-silence) as well as reduced loudspeaker-to-beam coupling performance when required (i.e. during far-end speech, double-talk and possibly double-silence).
The present invention relies on a good mechanism to detect the presence of near-end speech and/or far-end speech, This mechanism is not part of the present invention but would be well known to a person of ordinary skill in the art. In general, such a mechanism is available in conventional full-duplex acoustic echo cancellation algorithms that operate based on reliable decision logic regarding silence, single-talk and double-talk situations. Examples of such standalone mechanisms are set forth in U.S. Pat. No. 6,453,041 (Voice activity detection system and method, Erol Eryilmaz), and U.S. Pat. No. 5,963,901 (Method and device for voice activity detection in a communication device, Vahatalo et al).
Preferably, switching between the two sound pickup schemes is gradual rather than instantaneous. In particular, in the case where double-silence is handled by the “far-end speech” sound pickup scheme, when near-end speech starts or resumes in the near-end location, the system preferably switches gradually from the “far-end speech” sound-pickup scheme to the “near-end speech” sound pickup scheme. This results in a gradual audio focus towards the near-end talker. The speed of the transitions from one beamformer to the other may depend on the properties of the two sound pickup schemes used, as will be apparent to a person of ordinary skill in the art.
The design of the two sound-pickup schemes utilize standard beamforming techniques available in the literature (see Brandstein et al, supra). Generally, the design requirements are as follows. The “far-end” beam needs good coupling response to the loudspeaker signal (e.g. standard linear-constraint techniques combined with standard regularisation techniques to ensure good robustness). The “near-end” beam needs good spatial directivity towards its look direction in order to deliver enhanced speech quality to the far-end party. To minimize transition effects, both beams should present similar gain and frequency response in the look direction of interest.
Alternatives and variations of the invention are possible. For example, the two sound-pickup schemes do not specifically need to be beamformers, but can be any sound pickup schemes that provide spatial directivity in one case and coupling performance in the other case, while both offering full gain towards the near-end talker.
Another variation is to use a different “far-end” sound pickup scheme for each look direction instead of a single “far-end” beam common to all directions. The “far-end” sound pickup schemes would then have to present good coupling characteristics and some directivity towards their look directions. This approach solves the problem of compromise between coupling and directivity, but it does not solve the problem of having to deal with acoustic echo cancellation on a signal with time varying echo path, as discussed above.
The many features and advantages of the invention are apparent from the detailed specification and, thus, it is intended by the appended claims to cover all such features and advantages of the invention that fall within the sphere and scope of the invention. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction and operation illustrated and described, and accordingly all suitable modifications and equivalents may be resorted to, falling within the scope of the invention.
Number | Date | Country | Kind |
---|---|---|---|
0317158.4 | Jul 2003 | GB | national |
Number | Name | Date | Kind |
---|---|---|---|
4485272 | Duong et al. | Nov 1984 | A |
5963901 | Vahatalo et al. | Oct 1999 | A |
6449593 | Valve | Sep 2002 | B1 |
6453041 | Eryilmaz | Sep 2002 | B1 |
6535610 | Stewart | Mar 2003 | B1 |
6990193 | Beaucoup et al. | Jan 2006 | B2 |
Number | Date | Country |
---|---|---|
1 195 978 | Apr 2002 | EP |
110 18193 | Jan 1999 | JP |
Number | Date | Country | |
---|---|---|---|
20050018836 A1 | Jan 2005 | US |