Patent Grant
11765522
The present invention relates to listening devices comprising microphone arrays, such as directional hearing aids.
Speech understanding in noisy environments is a significant problem for the hearing-impaired. Hearing impairment is usually accompanied by a reduced time resolution of the sensorial system in addition to a gain loss. These characteristics further reduce the ability of the hearing-impaired to filter the target source from the background noise and particularly to understand speech in noisy environments.
Some newer hearing aids offer a directional hearing mode to improve speech intelligibility in noisy environments. This mode makes use of multiple microphones and applies beamforming technology to combine inputs from the microphones into a single, directional audio output channel. The output channel has spatial characteristics that increase the contribution of acoustic waves arriving from the target direction relative to those of the acoustic waves from other directions. Widrow and Luo survey the theory and practice of directional hearing aids in “Microphone arrays for hearing aids: An overview,” Speech Communication 39 (2003), pages 139-146, which is incorporated herein by reference.
US Patent Application Publication 2019/0104370, whose disclosure is incorporated herein by reference, describes a hearing aid apparatus including a case, which is configured to be physically fixed to a mobile telephone. An array of microphones are spaced apart within the case and are configured to produce electrical signals in response to acoustical inputs to the microphones. An interface is fixed within the case. Processing circuitry is fixed within the case and is coupled to receive and process the electrical signals from the microphones so as to generate a combined signal for output via the interface.
U.S. Pat. No. 10,567,888, whose disclosure is incorporated herein by reference, describes an audio apparatus including a neckband, which is sized and shaped to be worn around a neck of a human subject and includes left and right sides that rest respectively above the left and right clavicles of the human subject wearing the neckband. First and second arrays of microphones are disposed respectively on the left and right sides of the neckband and configured to produce respective electrical signals in response to acoustical inputs to the microphones. One or more earphones are worn in the ears of the human subject. Processing circuitry is coupled to receive and mix the electrical signals from the microphones in the first and second arrays in accordance with a specified directional response relative to the neckband so as to generate a combined audio signal for output via the one or more earphones.
There is provided, in accordance with some embodiments of the present invention, a system including a plurality of microphones, configured to generate different respective signals in response to acoustic waves arriving at the microphones, and a processor. The processor is configured to receive the signals and to combine the signals into multiple channels, which correspond to different respective directions relative to the microphones by virtue of each channel representing any portion of the acoustic waves arriving from the corresponding direction with greater weight, relative to others of the directions. The processor is further configured to calculate respective energy measures of the channels, to select one of the directions, in response to the energy measure for the channel corresponding to the selected direction passing one or more energy thresholds, and to output a combined signal representing the selected direction with greater weight, relative to others of the directions.
In some embodiments, the combined signal is the channel corresponding to the selected direction.
In some embodiments, the processor is further configured to indicate the selected direction to a user of the system.
In some embodiments, the processor is further configured to calculate one or more speech-similarity scores for one or more of the channels, respectively, each of the speech-similarity scores quantifying a degree to which a different respective one of the channels appears to represent speech, and the processor is configured to select the one of the directions in response to the speech-similarity scores.
In some embodiments, the processor is configured to calculate each of the speech-similarity scores by correlating first coefficients, which represent a spectral envelope of one of the channels, with second coefficients, which represent a canonical speech spectral envelope.
In some embodiments, the processor is configured to combine the signals into the multiple channels using blind source separation (BSS).
In some embodiments, the processor is configured to combine the signals into the multiple channels in accordance with multiple directional responses oriented in the directions, respectively.
In some embodiments, the processor is further configured to identify the directions using a direction-of-arrival (DOA) identifying technique.
In some embodiments, the directions are predefined.
In some embodiments, the energy measures are based on respective time-averaged acoustic energies of the channels, respectively, over a period of time.
In some embodiments,
the time-averaged acoustic energies are first time-averaged acoustic energies,
the processor is configured to receive the signals while outputting another combined signal corresponding to another one of the directions, and
at least one of the energy thresholds is based on a second time-averaged acoustic energy of the channel corresponding to the other one of the directions, the second time-averaged acoustic energy giving greater weight to earlier portions of the period of time relative to the first time-averaged acoustic energies.
In some embodiments, at least one of the energy thresholds is based on an average of the time-averaged, acoustic energies.
In some embodiments,
the time-averaged acoustic energies are first time-averaged, acoustic energies,
the processor is further configured to calculate respective second time-averaged acoustic energies of the channels over the period of time, the second time-averaged acoustic energies giving greater weight to earlier portions of the period of time, relative to the first time-averaged acoustic energies, and
at least one of the energy thresholds is based on an average of the second time-averaged acoustic energies.
In some embodiments,
the selected direction is a first selected direction and the combined signal is a first combined signal, and
the processor is further configured to:
In some embodiments, the processor is further configured to:
select a third one of the directions,
ascertain that the second selected, direction is more similar e third selected direction than is the first selected direction, and
output, instead of the second combined signal, a third combined signal representing both the first selected direction and the third selected direction with greater weight, relative to others of the directions.
There is further provided, in accordance with some embodiments of the present invention, a method including receiving, by a processor, a plurality of signals from different respective microphones, the signals being generated by the microphones in response to acoustic waves arriving at the microphones. The method further includes combining the signals into multiple channels, which correspond to different respective directions relative to the microphones by virtue of each channel representing any portion of the acoustic waves arriving from the corresponding direction with greater weight, relative to others of the directions. The method further includes calculating respective energy measures of the channels, selecting one of the directions, in response to the energy measure for the channel corresponding to the selected direction passing one or more energy thresholds, and outputting a combined signal representing the selected direction with greater weight, relative to others of the directions.
There is further provided, in accordance with some embodiments of the present invention, a computer software product including a tangible non-transitory computer-readable medium in which program instructions are stored. The instructions, when read by a processor, cause the processor to receive, from a plurality of microphones, respective signals generated by the microphones in response to acoustic waves arriving at the microphones, and to combine the signals into multiple channels, which correspond to different respective directions relative to the microphones by virtue of each channel representing any portion of the acoustic waves arriving from the corresponding direction with greater weight, relative to others of the directions. The instructions further cause the processor to calculate respective energy measures of the channels, to select one of the directions, in response to the energy measure for the channel corresponding to the selected direction passing one or more energy thresholds, and to output a combined signal representing the selected direction with greater weight, relative to others of the directions.
The present invention will be more fully understood from the following detailed description of embodiments thereof, taken together with the drawings, in which:
Embodiments of the present invention include a listening device for tracking speech. The listening device may function as a hearing aid for a hearing-impaired user, by amplifying speech over other sources of noise. Alternatively, the listening device may function as a “smart” microphone in a conference room or any other setting in which a speaker may be speaking in the presence of other noise.
The listening device comprises an array of microphones, each of which is configured to output a respective audio signal in response to received acoustic waves. The listening device further comprises a processor, configured to combine the audio signals into multiple channels corresponding to different respective directions from which the acoustic waves are arriving at the listening device. Subsequently to generating the channels, the processor selects the channel that is most likely to represent speech, rather than other noise. For example, the processor may calculate respective energy measures for the channels, and then select the channel having the highest energy measure. Optionally, the processor may require that the spectral envelope of the selected channel be sufficiently similar to the spectral envelope of a canonical speech signal. Subsequently to selecting the channel, the processor outputs the selected channel.
In some embodiments, the processor uses blind source separation (BSS) techniques to generate the channels, such that the processor need not necessarily identify any of the directions to which the channels correspond. In other embodiments, the processor uses a direction-of-arrival (DOA) identifying technique to identify the primary directions from which the acoustic waves are arriving, and then generates the channels by combining the signals in accordance with multiple different directional responses oriented in the identified directions, respectively. In yet other embodiments, the processor generates the channels by combining the signals in accordance with multiple directional responses oriented in different respective predefined directions.
Typically, the listening device is not redirected to a new channel unless the time-averaged amount of acoustic energy of the channel over a period of time exceeds one or more thresholds. By virtue of comparing the time-averaged energy to the thresholds, occurrences in which the listening device performs a spurious or premature redirection away from a speaker are reduced. The thresholds may include, for example, a multiple of a time-averaged amount of acoustic energy of the channel that is currently being output from the listening device.
Embodiments of the present invention further provide techniques for alternating between a single listening direction and multiple listening directions, so as to seamlessly follow conversations in which multiple speakers may speak simultaneously on occasion.
Reference is initially made to
Listening device 20 comprises multiple (e.g., four, eight, or more) microphones 22, each of which may comprise any suitable type of acoustic transducer known in the art, such as a microelectromechanical system (MEMS) device or miniature piezoelectric transducer. (The term “acoustic transducer” is used broadly, in the context of the present patent application, to refer to any device that converts acoustic waves into an electrical signal, or vice versa.) Microphones 22 are configured to receive (or “detect”) acoustic waves 36 and, in response to the acoustic waves, generate signals, referred to herein as “audio signals,” representing the time-varying amplitude of acoustic waves 36.
In some embodiments, as shown in
By way of example,
Typically, the pod further comprises a communication interface. For example, the pod may comprise an audio jack 26 and/or a Universal Serial Bus (USB) jack (not shown) for connecting headphones or earphones to the pod, such that a user may listen to the signal output by the pod (as described in detail below) via the headphones or earphones. (Thus, the listening device may function as a hearing aid.) Alternatively or additionally, the pod may comprise a network interface (not shown) for communicating the output signal over a computer network (e.g., the Internet), telephone network, or any other suitable communication network. (Thus, the listening device may function as a smart microphone for conference rooms and other similar settings.) Pod 21 is generally used while sitting on a table or another surface.
Alternatively to pod 21, listening device 20 may comprise any other suitable apparatus comprising any of the components described above. For example, the listening device may comprise a mobile-phone case, as described in US Patent Application Publication 2019/0104370, whose disclosure is incorporated herein by reference, a neckband, as described in U.S. Pat. No. 10,567,888, whose disclosure is incorporated herein by reference, a spectacle frame, a closed necklace, a belt, or an implement that is clipped to or embedded in the user's clothing. For each of these devices, the relative positions of the microphones are generally fixed, i.e., the microphones do not move relative to each other while the listening device is in use.
Listening device 20 further comprises a processor 34 and a memory 38, which typically comprises a high-speed nonvolatile memory array, such as a flash memory. In some embodiments, the processor and memory are implemented in single integrated circuit chip contained within the apparatus comprising the microphones, such as within pod 21, or externally to the apparatus, e.g., within headphones or earphones connected to the device. Alternatively, the processor and/or memory may be distributed over multiple chips, some of which may be located externally to the apparatus.
As described in detail below, by processing the audio signals received from the microphones, processor 34 generates an output signal—referred to hereinbelow as a “combined signal”—in which the audio signals are combined so as to represent the portion of the acoustic waves having the greatest amount of energy with greater weight, relative to other portions of the acoustic waves. Typically, the former are produced by a speaker, while the latter are produced by sources of noise; thus, the listening device is described herein as a “speech-tracking” listening device. As described above, the output signal may be output (in digital or analog form) from the listening device via any suitable communication interface.
In some embodiments, the processor generates the combined signal by applying any suitable blind source separation technique to the audio signals. In such embodiments, the processor need not necessarily identify the direction from which the most energetic portion of the acoustic waves is arriving at the listening device.
In other embodiments, the processor generates the combined signal by applying suitable beamforming coefficients to the audio signals so as to time-shift the signals, gain-adjust the various frequency bands of the signals, and then sum the signals, all this being done in accordance with a particular directional response. In some embodiments, this computation is performed in the frequency domain, by multiplying the respective Fast Fourier Transforms (FFTs) of the (digitized) audio signals by appropriate beam-forming coefficients, summing the FFTs, and then computing the combined signal as the inverse FFT of the sum. In other embodiments, this computation is performed. In the time domain, by applying, to the audio signals, the finite impulse response (FIR) filter of suitable beamforming coefficients. In any case, the combined signal is generated so as to increase the contribution of acoustic waves arriving from a target direction, relative to the contribution of acoustic waves arriving from other directions.
In some such embodiments, the direction which the directional response is oriented is defined by a pair of angles, including an azimuthal angle φ and a polar angle, in a coordinate system of the listening device. (The origin of the coordinate system may be located, for example, at a point that is equidistant to each of the microphones.) In other such embodiments, for ease of computation, differences in elevation are ignored, such that the direction is defined by an azimuthal angle φ for all elevations. In any case, by combining the audio signals in accordance with the directional response, the processor effectively forms a listening beam 23 oriented in the direction, such that the combined signal gives greater representation to acoustic waves originating within listening beam 23, relative to acoustic waves originating outside listening beam 23. (Listening beam 23 may have any suitable width.)
In some embodiments, the microphones output the audio signals in analog form. In such embodiments, processor 34 comprises an analog/digital (A/D) converter, which digitizes the audio signals. Alternatively, the microphones may output the audio signals in digital form, by virtue of A/D conversion circuitry integrated into the microphones. Even in such embodiments, however, the processor may comprise an A/D converter for converting the aforementioned combined signal to analog form, for output via an analog communication interface. (It is noted that in the context of the present application, including the claims, the same term may be used to refer to a particular signal in both its analog form and its digital form.)
Typically, processor 34 further comprises processing circuitry, such as a digital signal processor (DSP) or field programmable gate array (FPGA), for combining the audio signals. An example embodiment of suitable processing circuitry is the iCE40 FPGA by Lattice Semiconductor, Santa Clara, Calif.
Alternatively or additionally to the aforementioned circuitry, processor 34 may comprise a microprocessor, which is programmed in software or firmware to carry out at least some of the functions described herein. Such a microprocessor may comprise at least a central processing unit (CPU) and random access memory (RAM). Program code, including software programs, and/or data are loaded into the RAM for execution and processing by the CPU. The program code and/or data may be downloaded to the processor in electronic form, over a network, for example. Alternatively or additionally, the program code and/or data may be provided and/or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory. Such program code and/or data, when provided to the processor, produce a machine or special-purpose computer, configured to perform the tasks described herein.
In some embodiments, memory 38 stores multiple sets of beamforming coefficients corresponding to different respective predefined directions, and the listening device always listens in one of the predefined directions when performing directional hearing. In general, any suitable number of directions may be predefined. As a purely illustrative example, eight directions, corresponding to azimuthal angles of 0, 45, 90, 135, 180, 225, 270, and 315 degrees in the coordinate system of the listening device, may be predefined, and memory 38 may thus store eight corresponding sets of beamforming coefficients. In other embodiments, the processor calculates at least some sets of beamforming coefficients on the fly, such that the listening device may listen in any direction.
In general, the beamforming coefficients may be calculated—in advance of being stored in memory 38, or on the fly by the processor—using any suitable algorithm known in the art, such as any of the algorithms described in the above-mentioned article by Widrow and Luo. One specific example is a time delay (or delay-and-sum (DAS)) algorithm, which, for any particular direction, computes beamforming coefficients so as to combine the audio signals with time shifts equal to the propagation times of the acoustic waves between the microphone locations with respect to the particular direction. Other examples include Minimum Variance Distortionless Response (MVDR), Linear Constraint Minimum Variance (LCMV), General Sidelobe Canceller (GSC), and Broadband Constrained Minimum Variance (BCMV). Such beamforming algorithms, as well as other audio enhancement functions that can be applied by processor 34, are further described in the above-mentioned PCT International Publication WO 2017/158507.
It is noted that a set of beamforming coefficients de multiple subsets of coefficients for different respective frequency bands.
Reference is now made to
Each iteration of algorithm 25 begins at a sample-extracting step 42, at which a respective sequence of samples is extracted from each audio signal. Each sequence of samples may span, for example, 2-10 ms.
Subsequently to extracting the samples, the processor, at a signal-combining step 27, combines the signals—in particular, the respective sequences of samples extracted from the signals into multiple channels. The channels correspond to different respective directions relative to the listening device (or relative to the microphones) by virtue of each channel representing any portion of the acoustic waves arriving from the corresponding direction with greater weight, relative to other directions. However, the processor does not identify the directions; rather, the processor uses a blind source separation (BSS) technique to generate the channels.
In general, the processor may use any suitable BSS technique. One such technique, which applies independent component analysis (ICA) to the audio signals, is described in Choi, Seungjin, et al., “Blind source separation and independent component analysis: A review,” Neural Information Processing-Letters and Reviews 6.1 (2005): 1-57, which is incorporated herein by reference. Other such techniques may similarly use ICA; alternatively, they may apply principal component analysis (PCA) or neural networks to the audio signals.
Subsequently, for each channel, the processor calculates a respective energy measure at a first energy-measure-calculating step 29, and then compares the energy measure to one or more energy thresholds at an energy-measure-comparing step 31. Further details regarding these steps are provided below, in the subsection entitled “Calculating the energy measures and thresholds.”
Subsequently, at a channel-outputting step 33, the processor causes the listening device to output at least one channel for which the energy measure passes the thresholds. In other words, the processor outputs the channel to a communication interface of the listening device, such that the listening device outputs the channel via the communication interface.
In some embodiments, the listening device outputs only those channels that appear to represent speech. For example, subsequently to ascertaining that the energy measure of a particular channel passes the thresholds, the processor may apply a neural network or any other machine-learned model to the channel. The model may ascertain that the channel represents speech in response to the degree to which features of the channel, such as frequencies of the channel, are indicative of speech content. Alternatively, the processor may calculate a speech-similarity score for the channel, the score quantifying the degree to which the channel appears to represent speech, and then compare the score to a suitable threshold. The score may be calculated, for example, by correlating coefficients representing the spectral envelope of the channel with other coefficients representing a canonical speech spectral envelope, which represents the average spectral properties of speech in a particular language (and, optionally, dialect). Further details regarding this calculation are provided, below, in the subsection entitled “Calculating the speech-similarity score.”
In some embodiments, subsequently to selecting a channel for output, the processor identifies the direction corresponding to the selected channel. For example, for embodiments in which an ICA technique is used for BSS, the processor may calculate the direction from particular interim output of the technique, known as the “separation matrix,” and the respective locations of the microphones, as described, for example, in Mukai, Ryo, et al., “Real-time blind source separation and DOA estimation using small 3-D microphone array,” Proc. Int. Workshop on Acoustic Echo and Noise Control (IWAENC), 2005, whose disclosure is incorporated herein by reference. Subsequently, the processor may indicate the direction to the user(s) of the listening device, as described at the end of the present description.
Reference is now made to
By way of introduction, it is noted that algorithm 35 differs from algorithm 25 (
Each iteration of algorithm 35 begins with sample-extracting step 42, as described above with reference to
In performing DOA-identifying step 37, the processor may use any suitable DOA-identifying technique known in the art. One such technique, which identifies DOAs by correlating between the audio signals, is described in Huang, Yiteng, et al., “Real-time passive source localization: A practical linear-correction least-squares approach,” IEEE transactions on Speech and Audio Processing 9.8 (2001): 943-956, which is incorporated herein by reference, Another such technique, which applies ICA to the audio signals, is described in Sawada, Hiroshi et al., “Direction of arrival estimation for multiple source signals using independent component analysis,” Seventh International Symposium on Signal Processing and Its Applications, 2003 Proceedings, Vol. 2, IEEE, 2003, which is incorporated herein by reference. Yet another such technique, which applies a neural network to the audio signals, is described in Adavanne, Sharath et al., “Direction of arrival estimation for multiple sound sources using convolutional recurrent neural network,” 2018 26th European Signal Processing Conference (EUSIPCO), IEEE, 2018, which is incorporated herein by reference.
Subsequently, the processor, at a first directional-signal-computing step 39, computes respective directional signals for the identified DOAs. In other words, for each DOA, the processor combines the audio signals in accordance with a directional response oriented in the DOA, so as to generate a directional signal giving greater representation to sound arriving from the DOA, relative to other directions. In performing this functionality, the processor may calculate suitable beamforming coefficients on the fly, as described above with reference to
Next, at a second energy-measure-calculating step 41, the processor calculates a respective energy measure for each DOA (i.e., for each directional signal). The processor then compares each energy measure to one or more energy thresholds at energy-measure-comparing step 31. As noted above with reference to
Finally, at a first directing step 45, the processor directs the listening device to at least one DOA for which the energy measure passes the thresholds. For example, the processor may cause the listening device to output the directional signal, computed at first directional-signal-computing step 39, that corresponds to the DOA. Alternatively, the processor may use different beamforming coefficients to generate, for output by the listening device, another combined signal having a directional response oriented in the DOA.
As described above with reference to
An advantage of the aforementioned directional-hearing embodiments is that the directional response of the listening device may be oriented in any direction. In some embodiments, however, to reduce the computational load on the processor, the processor selects one of multiple predefined directions, and then orients the directional response of the listening device in the selected direction.
In such embodiments, the processor first generates multiple channels (again referred to as “directional signals”) {Xn}, n=1 . . . N, where N is the number of predefined directions. Each directional signal gives greater representation to sound arriving from a different respective one of the predefined directions.
Subsequently, the processor calculates respective energy measures for the directional signals, e.g., as further described below in the subsection entitled “Calculating the energy measures and thresholds.” The processor may further calculate one or more speech-similarity scores for one or more of the directional signals, e.g., as further described below in the subsection entitled “Calculating the speech-similarity score.” Subsequently, based on the energy measures and, optionally, the speech-similarity scores, the processor selects at least one of the predefined directions for the directional response of the listening device. The processor may then cause the listening device to output the directional signal corresponding to the selected predefined direction; alternatively, the processor may use different beamforming coefficients to generate, for output by the listening device, another signal having the directional response oriented in the selected predefined direction.
In some embodiments, the processor calculates a respective speech-similarity score for each of the directional signals. Subsequently, the processor computes respective speech-energy measures for the directional signals, based on the energy measures and the speech-similarity scores. For example, given a convention in which a higher energy measure indicates greater energy and a higher speech-similarity score indicates greater similarity to speech, the processor may calculate each speech-energy measure by multiplying the energy measure by the speech-similarity score. The processor may then select one of the predefined directions in response to the speech-energy measure for the direction passing one or more predefined speech-energy thresholds.
In other embodiments, the processor calculates a speech-similarity score for a single one of the directional signals, such as the directional signal having the highest energy measure or the directional signal corresponding to a current listening direction. Subsequently to calculating the speech-similarity score, the processor compares the speech-similarity score to a predefined speech-similarity threshold, and also compares each of the energy measures with one or more predefined energy thresholds. If the speech-similarity score passes the speech-similarity threshold, the processor may select, for the directional response of the listening device, at least one of the directions for which the energy measure passes the energy thresholds.
As yet another alternative, the processor may first identify the directional signals whose respective energy measures pass the energy thresholds. Subsequently, the processor may ascertain whether at least one of these signals represents speech, e.g., based on a speech-similarity score or machine-learned model, as described above with reference to
For further details, reference is now made to
Each iteration of algorithm 40 begins at sample-extracting step 42, at which a respective sequence of samples is extracted from each audio signal. Subsequently to extracting the samples, the processor, at a second directional-signal-computing step 43, computes, from the extracted samples, respective directional signals for the predefined directions.
Typically, to avoid aliasing, the number of samples in each extracted sequence is greater than the number K of samples in each directional signal. In particular, at each iteration, the processor extracts a sequence Yi of the 2K most recent samples from each ith audio signal. Subsequently, the processor computes the FFT Zi of each sequence Yi(Zi=FFT(Yi)). Next, for each nth predefined direction, the processor:
(a) computes the sum ΣiZi·*Bin, where (i) Bin is a vector of beamforming coefficients (of length 2K) for the ith audio signal and nth direction, and (ii) “·*” indicates component-wise multiplication, and
(b) computes the directional signal Xn as the latter K elements of the inverse FFT of the aforementioned sum (Xn=Xn′[K:2K−1], where Xn′=IFFT(Σi*Bin)).
(Alternatively, as noted above with reference to
Algorithm 40 is typically executed periodically with a period T equal to K/f, where f is the sampling frequency with which the analog microphone signals are sampled by the processor while digitizing the signals. Xn spans the time period spanned by the middle K samples of each sequence Yi. (There is thus a lag of approximately K/2f between the end of the time period spanned by Xn and the computation of Xn.)
Typically, T is between 2-10 ms. As a purely illustrative example, T may be 4 ms, f may be 16 kHz, and K may be 64.
Next, the processor calculates, at an energy-measure-calculating step 44, respective energy measures for the directional signals.
Subsequently to calculating the energy measures, the processor checks, at a first checking step 46, whether any one of the energy measures passes one or more predefined energy thresholds. If no energy measure passes the thresholds, the current iteration of algorithm 40 ends. Otherwise, the processor proceeds to a measure-selecting step 48, at which the processor selects the highest energy measure passing the thresholds that has not been selected yet. The processor then checks, at a second checking step 50, whether the listening device is already listening in the direction for which the selected energy measure was calculated. If not, the direction is added, at a direction-adding step 52, to a list of directions.
Subsequently, or if the listening device is already listening in the direction for which the selected energy measure was calculated, the processor checks, at a third checking step 54, whether any more energy measures should be selected. For example, the processor may check whether (i) at least one other not-yet-selected energy measure passes the thresholds, and (ii) the number of directions in the list is less than the maximum number of simultaneous listening directions. The maximum number of simultaneous listening directions, which is typically one or two, may be a hardcoded parameter, or it may be set by the user, e.g., using a suitable interface belonging to pod 21 (
If the processor ascertains that another energy measure should be selected, the processor returns to measure-selecting step 48. Otherwise, the processor proceeds to a fourth checking step 56, at which the processor checks whether the list contains at least one direction. If not, the current iteration ends. Otherwise, the processor, at a third speech-similarity-score-calculating step 58, calculates a speech-similarity score, based on one of the directional signals.
Subsequently to calculating the speech-similarity score, the processor checks, at a fifth checking step 60, whether the speech-similarity score passes a predefined speech-similarity threshold. For example, for embodiments in which a higher score indicates greater similarity, the processor may check whether the speech-similarity score exceeds the threshold. If yes, the processor, at a second directing step 62, directs the listening device to at least one of the directions in the list. For example, the processor may output the directional signal, corresponding to one of the directions in the list, that was already calculated, or the processor may generate a new directional signal for one of the directions in the list using different beamforming coefficients. Subsequently, or if the speech-similarity score does not pass the threshold, the iteration ends.
Typically, if the list contains a single direction, the speech-similarity score is computed for the directional signal corresponding to the single direction in the list. If the list contains multiple directions, the speech-similarity score may be computed for any one of the directional signals corresponding to these directions, or for the directional signal corresponding to a current listening direction. Alternatively, a respective speech-similarity score may be computed for each of the directions in the list, and the listening device may be directed to each of these directions provided that the speech-similarity score for the direction passes the speech-similarity threshold, or provided that a speech-energy score for the direction—computed, for example, by multiplying the speech-similarity score for the direction by the energy measure for the direction—passes a speech-energy threshold.
Typically, a listening direction is dropped, even without replacement with a new listening direction, if the energy measure for the listening direction does not pass the energy thresholds for a predefined threshold period of time (e.g., 2-10 s). In some embodiments, the listening direction is dropped only if at least one other listening direction remains.
It is emphasized that algorithm 40 is provided by way of example only. Other embodiments may reorder some of the steps in algorithm 40, and/or add or remove one or more steps. For example, the speech-similarity score, or respective speech-similarity scores for the directional signals, may be calculated prior to calculating the energy measures. Alternatively, no speech-similarity scores may be calculated at all, and the listening direction(s) may be selected in response to the energy measures w considering whether the corresponding directional signals appear to represent speech.
In some embodiments, the energy measures calculated during the execution of algorithm 25 (
(i) Calculate the energy En of Xn. This calculation may be performed in the time domain, e.g., per the formula En=Σi=1K−1(Xn[i]−Xn[i−1])2. Alternatively, the calculation of En may be performed in the frequency domain, optionally giving greater weight to typical speech frequencies such as frequencies within a range of 100-8000 Hz.
(ii) Calculate the time-averaged acoustic energy as Sn=αEn+(1−α)Sn′, where Sn′ is the time-averaged acoustic energy for Xn calculated during be previous iteration (i.e., the time-averaged acoustic energy of the previous sequence of samples extracted from Xn) and α is between 0 and 1. (The period of time over which Sn is calculated thus begins at the time corresponding to the first sample extracted from Xn during the first iteration of the algorithm, and ends at the time corresponding to the last sample extracted from Xn during the present iteration.)
In some embodiments, one of the energy thresholds is based on a time-averaged acoustic energy Lm for the mth channel, where the mth direction is a current listening direction different from the n direction. (In case there are multiple current listening directions, Lm is typically the lowest time-averaged acoustic energy from among all the current listening directions.) For example, the threshold may equal a multiple of Lm and a constant C1. Lm is typically calculated as described above for Sn; however, Lm gives greater weight to earlier portions of the period of time relative to Sn, by virtue of a being closer to 0. (As a purely illustrative example, α may be 0.1 for Sn and 0.005 for Lm.) Thus, Lm, may be thought of a “long-term time-averaged energy,” and Sn as a “short-term time-averaged energy.”
Alternatively or additionally, one of the energy thresholds may be based on an average of the short-term time-averaged acoustic energies,
where N is the number of channels. For example, the threshold may equal a multiple of this average and another constant C2.
Alternatively or additionally, one of the energy thresholds may be based on an average of the long-term time-averaged acoustic energies,
For example, the threshold may equal a multiple of this average and another constant C3.
In some embodiments, each speech-similarity score calculated during the execution of algorithm 25 (
Typically, the canonical coefficients are stored in memory 38 (
In some embodiments, the coefficients of the spectral envelope of Xn include mel frequency cepstral coefficients (MFCCs). These may be calculated, for example, by (i) calculating the Welch spectrum of the FFT of Xn and eliminating any direct current (DC) component thereof, (ii) transforming the Welch spectrum from a linear frequency scale to a mel-frequency scale, using a linear-to-mel filter bank, (iii) transforming the mel-frequency spectrum to a decibel scale, and (iv) calculating the MFCCs as the coefficients of a discrete cosine transform (DCT) of the transformed mel-frequency spectrum.
In such embodiments, the coefficients of the canonical envelope also include MFCCs. These may be calculated, for example, by eliminating the DC component from an LTASS, transforming the resulting spectrum to a mel-frequency scale as in step (ii) above, transforming the mel-frequency spectrum to a decibel scale as in step (iii) above, and calculating the MFCCs as the coefficients of the DCT of the transformed met-frequency spectrum as in step (iv) above. Given the set MX of MFCCs of Xn and the corresponding set MC of canonical MFCCs, the speech-similarity score may be calculated as ΣiMX[i]MC[i]/√{square root over (ΣiMX[i]2ΣiMC[i]2)}.
In some embodiments, the processor may direct the listening device to multiple directions simultaneously. In such embodiments, the processor—e.g., in channel-outputting step 33 (
In the event that a single direction is to be replaced, the processor may replace the listening direction having the minimum time-averaged acoustic energy over a period of time, such as the minimum short-term time-averaged acoustic energy. In other words, the processor may identify the minimum time-averaged acoustic energy for the current listening directions, and then replace the direction for which the minimum was identified.
Alternatively, the processor may replace the current listening direction that is most similar to the new direction, based on the assumption that a speaker previously speaking from the former direction is now speaking from the latter direction. For example, given a first current listening direction oriented at 0 degrees, a second current listening direction oriented at 90 degrees, and a new direction oriented at 80 degrees, the processor may replace the second current listening direction with the new direction (even if the energy from the second current listening direction is greater than the energy from the first current listening direction), since |80−90|=10 is less than |80−0|=80.
In some embodiments, the processor directs the listening device to multiple listening directions by summing the respective combined signals for the listening directions. Typically, in this summation, each combined signal is weighted by its relative short-term or long-term time-averaged energy. For example, given two combined signals Xn1 and Xn2, the combined signal for output may be calculated as
In other embodiments, the processor directs the listening device to multiple listening directions by combining the audio signals using a single set of beamforming coefficients that corresponds to the combination of the multiple listening directions.
Typically, the processor indicates each current listening direction to the user(s) of the listening device. For example, multiple indicator lights 30 (
It will be appreciated by persons skilled in the art that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof that are not in the prior art, which would occur to persons skilled in the art upon reading the foregoing description.
The present application claims the benefit of U.S. Provisional Application 62/876,691, entitled “Automatic determination of listening direction,” filed Jul. 21, 2019, whose disclosure is incorporated herein by reference.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/IB2020/056826 | 7/21/2020 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2021/014344 | 1/28/2021 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 3119903 | Rosemond et al. | Jan 1964 | A |
| 4904078 | Gorike | Feb 1990 | A |
| 5793875 | Lehr et al. | Aug 1998 | A |
| 7031483 | Boone et al. | Apr 2006 | B2 |
| 7099486 | Julstrom et al. | Aug 2006 | B2 |
| 7103192 | Bailey | Sep 2006 | B2 |
| 7369669 | Hagen et al. | May 2008 | B2 |
| 7369671 | Sacha | May 2008 | B2 |
| 7542580 | Burns | Jun 2009 | B2 |
| 7609842 | Sipkema et al. | Oct 2009 | B2 |
| 7735996 | Van Der Zwan et al. | Jun 2010 | B2 |
| 7809149 | Burns | Oct 2010 | B2 |
| 7822217 | Hagen et al. | Oct 2010 | B2 |
| 8116493 | Westermann | Feb 2012 | B2 |
| 3139801 | Sipkema et al. | Mar 2012 | A1 |
| 8494193 | Zhang et al. | Jul 2013 | B2 |
| 8611554 | Short et al. | Dec 2013 | B2 |
| 8744101 | Burns | Jun 2014 | B1 |
| 9113245 | Gelhard | Aug 2015 | B2 |
| 9282392 | Ushakov | Mar 2016 | B2 |
| 9288589 | Cheung | Mar 2016 | B2 |
| 9392381 | Park et al. | Jul 2016 | B1 |
| 9591410 | Short et al. | Mar 2017 | B2 |
| 9635474 | Kuster | Apr 2017 | B2 |
| 9641942 | Strelcyk et al. | May 2017 | B2 |
| 9763016 | Merks et al. | Sep 2017 | B2 |
| 9781523 | Kuster et al. | Oct 2017 | B2 |
| 9812116 | Ushakov | Nov 2017 | B2 |
| 9980054 | McCracken | May 2018 | B2 |
| 10102850 | Basye et al. | Oct 2018 | B1 |
| 10225670 | Feilner et al. | Mar 2019 | B2 |
| 10231065 | Udesen | Mar 2019 | B2 |
| 10353221 | Graff et al. | Jul 2019 | B1 |
| D865040 | Schaal et al. | Oct 2019 | S |
| D874008 | Kotzer et al. | Jan 2020 | S |
| 10567888 | Hertzberg et al. | Feb 2020 | B2 |
| 10582295 | Zhong et al. | Mar 2020 | B1 |
| 10721572 | Petersen et al. | Jul 2020 | B2 |
| 10805739 | Sjursen | Oct 2020 | B2 |
| 20040076301 | Algazi et al. | Apr 2004 | A1 |
| 20060013416 | Truong et al. | Jan 2006 | A1 |
| 20070038442 | Visser et al. | Feb 2007 | A1 |
| 20080192968 | Ho et al. | Aug 2008 | A1 |
| 20090323973 | Dyba | Dec 2009 | A1 |
| 20110091057 | Derkx et al. | Apr 2011 | A1 |
| 20110293129 | Dillen et al. | Dec 2011 | A1 |
| 20120128175 | Visser et al. | May 2012 | A1 |
| 20120215519 | Park | Aug 2012 | A1 |
| 20120224715 | Kikker | Sep 2012 | A1 |
| 20140093091 | Dusan et al. | Apr 2014 | A1 |
| 20140093093 | Dusan et al. | Apr 2014 | A1 |
| 20140270316 | Kopina et al. | Sep 2014 | A1 |
| 20150036856 | Pruthi et al. | Feb 2015 | A1 |
| 20150049892 | Petersen et al. | Feb 2015 | A1 |
| 20150201271 | Diethorn et al. | Jul 2015 | A1 |
| 20150230026 | Eichfeld et al. | Aug 2015 | A1 |
| 20150289064 | Jensen et al. | Oct 2015 | A1 |
| 20160111113 | Cho et al. | Apr 2016 | A1 |
| 20170272867 | Zisapel et al. | Sep 2017 | A1 |
| 20180146285 | Benattar et al. | May 2018 | A1 |
| 20180330747 | Ebenezer | Nov 2018 | A1 |
| 20180359294 | Brown et al. | Dec 2018 | A1 |
| 20190104370 | Zisapel et al. | Apr 2019 | A1 |
| 20190373355 | Lee et al. | Dec 2019 | A1 |
| 20200005770 | Lunner | Jan 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 205608327 | Sep 2016 | CN |
| 206115061 | Apr 2017 | CN |
| 206920741 | Jan 2018 | CN |
| 207037261 | Feb 2018 | CN |
| 208314369 | Jan 2019 | CN |
| 208351162 | Jan 2019 | CN |
| 209693024 | Nov 2019 | CN |
| 209803482 | Dec 2019 | CN |
| 1213304 | May 2018 | ES |
| 20130054898 | May 2013 | KR |
| 101786613 | Oct 2017 | KR |
| 102006414 | Aug 2019 | KR |
| 9960822 | Nov 1999 | WO |
| 2004016037 | Feb 2004 | WO |
| 2013169618 | Nov 2013 | WO |
| 2017158507 | Sep 2017 | WO |
| 2017171137 | Oct 2017 | WO |
| 2018127412 | Jul 2018 | WO |
| 2018234628 | Dec 2018 | WO |
| Entry |
|---|
| Widrow et al., “Microphone Arrays for Hearing Aids: An Overview”, Speech Communication, vol. 39, pp. 139-146, year 2003. |
| Bose Hearphones™, “Hear Better”, pp. 1-3, Feb. 19, 2017. |
| Veen et al., “Beamforming Techniques for Spatial Filtering”, CRC Press, pp. 1-23, year 1999. |
| “iCE40 Series MobileFPGA Family,” Product Information, Lattice Semiconductor, Santa Clara, Calif., pp. 1-2, last updated May 13, 2021, as downloaded from https://www.mouser.co.il/new/lattice-semiconductor/lattice-ice40-FPGA/. |
| Choi et al., “Blind Source Separation and Independent Component Analysis: A Review,” Neural Information Processing—Letters and Review, vol. 6, No. 1, pp. 1-57, year 2005. |
| Mukai et al., “Real-Time Blind Source Separation and DOA Estimation Using Small 3-D Microphone Array,” Proceedings of the International Workshop on Acoustic Echo and Noise Control (IWAENC), pp. 45-48, year 2005. |
| Huang et al., “Real-Time Passive Source Localization: A Practical Linear-Correction Least-Squares Approach,” IEEE Transactions on Speech and Audio Processing, vol. 9, No. 8, pp. 943-956, year 2001. |
| Sawada et al., “Direction of Arrival Estimation for Multiple Source Signals Using Independent Component Analysis,” EEE Proceedings of the Seventh International Symposium on Signal Processing and its Applications, vol. 2, pp. 1-4, year 2003. |
| Adavanne et al., “Direction of Arrival Estimation for Multiple Sound Sources Using Convolutional Recurrent Neural Network,” 26th European Signal Processing Conference (EUSIPCO), IEEE, pp. 1462-1466, year 2018. |
| Byrne et al., “An International Comparison of Long-Term Average Speech Spectra,” The Journal of the Acoustical Society of America, vol. 96, No. 4, pp. 2108-2120, year 1994. |
| Wikipedia, “Direction of Arrival,” pp. 1-2, last edited Nov. 15, 2020. |
| DiBiase, “A High-Accuracy, Low-Latency Technique for Talker Localization in Reverberant Environments Using Microphone Arrays,” Doctoral Thesis, Division of Engineering, Brown University, Providence, Rhode Island, pp. 1-122, year 2000. |
| International Application # PCT/IB2020/056826 Search Report dated Nov. 17, 2020. |
| AU Application # 2020316738 Office Action dated Dec. 16, 2022. |
| EP Application # 20844216.0 ESR dated Jun. 29, 2023. |
| Number | Date | Country | |
|---|---|---|---|
| 20220417679 A1 | Dec 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62876691 | Jul 2019 | US |
