Patent Application
20250106570
Many modern hearing aids support wireless streaming of audio from external sources (e.g. localized in the vicinity of the hearing aid user), such as from a TV adapter connected to the TV for transmitting TV-audio to one or more hearing aids, from remote microphones (partner microphones, table microphones, etc.) and from smartphones. Streaming the audio directly to the hearing aids improves the speech understanding but can degrade the perception of the spatial orientation of the sound sources as well as speech understanding if multiple speakers are present. An increased spatial orientation and possible externalization that spatial audio can yield can give a more natural perception of sound which resembles hearing without streaming from the target source.
Streaming of audio from external devices to one or more hearing aids has been dealt with in a number of references. Some examples are provided in the following.
EP3270608A1 deals with a hearing device comprising a direction estimator configured to estimate a head direction of a user of the hearing device, wherein the hearing device is configured to select and apply a processing scheme in the hearing device based on the estimated head direction.
US2013094683A1 deals with applying directional cues to a streamed signal in a binaural hearing aid system. The direction of arrival can be determined based on delay differences. Directional cues (e.g. HRTFs) may be added to the streamed signal.
EP3716642A1 deals with a hearing system, a hearing device and a multitude of audio transmitters. The hearing device comprises a selector/mixer controlled by a source selection control signal determined in dependence of a comparison of a beamformed signal provided by microphones of the hearing device and streamed sound signals received from audio transmitters in an environment around the user wearing the hearing device.
EP3013070A2 deals with sound source localization in a hearing aid system wherein streamed sound (e.g. from a wireless microphone or a TV adapter) is received by a hearing aid together with acoustically propagated sound from a target sound source. Movements of the head may be detected by a head tracker.
WO2010133246A1 deals with the use in a hearing aid of directional information from an acoustically propagated (target) signal to color a wirelessly propagated (typically cleaner) (target) signal (e.g. by applying HRTFs to the streamed signal). WO2010133246A1 further describes the ‘opposite’ situation a target signal is estimated based on the acoustically propagated signal, using the wirelessly propagated signal to ‘clean’ the acoustically propagated signal.
US2015003653A1 deals with determining a position of a hearing aid relative to a streaming source using a sensor, e.g. to track head position/orientation.
US2013259237A1 deals with a hearing assistance system and method for wireless RF audio signal transmission from at least one audio signal source to ear level receivers, wherein a close-to-natural hearing impression is to be achieved. Detects angle of incidence of a wireless signal by comparing signal strengths received at left and right ears (reflecting a current head direction relative to the transmitter) and application of signals at left and right ears reflecting the difference in signal strength.
US2014348331A1 relates to binaural processing in a hearing aid system (applying HRTFs on monaural (streamed) signals based on an orientation of the head of the user relative to the sound source).
In the following, some problems with the presentation of streaming audio in a hearing aid are outlined.
Example 1: The partner of the hearing aid user wears a partner microphone, but the sound from the partner microphone is usually streamed as a mono signal, and the hearing aid user will experience the sound being presented from within the user's head, and not have a spatial perception of where the partner is placed.
Example 2: The hearing aid user is watching TV and may receive a stereo signal from a TV-adapter, thus experiencing a surround like sound, but if the user turns the head, the sound picture follows the users head and will then no longer be perceived to be externalized. Additionally, if the hearing aid user watching TV would like to listen to another person in the room trying to get the user's attention, then when the user turns the head towards the other person then the streamed sound from the TV will “follow” the user and disturb the user's ability to hear the other person.
Example 3: In a conference call with multiple speakers, the sound from the far end speaker will be presented as a mono signal in both hearing aids, making it more difficult for the hearing aid user to separate the multiple speakers.
Example 4: With many different spoken notifications available in a hearing aid user interface, it can be difficult to distinguish the many different notifications from each other. By externalizing spoken notifications (i.e. by assigning each notification to a different point in space) the understanding and recognition of notifications might increase (especially during tougher listening environments, even if it is not possible to fully hear the notification it might be recognized based on the point of origin).
Example 5: Streamed sound sources can be placed at a given proximity to the user based on the distance between the user and the streaming source to provide the user with better spatial depth perception. For applications like streaming of TV-sound, it might be beneficial to be able to attenuate and change the streamed source volume based on distance between the streaming device (e.g. TV) and the user wearing the hearing aid). This will both create a natural feeling of the incoming sound but also provide the ability to seamlessly turn up/down the streamed source (and vice versa the hearing aid output) when moving around the room and to attend the hearing aid output sound (based on input from microphones of the hearing aid) in certain situations without having to pause/resume the streamed signal source. A distance between transmitting and receiving devices may e.g. be estimated by detecting a received signal strength in the receiving device and receiving a transmitted signal strength from the transmitting device. The Bluetooth parameter ‘High Accuracy Distance Measurement’ (HADM) may likewise be used.
Example 6: In a classroom with hearing impaired students, and multiple teachers with microphones, it can be difficult for the hearing-impaired student to locate and/or separate the multiple streamed microphone signals.
Our co-pending European patent application number EP23199298.3, entitled “A hearing aid comprising a wireless audio receiver and an own-voice detector”, filed with the European Patent Office on 25 Sep. 2023, and dealing with a strategy for (automatically) controlling the gain applied to a hearing aid microphone input when a direct audio input is received is incorporated herein by reference.
In the present disclosure it is proposed to make a solution for the hearing aid user that can ensure that streamed audio signals are presented to the user according to one or more of the following:
When humans in general try to localize sounds it can help to turn the head in order to detect small changes in latency and level difference between the ears as well as the spectral shaping of the incoming sound. Additionally, it can also help to turn the head to hear another person better, since the right ear is more sensitive to sounds arriving from 30-60 degrees to the right compared to straight ahead for the right ear. Finally, the human brain can cognitively easier separate multiple speakers if they are separated in space compared to collocated speakers. The present disclosure attempts to utilize (at least some of) these effects to enhance the user's spatial awareness and/or to improve speech understanding. Furthermore, the embodiments of the disclosure may allow hearing aid users to temporarily disengage from the audio stream and focus on hearing aid microphone input without having to stop or disconnect the active stream. This will provide the user with more seamless interaction with streaming sources. Implementation of the feature may be based on activity data like walking, distance measures such as Bluetooth signal strength or HADM (High Accuracy Distance Measurement), relative head direction compared to the signal source direction or amount of head turn in general (head is still->stream sound increased, high amount of head turn->HA sound increased).
In an aspect of the present application, a binaural hearing aid system is provided. The binaural hearing aid system comprises first and second hearing aids adapted for being located at or in left and right ears, respectively, of a user. Each of the first and second hearing aids comprises an input transducer for converting an acoustically propagated signal impinging on said input transducer to an electric sound input signal comprising a target signal from at least one target sound source and other signals from possible other sound sources in an environment around the user. At least one, e.g. each, of the first and second hearing aids further comprises a wireless receiver for receiving a wirelessly transmitted signal from an audio transmitter and for retrieving therefrom a streamed audio input signal comprising a target signal from at least one target sound source and optionally other signals from other sound sources in the environment around the audio transmitter. Each of the first and second hearing aids comprises an input gain controller for controlling a relative weight between said electric sound input signal and said streamed audio input signal and providing a weighted sum of said input signals. Each of the first and second hearing aids further comprises an output transducer configured to convert the weighted sum of the input signals, or a further processed version thereof, to stimuli perceivable as sound by the user. The binaural hearing aid system further comprises a position detector configured to provide an estimate of a current position of the at least one target sound source relative to the user's head and to provide a position detector control signal indicative thereof. The binaural hearing aid system may further comprise that at least one (e.g. both) of said input gain controllers of the first and second hearing aids is configured to provide said relative weight in dependence of said position detector control signal.
Thereby an improved hearing aid system may be provided. In particular, binaural processing in the binaural hearing aid system provides input gains to the microphone signal(s) and the streamed signal(s) related to the position of current target sound source(s) relative to the user.
The first and second hearing aids may comprise first and second earpieces forming part of or constituting the first and second hearing aids, respectively. The earpieces may be adapted to be located in an ear of the user, e.g. at least partially in an ear canal of the user, e.g. partially outside the ear canal (e.g. partially in concha) and partially in the ear canal.
The wireless receiver may alternatively be located in a separate processing device forming part of the binaural hearing aid system and e.g. configured to service both earpieces.
The input transducer may comprise a noise reduction algorithm configured to reduce noise in the resulting electric sound input signal (i.e. provide the electric sound input signal with reduced noise). Likewise, the wireless receiver may comprise a noise reduction algorithm configured to reduce noise in the resulting streamed audio input signal.
The input transducer may e.g. comprise multitude of microphones and a beamformer filter configured to provide the resulting electric sound input signal as a beamformed signal.
The at least one target sound source providing the target signal received by the wireless receiver may be the same as the at least one target sound source providing the target signal received by the input transducer (e.g. if the audio transmitter is a microphone unit). They may, however, also be different (e.g. if the audio transmitter is a TV-sound transmitter).
An estimate of head movement activity, e.g. a detection of head movement, may e.g. indicate a change of the user's attention from one target sound source to another. The environment around the user may e.g. comprise more than one target sound source, e.g. two. The environment around the user may e.g. comprise one or more target sound sources that move relative to the user over time. The user's attention may over time may shift from one target sound source to another. An acoustic scene may comprise two or more target sound sources that are in a ‘conversation-like’ interaction, e.g. involving a shifting of ‘the right to speak’ (turn-taking), so that the speakers do not speak simultaneously (or only have a small overlap, e.g. less than 2 s, of simultaneous speech). In a first period of time, where the user's head movement activity is relatively small, e.g. below a threshold, it may be assumed that the user's attention is to a specific first target sound source (having a first position relative to the user, corresponding to a first look direction of the user). At times, where the user's head movement activity is relatively large, e.g. above a threshold, it may be assumed that the user's attention changes from one target sound source to another. When the user's head movement activity is (again) relatively small, it may be assumed that the user's attention is on the target sound source (e.g. located at a second position, corresponding to a second (current) look direction of the user).
The estimate of the position of the at least one target sound source relative to the user's head may comprise an estimate of an angle between the current look direction of the user, and a direction from the user's head to the at least one target sound source.
The estimate of the look direction and the direction from the user's head to a target sound source may be estimated relative to a common reference direction. The current look direction and the direction from the user's head to a (or the) target sound source may be estimated relative to a (e.g. common) reference direction. The (e.g. common) reference direction may be a ‘normal forward-looking direction of a user’. In a typical scenario, the user looks at the target sound source of current interest to the user by orienting the head in the direction of the target sound source, e.g. either by turning the head alone or by including the torso, so that the current look direction is equal to the direction from the user to the target sound source. In other words, the angle between the direction to the target sound source of current interest to the user and the current look direction is equal to zero (or close to 0). Other target sound sources located elsewhere than the sound source of current interest (and e.g. assumed to (currently) be of minor interest to the user than the ‘sound source of current interest’) will exhibit an angle between the direction to said (other) target sound source in question and the current look direction of the user that is different from zero (e.g. more than a threshold angle different from zero, e.g. more than 10°).
‘A normal forward-looking direction of a user’ (cf. ‘NLD’ in
The reference direction may be a direction from the user to the transmitter, or a normal forward-looking position relative to the torso (cf. e.g. ‘TSO’ in
The position of the transmitter relative to the user may be approximated by a direction from the user (e.g. a wireless receiver worn by the user) to the transmitter, or a normal forward-looking direction of the user.
Tracking (estimating) the position of the target audio sound source relative to the orientation of the user's head may be used to control the amplification of standard amplified sound of the hearing aids (picked up by the input transducer(s) of the hearing aid) while streaming, in other words to determine the relative weight between the electric sound input signal and the streamed audio input signal. When the user, for example, is looking at the TV (including a TV-audio sound transmitter), then the ambient sound amplification may be automatically reduced (relative to the streamed sound), and when the user looks away from the TV, then the ambient sound amplification may be automatically increased (relative to the streamed sound).
The input gain controller may be configured to decrease the relative weight of the electric sound input signal with increasing angle. A simple implementation of this may be to have full streaming gain, (e.g. gain=1 (0 dB)), when the users head is pointing in the direction of the desired source (angle ˜0, within +/−30°), and to successively reduce the gain at larger deviations in angle (e.g. −3 dB at +/−45°, e.g. −6 dB at +/−60°, e.g. −12 dB at +/−90°, and up to −18 dB at more than +/−90° angles. Otherwise, a continuous dependence between gain and angle may be applied, e.g. with a maximum cap on attenuation (e.g. 6 dB).
The modification of the relative weights (gains) may be dependent on a reception control signal indicating that the at least one streamed audio input signal is currently being received, e.g. so that the weights are only modified, when a valid streamed audio input signal is retrieved.
The modification of the relative weights may further, or alternatively, be dependent on a voice control signal from a voice activity detector indicating the presence of a voice (e.g. the user's voice, or any voice, or other voices than the user's) in the electric sound input signal and/or in the streamed audio input signal. The input gain controller may be configured to only modify the weights when the streamed audio input signal comprises speech (e.g. is dominated by speech).
The modification of the relative weights may further or alternatively be dependent on a movement control signal from a movement detector indicating whether or not the user is moving. The input gain controller may be configured to only modify the weights when the user is NOT moving significantly (movement is below a threshold).
The position detector may comprise a head tracker configured to track an angle of rotation of the user's head compared to a reference direction to thereby estimate, or contribute to the estimation of, the position of the target sound source relative to the user's head. The angle of rotation of the user's head may e.g. be provided by a head tracker, e.g. based on 1D, 2D or 3D gyroscopes, and/or 1D, 2D or 3D accelerometers, and/or 1D, 2D or 3D magnetometers. Such devices are sometimes known under the common term ‘Inertial Measurements Units’ (IMUs), cf. e.g. EP3477964A1. The reference direction of the head tracker may e.g. be the ‘normal forward-looking direction of a user’.
The head tracker may comprise a combination of a gyroscope and an accelerometer, e.g. a combination of 1D, 2D or 3D gyroscopes, and 1D, 2D or 3D accelerometers.
The position detector may comprise an eye tracker allowing to estimate a current eye gaze angle of the user relative to a current orientation of the user's head to thereby finetune the estimation of the position of the target sound source relative to the user's head. The current eye gaze angle of the user relative to a current orientation of the user's head may be represented by an angle relative to the current angle of rotation of the user's head. The eye gaze angle may thus be used as a modification (fine-tuning) of the position of the target sound source relative to the user's head, e.g. estimated as a sum of the angle of rotation of the user's head and the eye gaze angle (counted with sign, so that an eye gaze in the same direction as a head rotation has the same sign, whereas an eye gaze in the opposite direction as a head rotation has the opposite sign, θpos=θhead+θeye). The eye tracker may by based on one or more electrodes in contact with the user's skin to pick up potentials from the eyeballs. The electrodes may be located on a surface of a housing of the first and second hearing aids and be configured to provide appropriate Electrooculography (EOG) signal, cf. e.g. EP3185590A1.
The estimate of the position of the target sound source relative to the user's head may be determined as a combination of a) an angle (θ) between a line from the position of the target sound source to the head (e.g. its mid-point) of the user and a line parallel to a normal forward-looking direction of a user (both lines being located in a horizontal plane) and b) a distance (D) between the target sound source and the user's head. In other words, the position of the target sound source may be expressed in polar coordinates as (D, θ), when the coordinate system has its origo in the (middle of the) user's head (see e.g.
The estimate of the current position of the at least one target sound source relative to the user's head comprises an estimate of a distance between the target sound source and the user's head.
The estimate of the current position of the at least one target sound source relative to the user's head may comprise an estimate of a distance between the audio transmitter and the wireless receiver. A distance between transmitting and receiving devices (of the hearing aid system) may e.g. be estimated by detecting a received signal strength (e.g. a “Received Signal Strength Indicator” (RSSI) or a “Received Channel Power Indicator” (RCPI)) in the receiving device and receiving a transmitted signal strength (or channel power) from the transmitting device. The Bluetooth parameter ‘High Accuracy Distance Measurement’ (HADM) may likewise be used.
A direction from the transmitter to the user (e.g. to a wireless receiver, e.g. of the binaural hearing aid system, worn by the user) may e.g. be estimated in the wireless receiver(s) of the binaural hearing aid system.
The angle (cf. angle θU in
The input gain controller may be configured to decrease the relative weight of the electric sound input signal with increasing distance. The input gain controller may alternatively be configured to increase the relative weight of the streamed audio input signal with increasing distance.
The estimate of a position of the target sound source relative to the user's head may be provided as a user input. The binaural hearing aid system may comprise a user interface (e.g. implemented in an auxiliary device in communication with or forming part of the binaural hearing aid system, see e.g.
Each of the first and second hearing aids may comprise a monaural audio signal processor configured to apply one or more processing algorithms to said weighted sum of said input signals and to provide a processed electric output signal in dependence thereof. The one or more processing algorithms may be configured to compensate for a hearing impairment of the user.
The position detector may be configured to estimate a direction of arrival of sound from the target sound source in dependence of one or more of the electric sound input signal and the streamed audio input signal. The direction of arrival of sound from the target sound source may be equal to the angle of the direction from the user's head to the target sound source relative to a reference direction, e.g. a normal forward-looking direction of a user, cf. e.g.
The binaural hearing aid system may comprise a binaural audio signal processor configured to apply binaural gains to the streamed audio input signals of the first and second hearing aids. The binaural audio signal processor may be configured to provide respective first and second binaurally processed electric output signals comprising said streamed audio input signals of the first and second hearing aids after said binaural gains have been applied.
The binaural audio signal processor may be configured to control the binaural gains applied to the streamed audio input signal of the respective first and second hearing aids in dependence of the estimate of the position of the target sound source relative to the user's head. Thereby the first and second binaurally processed electric output signals providing a spatial sense of origin of the target sound source external to the user's head may be provided.
The binaural hearing aid system may comprise a separate processing device comprising the monaural and/or binaural audio signal processor and/or the wireless receiver(s). Each of the first and second hearing aids, e.g. the first and second earpieces, may comprise a wireless transceiver adapted for exchanging data, e.g. audio or other data, with the separate processing device.
The binaural hearing aid system may be configured to provide the respective first and second binaurally processed electric output signals in dependence of one or more detectors. The one or more detectors may comprise one or more of a wireless reception detector, a look direction detector (estimator), a distance detector (estimator), a voice activity detector (estimator), e.g. a general voice activity detector (e.g. a speech detector), and/or an own voice detector, a movement detector (providing a motion control signal indicative of a user's current motion), a brain wave detector, etc.
‘Spatial information’ (or ‘spatial cues’) providing a ‘spatial sense of origin’ to the user may comprise acoustic transfer functions from the target position (i.e. the position of the target sound source) to each of the first and second hearing aids (e.g. earpieces) when located at the first and second ears, respectively of the user (or relative acoustic transfer functions from one of the first and second earpieces (e.g. a microphone thereof) to the other, when sound impinges from the target position). The spatial information may e.g. be generated in the audio transmitter, based on head orientation data, measured in the hearing aid system and forwarded to the transmitter via a ‘back link’ from the hearing aid system to the audio transmitter. Hence, the streamed audio signal from the audio transmitter may include the spatial information. The streamed audio signal may e.g. be forwarded to the binaural hearing aid system as a stereo signal (e.g. different signals to first and second hearing aids). This could e.g. be relevant if the audio transmitter forms part of a remote microphone array, or a device comprising a microphone array (e.g. a table microphone, cf. e.g.
The spatial orientation data (e.g. head orientation data, or spatial cues) may be applied in the form of head-related (acoustic) transfer functions (HRTF) for acoustic propagation of sound from a sound source at a given position around the user to the different input transducers of the hearing aids of the hearing aid system (e.g. to one or more input transducers located at first and second ears of the user). The head-related transfer functions may be approximated by the level-difference between the two ears. The head-related transfer functions may be approximated by the latency-difference between the two ears. The head-related transfer functions may be represented by frequency dependent level- and latency-differences between the two ears. The head-related transfer functions may be implemented by application of specific (e.g. complex) binaural gain modifications to the signals presented by the first and second hearing aids (e.g. earpieces) at the left and right ears. The real and imaginary parts of the binaural complex gain modifications may represent the level differences (real part of gain) and latency differences (imaginary part of gain). In case of more than one audio signal is received by the binaural hearing aid system from respective more than one audio transmitters (or on the transmitter picking up differently located target sound sources, e.g. a table microphone), relevant HRTFs for each of the positions of the more than one audio transmitters (or target sound sources) may be applied to the corresponding more than one audio signal before being presented to the user. Thereby a spatial sense of origin external to said user's head of the one or more target sound sources corresponding to the sound provided by the audio transmitter(s) to the binaural hearing aid system may be provided. A resulting signal comprising appropriate acoustic transfer functions (HRTFs) (or impulse responses (HRIRs)) may be provided as a linear weighted combination of the signals from each target sound source, where the weights are the appropriate acoustic transfer functions (or impulse responses) for the respective sound source locations relative to the user. This may be accomplished by identifying the position of the currently present target sound sources over time as proposed by the present invention, and applying the appropriate HRTFs to various signals currently present in streamed audio input signal. To take into account also the assumed current interest of the user in the target sound sources present at a given point in time, the respective weights may also comprise an estimate of the respective priorities (e.g. determined according to the present disclosure) of these target sound sources.
The binaural hearing aid system may comprise a wireless reception detector configured to provide a reception control signal indicating whether or not the at least one streamed audio input signal comprising said target signal and optionally other signals from other sound sources in the environment around the user is currently received.
The target sound source may comprise sound from a television (TV) transmitted to the binaural hearing aid system via a TV-sound transmitter located together with the TV and/or a sound from one or more person(s) transmitted to the binaural hearing aid system via a microphone unit located at or near the person or persons in question. A scenario, where the user of the binaural hearing aid system is in conversation with two persons, each wearing a partner microphone unit, or sitting around a table microphone unit, configured to transmit sound from the person(s) in question to the binaural hearing aid system is illustrated in
The input transducer may comprise a noise reduction algorithm configured to reduce noise in the resulting electric sound input signal and/or wherein the input transducer comprises a multitude of microphones and a beamformer filter configured to provide the resulting electric sound input signal as a beamformed signal in dependence of signals from said multitude of microphones. Likewise, the wireless receiver may comprise a noise reduction algorithm configured to reduce noise in the resulting streamed audio input signal.
The (e.g. each of the) first and second hearing aids are constituted by or comprises an air-conduction type hearing aid, a bone-conduction type hearing aid, a cochlear implant type hearing aid, or a combination thereof.
In a further aspect of the present application, a binaural hearing aid system is provided. The binaural hearing aid system comprises
Thereby an improved hearing aid system may be provided. In particular binaural processing in the binaural hearing aid system provides spatial cues related to the current target sound source(s) of interest to the user.
The position detector may be configured to track the position of the user's head relative to the audio transmitter over time. The position detector may be configured to track the position of the user's head relative to the audio transmitter at least from one time instant to the next, preferably over a certain time range, e.g. of the order of seconds. The tracking (or the detector control signal) may be smoothed over time, e.g. to avoid or minimize reaction to small (short) movement changes.
The position detector may be configured to provide that the position detector control signal is indicative of at least one of a) a current distance between the target sound source and the user's head and b) a current angle between a direction from the user's head to the target sound source and a current look direction of the user.
A modifying level or gain applied to the first and second binaurally processed electric output signals may be determined in dependence of a current distance between the target sound source and the user's head, so that the modifying level or gain increases with decreasing distance and decreases with increasing distance, at least within a certain level or gain modification range.
A modifying level or gain applied to the first and second binaurally processed electric output signals may be determined in dependence of said current angle between a direction from the user's head to the target sound source and a current look direction of the user, so that said modifying level or gain increases with decreasing absolute value of said angle and decreases with increasing absolute value of said angle, at least within a certain level or gain modification range.
The modifying level or gain applied to the first and second binaurally processed electric output signals may be determined in dependence of the current position of the user's head relative to the audio transmitter, e.g. the current angle between a direction from the user's head to the target sound source and a current look direction of the user and the current distance between the target sound source and the user's head.
In a further aspect of the present application, a binaural hearing aid system is provided. The binaural hearing aid system comprises
The binaural audio signal processor is further configured to apply binaural spatial processing to said at least one streamed audio input signal and to provide said respective first and second binaurally processed electric output signals providing a spatial sense of origin external to said user's head of said target sound source in dependence of one or more of
The first and second hearing aids may comprise first and second earpieces forming part of or constituting said first and second hearing aids, respectively. The earpieces may be adapted to be located in an ear of the user, e.g. at least partially in an ear canal of the user, e.g. partially outside the ear canal (e.g. partially in concha) and partially in the ear canal.
The at least one wireless receiver may be located in a separate processing device forming part of the binaural hearing aid system and configured to service both earpieces. Each of the first and second hearing aids may comprise a wireless receiver (together forming part of, such as constituting ‘the at least one wireless receiver’).
The ‘spatial information’ (or ‘spatial cues’) providing the ‘spatial sense of origin’ to the user may comprise acoustic transfer functions from the target position (i.e. the position of the target sound source) to each of the first and second earpieces when located at the first and second ears, respectively of the user (or relative acoustic transfer functions from one of the first and second earpieces (e.g. microphones thereof) to the other, when sound impinges from the target position). The spatial information may e.g. be generated in the transmitter, based on head orientation data, measured in the hearing aid system and forwarded to the transmitter via a ‘back link’ from the hearing aid system to the transmitter. Hence, the streamed audio signal from the transmitter may include the spatial information. The streamed audio signal may e.g. be forwarded to the binaural hearing aid system as a stereo signal. This could e.g. be relevant if the transmitter forms part of a remote microphone array, or a device comprising a microphone array (e.g. a table microphone). The spatial information may be generated in the binaural hearing system, e.g. in a separate processing device or in each of the first and second hearing aids, or in combination between the transmitter and the binaural hearing aid system.
An estimate of head movement activity, e.g. a detection of head movement, may e.g. indicate a change of the user's attention from one target sound source to another. The environment around the user may e.g. comprise more than one target sound source, e.g. two. The environment around the user may e.g. comprise one or more target sound sources that move relative to the user over time. The user's attention may over time shift from one target sound source to another. An acoustic scene may comprise two or more target sound sources that are in a ‘conversation-like’ interaction, e.g. involving a shifting of ‘the right to speak’ (turn-taking), so that the speakers do not speak simultaneously (or only have a small overlap of simultaneous speech). In a first period of time, where the user's head movement activity is relatively small, it may be assumed that the user's attention is to a specific first target sound source (having a first position relative to the user, corresponding to a first look direction of the user). At times, where the user's head movement activity is relatively large, a change of the user's attention from one target sound source to another. When the user's head movement activity is (again) relatively small, it may be assumed that the user's attention is on the target sound source (e.g. located at a second position, corresponding to a second (current) look direction of the user).
The binaural hearing aid system may comprise a monaural audio signal processor configured to apply one or more processing algorithms to said first and second electric sound input signals, respectively, and optionally to said streamed audio input signal, or to a signal or signals originating therefrom. The monaural audio signal processor may comprise first and second monaural audio signal processors. The first and second monaural audio signal processor may form part of the binaural audio signal processor. The first and second monaural audio signal processor may be located in the first and second hearing aids, e.g. in the first and second earpieces, respectively, or in a separate processing device. The first monaural audio signal processor may be configured to apply one or more processing algorithms to the first electric sound input signal and, optionally, to the streamed audio input signal(s), or to a signal or signals originating therefrom, e.g. to compensate for a hearing impairment of the user. Likewise, the second monaural audio signal processor may be configured to apply one or more processing algorithms to the second electric sound input signal and, optionally, to the streamed audio input signal(s), or to a signal or signals originating therefrom. The binaural audio signal processor may be configured to provide binaural gains adapted to modify monaural gains provided by said first and second monaural processors for said first and second electric sound input signals and, optionally, said streamed audio input signal(s), or to a signal or signals originating therefrom. The binaural gains may e.g. be constituted by or comprise gains that provide said spatial sense of origin of said target sound source in said first and second binaurally processed electric output signals.
The binaural audio signal processor may be configured to estimate a direction of arrival of sound from the target sound source in dependence of the streamed audio input signal and the first and second electric sound input signals. A direction of arrival of sound from a target sound source may e.g. be estimated as disclosed in EP3285500A1.
The binaural audio signal processor may be configured to control the gain applied to the at least one streamed audio signal in dependence of the estimate of the position of the target sound source relative to the user's head. Thereby the first and second binaurally processed electric output signals providing a spatial sense of origin of the target sound source external to the user's head may be provided. E.g. when the user is looking in the direction of the source of the streamed sound, the streamed sound amplification is increased relative to the acoustically propagated sound; and vice versa, when the user is looking away from the source of the streamed sound, the streamed sound amplification is decreased relative to the acoustically propagated sound.
The binaural hearing aid system may comprise a separate processing device comprising the binaural audio signal processor and/or the at least one wireless receiver.
Each of the first and second hearing aids, e.g. the first and second earpieces, of the binaural hearing aid system may comprise a wireless transceiver adapted for exchanging data, e.g. audio or other data, with the separate processing device.
The binaural hearing aid system may be configured to provide the respective first and second binaurally processed electric output signals in (further) dependence of one or more detectors. The one or more detectors may comprise one or more of a wireless reception detector, a level detector, a look direction detector (estimator), a distance detector (estimator), a voice activity detector (estimator), e.g. a general voice activity detector (e.g. a speech detector), and/or an own voice detector, a movement detector, a brain wave detector, etc.
The binaural hearing aid system may comprise a wireless reception detector configured to provide a reception control signal indicating whether or not the at least one streamed audio input signal comprising said target signal and optionally other signals from other sound sources in the environment around the user is currently received.
The binaural hearing aid system may comprise a look direction detector configured to provide a look direction control signal indicative of a current look direction of the user relative to a direction to the position of the target sound source. The look direction detector may e.g. comprise one or more of a gyroscope, an accelerometer, and a magnetometer, and a detector of direction of arrival (DOA) of wireless signals.
The binaural hearing aid system may comprise a motion sensor providing a motion control signal indicative of a user's current motion.
The levels of the first and second binaurally processed electric output signals may be modified in dependence of a difference between a current look direction and a direction to the position of the target sound source. The levels may be modified by applying a (real) gain to the magnitude of the signal in question. The modification may b frequency dependent. The levels of the first and second binaurally processed electric output signals may be modified in dependence of the look direction control signal indicative of a current look direction of the user relative to a direction to the position of the target sound source. The modification of the levels may be dependent on the reception control signal indicating that the at least one streamed audio input signal is currently being received. The levels may be increased the smaller the difference between the current look direction and the direction to the position of the target sound source and decreased the larger the difference between the current look direction and the direction to the position of the target sound source. The levels may e.g. be modified within a range, e.g. between a maximum and a minimum level modification, e.g. limited to 6 dB.
The levels of the first and second binaurally processed electric output signals may be modified in dependence of a current distance between the target sound source and the user's head. The levels may be increased or decreased, the smaller or larger, respectively, the distance between the target sound source and said user's head. The levels of the first and second binaurally processed electric output signals may be modified in dependence of the distance control signal indicative of a current distance between the target sound source and the user's head. The modification of the levels may further be dependent on the reception control signal indicating that the at least one streamed audio input signal is currently being received. The modification of the levels may further be dependent on the look direction control signal being indicative of the current look direction being equal to or within a certain range (e.g. angle Δθ, e.g. +/−5°) of the direction to the position of the target sound source.
The modification of the levels may further or alternatively be dependent on a voice control signal from a voice activity detector indicating the presence of a voice (e.g. the user's voice, or any voice) in the first and second electric sound input signals.
The modification of the levels may further or alternatively be dependent on a movement control signal from a movement detector indicating whether or not the user is moving.
The target sound source may comprise sound from a television (TV) transmitted to the binaural hearing aid system via a TV-sound transmitter located together with the TV and/or a sound from one or more person(s) transmitted to the binaural hearing aid system via a microphone unit located at the person or persons in question on a table or a carrier located near said person or persons. A scenario, where the user of the binaural hearing aid system is in conversation with two persons, each wearing a partner microphone unit configured to transmit sound from the person in question to the binaural hearing aid system is illustrated in
The binaural hearing aid system may be configured to track the position of the user relative to the audio transmitter providing said target signal and providing said spatial sense of origin of said target sound source external to said user's head by applying head-related transfer functions to the first and second binaurally processed electric output signals. The head-related transfer functions (HRTF) may be approximated by the level difference between the two ears. The head-related transfer functions may be approximated by the latency difference between the two ears. The head-related transfer functions may be represented by frequency dependent level and latency differences between the two ears. In case of more than one audio signal is received by the binaural hearing aid system from respective more than one audio transmitters, relevant HRTFs for each of the positions of the more than one audio transmitters may be applied to the corresponding more than one audio signal before being presented to the user. Thereby a spatial sense of origin external to said user's head of the one or more target sound sources corresponding to the sound provided by the more than one audio transmitters to the binaural hearing aid system.
The first and second hearing aids may be constituted by or comprise an air-conduction type hearing aid, a bone-conduction type hearing aid, a cochlear implant type hearing aid, or a combination thereof.
In a further aspect, hearing aid system comprising a hearing aid and an audio transmitter is provided. The hearing aid and the audio transmitter are being configured to exchange data between them (e. comprising appropriate antenna and transmitter-receiver circuitry).
The hearing aid comprises
The hearing aid may further comprise an input transducer for converting an acoustically propagated signal impinging on the input transducer to an electric sound input signal comprising a target signal from at least one target sound source and other signals from possible other sound sources in an environment around the user.
The audio transmitter may e.g. comprise a television- (TV) or other video-sound transmitter configured to receive and transmit sound from a TV or other video device to the hearing aid. The audio transmitter may e.g. comprise a microphone unit configured to pick up and transmit sound from oner or more target sound sources in the environment of the microphone unit. The TV- (or video-) sound transmitter may e.g. be located together with (or integrated in) the TV (or video device). The target sound source may comprise the sound from the TV or video device transmitted to the hearing aid via the TV- or video sound transmitter. The microphone unit may be configured to be located at or near a person or a group of persons (e.g. constituting target sound source(s)). The target sound source may comprise sound from one or more person(s) transmitted to the hearing aid via the microphone unit, when located at or near the person or persons in question.
The estimate of the position of the at least one target sound source relative to the user's head may comprise an estimate of an angle between a reference direction, and a direction from the user's head to the at least one target sound source. Thereby a priority between two (or more) sound sources may be implemented in the audio transmitter (e.g. constituting or forming part of a microphone unit, e.g. a table microphone unit (e.g. a ‘speakerphone)’). The reference direction may e.g. be a normal forward-looking direction of a user, cf. e.g.
The transmitter gain may comprise spatial information representing the current position of the at least one target sound source relative to the user's head. In other words, spatial information is generated in the audio transmitter, based on head orientation data, measured in the hearing aid and forwarded to the transmitter via a ‘back link’ from the hearing aid to the audio transmitter. Hence, the streamed audio signal from the audio transmitter may include the spatial information. The streamed audio signal may e.g. be forwarded to a binaural hearing aid system comprising left and right hearing aids as a stereo signal. This could e.g. be relevant if the audio transmitter forms part of a remote microphone unit comprising a microphone array (e.g. a table microphone), e.g. involving more than one, e.g. intermittently talking, target sound sources (e.g. persons) at different locations around the microphone unit.
A prioritization between the electric sound input signal picked up by the respective input transducers of the first and second hearing aids and the streamed audio input signal in dependence of the position detector control signal may be provided by respective input gain controllers of the first and second hearing aids, e.g. as respective weighted sums (out1, out2) of the input signals and, respectively. The hearing aid may comprise an input gain controller for controlling a relative weight between the electric sound input signal and the streamed audio input signal and providing a weighted sum of the input signals.
The hearing aid may be adapted to provide a frequency dependent gain and/or a level dependent compression and/or a transposition (with or without frequency compression) of one or more frequency ranges to one or more other frequency ranges, e.g. to compensate for a hearing impairment of a user. The hearing aid may comprise a signal processor for enhancing the input signals and providing a processed output signal.
The hearing aid may comprise an output unit for providing a stimulus perceived by the user as an acoustic signal based on a processed electric signal. The output unit may comprise a number of electrodes of a cochlear implant (for a CI type hearing aid) or a vibrator of a bone conducting hearing aid. The output unit may comprise an output transducer. The output transducer may comprise a receiver (loudspeaker) for providing the stimulus as an acoustic signal to the user (e.g. in an acoustic (air conduction based) hearing aid). The output transducer may comprise a vibrator for providing the stimulus as mechanical vibration of a skull bone to the user (e.g. in a bone-attached or bone-anchored hearing aid). The output unit may (additionally or alternatively) comprise a transmitter for transmitting sound picked up-by the hearing aid to another device, e.g. a far-end communication partner (e.g. via a network, e.g. in a telephone mode of operation, or in a headset configuration).
The hearing aid may comprise an input unit for providing an electric input signal representing sound. The input unit may comprise an input transducer, e.g. a microphone, for converting an input sound to an electric input signal. The input unit may comprise a wireless receiver for receiving a wireless signal comprising or representing sound and for providing an electric input signal representing said sound.
The wireless receiver and/or transmitter may e.g. be configured to receive and/or transmit an electromagnetic signal in the radio frequency range (3 kHz to 300 GHz). The wireless receiver and/or transmitter may e.g. be configured to receive and/or transmit an electromagnetic signal in a frequency range of light (e.g. infrared light 300 GHz to 430 THz, or visible light, e.g. 430 Hz to 770 THz).
The hearing aid may comprise a directional microphone system adapted to spatially filter sounds from the environment, and thereby enhance a target acoustic source among a multitude of acoustic sources in the local environment of the user wearing the hearing aid. The directional system may be adapted to detect (such as adaptively detect) from which direction a particular part of the microphone signal originates. This can be achieved in various different ways as e.g. described in the prior art. In hearing aids, a microphone array beamformer is often used for spatially attenuating background noise sources. The beamformer may comprise a linear constraint minimum variance (LCMV) beamformer. Many beamformer variants can be found in literature. The minimum variance distortionless response (MVDR) beamformer is widely used in microphone array signal processing. Ideally the MVDR beamformer keeps the signals from the target direction (also referred to as the look direction) unchanged, while attenuating sound signals from other directions maximally. The generalized sidelobe canceller (GSC) structure is an equivalent representation of the MVDR beamformer offering computational and numerical advantages over a direct implementation in its original form.
The hearing aid may comprise antenna and transceiver circuitry allowing a wireless link to an entertainment device (e.g. a TV-set), a communication device (e.g. a telephone), a wireless microphone, or another hearing aid, etc. The hearing aid may thus be configured to wirelessly receive a direct electric input signal from another device. Likewise, the hearing aid may be configured to wirelessly transmit a direct electric output signal to another device. The direct electric input or output signal may represent or comprise an audio signal and/or a control signal and/or an information signal.
In general, a wireless link established by antenna and transceiver circuitry of the hearing aid can be of any type. The wireless link may be a link based on near-field communication, e.g. an inductive link based on an inductive coupling between antenna coils of transmitter and receiver parts. The wireless link may be based on far-field, electromagnetic radiation. Preferably, frequencies used to establish a communication link between the hearing aid and the other device is below 70 GHz, e.g. located in a range from 50 MHz to 70 GHz, e.g. above 300 MHz, e.g. in an ISM range above 300 MHz, e.g. in the 900 MHz range or in the 2.4 GHz range or in the 5.8 GHz range or in the 60 GHz range (ISM=Industrial, Scientific and Medical, such standardized ranges being e.g. defined by the International Telecommunication Union, ITU). The wireless link may be based on a standardized or proprietary technology. The wireless link may be based on Bluetooth technology (e.g. Bluetooth Low-Energy technology, e.g. LE Audio), or Ultra WideBand (UWB) technology.
The hearing aid may be or form part of a portable (i.e. configured to be wearable) device, e.g. a device comprising a local energy source, e.g. a battery, e.g. a rechargeable battery. The hearing aid may e.g. be a low weight, easily wearable, device, e.g. having a total weight less than 100 g, such as less than 20 g.
The hearing aid may comprise a ‘forward’ (or ‘signal’) path for processing an audio signal between an input and an output of the hearing aid. A signal processor may be located in the forward path. The signal processor may be adapted to provide a frequency dependent gain according to a user's particular needs (e.g. hearing impairment). The hearing aid may comprise an ‘analysis’ path comprising functional components for analyzing signals and/or controlling processing of the forward path. Some or all signal processing of the analysis path and/or the forward path may be conducted in the frequency domain, in which case the hearing aid comprises appropriate analysis and synthesis filter banks. Some or all signal processing of the analysis path and/or the forward path may be conducted in the time domain.
An analogue electric signal representing an acoustic signal may be converted to a digital audio signal in an analogue-to-digital (AD) conversion process, where the analogue signal is sampled with a predefined sampling frequency or rate fs, fs being e.g. in the range from 8 kHz to 48 kHz (adapted to the particular needs of the application) to provide digital samples xn (or x[n]) at discrete points in time tn (or n), each audio sample representing the value of the acoustic signal at tn by a predefined number Nb of bits, Nb being e.g. in the range from 1 to 48 bits, e.g. 24 bits. Each audio sample is hence quantized using Nb bits (resulting in 2Nb different possible values of the audio sample). A digital sample x has a length in time of 1/fs, e.g. 50 μs, for fs=20 kHz. A number of audio samples may be arranged in a time frame. A time frame may comprise 64 or 128 audio data samples. Other frame lengths may be used depending on the practical application.
The hearing aid may comprise an analogue-to-digital (AD) converter to digitize an analogue input (e.g. from an input transducer, such as a microphone) with a predefined sampling rate, e.g. 20 kHz. The hearing aids may comprise a digital-to-analogue (DA) converter to convert a digital signal to an analogue output signal, e.g. for being presented to a user via an output transducer.
The hearing aid, e.g. the input unit, and or the antenna and transceiver circuitry may comprise a transform unit for converting a time domain signal to a signal in the transform domain (e.g. frequency domain or Laplace domain, Z transform, wavelet transform, etc.). The transform unit may be constituted by or comprise a TF-conversion unit for providing a time-frequency representation of an input signal. The time-frequency representation may comprise an array or map of corresponding complex or real values of the signal in question in a particular time and frequency range. The TF conversion unit may comprise a filter bank for filtering a (time varying) input signal and providing a number of (time varying) output signals each comprising a distinct frequency range of the input signal. The TF conversion unit may comprise a Fourier transformation unit (e.g. a Discrete Fourier Transform (DFT) algorithm, or a Short Time Fourier Transform (STFT) algorithm, or similar) for converting a time variant input signal to a (time variant) signal in the (time-)frequency domain. The frequency range considered by the hearing aid from a minimum frequency fmin to a maximum frequency fmax may comprise a part of the typical human audible frequency range from 20 Hz to 20 kHz, e.g. a part of the range from 20 Hz to 12 kHz. Typically, a sample rate fs is larger than or equal to twice the maximum frequency fmax, fs≥2fmax. A signal of the forward and/or analysis path of the hearing aid may be split into a number NI of frequency bands (e.g. of uniform width), where NI is e.g. larger than 5, such as larger than 10, such as larger than 50, such as larger than 100, such as larger than 500, at least some of which are processed individually. The hearing aid may be adapted to process a signal of the forward and/or analysis path in a number NP of different frequency channels (NP≤NJ). The frequency channels may be uniform or non-uniform in width (e.g. increasing in width with frequency), overlapping or non-overlapping.
The hearing aid may be configured to operate in different modes, e.g. a normal mode and one or more specific modes, e.g. selectable by a user, or automatically selectable. A mode of operation may be optimized to a specific acoustic situation or environment, e.g. a communication mode, such as a telephone mode. A mode of operation may include a low-power mode, where functionality of the hearing aid is reduced (e.g. to save power), e.g. to disable wireless communication, and/or to disable specific features of the hearing aid.
The hearing aid may comprise a number of detectors configured to provide status signals relating to a current physical environment of the hearing aid (e.g. the current acoustic environment), and/or to a current state of the user wearing the hearing aid, and/or to a current state or mode of operation of the hearing aid. Alternatively or additionally, one or more detectors may form part of an external device in communication (e.g. wirelessly) with the hearing aid. An external device may e.g. comprise another hearing aid, a remote control, and audio delivery device, a telephone (e.g. a smartphone), an external sensor, etc.
One or more of the number of detectors may operate on the full band signal (time domain). One or more of the number of detectors may operate on band split signals ((time-) frequency domain), e.g. in a limited number of frequency bands.
The number of detectors may comprise a level detector for estimating a current level of a signal of the forward path. The detector may be configured to decide whether the current level of a signal of the forward path is above or below a given (L-)threshold value. The level detector operates on the full band signal (time domain). The level detector operates on band split signals ((time-) frequency domain).
The hearing aid may comprise a voice activity detector (VAD) for estimating whether or not (or with what probability) an input signal comprises a voice signal (at a given point in time). A voice signal may in the present context be taken to include a speech signal from a human being. It may also include other forms of utterances generated by the human speech system (e.g. singing). The voice activity detector unit may be adapted to classify a current acoustic environment of the user as a VOICE or NO-VOICE environment. This has the advantage that time segments of the electric microphone signal comprising human utterances (e.g. speech) in the user's environment can be identified, and thus separated from time segments only (or mainly) comprising other sound sources (e.g. artificially generated noise). The voice activity detector may be adapted to detect as a VOICE also the user's own voice. Alternatively, the voice activity detector may be adapted to exclude a user's own voice from the detection of a VOICE.
The hearing aid may comprise an own voice detector for estimating whether or not (or with what probability) a given input sound (e.g. a voice, e.g. speech) originates from the voice of the user of the system. A microphone system of the hearing aid may be adapted to be able to differentiate between a user's own voice and another person's voice and possibly from NON-voice sounds.
The number of detectors may comprise a movement detector, e.g. an acceleration sensor. The movement detector may be configured to detect movement of the user's facial muscles and/or bones, e.g. due to speech or chewing (e.g. jaw movement) and to provide a detector signal indicative thereof.
The hearing aid may comprise a classification unit configured to classify the current situation based on input signals from (at least some of) the detectors, and possibly other inputs as well. In the present context ‘a current situation’ may be taken to be defined by one or more of
The classification unit may be based on or comprise a neural network, e.g. a trained neural network.
The hearing aid may comprise an acoustic (and/or mechanical) feedback control (e.g. suppression) or echo-cancelling system. Adaptive feedback cancellation has the ability to track feedback path changes over time. It is typically based on a linear time invariant filter to estimate the feedback path but its filter weights are updated over time. The filter update may be calculated using stochastic gradient algorithms, including some form of the Least Mean Square (LMS) or the Normalized LMS (NLMS) algorithms. They both have the property to minimize the error signal in the mean square sense with the NLMS additionally normalizing the filter update with respect to the squared Euclidean norm of some reference signal.
The hearing aid may further comprise other relevant functionality for the application in question, e.g. compression, noise reduction, etc.
The hearing aid may comprise a hearing instrument, e.g. a hearing instrument adapted for being located at the ear or fully or partially in the ear canal of a user, e.g. a headset, an earphone, an ear protection device or a combination thereof. A hearing system may comprise a speakerphone (comprising a number of input transducers and a number of output transducers, e.g. for use in an audio conference situation), e.g. comprising a beamformer filtering unit, e.g. providing multiple beamforming capabilities.
In a further aspect, a hearing system comprising a hearing aid system as described above, in the ‘detailed description of embodiments’, and in the claims, AND an auxiliary device is moreover provided.
The hearing system may be adapted to establish a communication link between the hearing aid and the auxiliary device to provide that information (e.g. control and status signals, possibly audio signals) can be exchanged or forwarded from one to the other.
The auxiliary device may comprise a remote control, a smartphone, or other portable or wearable electronic device, such as a smartwatch or the like.
The auxiliary device may be constituted by or comprise a remote control for controlling functionality and operation of the hearing aid(s). The function of a remote control may be implemented in a smartphone, the smartphone possibly running an APP allowing to control the functionality of the audio processing device via the smartphone (the hearing aid(s) comprising an appropriate wireless interface to the smartphone, e.g. based on Bluetooth or some other standardized or proprietary scheme).
The auxiliary device may be constituted by or comprise an audio gateway device adapted for receiving a multitude of audio signals (e.g. from an entertainment device, e.g. a TV or a music player, a telephone apparatus, e.g. a mobile telephone or a computer, e.g. a PC) and adapted for selecting and/or combining an appropriate one of the received audio signals (or combination of signals) for transmission to the hearing aid.
The auxiliary device may be constituted by or comprise another hearing aid. The hearing system may comprise two hearing aids adapted to implement a binaural hearing system, e.g. a binaural hearing aid system.
In a further aspect, a non-transitory application, termed an APP, is furthermore provided by the present disclosure. The APP comprises executable instructions configured to be executed on an auxiliary device to implement a user interface for a hearing aid or a hearing system described above in the ‘detailed description of embodiments’, and in the claims. The APP may be configured to run on cellular phone, e.g. a smartphone, or on another portable device allowing communication with said hearing aid or said hearing system.
In the present context, a hearing aid, e.g. a denoted a hearing instrument, refers to a device, which is adapted to improve, augment and/or protect the hearing capability of a user by receiving acoustic signals from the user's surroundings, generating corresponding audio signals, possibly modifying the audio signals and providing the possibly modified audio signals as audible signals to at least one of the user's ears. Such audible signals may e.g. be provided in the form of acoustic signals radiated into the user's outer ears, acoustic signals transferred as mechanical vibrations to the user's inner ears through the bone structure of the user's head and/or through parts of the middle ear as well as electric signals transferred directly or indirectly to the cochlear nerve of the user.
The hearing aid may be configured to be worn in any known way, e.g. as a unit arranged behind the ear with a tube leading radiated acoustic signals into the ear canal or with an output transducer, e.g. a loudspeaker, arranged close to or in the ear canal, as a unit entirely or partly arranged in the pinna and/or in the ear canal, as a unit, e.g. a vibrator, attached to a fixture implanted into the skull bone, as an attachable, or entirely or partly implanted, unit, etc. The hearing aid may comprise a single unit or several units communicating (e.g. acoustically, electrically or optically) with each other. The loudspeaker may be arranged in a housing together with other components of the hearing aid, or may be an external unit in itself (possibly in combination with a flexible guiding element, e.g. a dome-like element).
A hearing aid may be adapted to a particular user's needs, e.g. a hearing impairment. A configurable signal processing circuit of the hearing aid may be adapted to apply a frequency and level dependent compressive amplification of an input signal. A customized frequency and level dependent gain (amplification or compression) may be determined in a fitting process by a fitting system based on a user's hearing data, e.g. an audiogram, using a fitting rationale (e.g. adapted to speech). The frequency and level dependent gain may e.g. be embodied in processing parameters, e.g. uploaded to the hearing aid via an interface to a programming device (fitting system), and used by a processing algorithm executed by the configurable signal processing circuit of the hearing aid.
A ‘hearing system’ refers to a system comprising one or two hearing aids, and a ‘binaural hearing system’ refers to a system comprising two hearing aids and being adapted to cooperatively provide audible signals to both of the user's ears. Hearing systems or binaural hearing systems may further comprise one or more ‘auxiliary devices’, which communicate with the hearing aid(s) and affect and/or benefit from the function of the hearing aid(s). Such auxiliary devices may include at least one of a remote control, a remote microphone, an audio gateway device, an entertainment device, e.g. a music player, a wireless communication device, e.g. a mobile phone (such as a smartphone) or a tablet or another device, e.g. comprising a graphical interface. Hearing aids, hearing systems or binaural hearing systems may e.g. be used for compensating for a hearing-impaired person's loss of hearing capability, augmenting or protecting a normal-hearing person's hearing capability and/or conveying electronic audio signals to a person. Hearing aids or hearing systems may e.g. form part of or interact with public-address systems, active ear protection systems, handsfree telephone systems, car audio systems, entertainment (e.g. TV, music playing or karaoke) systems, teleconferencing systems, classroom amplification systems, etc.
Embodiments of the disclosure may e.g. be useful in applications such as applications.
The aspects of the disclosure may be best understood from the following detailed description taken in conjunction with the accompanying figures. The figures are schematic and simplified for clarity, and they just show details to improve the understanding of the claims, while other details are left out. Throughout, the same reference numerals are used for identical or corresponding parts. The individual features of each aspect may each be combined with any or all features of the other aspects. These and other aspects, features and/or technical effect will be apparent from and elucidated with reference to the illustrations described hereinafter in which:
The figures are schematic and simplified for clarity, and they just show details which are essential to the understanding of the disclosure, while other details are left out. Throughout, the same reference signs are used for identical or corresponding parts.
Further scope of applicability of the present disclosure will become apparent from the detailed description given hereinafter. However, it should be understood that the detailed description and specific examples, while indicating preferred embodiments of the disclosure, are given by way of illustration only. Other embodiments may become apparent to those skilled in the art from the following detailed description.
The detailed description set forth below in connection with the appended drawings is intended as a description of various configurations. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. Several aspects of the apparatus and methods are described by various blocks, functional units, modules, components, circuits, steps, processes, algorithms, etc. (collectively referred to as “elements”). Depending upon particular application, design constraints or other reasons, these elements may be implemented using electronic hardware, computer program, or any combination thereof.
The electronic hardware may include micro-electronic-mechanical systems (MEMS), integrated circuits (e.g. application specific), microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), gated logic, discrete hardware circuits, printed circuit boards (PCB) (e.g. flexible PCBs), and other suitable hardware configured to perform the various functionality described throughout this disclosure, e.g. sensors, e.g. for sensing and/or registering physical properties of the environment, the device, the user, etc. Computer program shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
The present application relates to the field of hearing aids, in particular to hearing aids or hearing aid systems configured to received one or more streamed audio signals.
In the present disclosure, it is proposed to (perceptually) place the streamed audio in the correct (or appropriate) direction in space. To do this the streamed audio signal can be applied a Head-Related Transfer Function (HRTF). HRTF(L/R, D, f) is a function of side of the head (Left and Right, L/R), the direction (D) in space and frequency (f) as dependent variables. So, a sound arriving from the right side of the user will arrive later at the left ear than the right ear, and at lower sound pressure level (SPL) at the left ear than the right ear and the SPL difference will be larger at higher frequencies. Panning (comprising a model with head shadow included) may alternatively or additionally be used.
An example of detecting the position of the user relative to the external microphone array (MA), may be to use an own voice detector in the hearing aid as input to the external microphone array to detect the angle relative to the user, by correlating to the microphone array beam direction.
Alternatively, or additionally, the external microphone array (MA) may be configured to emit an ultra-sonic signal that the hearing aid microphones pick up, and wherein the hearing aid (or hearing aid system) is configured to use to determine the user head orientation relative to the external microphone array.
Alternatively, or additionally, the external microphone array (MA) may be configured to use the Bluetooth 5.1 parameter ‘angle of arrival’ (AOA). Using a Bluetooth connection between the microphone array (transmitter) and the hearing aid(s) of the binaural hearing aid system (receiver), a constant tone extension can be added to the communication between the transmitter and receiver. When receiving a constant-phase; you can measure the delta-phase, when switching between different antennas. With the knowledge of the antenna distance, the angle from where the signal arrives can be calculated. The formula for estimating the angle of arrival based on the phase difference is given by
where θ is the angle of arrival, ψ is the phase difference, λ is the wavelength, and d is the antenna distance.
Furthermore, the distance between the transmitter and receiver, and thereby the distance between the hearing aid user and the connected device, can be determined by time of flight (ToF) (time difference between the signal being sent and received) defined by the propagation speed of the signal. This can be used to determine the exact position (e.g. the distance, D) of the user relative to the streaming device which in turn can be used as input for the spatial processing of the streamed sound.
In a further aspect of the present disclosure, some of the use cases for streaming are enhanced by measuring the relative angle of the hearing aid user relative to the streaming source. This is particularly useful for the table microphone- (MA) and partner microphone (PM)-use cases (
The use case with a table microphone (MA) described in
The system may be configured to support the surround codecs on the market (Dolby, DTS, B-Format first-order Ambisonics, Opus). From multichannel to two channel surround sound.
In the following
The binaural hearing (aid) system further comprises at least one wireless receiver (Rx) for receiving a wirelessly transmitted signal from an audio transmitter (AT) and for retrieving therefrom at least one streamed audio input signal (saux) comprising the target signal from the target sound source (S) and optionally other signals (or signal components) from the other sound sources (NL, ND) in the environment around the user. The audio communication link between the audio transmitter (AT) and the binaural hearing aid system (here the audio receiver (Rx))—indicated by a bold dashed arrow from transmitter (AT) to receiver (Rx) in
The binaural hearing (aid) system further comprises a binaural audio signal processor (AUD-PRO) configured to receive the at least one streamed audio signal (saux) and the first and second electric sound input signals (in1, in2) (or signals originating therefrom) and to provide the respective first and second binaurally processed electric output signals (out1, out2) in dependence thereof. The binaural audio signal processor (BIN-PRO) is further configured to apply binaural spatial processing to the at least one streamed audio input signal (saux) (or to a signal or signals originating therefrom) and to provide said respective first and second binaurally processed electric output signals (out1, out2), which provide a spatial sense of origin external to said user's head of the target sound source, in dependence of one or more of A) the at least one streamed audio input signal (saux) and the first and second electric sound input signals (in1, in2), and B) said at least one streamed audio input signal (saux) and an estimate of a position (D, θ) of the target sound source (S) relative to the user's head (U) (cf. e.g.
Each of the input transducers (IT1, IT2) may comprise a noise reduction algorithm configured to reduce noise in the resulting electric sound input signal (in1, in2). Likewise, the wireless receiver (Rx) or the wireless receivers (Rx1, Rx2) may comprise a noise reduction algorithm configured to reduce noise in the resulting streamed audio input signal (saux; saux1, saux2).
Each of the input transducers (IT1, IT2) may e.g. comprise a multitude of microphones and a beamformer filter configured to provide the resulting electric sound input signal (in1, in2) as a beamformed signal.
The binaural audio signal processor (AUD-PRO) may form part of one or both of the first and second earpieces (EP1, EP2) or be located (e.g. mainly, e.g. apart from a selector or mixer of two audio signals located in the respective earpieces, see e.g. units SEL-MIX1, SEL-MIX2 in
The embodiment of a binaural hearing aid system of
The one or more detectors (DET) may comprise one or more of a wireless reception detector (WRD), a position detector (PD), a voice activity detector (estimator) (VAD), e.g. a general voice activity detector (e.g. a speech detector), and/or an own voice detector (OVD), a movement detector (MD), a brain wave detector (BWD), etc.
An example of a detection unit (DET) comprising one or more detectors is illustrated in
The position detector (PD) is configured to estimate a position (TPOS) of the target sound source relative to the user's head (e.g. to the earpieces (or hearing aids) of the binaural hearing aid system). The estimate of the position of the target sound source relative to the user's head may be determined as a combination of a) an angle (θ) between a1) a line from the position of the target sound source (S) to the head (U, e.g. its mid-point) of the user and a2) a reference direction, e.g. a line parallel to a normal forward-looking direction of a user (both lines being located in a horizontal plane) and/or b) a distance (D) between the target sound source (S) and the user's head (U) (e.g. its mid-point, or to each of the first and second input transducers (IT1, IT2, respectively)). In other words, the position (xs, ys) of the target sound source (S) may be expressed in polar coordinates as (D, θ), when the coordinate system has its origo in the (middle of the) user's head (see e.g.
An estimate of the position (xs, ys; D, θ) of the target sound source (S) relative to the user's head may be fully or partially determined as (approximated by) an angle (θ) relative to a reference direction, e.g. a normal forward-looking direction of a user, cf. e.g.
The position detector (PD) may comprise a look direction detector (LDD) (e.g. a head tracker) configured to provide a look direction control signal (LDCS) indicative of a current look direction (LDIR) of the user relative to a direction to the position ((D, θ) of the target sound source (S), in practice the angle θ in
The position detector (PD), e.g. the look direction detector (LDD), may comprise an eye tracker allowing to estimate a current eye gaze angle of the user relative to a current orientation of the user's head to thereby finetune the estimation of the position of the target sound source relative to the user's head. The current eye gaze angle of the user relative to a current orientation of the user's head may be represented by an angle relative to the current angle of rotation of the user's head. The eye gaze angle may thus be used as a modification (fine-tuning) of the position of the target sound source relative to the user's head, e.g. estimated as a sum of the angle of rotation of the user's head and the eye gaze angle (counted with sign, so that an eye gaze in the same direction as a head rotation has the same sign, whereas an eye gaze in the opposite direction of a head rotation has the opposite sign, θpos=θhead+θeye). The eye tracker may by based on one or more electrodes in contact with the user's skin to pick up potentials from the eyeballs. The electrodes may be located on a surface of a housing of the first and second hearing aids (e.g. the earpieces) and be configured to provide appropriate Electrooculography (EOG) signals, cf. e.g. EP3185590A1.
The position detector (PD) may comprise a direction of arrival detector (DOAD) configured to estimate a direction of arrival (DOA) of sound from the target sound source (S) in dependence of the streamed audio input signal (saux) and the first and the second electric sound input signals (in1, in2). A direction of arrival of sound from a target sound source may e.g. be estimated as disclosed in EP3285500A1.
An estimate of the position of the target sound source (S) relative to the user's head (U) may be fully or partially determined as (approximated by) a distance (D) between the target sound source and the user's head. The position detector (PD) may comprise a distance detector (estimator) (DD) providing a distance control signal (DE) indicative of a current estimate of a distance (D) between the position of the target sound source and the user's head. A distance between transmitting and receiving devices may e.g. be estimated by detecting a received signal strength (e.g. a “Received Signal Strength Indicator” (RSSI) or a “Received Channel Power Indicator” (RCPI)) in the receiving device (e.g. Rx in
The detector control signal(s) (‘det’ in
The estimate of a position of the target sound source (S) relative to the user's head (U) may be provided as a user input. The binaural hearing aid system may comprise a user interface (e.g. implemented in an auxiliary device (e.g. a separate processing device of the system) in communication with or forming part of the binaural hearing aid system, see e.g.
The wireless reception detector (WRD) may be configured to provide a reception control signal (RCS) indicating whether or not the at least one streamed audio input signal (saux) comprising the target signal (and possibly second other signals from other sound sources in the environment around the user) is currently received. The wireless reception detector may form part of the wireless receiver (Rx; Rx1, Rx2), which, in dependence of the wireless communication protocol used (e.g. Bluetooth), may provide a ‘no signal’ indicator in case no valid (e.g. Bluetooth) signal is received by the receiver (Rx). The reception control signal (RCS) may be based on the received wireless signal (denoted s′aux in
The binaural audio signal processor (AUD-PRO) of the embodiment of
Tracking the position of the target audio sound source (S) relative to the orientation of the user's head (U) may be used to control the amplification of standard amplified sound of the hearing aids while streaming. When a user, for example, is looking at the TV (including a TV-audio sound transmitter), then the ambient sound amplification (based on microphone inputs) may be automatically reduced, and when the user looks away from the TV, then the ambient sound amplification may be automatically increased.
The binaural audio signal processor (AUD-PRO) is configured to control the gain applied to the at least one streamed audio signal in dependence of the estimate of the position of the target sound source (S) relative to the user's head (U). Thereby the first and second binaurally processed electric output signals (saux,b1, saux,b2) or (out1, out2) providing a spatial sense of origin of the target sound source external to said user's head may be provided. E.g. when a user is looking in the direction of the target sound source (S) of the streamed sound, then the streamed sound amplification may be increased (and vice versa, e.g. decreased, if the look direction of the user deviates from the direction to the target sound source).
The levels of the first and second binaurally processed electric output signals (saux,b1, saux,b2) may be modified in dependence of a difference between a current look direction (LDIR) and a direction (D) to the position of the target sound source (S, cf. e.g. angle θsq in
The levels of said first and second binaurally processed electric output signals (saux,b1, saux,b2) may be modified in dependence of a current distance (D) between said target sound source (S) and the user's head (U). The levels may be increased or decreased, the smaller or larger, respectively, the distance (D) between the target sound source (S) and the user's head (U). The levels of the first and second binaurally processed electric output signals may be modified in dependence of the distance control signal (DE) indicative of a current distance (D) between the target sound source (S) and the user's head (U). The modification of the levels may further be dependent on the reception control signal (RCS) indicating whether the at least one streamed audio input signal (saux) is currently being received. The modification of the levels may further be dependent on the look direction control signal (LDCS=θ) being indicative of the current look direction (LDIR) being equal to or within a certain distance (e.g. angle Δθ, e.g. +/−5°) of the direction (D) to the position of the target sound source (S).
The modification of the levels may further, or alternatively, be dependent on a voice control signal from a voice activity detector indicating the presence of a voice (e.g. the user's voice, or any voice) in the first and second electric sound input signals (x1, x2).
The modification of the levels may further, or alternatively, be dependent on a movement control signal from a movement detector indicating whether or not the user is moving.
The binaural hearing aid system may comprise a separate processing device comprising the binaural audio signal processor (AUD-PRO) and/or the at least one wireless receiver (Rx).
Each of the first and second earpieces (EP1, EP2) may comprise a wireless transceiver adapted for exchanging data, e.g. audio or other data, with the separate processing device (and/or directly between each other).
The binaural audio signal processor (AUD-PRO) may form part of one or both of the first and second earpieces (EP1, EP2) (cf. e.g.
The first and second monaural audio signal processors (M-PRO1, M-PRO2) may, as shown in the embodiment of
The first and second monaural audio signal processors (M-PRO1, M-PRO2) may be configured to estimate a direction of arrival (DOA) of sound from the target sound source (S) independently. In other words, DOA1 is determined in M-PRO1 (e.g. in EP1) in dependence of saux and in1, and DOA2 is determined in M-PRO2 (e.g. in EP2) in dependence of saux and in2. The direction of arrival of sound from the target sound source may be equal to the angle of the direction to the target sound source relative to a normal forward-looking direction of a user, cf. e.g.
The target sound source (S) may e.g. be sound from a television (TV) transmitted to the binaural hearing aid system via a TV-sound transmitter (ED) located together with the TV (see e.g.
The binaural hearing aid system further comprises a position detector (DET) configured to provide an estimate of a current position of the at least one target sound source (S) relative to the user's head and to provide a position detector control signal (det) indicative thereof. At least one (e.g. each) of the input gain controllers (IGC1; IGC2) of the first and second hearing aids (HIl, HIr) is configured to provide the relative weight between said electric sound input signal (in1; in2) and said streamed audio input signal (saux1; saux2) in dependence of the position detector control signal (det).
The first and second hearing aids (HIl, HIr) may comprise first and second earpieces (EP1, EP2 as in
where the weights may be determined based on an angel of a current look direction of the user with a reference direction and/or an estimated distance between the user and the target sound source.
The binaural hearing aid system comprises a wireless transmitter for transmitting data to the audio transmitter (AT). The binaural hearing aid system is configured to transmit the detector control signal (det3), e.g. the position detector control signal, to the audio transmitter. The audio transmitter may comprise a transmit processor (cf. e.g. ‘PRI’ in
A further prioritization between the electric sound input signal (in1; in2) picked up by the respective input transducers (IT1, IT2) of the first and second hearing aids (HIl, HIr) and the streamed audio input signal (saux1; saux2) in dependence of the detector control signal (det; det1, det2) may be provided by respective input gain controllers (IGC1, IGC2) of the first and second hearing aids (HIl, HIr), e.g. as respective weighted sums (out1, out2) of the input signals (in1, saux1) and (in2, saux2), respectively.
In the context of
Thereby a priority (reflected by gains G1, G2=f(θSq(t), q=1, 2, t indicating a time variance of θSq) between two (or more) sound sources is implemented in the audio transmitter (e.g. constituting or forming part of a microphone unit, e.g. a table microphone unit (e.g. a ‘speakerphone)’). The direction to a given target sound source may be compared to a current look direction of a user to thereby evaluate a current interest of the user in said target sound source. In a typical scenario, the user looks at the target sound source of current interest to the user by orienting the head in the direction of the target sound source, e.g. either by turning the head alone, or by including the torso, so that the current look direction is equal to the direction from the user to the target sound source of interest. In other words, the angle between the direction to the target sound source and the current look direction is zero (or close to 0). A top priority should hence be associated with a minimum angle between to the current look direction and the target sound source of current interest. An algorithm providing a maximum gain to the signal transmitted to the binaural hearing aid system sound source associated with a minimum angle and a minimum gain to all other target sound sources provided by the audio transmitter (AT) may e.g. be implemented.
The binaural hearing aid system may comprise a motion sensor providing a motion control signal indicative of a user's current motion. The binaural hearing aid system may be configured to track the position of the user relative to the audio transmitter (AT) providing the target signal (saux) and to provide the spatial sense of origin of the target sound source external to said user's head by applying head-related transfer functions to the first and second binaurally processed electric output signals. The head-related transfer functions (HRTF) may be approximated by the level difference between the two ears. The head-related transfer functions may be approximated by the latency difference between the two ears. The head-related transfer functions may be represented by frequency dependent level and latency differences between the two ears. In case of more than one audio signal is received by the binaural hearing aid system from respective more than one audio transmitters, relevant HRTFs for each of the positions of the more than one audio transmitters may be applied to the corresponding more than one audio signal before being presented to the user. Thereby a spatial sense of origin external to the user's head of the one or more target sound sources corresponding to the sound provided by the more than one audio transmitters to the binaural hearing aid system may be provided.
The binaural hearing aid system may comprise first and second hearing aids. The first and second hearing aids may comprise the first and second earpieces (EP1, EP2), respectively. The first and second hearing aids may be constituted by or comprise air-conduction type hearing aids, bone-conduction type hearing aids, cochlear implant type hearing aids, or a combination thereof.
According to the present disclosure, it is possible to place a streamed sound signal at a desired angle in space. E.g. if a user is having a phone conference with two speakers, the one speaker can be (perceptually) placed at −45 degrees in the horizontal space, and the other speaker at +45 degrees (cf. e.g.
where ‘t’ represents time, “HRIR*Signal” represents the convolution of the impulse responses ‘HRIR’ and the ‘Signal’. Likewise, to obtain the same effect, the corresponding transfer functions ‘HRTF’ can be multiplied with the ‘Signal’ in the (time-)frequency domain (k,m) (where k and m are frequency and time indices, respectively). SignalSpkr1 and SignalSpkr2 represent the wirelessly received ‘at least one streamed audio input signals’ from the respective transmitters (of the (here) two speakers (of the phone conference)).
The offset (or reference direction) of the head orientation can either be:
The HRTF and/or HRIR function can be selected from a predefined set of transfer functions (HRTF) and/or impulse responses (HRIR) stored in a lookup table depending on input angle, frequency and distance. Alternatively, the binaural signals can be calculated with a parametric model, that includes level and latency difference between the two ears as a function of angle, frequency and distance. A parametric model may be easier to implement in a hearing aid system with limited memory and processing power. An example of such a system may be formulated as
where x represents multiplication, and Signal(t+τL,angle) and Signal(t+τR,angle) are the (model) wirelessly received signals from a given target sound source at time t plus the (acoustic) delay τ appropriate for the angle θ at the left and right ears (see e.g.
Likewise, the coefficients τL, angle (=τL(θ)) and TR, angle (=τR(θ)) is the delay difference to be applied to the left and right channel of the streaming signal based on the estimated/desired angle of the signal relative to the head position of the hearing aid user.
This solution ensures low implementation complexity, processing power and memory cost by applying broadband signal gain and delay differences whilst providing some level of spatial orientation perception. Spatial depth perception can be added by inclusion of a distance-based modification (attenuation) to the coefficient values, and room acoustics can also be added by reverberance based modifications.
Solution complexity and perception of target sources may also be enhanced by adding spectral/frequency-based modifications to the gain and delay changes between the right and left ear. A way to achieve this is to expand each coefficient to a vectorized set of values at a discrete number of frequencies providing frequency-based variations in the gain and delay difference between left and right ear. Another way to achieve this could be to apply head-related impulse responses to the left and right signal at a discrete number of angles, as formulated below.
The head-related impulse responses (HRIR) provide the appropriate spatial sensation for the user, and the gains, GL and GR, are the additionally applied gain in order to better hear the desired source. (Usually the source is in front of the user). This enables the system to attenuate other sound sources in the space without removing them completely and maintaining their position in space.
If the streamed sound includes multiple target sources or speakers which should be separated, this can be done by addition of each target source to the signal to left and right ear with individual reference angle input to the gain and delay coefficients as well as the HRIR.
In a specific example, it is proposed to apply additional gain to the streamed sound and/or the hearing aid output dependent of the user head orientation. A simple implementation of this may be to have full streaming gain, (e.g. gain=1 (0 dB)), when the users head is pointing in the direction of the desired source, +/−30°, and to successively reduce the gain at larger deviations in angle (e.g. −3 dB at +/−45°, e.g. −6 dB at +/−60°, e.g. −12 dB at +/−90°, and up to −18 dB at more than +/−90° angles. Alternatively, the implementation may be a more continuously dependence of angle. An example of this is illustrated in
This system can be described as an attention-based gain control system where the signal of interest, either the streamed signal or the hearing aid output is amplified whilst the other is attenuated, in order to achieve optimal listening conditions based on intend. This can be exemplified by adding a coefficient for overall gain which is applied to both the right and left channel of the streamed sound source as well as the hearing aid, HI, output. This can be exemplified by addition of the HI output (based on the microphone signals) to the streamed signal source shown in the equations above, renamed to SignalL, Stream and SignalR, Stream in the formulation below.
The coefficients GStream, Attention and GHI, Attention are the gain/level to be applied to the streamed audio signal and HI output signal based on the attention/engagement of hearing aid user estimated by angle of streaming source relative to the head position of the hearing aid user.
It is further proposed is to track the user's orientation of the head in order to place the streamed sounds in space and keep them in the spatial location when the user turns the head, see e.g.
Head tracking can be measured by a three axis coordinate system (x, y, z) with origin in the center of the user's head. Rotational force of the head is expressed as the rotation around each of these axes and can be named by the terms yaw, pitch, and roll, as illustrated in the
Head tracking using the motion sensing technologies accelerometer, gyroscope and magnetometer can be done in several different ways. Some algorithms include only a single of the above-mentioned motion sensing technologies, while others require multiple. These include but are not limited to:
Complimentary filter implementation example: The complementary filter fuses the accelerometer and integrated gyro data by passing the former through a 1st-order low pass and the latter through a 1st-order high pass filter and adding the outputs. The angular output for each of the three axes can be calculated separately by:
where θk is the angle (between x and y, x and z or y and z)>), ωk is the high pass filtered gyro data, ak is the low pass filtered accelerometer data, k is a sample index-index, Δt is the time between samples, and the filter scalin factor α can be calculated by:
where T determines the filter cut-off frequencies.
Machine learning algorithms: An example of a Neural Network trained to determine the head orientation angle θ relative to the target direction may comprise Extract data from one or more sensors and use the extracted parameters as inputs and determine the angle θ as an output using a pretrained Neural Network. A gyroscope is the optimal sensor for detecting head orientation but is also very power hungry for a hearing system. Accelerometers are really good at detecting the gravitational pull, but not good at detecting head orientation in the horizontal plane. Here machine learning may help to extract information of the head orientation based on accelerometer data. Combining accelerometer data with magnetometer data, may improve the performance of a machine learning model.
An example for how to train a machine learning model may be to collect data from a prototype set of hearing aids including both gyroscope, accelerometer and/or magnetometer. Based on this, a well-known and commonly used algorithm for orientation estimation, such as the Madgwick filter implementation, can be utilized in order to estimate the “true” orientation and be used as the response/target value when training machine learning models. In order to have a model for head orientation based on accelerometer data, the model may comprise raw measurements from one or more axis of the accelerometer as well as computed values based on features of the data. Examples of feature data include either raw or filtered signal point metrics, signal distance metrics, signal statistics, signal spectrum, and other signal characteristics. Signals can both consist of data from a single axis or by any combination of the 3 available axes. The machine learning model can use the signals and features either sample by sample or in sequences based on the implemented model structure. The model can either be configured as a discrete classification model or a continuous regression model based on solution intent. A specific example may comprise sequential signal data used in a 2 stage Convolution Neural Network (CNN) for discrete classification of angular data.
A choice of which head tracking algorithm to use may be based on the available motion sensing hardware technology in the device used as well as desired implementation complexity and computational load.
A definition of the rotational movement parameters pitch, yaw and roll relative to the x, y and z axis of an orthogonal coordinate system is illustrated in
Referring to the placement of the orthogonal coordinate system relative to a user's head in
Pitch is defined as a rotation of the head around the y-axis (e.g. imposed by nodding (moving the head in the x-z-plane)). Can be measured by either a single or a pair of hearing aid devices. A gyroscope in a hearing aid device can measure it directly. Measurements from a pair of gyroscopes in each their hearing aid device can be averaged to provide higher precision. An accelerometer will measure the direction of the gravity field and the pitch can then be determined by calculation of the difference between the actual directions of the gravity and a previous determined ‘normal’ direction i.e. the established z-axis. If two hearing aids both estimate pitch, they can combine their results for better precision.
Yaw is defined as a rotation of the head around the z-axis (e.g. imposed by moving the head from side to side in a horizontal (x-y) plane). Can be measured by either a single or a pair of hearing aid devices. A gyroscope in a hearing aid device can measure it directly. Measurements from a pair of gyroscopes, one in each hearing aid device can be compared (e.g. averaged) to provide higher precision. With an accelerometer there are two ways to estimate yaw or more exact angular velocity ω.
Roll is defined as a rotation of the head around the x-axis (e.g. imposed by moving the head from side to side in a vertical (y-z) plane).
The hearing aid (HI) exemplified in
The hearing aid (HI) comprises a directional microphone system (beamformer filtering unit (BFU)) adapted to enhance a target acoustic source among a multitude of acoustic sources in the local environment of the user wearing the hearing aid. The directional system may be adapted to detect (such as adaptively detect) from which direction a particular part of the microphone signal (e.g. a target part and/or a noise part) originates and/or to receive inputs from a user interface (UI, e.g. a remote control or a smartphone) regarding the present target direction (cf. auxiliary device (AUX) in the lower part of
The hearing aid of
The hearing aid (HI) according to the present disclosure may comprise a user interface (UI), e.g. as shown in
A neural network may be used to determine the head orientation of the user of a hearing aid or hearing aid system.
In the following, some combinations of applications of the present concepts in a hearing aid are mentioned:
It is intended that the structural features of the devices described above, either in the detailed description and/or in the claims, may be combined with steps of the method, when appropriately substituted by a corresponding process.
As used, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well (i.e. to have the meaning “at least one”), unless expressly stated otherwise. It will be further understood that the terms “includes,” “comprises,” “including,” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will also be understood that when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, but an intervening element may also be present, unless expressly stated otherwise. Furthermore, “connected” or “coupled” as used herein may include wirelessly connected or coupled. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. The steps of any disclosed method are not limited to the exact order stated herein, unless expressly stated otherwise.
It should be appreciated that reference throughout this specification to “one embodiment” or “an embodiment” or “an aspect” or features included as “may” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. Furthermore, the particular features, structures or characteristics may be combined as suitable in one or more embodiments of the disclosure. The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects.
The claims are not intended to be limited to the aspects shown herein but are to be accorded the full scope consistent with the language of the claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more.
