Patent Grant
12058496
The present disclosure relates to hearing aids, in particular to the fitting of a hearing aid to a specific user's hearing impairment, specifically to an increased (and, optionally, continuous) personalization of the fitting procedure. In addition to the term ‘a hearing aid’, the term ‘a hearing instrument’ is used in parts of the present disclosure with no intended difference in meaning.
CN111800720A deals with the transmission of audio data to a cloud server providing classification of the sound environment represented by the audio data. Based on the sound scene classification, time and location, a number of predefined settings of the hearing aid are selected.
The present disclosure describes personalized preference learning with simulation and adaptation, e.g. in a double artificial intelligence (AI) loop. The present disclosure relates to a hearing system and a method relying on an initial (, and thereafter e.g. continued,) interaction between a simulation model of a physical environment comprising a specific hearing aid and the physical environment comprising a particular user wearing the specific hearing aid. The simulation model is mainly focused on determining a personalized parameter setting for one or more audio processing algorithms used in the particular hearing aid to process input signals according to the user's needs (e.g. including to compensate for the user's hearing impairment). A ‘personalized parameter setting’ is intended to mean a parameter setting that allows the user to benefit optimally from the processing of an audio signal picked up in a given acoustic environment. In other words, a personalized parameter setting may be a parameter setting that provides a compromise between an optimal compensation for the user's hearing impairment (e.g. to provide maximum intelligibility of speech) while considering the user's personal properties and intentions in a current acoustic environment.
Present user preference learning tools infer the personalized preferences from applying AI (e.g. using machine learning (ML) techniques) to the combination of.
Due to safety and regulatory processes, individuals can only change parameter settings within a limited parameter space defined by the programs put in the hearing instrument by the audiologist. Moreover, updating the programs require a scheduled physical or virtual meeting where the audiologist connects to the hearing instruments and adjusts or replace programs.
Thus, the reality is that current preference learning offerings are not capable of exploring the (parameter) settings space of the hearing instruments sufficiently as the users cannot test all the possible combinations of parameter settings (especially as different parameter settings may be relevant in different sound environments, but also even in the same sound environment if for example the intent, capabilities, or activity is different), because the audiologist is not available 24/7. Moreover, even if the audiologist was available 24/7 it is a rather cumbersome process to schedule even a virtual fitting session whilst communicating in a complex environment and even more cumbersome, if the user wishes to experiment with more than a few parameter settings in each sound environment.
In the present context, the term ‘a sound scene or a sound environment’ is taken to mean a description/characterization of an acoustic scene, e.g.
In the present context, the term ‘intent’ is taken to mean a description/characterization of what the wearer of a hearing instrument intends to do in a given sound environment. E.g. at a cocktail party, the wearers' intent can vary, e.g. among other change between 1) speaking to a person next to them, 2) listen for what is happening around them, or 3) attending to the background music.
In the present context, the term ‘situation’ is taken to mean a combination of an ‘intent’ and ‘a sound scene or a sound environment’. Thus, the above three examples of a user's possible intent in a given sound environment (here, D) ‘cocktail party’) constitute three different situations even if the sound environment is the same.
In the present context the term ‘settings’ is taken to refer to ‘parameter settings’ of a hearing aid program or a processing algorithm. The term ‘hearing aid settings’ may include a set of ‘parameter settings’ covering parameter settings for a multitude of hearing aid programs or processing algorithms.
In the present disclosure, the current solutions for obtaining personalized preferences from applying AI and ML to the aforementioned data types are proposed to be extended by adding at least one (e.g. a majority, or all) of four further steps (cf. I, II, III, IV, below) to the current process where manufacturers provide standard settings, audiologists fine-tune standard settings or start from scratch, and hearing instrument wearers report back to audiologist about preferences or where preferences are monitored through data logging (possibly extended with bio-signals, e.g. EEG, temperature, etc.).
The simulation model may be considered as a digital model of the hearing aid (e.g. the hearing aid worn by the particular user (or a hearing aid that may be a candidate for an alternative hearing aid for the particular user))— thus a digital model replica of a hearing aid that works on sound files. This means that the processing parameters may be EXACTLY the same as those of the hearing aid (or candidate hearing aid) of the particular user (only that their current values may be optimized by the iterative use of the simulation model.
A foreseen benefit of embodiments of a hearing system and method according to the present disclosure is that the end-user (the particular user wearing the hearing aid) or the HCP does not have to search the big parameter space and thus try many small steps themselves, but that the simulation model will find new optimal programs/parameter settings for them.
A Hearing System Comprising a Hearing Aid:
In an aspect of the present application, a hearing system is provided. The hearing system comprises
The Processing Device Comprises
The hearing system may further be configured to feed said time segments of said electric input signal and data representing corresponding user intent (or data representative thereof) from said data logger to said simulation model via said communication interface to thereby allow said simulation model to optimize said specific parameter setting with data from said hearing aid and said user.
The simulation model may be configured to optimize the specific parameter setting with data from the hearing aid and the user in an iterative procedure wherein a current parameter setting for the simulation model of the hearing aid is iteratively changed in dependence of a cost function, and wherein the optimized simulation-based hearing aid setting is determined as the parameter setting optimizing the cost function.
The cost function may comprise a speech intelligibility measure, or other auditory perception measure, e.g. listening effort (e.g. cognitive load).
Thereby an improved hearing aid may be provided.
The processing device may form part of or constitute a fitting system. The processing device may be implemented in a computer, e.g. a laptop, or tablet computer. The processing device may be configured to execute a fitting software for adapt parameters of the hearing aid to the user's needs (e.g. managed by a hearing care professional (HCP)). The processing device may be or comprise a portable electronic device comprising a suitable user interface (e.g. a display and a keyboard, e.g. integrated in a touch sensitive display), e.g. a dedicated processing device for the hearing aid. The portable electronic device may be a smartphone (or similar communication device). The user interface of the processing device may comprise a touch sensitive display in communication with an APP configured to be executed on the smartphone. The APP may comprise (or have access to) fitting software for personalizing settings of the hearing aid to the user's needs. The APP may comprise (or have access to) the simulation model.
The simulation model may e.g. be configured to determine a personalized parameter setting for one or more audio processing algorithms used in the particular hearing aid to process input signals according to the user's needs (e.g. including to compensate for the user's hearing impairment).
The user interface of the hearing aid may comprise an APP configured to be executed on a portable electronic device. The user interface of the hearing aid may comprise a touch sensitive display in communication with an APP configured to be executed on the smartphone. The user interface of the hearing aid and the user interface of the processing device may be implemented in the same device, e.g. the processing device.
The hearing system may be configured to provide that at least a part of the functionality of the processing device is accessible (or provided) via a communication network. The communication interface between the processing device and the hearing aid may be implemented as a network interface, e.g. an interface to the Internet. Thereby at least a part of the functionality of the processing device may be accessible (provided) as a cloud service (e.g. to be executed on a remote server). Thereby a larger processing power to the processing device (e.g. to execute the simulation model, and/or to log data) may be provided. Since the update of processing parameters may not be timing critical, the delay of a cloud service may be acceptable. The communication with the cloud service may be performed via an APP of a smartphone, e.g. forming part of the user interface of the hearing aid. The APP may be configured to buffer data from the data logger before being transmitted to the cloud service (see e.g.
The hearing system may be configured to determine a simulation-based hearing aid setting in dependence of
The set of recorded sound segments may e.g. be mixed according to general environments, e.g. based on prior knowledge and aggregated data logging across different users and/or on individualised environments based on logged data of the user. The hearing system is configured to determine a simulation-based hearing aid setting solely on the hearing profile of the user and model data (e.g. including recorded sound segments) and to use this simulation-based hearing aid setting during an initial (learning) period, where data during normal use of the hearing aid when worn by the particular user for which it is to be personalized can be gathered. Thereby an automized (learning) hearing system may be provided.
The simulation model may comprise a model of acoustic scenes. The model of acoustic scenes may be configured to generate a variety of acoustic scenes from different time segments of electric input signals, where e.g. (relatively) clean target signals (e.g. speech or music or other sound sources) are mixed with different noise types (and levels).
The learning algorithm may be configured to determine said specific parameter setting for said hearing aid in dependence of a variety of different acoustic scenes created by mixing said time segments of the electric input signals in accordance with said model of acoustic scenes. The acoustic scenes may e.g. include general scenes that span standardized acoustic scenes and/or individual (personalized) acoustic scenes according to the logged data from the hearing aid.
The hearing aid system may comprise at least one detector or sensor for detecting a current property of the user or of the environment around the user. The at least one detector or sensor may comprise a movement sensor, e.g. an accelerometer to indicate a current movement of the user. The at least one detector or sensor may comprise a temperature sensor to indicate a current temperature of the user and/or of the environment around the user. The at least one detector or sensor may comprise sensor to bio-signal from the user's body, e.g. an EEG-signal, e.g. for extracting a user's current intent, and/or estimating a user's current mental or cognitive load.
The hearing aid system may be configured to provide that current data from the at least one detector or sensor are stored in the datalogger and associated with other current data stored in the data logger. The sensor/detector data may e.g. be stored together with the user's intent or classification of the current acoustic environment, or with data representing the current acoustic environment, e.g. a time segment of an electric input signal (e.g. a microphone signal), or a signal derived therefrom.
The hearing aid may be constituted by or comprise an air-conduction type hearing aid, a bone-conduction type hearing aid, a cochlear implant type hearing aid, or a combination thereof.
The hearing aid may be adapted to provide a frequency dependent gain and/or a level dependent compression and/or a transposition (with or without frequency compression) of one or more frequency ranges to one or more other frequency ranges, e.g. to compensate for a hearing impairment of a user. The hearing aid may comprise a signal processor for enhancing the input signals and providing a processed output signal.
The hearing aid may comprise an output unit for providing a stimulus perceived by the user as an acoustic signal based on a processed electric signal. The output unit may comprise a number of electrodes of a cochlear implant (for a CI type hearing aid) or a vibrator of a bone conducting hearing aid. The output unit may comprise an output transducer. The output transducer may comprise a receiver (loudspeaker) for providing the stimulus as an acoustic signal to the user (e.g. in an acoustic (air conduction based) hearing aid). The output transducer may comprise a vibrator for providing the stimulus as mechanical vibration of a skull bone to the user (e.g. in a bone-attached or bone-anchored hearing aid). The output unit may (additionally or alternatively) comprise a transmitter for transmitting sound picked up-by the hearing aid to another device, e.g. a far-end communication partner (e.g. via a network, e.g. in a telephone mode of operation, or in a headset configuration).
The hearing aid may comprise an input unit for providing an electric input signal representing sound. The input unit may comprise an input transducer, e.g. a microphone, for converting an input sound to an electric input signal. The input unit may comprise a wireless receiver for receiving a wireless signal comprising or representing sound and for providing an electric input signal representing said sound. The wireless receiver may e.g. be configured to receive an electromagnetic signal in the radio frequency range (3 kHz to 300 GHz). The wireless receiver may e.g. be configured to receive an electromagnetic signal in a frequency range of light (e.g. infrared light 300 GHz to 430 THz, or visible light, e.g. 430 THz to 770 THz).
The hearing aid may comprise a directional microphone system adapted to spatially filter sounds from the environment, and thereby enhance a target acoustic source among a multitude of acoustic sources in the local environment of the user wearing the hearing aid. The directional system may be adapted to detect (such as adaptively detect) from which direction a particular part of the microphone signal originates. This can be achieved in various different ways as e.g. described in the prior art. In hearing aids, a microphone array beamformer is often used for spatially attenuating background noise sources. Many beamformer variants can be found in literature. The minimum variance distortionless response (MVDR) beamformer is widely used in microphone array signal processing. Ideally the MVDR beamformer keeps the signals from the target direction (also referred to as the look direction) unchanged, while attenuating sound signals from other directions maximally. The generalized sidelobe canceller (GSC) structure is an equivalent representation of the MVDR beamformer offering computational and numerical advantages over a direct implementation in its original form.
The hearing aid may comprise antenna and transceiver circuitry allowing a wireless link to an entertainment device (e.g. a TV-set), a communication device (e.g. a telephone), a wireless microphone, or another hearing aid, etc. The hearing aid may thus be configured to wirelessly receive a direct electric input signal from another device. Likewise, the hearing aid may be configured to wirelessly transmit a direct electric output signal to another device. The direct electric input or output signal may represent or comprise an audio signal and/or a control signal and/or an information signal.
In general, a wireless link established by antenna and transceiver circuitry of the hearing aid can be of any type. The wireless link may be a link based on near-field communication, e.g. an inductive link based on an inductive coupling between antenna coils of transmitter and receiver parts. The wireless link may be based on far-field, electromagnetic radiation. Preferably, frequencies used to establish a communication link between the hearing aid and the other device is below 70 GHz, e.g. located in a range from 50 MHz to 70 GHz, e.g. above 300 MHz, e.g. in an ISM range above 300 MHz, e.g. in the 900 MHz range or in the 2.4 GHz range or in the 5.8 GHz range or in the 60 GHz range (ISM=Industrial, Scientific and Medical, such standardized ranges being e.g. defined by the International Telecommunication Union, ITU). The wireless link may be based on a standardized or proprietary technology. The wireless link may be based on Bluetooth technology (e.g. Bluetooth Low-Energy technology), or Ultra WideBand (UWB) technology.
The hearing aid may be or form part of a portable (i.e. configured to be wearable) device, e.g. a device comprising a local energy source, e.g. a battery, e.g. a rechargeable battery.
The hearing aid may comprise a ‘forward’ (or ‘signal’) path for processing an audio signal between an input and an output of the hearing aid. A signal processor may be located in the forward path. The signal processor may be adapted to provide a frequency dependent gain according to a user's particular needs (e.g. hearing impairment). The hearing aid may comprise an ‘analysis’ path comprising functional components for analyzing signals and/or controlling processing of the forward path. Some or all signal processing of the analysis path and/or the forward path may be conducted in the frequency domain, in which case the hearing aid comprises appropriate analysis and synthesis filter banks. Some or all signal processing of the analysis path and/or the forward path may be conducted in the time domain.
The hearing aid may be configured to operate in different modes, e.g. a normal mode and one or more specific modes, e.g. selectable by a user, or automatically selectable. A mode of operation may be optimized to a specific acoustic situation or environment. A mode of operation may include a low-power mode, where functionality of the hearing aid is reduced (e.g. to save power), e.g. to disable wireless communication, and/or to disable specific features of the hearing aid.
The hearing aid may comprise a number of detectors configured to provide status signals relating to a current physical environment of the hearing aid (e.g. the current acoustic environment), and/or to a current state of the user wearing the hearing aid, and/or to a current state or mode of operation of the hearing aid. Alternatively or additionally, one or more detectors may form part of an external device in communication (e.g. wirelessly) with the hearing aid. An external device may e.g. comprise another hearing aid, a remote control, and audio delivery device, a telephone (e.g. a smartphone), an external sensor, etc.
One or more of the number of detectors may operate on the full band signal (time domain). One or more of the number of detectors may operate on band split signals ((time-) frequency domain), e.g. in a limited number of frequency bands.
The number of detectors may comprise a level detector for estimating a current level of a signal of the forward path. The detector may be configured to decide whether the current level of a signal of the forward path is above or below a given (L-)threshold value. The level detector operates on the full band signal (time domain). The level detector operates on band split signals ((time-) frequency domain).
The hearing aid may comprise a voice activity detector (VAD) for estimating whether or not (or with what probability) an input signal comprises a voice signal (at a given point in time). A voice signal may in the present context be taken to include a speech signal from a human being. It may also include other forms of utterances generated by the human speech system (e.g. singing). The voice activity detector unit may be adapted to classify a current acoustic environment of the user as a VOICE or NO-VOICE environment. This has the advantage that time segments of the electric microphone signal comprising human utterances (e.g. speech) in the user's environment can be identified, and thus separated from time segments only (or mainly) comprising other sound sources (e.g. artificially generated noise). The voice activity detector may be adapted to detect as a VOICE also the user's own voice. Alternatively, the voice activity detector may be adapted to exclude a user's own voice from the detection of a VOICE.
The hearing aid may comprise an own voice detector for estimating whether or not (or with what probability) a given input sound (e.g. a voice, e.g. speech) originates from the voice of the user of the system. A microphone system of the hearing aid may be adapted to be able to differentiate between a user's own voice and another person's voice and possibly from NON-voice sounds.
The number of detectors may comprise a movement detector, e.g. an acceleration sensor. The movement detector may be configured to detect movement of the user's facial muscles and/or bones, e.g. due to speech or chewing (e.g. jaw movement) and to provide a detector signal indicative thereof.
The hearing aid may comprise a classification unit configured to classify the current situation based on input signals from (at least some of) the detectors, and possibly other inputs as well. In the present context ‘a current situation’ may be taken to be defined by one or more of
The classification unit may be based on or comprise a neural network, e.g. a trained neural network.
The hearing aid may comprise an acoustic (and/or mechanical) feedback control (e.g. suppression) or echo-cancelling system.
The hearing aid may further comprise other relevant functionality for the application in question, e.g. compression, noise reduction, etc.
The hearing aid may comprise a hearing instrument, e.g. a hearing instrument adapted for being located at the ear or fully or partially in the ear canal of a user, e.g. a headset, an earphone, an ear protection device or a combination thereof.
Use:
In an aspect, use of a hearing aid as described above, in the ‘detailed description of embodiments’ and in the claims, is moreover provided. Use may be provided in a system comprising one or more hearing aids (e.g. hearing instruments), headsets, ear phones, active ear protection systems, etc., e.g. in handsfree telephone systems, teleconferencing systems (e.g. including a speakerphone), public address systems, karaoke systems, classroom amplification systems, etc.
A Method of Determining a Hearing Aid Setting:
In an aspect, a method of determining hearing aid setting comprising a parameter setting, or a set of parameter settings, for a specific hearing aid of a particular user, the method comprising S1. Providing a simulation-based hearing aid setting in dependence of
S2. Transferring the simulation-based hearing aid setting to an actual version of said specific hearing aid.
S3. Using the simulation-based hearing aid setting on said actual hearing aid, when worn by the user.
S4. Logging data from the actual hearing aid, said data including data representing encountered sound environments and the user's classification thereof.
S5. Transferring the logged data to the simulation model.
S6. Optimizing said simulation-based hearing aid setting determined in step S1 based on said logged data, optionally mixed with said recorded sound segments,
S7. Transferring the optimized simulation-based hearing aid setting to the actual version of said specific hearing aid is furthermore provided by the present application.
It is intended that some or all of the structural features of the system and device described above, in the ‘detailed description of embodiments’ or in the claims can be combined with embodiments of the method, when appropriately substituted by a corresponding process and vice versa. Embodiments of the method have the same advantages as the corresponding system and device.
The method may comprise that steps S4-S7 are repeated (e.g. continually, e.g. with a specific frequency or triggered by specific events, or manually initiated (e.g. by the user or by a HCP).
The method may comprise that step S4 further comprises logging data from one or more of the activities of the user, the intent of the user, and the priorities of the user (in the given acoustic environment), see e.g.
In an aspect, a method of determining a hearing aid setting comprising a parameter setting, or set of parameter settings, for a specific hearing aid of a particular user is provided by the present disclosure. The method comprises:
An embodiment of the method is illustrated in
Step S1 can be influenced by logging data obtained with the same hearing aid or with another hearing aid without it having been part of the loop.
Meta-data of the hearing aid may e.g. be data derived by the hearing aid from input sound to the hearing aid. Meta-data of the hearing aid may e.g. comprise input signal levels (e.g. provided by a level detector connected to an electric input signal provided by a microphone (or to a processed version thereof). Meta-data of the hearing aid may e.g. comprise quality measures of an input signal to the hearing aid, e.g. a signal to noise ratio (SNR) of an electric input signal provided by a microphone (or of a processed version thereof), e.g. estimates of the persons own voice activity, internal and proprietary processing parameters from the hearing aid algorithms, estimates of effort, estimates of intelligibility, estimates of head and body movements, actual recordings of the microphone signal, sound scene classifications. The meta-data of a hearing aid may e.g. be logged continuously, or taken at certain occasions, e.g. triggered by a specific event or criterion (e.g. exceeding a threshold), or be user-triggered.
The method comprises two loops: An inner loop comprising steps S2-S6, and an outer loop comprising steps S1-S11.
The simulation model of the hearing aid may represent the user's hearing aid or another hearing aid, e.g. a hearing aid style that may be considered as a useful alternative for the user.
The simulation model is a digital simulation of a hearing aid that processes sound represented in digital format with a (current, but configurable) set of hearing aid settings. It takes sounds, either direct recordings from the users hearing aid, or sounds generated by mixing sounds from the database according to the users Meta-data and settings, as inputs and provides sound as an output.
A Computer Readable Medium or Data Carrier:
In an aspect, a tangible computer-readable medium (a data carrier) storing a computer program comprising program code means (instructions) for causing a data processing system (a computer) to perform (carry out) at least some (such as a majority or all) of the (steps of the) method described above, in the ‘detailed description of embodiments’ and in the claims, when said computer program is executed on the data processing system is furthermore provided by the present application.
By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Other storage media include storage in DNA (e.g. in synthesized DNA strands). Combinations of the above should also be included within the scope of computer-readable media. In addition to being stored on a tangible medium, the computer program can also be transmitted via a transmission medium such as a wired or wireless link or a network, e.g. the Internet, and loaded into a data processing system for being executed at a location different from that of the tangible medium.
A Computer Program:
A computer program (product) comprising instructions which, when the program is executed by a computer, cause the computer to carry out (steps of) the method described above, in the ‘detailed description of embodiments’ and in the claims is furthermore provided by the present application.
A Data Processing System:
In an aspect, a data processing system comprising a processor and program code means for causing the processor to perform at least some (such as a majority or all) of the steps of the method described above, in the ‘detailed description of embodiments’ and in the claims is furthermore provided by the present application.
An App:
In a further aspect, a non-transitory application, termed an APP, is furthermore provided by the present disclosure. The APP comprises executable instructions configured to be executed on an auxiliary device to implement a user interface for a hearing aid or a hearing system described above in the ‘detailed description of embodiments’, and in the claims. The APP may be configured to run on cellular phone, e.g. a smartphone, or on another portable device (e.g. the processing device) allowing communication with said hearing aid or said hearing system.
The aspects of the disclosure may be best understood from the following detailed description taken in conjunction with the accompanying figures. The figures are schematic and simplified for clarity, and they just show details to improve the understanding of the claims, while other details are left out. Throughout, the same reference numerals are used for identical or corresponding parts. The individual features of each aspect may each be combined with any or all features of the other aspects. These and other aspects, features and/or technical effect will be apparent from and elucidated with reference to the illustrations described hereinafter in which:
The figures are schematic and simplified for clarity, and they just show details which are essential to the understanding of the disclosure, while other details are left out. Throughout, the same reference signs are used for identical or corresponding parts.
Further scope of applicability of the present disclosure will become apparent from the detailed description given hereinafter. However, it should be understood that the detailed description and specific examples, while indicating preferred embodiments of the disclosure, are given by way of illustration only. Other embodiments may become apparent to those skilled in the art from the following detailed description.
The detailed description set forth below in connection with the appended drawings is intended as a description of various configurations. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. Several aspects of the apparatus and methods are described by various blocks, functional units, modules, components, circuits, steps, processes, algorithms, etc. (collectively referred to as “elements”). Depending upon particular application, design constraints or other reasons, these elements may be implemented using electronic hardware, computer program, or any combination thereof.
The electronic hardware may include micro-electronic-mechanical systems (MEMS), integrated circuits (e.g. application specific), microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), gated logic, discrete hardware circuits, printed circuit boards (PCB) (e.g. flexible PCBs), and other suitable hardware configured to perform the various functionality described throughout this disclosure, e.g. sensors, e.g. for sensing and/or registering physical properties of the environment, the device, the user, etc. Computer program shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
The present application relates to the field of hearing aids, in particular to personalizing processing of a hearing aid to its current user.
In the present disclosure, the current solutions for obtaining personalized preferences from applying AI and ML to the aforementioned data types are proposed to be extended by adding at least one (e.g. a majority, or all) of four further steps (cf. I, II, III, IV, below) to the current process where manufacturers provide standard settings, audiologists fine-tune standard settings or start from scratch, and hearing instrument wearers report back to audiologist about preferences or where preferences are monitored through data logging (possibly extended with bio-signals, e.g. EEG, temperature, etc.).
I. A First Step May Comprise Determining and Verifying a Simulation-Based Hearing Aid Setting:
Ia: Simulation Based Optimization of Prescribed Hearing Aid Settings with Respect to Speech Intelligibility or Other Domains Like Audibility, Comfort, Spatial Clarity, Etc.
Consider a hearing loss and outcome simulation engine (one particular embodiment is denoted FADE (described in [Schädler et al.; 2018], [Schadler et al.; 2016]), which handles hearing loss simulation, processing simulation, and estimation of intelligibility (involving automatic speech recognition), which is used as the example embodiment hereafter). The simulation engine FADE takes a set of recorded and transcribed sentences (e.g. both audio and text is available), a set of background noises (as audio), parameters describing an individual's hearing loss, an instance of a hearing aid (either physical instance or a digital equivalent) fitted to the individual hearing loss. The process starts by processing sounds from a database with prescribed settings and passing this mixture through the hearing loss and hearing outcome simulation, where FADE predicts the speech understanding performance. Analyzing the impact on the performance as a function of the hearing aid settings, a preference recommender learning tool then optimizes the settings of the hearing aid instance so that the automatic speech recognizer gets the best understanding (as predicted by FADE) for a particular hearing loss.
Ib: Check Optimized Hearing Aid Settings on Actual Hearing Aid(s) when Worn by the User).
The optimized settings may be subject to approval by the audiologist or directly. The optimized settings from the step Ia are then transferred to actual hearing aids worn by the individuals (e.g. a particular user). And here the traditional analytical method that combines context and ratings is used to confirm or reject whether the optimized settings are indeed optimal taking usage patterns into account.
II. A Second Step May Comprise Optimization of Hearing Aid Settings Based on Data from Actual Use.
IIa: Optimization of Hearing Aid Settings Based on Behavioral Speech- and Non-Speech-Auditory Performance Measures.
A new range of optimization metrics independent of the automatic speech recognizer used in FADE is introduced. These optimization metrics combine behavioral speech and non-speech auditory performance measures, e.g. detection thresholds for spectro-temporal modulation (STM) (like Audible Contrast Threshold (ACT)) or spectral contrasts (ripples or frequency resolution tests), transmission of auditory salient cues (interaural level, time, and phase cues, etc.), or correlated psychophysiological measures, such as EEG or objective measures of listening effort and sound quality (cf. e.g. validation step 2A in
IIb: Optimization of Hearing Aid Settings Based on User Preferences.
We also introduce a new set of scales and criteria with which the individual hearing aid user can choose to report their preferences in a given situation. In one situation, e.g., it is not the perceived speech recognition that the hearing aid user decides is of importance; instead the user reports on clarity of the sound scene, and this metric may hereafter be given more weight in the simulation of the present sound scene and possibly in similar scenes, cf. e.g. validation step 2 (2A, 2B) in
III. A Third Step May Provide Feedback to the Simulation Model of Logged Data Captured During Wear of Hearing Aid(s) by the User which May Spawn a New Round of Optimization with the Simulated Sound Scenes that Statistically Match the Encountered Scenes.
A third step may comprise that data logged from hearing aids that describe sound scenes in level, SNR, etc., are used to augment the scenes, which are used for the simulation and optimization of hearing aid settings, cf. e.g. validation step 3 in
IV. A Third Step May Provide Optimization of Hearing Aid Settings Based on Personality Traits.
A fourth step may comprise that the simulation model estimates personality traits of each individual from questionnaires or indirectly from data and uses this in the optimization of hearing aid settings. The estimated personality traits may further be used during testing and validating the proposed settings. Recently an interesting finding how especially neuroticism and extraversion among the Big5 (here the 5 most probable of the 5 most frequently occurring) personality traits impact the acceptance of noise, performance in noise, and perceived performance in noise (cf. e.g. [Wöstmann et al.; 2021], and regarding the ‘Big Five personality traits’, see e.g. Wikipedia at https://en.wikipedia.org/wiki/Big_Fivepersonalitytraits), cf. e.g. validation step 4 in
The general function of the method and hearing system illustrated in
An aim of the hearing system and method is to determine a personalized parameter setting for one or more audio processing algorithms used in the particular hearing aid to process input signals according to the user's needs (e.g. including to compensate for the user's hearing impairment). A ‘personalized parameter setting’ is intended to mean a parameter setting that allows the user to benefit optimally from the processing of an audio signal picked up in a given acoustic environment. In other words, a personalized parameter setting may be a parameter setting that provides a compromise between an optimal compensation for the user's hearing impairment (e.g. to provide maximum intelligibility of speech) while considering the user's personal properties and intentions in a current acoustic environment.
The embodiment of a hearing system shown in
The hearing system comprises a communication interface between the processing device (hosting the model of the physical environment) and the hearing aid of the particular user to allow the processing device and the hearing aid to exchange data between them (cf. arrows ‘S7’) from ‘Model of physical environment’ (processing device) to ‘Physical environment’ (hearing aid, or an intermediate device in communication with the hearing aid)).
A HCP may be involved in the transfer of the model based hearing aid setting to the actual hearing aid, e.g. in a fitting session (cf. ‘Hearing care professional’, and callouts indicating an exchange of information between the HCP and the user of the hearing aid, cf. ‘Particular user’ in
When the simulation-based hearing aid setting has been transferred to the actual version of said specific hearing aid and applied to the appropriate processing algorithms, the user wears the hearing aid in a learning period where data are logged. The logged data may e.g. include data representing encountered sound environments (e.g. time segments of an electric input signal, or signals or parameters derived therefrom, e.g. as meta-data) and the user's classification thereof and/or the user's intent when present in given sound environment. After a period of time (or continuously, or according to a predefined scheme, or at a session with a HCP), data are transferred from the data logger to the simulation model via the communication interface (cf. arrow ‘Validation’ in
The 2nd loop can be repeated continuously or with a predefined frequency, or triggered by specific events (e.g. power-up, data logger full, consultation with HCP (e.g. initiated by HCP), initiated by the user via a user interface, etc.).
These data are schematically illustrated in
Current Process Example
User Alice schedules and appointment for hearing aid fitting with audiologist Bob, and have her hearing measured and characterized by standard procedures like audiograms, questionnaires, specific speech tests, and in-clinic simulation of scenes and settings.
Alice then leaves Bob having one (or a few) distinct hearing aid settings on her hearing instruments and starts using the hearing instruments in her everyday situations.
After a while Alice returns to Bob for a follow-up session where they talk about the situations that Alice has encountered both the good and less good experiences. Based on this dialogue, and possibly assisted by looking at usage data (duration, sound environments, and relative use of the different settings) as well as experience and insights of Bob, Bob then adjust the settings in the hearing instrument so that the palette of settings better matches what Bob believes will benefit Alice. However, Bob is not aware of an update to the noise reduction and is therefore not capable of utilizing this to increase the benefits of the hearing instruments.
Alice now returns to using her hearing instruments in her everyday situations.
After another while Alice returns to Bob again and goes through the same process as last time. Still, Bob is not aware of an update to the noise reduction and is therefore not capable of utilizing this to the full extent.
Process Example According to the Present Disclosure
User Alice schedules and appointment for hearing aid fitting with audiologist Bob, and have her hearing measured and characterized by standard procedures like audiograms, questionnaires, specific speech tests, and in-clinic simulation of scenes and settings.
Alice then leaves Bob having one (or a few) distinct hearing aid settings on her hearing instruments and starts using the hearing instruments in her everyday situations.
While Alice uses the hearing instruments, the hearing instruments and the APP (e.g. implemented on a smartphone or other appropriate processing comprising display and data entry functionality) collects data about the sound environments and possibly intents of Alice in those situations (cf. ‘Data logger’ in
Meanwhile, the cloud service simulates sound environments and situations with the data that describes her hearing, her sound environments, intents, and priorities collected with the smartphone and the hearing instruments. The simulation model may be implemented as one part of the cloud service where logged data are used as inputs to the model related to the situations to be simulated. Another part of the cloud service may be the analysis of the metrics to learn the preference for the tested settings (cf. e.g. validation step 2 (2A, 2B) in
When Alice returns to Bob for a follow-up session they talk about the situations that Alice has encountered—both the good and less good experiences. Based on this dialogue Bob reviews the proposals of optimal settings and selects the ones which in his experience together with the description of the situations fit Alice's needs and situations the best. Since the devices were given to Alice, the noise reduction was updated and the optimization suggested a setting that utilizes this. The hearing instrument(s) may e.g. be (firmware-)updated during use, e.g. when recharged. The hearing instrument(s) may e.g. be firmware updated out of this cycle (e.g. at a (physical or remote) consultation with a hearing care professional). The hearing instrument(s) may not need to have firmware updates if a “new” feature is just launched by enabling a feature in the fitting software.
When Alice returns to Bob for another follow-up session, they can also see which of the individual settings that Alice rated as good and which ones she has used either a lot or for specific situations.
Further Examples
Embodiments of the present disclosure may include various combinations of the following features:
The hearing care professional (HCP) has access to a fitting system comprising the model of the physical environment including the AI-simulation model. A number of interfaces between the fitting system and the hearing aid and an associated processing device serving the hearing aid, e.g. a smartphone (running an APP forming part of a user interface for the hearing aid, denoted ‘HA-User interface (APP)’ in
In the embodiment of a hearing system shown in
Thereby, a highly flexible hearing system capable of providing an initial simulation-based hearing aid setting, which can be personalized during use of the hearing aid can be provided. By having access to processing power at different levels, partly in the hearing aid, partly on the handheld or portable processing device, and partly on a network server, the hearing system is capable of utilizing computationally demanding tasks, e.g. involving artificial intelligence, e.g. learning algorithms based on machine learning techniques, e.g. neural networks. Processing tasks may hence be allocated to an appropriate processor taking into account computational intensity AND timing of the outcome of the processing task to provide a resulting output signal to the user with an acceptable quality and latency.
The method may comprise some or all of the following steps (S1-S7).
The specific hearing aid may e.g. be of a specific style (e.g. a ‘receiver in the ear’ style having a loudspeaker in the ear canal and a processing part located at or behind pinna, or any other known hearing aid style). The specific hearing aid may be a further specific model of the style that the particular user is going to wear (e.g. exhibiting particular audiological features (e.g. regarding noise reduction/directionality, connectivity, access to sensors, etc.), e.g. according to a specific price segment (e.g. a specific combination of features)).
S1. Providing a simulation-based hearing aid setting in dependence of
The hearing profile may e.g. comprise an audiogram (showing a hearing threshold (or hearing loss) versus frequency for the (particular) user. The hearing profile may comprise further data related to the user's hearing ability (e.g. frequency and/or level resolution, etc.). A simulation model of the specific hearing aid may e.g. be configured to allow a computer simulation of the forward path of the hearing aid from an input transducer to an output transducer to be made. The set of recorded sound segments may e.g. comprise recorded and transcribed sentences (e.g. making both audio and text available), and a set of background noises (as audio). Thereby a multitude of electric input signals may be generated by mixing recorded sentences (of known content) with different noise types and levels of noise (relative to the target signal (sentence)). The simulation model may e.g. include an automatic speech recognition algorithm that estimates the content of the (noisy) sentences. Since the contents are known, an estimate of the intelligibility of each (noisy sentence) can be estimated. The simulation model may e.g. allow the simulation-based hearing aid setting to be optimized with respect to speech intelligibility. An optimal hearing aid setting for the particular user may e.g. be determined by optimizing the processing parameters of the simulation model in an iterative procedure in dependence of the recorded sound segments, the hearing profile, the simulation model, and a cost function (see e.g.
S2. Transferring the simulation-based hearing aid setting to an actual version of said specific hearing aid.
The simulation model may e.g. run on a specific processing device, e.g. a laptop or tablet computer or a portable device, e.g. a smart phone. The processing device and the actual hearing aid may comprise antenna and transceiver circuitry allowing the establishment of a wireless link between them to provide that an exchange of data between the hearing aid and the processing device can be provided. The simulation-based hearing aid setting may be applied to a processor of the hearing aid and used to process the electric input signal provided by one or more input transducers (e.g. microphones) to provide a processed signal intended for being presented to the user, e.g. via an output transducer of the hearing aid. The actual hearing aid may have a user-interface, e.g. implemented as an APP of a portable processing device, e.g. a smartphone. The user interface may be implemented on the same device as the simulation model. The user interface may be implemented on another device than the simulation model.
S3. Using the simulation-based hearing aid setting on said actual hearing aid, when worn by the user.
The simulation-based hearing aid setting is determined solely based on the hearing profile of the user and model data (e.g. including recorded sound segments). This simulation-based hearing aid setting is intended for use during an initial (learning) period, where data during normal use of the hearing aid, when worn by the particular user for which it is to be personalized, can be captured. Thereby an automized (learning) hearing system may be provided.
S4. Logging data from the actual hearing aid, said data including data representing encountered sound environments and the user's classification thereof.
A user interface, e.g. comprising an APP executed on a portable processing device, may be used as an interface to the hearing aid (and thus to the processing device). Thereby the user's inputs may be captured. Such inputs may e.g. include the user's intent in a given sound environment, and/or a classification of such sound environment. The step S4 may e.g. further comprise logging data from the activities of the user, the intent of the user, and the priorities of the user. The latter feature is shown in
S5. Transferring the logged data to the simulation model.
Thereby data from the user's practical use of the hearing aid can be considered by the simulation model (validation).
S6. Optimizing said simulation-based hearing aid setting based on said logged data.
A 2nd loop of the learning algorithm is executed using input data from the hearing aid reflecting acoustic environments experienced by the user while wearing the hearing aid (optionally mixed with recorded sound segments with known characteristics, see e.g. step S1), and the user's evaluation of these acoustic environments and/or his or her intent while being exposed to said acoustic environments. Again, an optimal hearing aid setting for the particular user may be determined by optimizing the processing parameters of the simulation model in an iterative procedure in dependence of the user logged and possibly pre-recorded sound segments, the hearing profile, the simulation model, and a cost function, e.g. related to an estimated speech intelligibility (see e.g.
S7. Transferring the optimized simulation-based hearing aid setting to the actual version of said specific hearing aid
The optimized simulation-based hearing aid setting thus represents a personalized setting of parameters that builds on the initial model data and data extracted from the user's wear of the hearing aid in the acoustic environment that he or she encounters during normal use.
Steps S4-S7 may be repeated, e.g. according to a predefined or adaptively determined scheme, or initiated via a user interface (as indicated by the dashed arrow from step S7 to step S4) or continuously.
sound environments . . . ’) in
An Exemplary Method of Determining a Hearing Aid Setting:
The method is configured to determine a set of parameter settings (setting(s) for brevity in the following) for a specific hearing aid of a particular user covering encountered listening situations. The steps S1-S11 of the method are described in the following:
S1. Meta-data charactering the encountered sound environments and listening situations (from HA data logging) leading to a set of simulated sound environments and listening situations from mixing sounds from a database.
S2. A digital simulation model of the user's own hearing aid that processes the sounds from S1 according to a current set of parameter settings.
S3. A digital simulation of the user's hearing loss based on the hearing profile of the user that simulates the direct impact on the sound due to e.g. deterioration from limited audibility, limited spectral resolution, etc.
S4. An AI-Hearing model that simulates the perception of the impaired hearing, e.g. 1) speech intelligibility based on automatic speech recognizers or metrics like E-STOI, listening effort, comfort based on established metrics.
S5. An optimization of outcomes from S4, e.g. maximization of intelligibility or comfort or sound quality, or minimization of listening effort updating the parameter settings of S2.
S6. Repetition of steps S2-S6 until convergence or set performance is reached (see arrow in
S7. Transferring the optimized simulation-based hearing aid setting(s) to the actual version of said specific hearing aid.
S8. Using the simulation-based hearing aid setting on said actual hearing aid, when worn by the user.
S9. Logging data from the actual hearing aid, said data including data representing encountered sound environments and the user's classification thereof.
S10. Transferring the logged data to the simulation model.
S11. Optimizing said simulation-based hearing aid setting based on said logged data following S1-S7 (see arrow in
S1 can be influenced by logging data obtained with same hearing aid or other hearing aid without it having been part of the loop.
The method comprises two loops: An ‘inner loop’: S2-S6 (denoted S6 in
The simulation model of the hearing aid (user's or other) is a digital simulation of a hearing aid that processes sound represented in digital format with a set of hearing aid settings. It takes sounds (e.g. provided as meta-data) and current (adaptable) settings as input and outputs sound.
Embodiments of the disclosure may e.g. be useful in applications such as fitting of a hearing aid or hearing aids to a particular user.
It is intended that the structural features of the devices described above, either in the detailed description and/or in the claims, may be combined with steps of the method, when appropriately substituted by a corresponding process.
As used, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well (i.e. to have the meaning “at least one”), unless expressly stated otherwise. It will be further understood that the terms “includes,” “comprises,” “including,” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will also be understood that when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, but an intervening element may also be present, unless expressly stated otherwise. Furthermore, “connected” or “coupled” as used herein may include wirelessly connected or coupled. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. The steps of any disclosed method are not limited to the exact order stated herein, unless expressly stated otherwise.
It should be appreciated that reference throughout this specification to “one embodiment” or “an embodiment” or “an aspect” or features included as “may” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. Furthermore, the particular features, structures or characteristics may be combined as suitable in one or more embodiments of the disclosure. The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects.
The claims are not intended to be limited to the aspects shown herein but are to be accorded the full scope consistent with the language of the claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more.
| Number | Date | Country | Kind |
|---|---|---|---|
| 21190156 | Aug 2021 | EP | regional |
| Number | Name | Date | Kind |
|---|---|---|---|
| 20120183165 | Foo et al. | Jul 2012 | A1 |
| 20220279296 | Davis | Sep 2022 | A1 |
| 20230037356 | Pontoppidan | Feb 2023 | A1 |
| 20230290333 | Tiefenau | Sep 2023 | A1 |
| 20230421974 | Luo | Dec 2023 | A1 |
| Number | Date | Country |
|---|---|---|
| 111800720 | Oct 2020 | CN |
| 1708543 | Oct 2006 | EP |
| WO 2021144964 | Jul 2021 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20230037356 A1 | Feb 2023 | US |
