Patent Grant
10685663
The exemplary and non-limiting embodiments relate generally to speech capture and audio signal processing, particularly headphone, and microphone signal processing.
Recognition of the sound from person's mouth (for example, speech, singing, etc.) using in-ear microphone and conventional signal processing is difficult because of the complexity of noisy systems. Audio, particularly speech, may be recorded and output via headphone and/or microphones. Signal processing for in-ear recording of audio may include application of an artificial bandwidth extension (ABE).
Certain abbreviations that may be found in the description and/or in the Figures are herewith defined as follows:
3GPP Third Generation Partnership Project
5G 5th generation mobile networks (or wireless systems)
gNB gNodeB
LTE Long Term Evolution
MM Mobility Management
MTC machine type communications
NR New Radio
SGW Serving GW
This section is intended to include examples and is not intended to be limiting.
In an example of an embodiment, a method is disclosed that includes accessing, by at least one processing device, a real noise-free audible signal including at least one real in-ear microphone audible signal and at least one real external microphone audible signal and at least one noise signal; training a generative network to generate an external microphone signal from an in-ear microphone signal based on the at least one real in-ear microphone audible signal and the at least one real external microphone audible signal; and outputting the generative network.
In an example of an embodiment, a method is disclosed that includes receiving, by an outside-the-ear microphone, a room sound transfer of at least one audio signal of interest and at least one noise signal; receiving, by an in-ear microphone, an in-body transfer of at least one audio signal of interest and the at least one noise signal, and an incoming audio signal; performing incoming audio cancellation on an output of the in-ear microphone; and performing deep learning inference based on an output of the incoming audio cancellation, an output of the outside-the-ear microphone and a pre-trained deep learning model to determine a noise-free (for example, clean) natural sound.
An example of an apparatus includes at least one processor; and at least one non-transitory memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to access a real noise-free audible signal including at least one real in-ear microphone audible signal and at least one real external microphone audible signal and at least one noise signal; train a generative network to generate an external microphone signal from an in-ear microphone signal based on the at least one real in-ear microphone audible signal and the at least one real external microphone audible signal; and output the generative network.
An example of an apparatus includes at least one processor; and at least one non-transitory memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to receive, by an outside-the-ear microphone, a room sound transfer of at least one audio signal of interest and at least one noise signal; receive, by an in-ear microphone, an in-body transfer of at least one audio signal of interest and the at least one noise signal, and an incoming audio signal; perform incoming audio cancellation on an output of the in-ear microphone; and perform deep learning inference based on an output of the incoming audio cancellation, an output of the outside-the-ear microphone and a pre-trained deep learning model to determine a clean natural sound.
The foregoing and other aspects of embodiments of this invention are made more evident in the following Detailed Description, when read in conjunction with the attached Drawing Figures, wherein:
The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments. All of the embodiments described in this Detailed Description are exemplary embodiments provided to enable persons skilled in the art to make or use the invention and not to limit the scope of the invention which is defined by the claims.
In the example embodiments as described herein a method and apparatus may perform speech capture that provides accurate and real-time audible (for example, speech) signal modeling and enhancement in order to achieve natural speech recording and transfer by deep learning and deep generative modeling using at least an in-ear microphone signal. Deep learning is a class of machine learning algorithms that uses a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer may use the output from the previous layer as input. Deep learning systems may learn in supervised (for example, classification) and/or unsupervised (for example, pattern analysis) manners. Deep learning systems may learn multiple levels of representations that correspond to different levels of abstraction; and the levels in deep learning may form a hierarchy of concepts. A Deep Generative model is a generative model that is implemented using deep learning.
Turning to
The gNB (NR/5G Node B but possibly an evolved NodeB) 170 is a base station (e.g., for LTE, long term evolution) that provides access by wireless devices such as the UE 110 to the wireless network 100. The gNB 170 includes one or more processors 152, one or more memories 155, one or more network interfaces (N/W I/F(s)) 161, and one or more transceivers 160 interconnected through one or more buses 157. Each of the one or more transceivers 160 includes a receiver, Rx, 162 and a transmitter, Tx, 163. The one or more transceivers 160 are connected to one or more antennas 158. The one or more memories 155 include computer program code 153. The gNB 170 includes a ZZZ module 150, comprising one of or both parts 150-1 and/or 150-2, which may be implemented in a number of ways. The ZZZ module 150 may be implemented in hardware as ZZZ module 150-1, such as being implemented as part of the one or more processors 152. The ZZZ module 150-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the ZZZ module 150 may be implemented as ZZZ module 150-2, which is implemented as computer program code 153 and is executed by the one or more processors 152. For instance, the one or more memories 155 and the computer program code 153 are configured to, with the one or more processors 152, cause the gNB 170 to perform one or more of the operations as described herein. The one or more network interfaces 161 communicate over a network such as via the links 176 and 131. Two or more gNBs 170 (or gNBs and eNBs) communicate using, e.g., link 176. The link 176 may be wired or wireless or both and may implement, e.g., an X2 interface.
The one or more buses 157 may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, wireless channels, and the like. For example, the one or more transceivers 160 may be implemented as a remote radio head (RRH) 195, with the other elements of the gNB 170 being physically in a different location from the RRH, and the one or more buses 157 could be implemented in part as fiber optic cable to connect the other elements of the gNB 170 to the RRH 195.
It is noted that description herein indicates that “cells” perform functions, but it should be clear that the gNB that forms the cell will perform the functions. The cell makes up part of a gNB. That is, there can be multiple cells per gNB.
The wireless network 100 may include a network control element (NCE) 190 that may include MME (Mobility Management Entity)/SGW (Serving Gateway) functionality, and which provides connectivity with a further network, such as a telephone network and/or a data communications network (e.g., the Internet). The gNB 170 is coupled via a link 131 to the NCE 190. The link 131 may be implemented as, e.g., an Si interface. The NCE 190 includes one or more processors 175, one or more memories 171, and one or more network interfaces (N/W I/F(s)) 180, interconnected through one or more buses 185. The one or more memories 171 include computer program code 173. The one or more memories 171 and the computer program code 173 are configured to, with the one or more processors 175, cause the NCE 190 to perform one or more operations.
The wireless network 100 may implement network virtualization, which is the process of combining hardware and software network resources and network functionality into a single, software-based administrative entity, a virtual network. Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is categorized as either external, combining many networks, or parts of networks, into a virtual unit, or internal, providing network-like functionality to software containers on a single system. Note that the virtualized entities that result from the network virtualization are still implemented, at some level, using hardware such as processors 152 or 175 and memories 155 and 171, and also such virtualized entities create technical effects.
The computer readable memories 125, 155, and 171 may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The computer readable memories 125, 155, and 171 may be means for performing storage functions. The processors 120, 152, and 175 may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on a multi-core processor architecture, as non-limiting examples. The processors 120, 152, and 175 may be means for performing functions, such as controlling the UE 110, gNB 170, and other functions as described herein.
In general, the various embodiments of the user equipment 110 can include, but are not limited to, cellular telephones such as smart phones, tablets, personal digital assistants (PDAs) having wireless communication capabilities, portable computers having wireless communication capabilities, image capture devices such as digital cameras having wireless communication capabilities, gaming devices having wireless communication capabilities, music storage and playback appliances having wireless communication capabilities, Internet appliances permitting wireless Internet access and browsing, tablets with wireless communication capabilities, as well as portable units or terminals that incorporate combinations of such functions.
Some example embodiments herein may be implemented in software (executed by one or more processors), hardware (e.g., an application specific integrated circuit), or a combination of software and hardware. In an example of an embodiment, the software (e.g., application logic, an instruction set) is maintained on any one of various conventional computer-readable media. In the context of this document, a “computer-readable medium” may be any media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer, with one example of a computer described and depicted, e.g., in
The current architecture in LTE networks is fully distributed in the radio and fully centralized in the core network. The low latency requires bringing the content close to the radio which leads to local break out and multi-access edge computing (MEC). 5G may use edge cloud and local cloud architecture. Edge computing covers a wide range of technologies such as wireless sensor networks, mobile data acquisition, mobile signature analysis, cooperative distributed peer-to-peer ad hoc networking and processing also classifiable as local cloud/fog computing and grid/mesh computing, dew computing, mobile edge computing, cloudlet, distributed data storage and retrieval, autonomic self-healing networks, remote cloud services and augmented reality. In radio communications, using edge cloud may mean node operations to be carried out, at least partly, in a server, host or node operationally coupled to a remote radio head or base station comprising radio parts. It is also possible that node operations will be distributed among a plurality of servers, nodes or hosts. It should also be understood that the distribution of labor between core network operations and base station operations may differ from that of the LTE or even be non-existent. Some other technology advancements probably to be used are Software-Defined Networking (SDN), Big Data, and all-IP, which may change the way networks are being constructed and managed.
Having thus introduced one suitable but non-limiting technical context for the practice of the example embodiments of this invention, the example embodiments will now be described with greater specificity.
As shown in
According to example embodiments, the systems and methods described herein may enhance, remove/reduce or manage the sound pressure level of a person's (own) voice when recording sound using a wearable microphone system. Similarly, as described herein below, ABE may be applied to signals such as subsampled signal 220 to determine a recovered signal 230, which may substantially correspond to the original high resolution signal 210.
Deep learning may provide for both artificial band width extension (ABE) and noise reduction (for example, denoising). In-ear voice capture may require a different setup than external microphones because of the 1) recording in a closed or partly open cavity (low-pass filtering effect which requires ABE to be solved), 2) noise (external noise, internal body noises, for example, breath and heart) and 3) changing response due to differences in producing sound (different vowels and consonants). The example embodiments described herein may counteract the low-pass filtering effect by high-pass filtering, for example, filtering with the inverse of the low-pass filter.
In addition to directly (or, right) outside the user's ear, the “outside-the-ear microphone” 330 may alternatively be located close to the user's mouth (for example in the headset wire). Although
Each of the headsets 340 may be comprised of at least one microphone, such as the in-ear microphone (340). The headsets 320 may form a connection to other headsets, for example, via mobile phones (and associated networks). The headsets 320 may include at least one processor, at least one memory storage device and an energy storage and/or energy source. The headsets 320 may include machine readable instructions, for example, instructions for implementing a deep learning process.
A combination of device (for example, headset 320-L and 320-R, including in-ear microphones 340 and outside-the-ear microphones 330) and machine readable instructions (for example, software) may be used to perform speech capture that provides accurate and real-time speech signal modeling and enhancement in order to achieve natural speech recording and transfer by deep learning and deep generative modeling using at least an in-ear microphone signal.
According to an example embodiment, deep learning based training headset 320 may include at least one in-ear microphone 340 and one outside-the-ear microphone 330. Deep learning based training headset 320 may process instructions to adjust audio for different conditions (for example, background noise conditions: type of noise (babble noise, traffic noise, music), and noise level, etc.), different people (for example, aural characteristics of voices including pitch, volume, resonance, etc.) and different types of sounds (for example, languages, singing, etc.).
According to an example scenario, the deep learning based training headset 320 may be used in a quiet location. In this scenario, the deep learning based training headset 320 may be trained for a plugged or, alternatively, an open headset. A plugged earbud or earplug completely seals the ear canal. An open headset does not seal the ear canal completely and may let in background noise to a (for example, much) greater extent than a plugged headset. The deep learning based training headset 320 may be trained for instances in which there may be sound from in-ear-speaker or, alternatively, no sound from in-ear-speaker. According to another example scenario, the deep learning based training headset 320 may be trained for a noisy environment.
Example embodiments may allow in-ear capture of person's own voice. The quality of in-ear recording of user's own voice using, for example, closed or almost closed headset, may be poor because of low pass filtering effect of the ear channel. The main resonance (quarter of the wavelength for open ear and half of the wavelength for blocked ear channel) may be approximately 2-3 kHz (open) or 4-6 kHz (blocked). The response in-ear canal depends on the content of the speech, for example, different vowels and consonants correspond to different geometry of the mouth, which affects the response function.
As shown in
According to an example embodiment, the systems may recognize the sound signal coming from person's own mouth using in-ear microphone of the headphone and deep learning algorithm.
As shown in
In instances in which the example embodiments are applied, and the in-ear microphone signal is the input (to the neural network) 610, the output signal may have a spectrogram similar to the outside-the-ear microphone 620.
As shown in
Communication in a noisy location may be enabled by in-ear voice capture of each person's own talk. As a first person (head 1910) talks (for example, speaks) into the in-ear microphone 340-1R, the in-ear microphone 340-1R may capture aspects of the sound source 410-1 (for example, time, frequency, pressure, etc.). Sound waves may be represented using complex exponentials. An associated processor may implement the deep learning based model to clean the signal to approximate natural speech. The signal may be transported to the headset of person 2 (head 2, 920) (for example, via mobile phones, M1930 and M2940). Similarly, the same system may be applied in the headset of person 2 (head 2, 920) for sound source 410-2.
This system may be implemented in use cases, such as communication in a noisy situation in-ear recording of the first user's voice transferred to other people's headphones, where the received signal is played and the voice of the second user (the listener of the first user) may be reduced.
When trained, for example using example embodiments such as described herein below with respect to
The system 1000 may take several inputs, such as a) noisy speech signal (or other signal of interest) through outside-the-ear microphone 1005, b) noisy speech through in-ear microphone 1010, c) incoming audio 1035 through in-ear microphone and d) pre-trained deep learning model 1055.
With regard to
Deep learning inference 1050 may implement different methods for training the deep learning model, such as shown in
In this example embodiment, the deep learning model may be trained using recorded, synchronized noiseless (clean) speech signals 1105 from both the in-ear 1015 (X: in-ear microphone speech 1115) and the outside-the-ear (for example, external) microphones 1020 (Y: external microphone speech 1110). Deep learning inference 1050 may train a deep learning system in which the input X˜ is the noisy speech signal 1130 from in-ear microphone, and output Y{circumflex over ( )} is the most probable clean speech signal 1155 that would have produced the observed in-ear signal X 1115. Deep learning inference 1050 may generate input X˜, the noisy speech signal 1130, based on combining in-ear microphone speech 1115 and approximated random in-ear response 1125 (which may be determined from a data store noise 1010 that includes an approximated random room response.
Deep learning inference 1050 may augment the clean speech signal X 1115 with a parametrized noise database 1010, but keep the target Y noiseless so that the network learns to produce the most likely consistent Y{circumflex over ( )} from the input X. This may include selection at random (select/real/fake randomly) 1180 between a real sample X˜, Y pair 1140 and a fake sample pair X˜, Y{circumflex over ( )}, 1160, which may have been determined by conditioned generator neural network G 1150. A real sample pair may be defined as a pair of signals, the noisy in-ear speech X˜ and the external mic speech Y, which are actually recorded using the microphones and not “fake” samples generated using the conditioned generator neural network G. Generator network G 1150 may receive latent variables z 1145 and gradients of error for training networks D and G, which may be determined by discriminator network D 1175. Generator network G 1150 may generate a clean speech signal Y{circumflex over ( )} 1155. Thereafter, clean speech signal Y{circumflex over ( )} 1155 may be paired with X˜, the noisy speech signal 1130 to create the fake sample X˜, Y{circumflex over ( )} pair 1160.
The (for example, conditioned) generator network G 1150 may be trained simultaneously with a discriminator network D 1175 as shown in
These gradients of error 1170 may be input to the generator network G 1150 and used in training the generator network G 1150 to generate an external microphone signal from an in-ear microphone signal (for example, clean speech signal 1155) based on the at least one real in-ear microphone audible signal and the at least one real external microphone audible signal. Deep learning inference 1050 may utilize any variant of Generative Adversarial Network (GAN), including Deep Regret Analytic Generative Adversarial Network (DRAGAN), Wasserstein Generative Adversarial Network (WGAN) or Progressive Growing of GANs, etc. Although
The input to the network may be raw signal, or any kind of time-spectrum representation, such as short-term Fourier transforms (STFTs).
According to an example embodiment, deep learning inference 1050 may train to adaptively utilize both inner and outer microphones. This example embodiment may extend the example embodiment presented above in
The external microphone signal may be (for example, selected, assessed, as) a good signal in instances in which there is very little noise. On the other hand, if the environment is extremely noisy, the internal microphone (with the approximated transfer function in-ear→external) may need (for example, provides a better approximation of clean speech) to be used. In many instances, the optimal result may be achieved using both signals. The example embodiments provide a method of using a neural network to adaptively utilize both signals in approximately optimal way. Note that during training the inputs to network G are noisy in-ear microphone signal X˜, noisy external microphone signal Y˜ and the output is the prediction of the most probable consistent clean external signal Y{circumflex over ( )} 1155.
The training may be implemented in a quiet environment with both mic signals (in-ear microphone and outside the ear microphone). The example embodiments may detect the noise level, and decide when to start recording data for the personalized training.
During training, both microphone signals may be required. In some instances a domain transfer training is possible without simultaneous microphone recordings (for example, in a manner similar to cycle Generative Adversarial Network (CycleGAN)), but the generator quality may be worse than that generated from both microphone signals.
As shown in the example embodiment, a system or device, for example deep learning inference 1050 may learn inverse time-dynamic transfer functions and generate large training sets from normal speech data. Deep learning inference 1050 may receive recorded, synchronized noiseless (clean) speech signals 1105 from both the in-ear 1015 (X: in-ear microphone speech 1115) and the outside (for example, external) microphones 1020 (Y: external microphone speech 1110). Generator network G 1150 may receive latent space z (for example, latent variables) 1205 and output fake sample pair 1160. A switch 1210 may receive real sample pair 1140 and fake sample pair 1160 and output to discriminator network 1175, which may determine a real/fake output 1165. The discriminator network may learn to distinguish between generated signals from generator network and real signals.
Deep learning training may require (for example, utilize) large representative databases in order to properly implement the deep learning process. The example embodiments may generate training data for a system, such as the one presented in
At block 1310, a device, for example UE 110 or other device in network 100, may access a clean speech signal(s) with multiple microphones and noise. The microphones may include external microphones and in-ear-microphones.
At block 1320, UE 110 may train generative model G (and potentially discriminative model D, if using generative adversarial network).
At block 1330, UE 110 may output generative model G. Generative model G may include a conditioned generative network, such as described with respect to
At block 1410, a device, for example UE 110, may receive (or access, etc.) in-ear microphone speech 1115 and external microphone speech 1110, for example, from a database of clean speech 1105. The speech signals may comprise synchronized noiseless (clean) speech signals from both the in-ear and the external microphone. For example, UE 110 may access corresponding samples of in-ear microphone speech and external (for example, outside-the-ear) microphone speech, which may be hey paired in this instance.
At block 1420, UE 110 may transmit (and/or determine) a real sample pair 1140 based on the in-ear microphone speech 1115.
At block 1430, UE 110 may process the in-ear microphone speech via a conditioned generator network to determine a fake sample pair.
At block 1440, UE 110 may process the real sample pair and the fake sample pair via discriminator network to determine a real/fake speech, for example, via a discriminator network. D network may be used for training (to get the gradients of error for training the G network). The gradient in this instance is a multi-variable generalization of the derivative.
At block 1510, a device, for example UE 110, may access potentially noisy signal from at least one microphone.
At block 1520, UE 110 may use a pre-trained generative model GT to generate clean natural sound.
At block 1530, UE 110 may output the clean natural sound.
At block 1610, a device, for example UE 110, may receive at least one of a noisy speech (or other audio) signal through an outside-the-ear microphone, a noisy speech (or other audio) signal through an in-ear microphone, incoming audio through an in-ear microphone and a pre-trained deep learning model. The UE may require at least one input plus pre-trained model.
At block 1420, UE 110 may perform an in-body sound transfer of the speech (or other signal of interest) and noise to an in-ear microphone. The in-ear microphone may also receive incoming audio.
At block 1430, incoming audio cancellation may be performed on the output of the in-ear microphone.
At block 1440, UE 110 may perform a room sound transfer of the speech (or other signal of interest) and noise to an outside-the-ear microphone.
At block 1450, UE 110 may perform deep learning inference on the outputs of the incoming audio cancellation and the outside-the-ear microphone to determine and output clean natural speech.
Without in any way limiting the scope, interpretation, or application of the claims appearing below, a technical effect of one or more of the example embodiments disclosed herein is to enable a speech capture solution that provides accurate and real-time speech signal modeling and enhancement in order to achieve natural speech recording and transfer by deep learning and deep generative modeling using at least an in-ear microphone signal.
An example embodiment may provide a method comprising accessing, by at least one processing device, a real noise-free audible signal including at least one real in-ear microphone audible signal and at least one real external microphone audible signal and at least one noise signal, training a generative network to generate an external microphone signal from an in-ear microphone signal based on the at least one real in-ear microphone audible signal and the at least one real external microphone audible signal; and outputting the generative network.
In accordance with an example embodiment as described in paragraphs above, accessing, by at least one processing device, at least one in-ear microphone speech signal and at least one external microphone speech signal; transmitting at least one real sample pair based on the at least one in-ear microphone speech signal; generating at least one fake pair based on processing the at least one in-ear microphone speech signal via a conditioned generator network; and processing the at least one real sample pair and the at least one fake sample pair via a discriminator network to determine whether real/fake.
In accordance with an example embodiment as described in paragraphs above, wherein the at least one processing device is part of a wearable microphone apparatus.
In accordance with an example embodiment as described in paragraphs above, wherein the wearable microphone system further comprises one of more of: at least one in-ear microphone; at least one in-ear speaker; a connection to at least one other wearable microphone system; at least one processor; and at least one memory storage device.
In accordance with an example embodiment as described in paragraphs above, wherein the at least one processing device further comprises: at least one in-ear microphone and at least one outside-the-ear microphone.
In accordance with an example embodiment as described in paragraphs above, wherein the at least one external microphone speech sample and the at least one external microphone speech sample are selected to include at least one of: different people; different types of sounds; a quiet environment including a plugged or an open headset; a quiet environment including sound from an in-ear speaker and no sound from an in-ear speaker; and a noisy environment.
In accordance with an example embodiment as described in paragraphs above, wherein where an input X˜ of the at least one processing device is a noisy speech signal from the at least one in-ear microphone, and an output Y{circumflex over ( )} is a most probable clean sound signal that would have produced an observed in-ear signal X.
In accordance with an example embodiment as described in paragraphs above, wherein the conditioned generator network comprises at least one of a generative adversarial network, a deep regret analytic generative adversarial network, a Wasserstein generative adversarial network and a progressive growing of generative adversarial networks.
In accordance with an example embodiment as described in paragraphs above, wherein the conditioned generator network comprises at least one of an auto-encoder and an autoregressive model.
An example embodiment may provide a method comprising receiving, by an outside-the-ear microphone, a room sound transfer of at least one audio signal of interest and at least one noise signal, receiving, by an in-ear microphone, an in-body transfer of at least one audio signal of interest and the at least one noise signal, and an incoming audio signal; performing incoming audio cancellation on an output of the in-ear microphone; and performing deep learning inference based on an output of the incoming audio cancellation, an output of the outside-the-ear microphone and a pre-trained deep learning model to determine a clean natural sound.
In accordance with an example embodiment as described in paragraphs above, transmitting the clean natural sound, wherein the clean natural sound is configured to be received and played by a second headphone.
In accordance with an example embodiment as described in paragraphs above, wherein the clean natural sound comprises human speech.
An example embodiment may be provided in an apparatus comprising at least one processor; and at least one non-transitory memory including computer program code, the at least one non-transitory memory and the computer program code may be configured to, with the at least one processor, cause the apparatus to: access at least one in-ear microphone speech signal and at least one external microphone speech signal; transmit at least one real sample pair based on the at least one in-ear microphone speech signal; generate at least one fake pair based on processing the at least one in-ear microphone speech signal via a conditioned generator network; and process the at least one real sample pair and the at least one fake sample pair via a discriminator network to determine whether real/fake.
In accordance with an example embodiment as described in paragraphs above, wherein the apparatus is part of a wearable microphone apparatus.
In accordance with an example embodiment as described in paragraphs above, wherein the wearable microphone system further comprises one of more of: at least one in-ear microphone; at least one in-ear speaker; a connection to at least one other wearable microphone system; at least one processor; and at least one memory storage device.
In accordance with an example embodiment as described in paragraphs above, wherein the apparatus further comprises: at least one in-ear microphone and at least one outside-the-ear microphone.
In accordance with an example embodiment as described in paragraphs above, wherein the at least one external microphone speech sample and the at least one external microphone speech sample are selected to include at least one of: different people; different types of sounds; a quiet environment including a plugged or an open headset; a quiet environment including sound from an in-ear speaker and no sound from an in-ear speaker; and a noisy environment.
In accordance with an example embodiment as described in paragraphs above, wherein an input X˜ of the apparatus is a noisy speech signal from the at least one in-ear microphone, and an output Y{circumflex over ( )} is a most probable clean sound signal that would have produced an observed in-ear signal X.
An example embodiment may be provided in an apparatus comprising at least one processor; and at least one non-transitory memory including computer program code, the at least one non-transitory memory and the computer program code may be configured to, with the at least one processor, cause the apparatus to: access a real noise-free audible signal including at least one real in-ear microphone audible signal and at least one real external microphone audible signal and at least one noise signal, train a generative network to generate an external microphone signal from an in-ear microphone signal based on the at least one real in-ear microphone audible signal and the at least one real external microphone audible signal; and output the generative network.
In accordance with an example embodiment as described in paragraphs above, receive, by an outside-the-ear microphone, a room sound transfer of at least one audio signal of interest and at least one noise signal; receive, by an in-ear microphone, an in-body transfer of at least one audio signal of interest and the at least one noise signal, and an incoming audio signal; perform incoming audio cancellation on an output of the in-ear microphone; and perform deep learning inference based on an output of the incoming audio cancellation, an output of the outside-the-ear microphone and a pre-trained deep learning model to determine a clean natural sound.
In accordance with an example embodiment as described in paragraphs above, wherein the at least one non-transitory memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to perform transmit the clean natural sound, wherein the clean natural sound is configured to be received and played by a second headphone.
In accordance with another example, an example apparatus comprises: means for accessing a real noise-free audible signal including at least one real in-ear microphone audible signal and at least one real external microphone audible signal and at least one noise signal, means for training a generative network to generate an external microphone signal from an in-ear microphone signal based on the at least one real in-ear microphone audible signal and the at least one real external microphone audible signal; and means for outputting the generative network.
In accordance with an example embodiment as described in paragraphs above, means for accessing, by at least one processing device, at least one in-ear microphone speech signal and at least one external microphone speech signal; means for transmitting at least one real sample pair based on the at least one in-ear microphone speech signal; means for generating at least one fake pair based on processing the at least one in-ear microphone speech signal via a conditioned generator network; and means for processing the at least one real sample pair and the at least one fake sample pair via a discriminator network to determine whether real/fake.
In accordance with an example embodiment as described in paragraphs above, wherein the apparatus is part of a wearable microphone apparatus.
In accordance with an example embodiment as described in paragraphs above, wherein the wearable microphone system further comprises one of more of: at least one in-ear microphone; at least one in-ear speaker; a connection to at least one other wearable microphone system; at least one processor; and at least one memory storage device.
In accordance with an example embodiment as described in paragraphs above, wherein the apparatus further comprises at least one in-ear microphone and at least one outside-the-ear microphone.
In accordance with an example embodiment as described in paragraphs above, wherein the at least one external microphone speech sample and the at least one external microphone speech sample are selected to include at least one of: different people; different types of sounds; a quiet environment including a plugged or an open headset; a quiet environment including sound from an in-ear speaker and no sound from an in-ear speaker; and a noisy environment.
In accordance with an example embodiment as described in paragraphs above, wherein where an input X˜ of the at least one processing device is a noisy speech signal from the at least one in-ear microphone, and an output Y{circumflex over ( )} is a most probable clean sound signal that would have produced an observed in-ear signal X.
In accordance with an example embodiment as described in paragraphs above, wherein the conditioned generator network comprises at least one of a generative adversarial network, a deep regret analytic generative adversarial network, a Wasserstein generative adversarial network and a progressive growing of generative adversarial networks.
In accordance with another example, an example apparatus comprises: means for receiving, by an outside-the-ear microphone, a room sound transfer of at least one audio signal of interest and at least one noise signal; means for receiving, by an in-ear microphone, an in-body transfer of at least one audio signal of interest and the at least one noise signal, and an incoming audio signal; means for performing incoming audio cancellation on an output of the in-ear microphone; and means for performing deep learning inference based on an output of the incoming audio cancellation, an output of the outside-the-ear microphone and a pre-trained deep learning model to determine a noise-free natural sound.
An example apparatus may be provided in a non-transitory program storage device, such as memory 125 shown in
An example apparatus may be provided in a non-transitory program storage device, such as memory 125 shown in
Embodiments herein may be implemented in software (executed by one or more processors), hardware (e.g., an application specific integrated circuit), or a combination of software and hardware. In an example embodiment, the software (e.g., application logic, an instruction set) is maintained on any one of various conventional computer-readable media. In the context of this document, a “computer-readable medium” may be any media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer, with one example of a computer described and depicted, e.g., in
If desired, the different functions discussed herein may be performed in a different order and/or concurrently with each other. Furthermore, if desired, one or more of the above-described functions may be optional or may be combined.
Although various aspects are set out above, other aspects comprise other combinations of features from the described embodiments, and not solely the combinations described above.
It is also noted herein that while the above describes example embodiments, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications which may be made without departing from the scope of the present invention.
Although various aspects of the invention are set out in the independent claims, other aspects of the invention comprise other combinations of features from the described embodiments and/or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims.
It is also noted herein that while the above describes example embodiments, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications which may be made without departing from the scope of the present invention as defined in the appended claims.
In general, the various embodiments may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto. While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
Embodiments may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.
The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments. All of the embodiments described in this Detailed Description are exemplary embodiments provided to enable persons skilled in the art to make or use the invention and not to limit the scope of the invention which is defined by the claims.
The foregoing description has provided by way of example and non-limiting examples a full and informative description of the best method and apparatus presently contemplated by the inventors for carrying out the invention. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this invention will still fall within the scope of this invention.
It should be noted that the terms “connected,” “coupled,” or any variant thereof, mean any connection or coupling, either direct or indirect, between two or more elements, and may encompass the presence of one or more intermediate elements between two elements that are “connected” or “coupled” together. The coupling or connection between the elements can be physical, logical, or a combination thereof. As employed herein two elements may be considered to be “connected” or “coupled” together by the use of one or more wires, cables and/or printed electrical connections, as well as by the use of electromagnetic energy, such as electromagnetic energy having wavelengths in the radio frequency region, the microwave region and the optical (both visible and invisible) region, as several non-limiting and non-exhaustive examples.
Furthermore, some of the features of the preferred embodiments of this invention could be used to advantage without the corresponding use of other features. As such, the foregoing description should be considered as merely illustrative of the principles of the invention, and not in limitation thereof.
| Number | Name | Date | Kind |
|---|---|---|---|
| 9253560 | Goldstein | Feb 2016 | B2 |
| 9640194 | Nemala et al. | May 2017 | B1 |
| 20080112569 | Asada | May 2008 | A1 |
| 20110135106 | Yehuday et al. | Jun 2011 | A1 |
| 20120084084 | Zhu | Apr 2012 | A1 |
| 20150063575 | Tan | Mar 2015 | A1 |
| 20160351203 | Tan et al. | Dec 2016 | A1 |
| 20170178668 | Kar et al. | Jun 2017 | A1 |
| 20170249954 | Kim | Aug 2017 | A1 |
| 20180367882 | Watts | Dec 2018 | A1 |
| 20190037298 | Reily | Jan 2019 | A1 |
| 20190043491 | Kupryjanow | Feb 2019 | A1 |
| 20190080710 | Zhang | Mar 2019 | A1 |
| 20190130926 | Giri | May 2019 | A1 |
| 20190209038 | Saab | Jul 2019 | A1 |
| 20190222691 | Shah | Jul 2019 | A1 |
| Entry |
|---|
| Pascual, S. et al. Segan: Speech Enhancement Generative Adversarial Network. In: arXiv.org [online], Jun. 9, 2017 [retrieved on Jul. 1, 2019-07]. Retrieved from https://arxiv.org/abs/1703.09452, abstract; sections 1,3,4.1-4.2, 5.1; fig 2. |
| Sriram, A. et al. Robust Speech Recognition Using Generative Adversarial Networks. In: arXiv.org [online], Nov. 5, 2017, [retrieved on Jul. 1, 2019]. Retrieved from https://arxiv.org/abs/1711.01567, abstract; sections 3.1-3.2; eq. 1-2; Alg. 1. |
| Creswell, A. et al. Generative Adversarial Networks: An Overview. In: arXiv.org [online], Oct. 19, 2017, [retrieved on Jul. 1, 2019]. Retrieved from https://arxiv.org/abs/1710.07035, abstract, sections III.B, III.E. |
| “The Future of Voice Computing is in the Ear” https://www.smartear.ai/ [retrieved Apr. 19, 2018]. |
| Patrick Kechichian and Sriram Srinivasam “Model-based Speech Enhancement Using a Bone-Conducted Signal” Feb. 23, 2012 http://asa.scitation.org/doi/pdf/10.1121/1.3687014. |
| Mingzi Li “Multisensory Speech Enhancement in Noisy Environments Using Bone-Conducted and Air-Conducted Mircophones” Nov. 2013 <http://webee.technion.ac.il/people/IsraelCohen/Info/Graduates/PDF/MingziLi_MSc_2013.pdf >. |
| Juian Horsey “Ripplebuds Noise Blocking Earbuds Fitted with In-ear Mic” Mar. 22, 2016 <https://www.geeky-gadgets.com/ripplebuds-noise-blocking-earbuds-fitted-with-in-ear-mic-Mar. 22, 2016/>. |
| “In-ear Voice Capture” http://think-a-move.com/page_id=14 [retrieved Apr. 19, 2018]. |
| Number | Date | Country | |
|---|---|---|---|
| 20190325887 A1 | Oct 2019 | US |
