Patent Application
20240105199
The present invention relates to an apparatus for integrating and eliminating noise and echo signals based on a multi-channel using a deep neural network, and more particularly, after separately estimating a noise signal and an echo signal using a plurality of sequentially connected artificial neural networks, The present invention relates to a technology capable of more accurately estimating only a user's voice signal by applying an attention mechanism to estimated information.
Speech communication means a technology that transmits the user's uttered voice to the other party for mutual communication between voice communication users. Specifically, speech communication is used in various fields such as a conference call, a video call, and a video conference as well as a widely used telephone.
In voice communication, only the clear voice signal of the speaker must be delivered in order to deliver accurate meaning to the other party. However, in situations where two or more speakers utter at the same time, or in the case where the previous speaker's utterance is input again into the microphone and playback from the speaker and input from the microphone are repeated, or if noise generated by the surrounding environment is input into the microphone, since only the user's voice is not input into the microphone, there is a problem in that the user's voice is not accurately transmitted to the other party.
Accordingly, recently, technologies for an acoustic echo canceller (AEC) that cancels the echo of sound have been developed in various ways. An acoustic echo canceller can remove acoustic echo, which is a phenomenon in which one's own voice is heard again by directly or indirectly re-inputting the audio signal from the speaker in a video call, video conference, etc. (through reflection from walls or surrounding objects) into the microphone.
For the acoustic echo canceller to efficiently cancel the acoustic echo, it is important to accurately estimate a room impulse response (RIR) path where the acoustic echo is generated. The acoustic echo canceller generally estimates an acoustic echo generation path (RIR) using an adaptive filter and generate an estimated acoustic echo signal. The acoustic echo canceller removes the acoustic echo by subtracting the estimated acoustic echo signal from the actual acoustic echo signal.
Methods for updating the adaptive filter coefficients of the adaptive filter for estimating the acoustic echo generation path (RIR) include a method using a recursive least square (RLS) algorithm, a method using a least mean square (LMS) algorithm, and a method using normalized least mean square (NLMS) algorithm, and a method using the Affine Projection algorithm.
In addition, with the recent development of artificial neural network technology, various technologies for synthesizing voices or recognizing voices using artificial neural networks have been developed. For example, a method of directly estimating the acoustic echo using a deep neural network or a convolutional recurrent neural network in deep learning has been developed.
However, most conventional technologies to date have removed acoustic echoes in the frequency domain by using a convolutional recurrent neural network, which is a type of deep learning technique. When acoustic echo is canceled in the frequency domain, the phase of the input signal is not directly reflected, so echo cancellation is performed by estimating real and imaginary values corresponding to complex values of the phase. Therefore, there is a problem in that the performance of echo cancellation is somewhat deteriorated because it is not a direct phase value of the input signal.
Therefore, multi-channel based noise and echo signal integrated cancellation device using deep neural network according to an embodiment is an invention designed to solve the above-described problems, and the invention relates to a technique capable of efficiently removing a noise signal and an echo signal among signals by using a noise signal and an echo signal separately estimated to be input to a microphone.
More specifically, purpose of the present invention is to provide a signal estimation device repeatedly and separately estimates a noise signal and an echo signal using a plurality of sequentially connected artificial neural networks, and then accurately extracts only the user's voice information based on the estimated information by applying attention mechanism.
A multi-channel based noise and echo signal integrated cancellation device using deep neural network according to an embodiment comprises a plurality of microphone encoders that receive a plurality of microphone input signals including an echo signal, and a speaker's voice signal, convert the plurality of microphone input signals into a plurality of conversion information, and output the plurality of conversion information, a channel convert unit for compressing the plurality of pieces of conversion information and converting them into first input information having a size of a single channel and outputting the converted first input information, a far-end signal encoder that receives a far-end signal, converts the far-end signal into second input information, and outputs the converted second input information, an attention unit outputting weight information by applying an attention mechanism to the first input information and the second input information, a pre-learned first artificial neural network taking third input information, which is the sum information of the weight information and the second input information, as input information, and first output information including mask information for estimating the voice signal from the second input information as output information and a voice signal estimator configured to output an estimated voice signal obtained by estimating the voice information based on the first output information and the second input information.
The microphone encoder converts the microphone input signal in the time-domain into a signal in the latent-domain.
The multi-channel based noise and echo signal integrated cancellation device using deep neural network further comprise a decoder for converting the estimated speech signal in the latent domain into an estimated speech signal in the time domain.
The attention part analyzes a correlation between the first input information and the second input information and outputs the weight information based on the analyzed result.
The attention part estimates the echo signal based on information on the far-end signal included in the first input information and outputs the weight information based on the estimated echo signal.
A multi-channel based noise and echo signal integrated cancellation device using deep neural network according to another embodiment comprises a plurality of microphone encoders that receive a plurality of microphone input signals including an echo signal, and a speaker's voice signal, convert the plurality of microphone input signals into a plurality of conversion information, and output the converted information, a channel convert unit for compressing the plurality of pieces of conversion information and converting them into first input information having a size of a single channel and outputting the converted first input information, a far-end signal encoder that receives a far-end signal, converts the far-end signal into second input information, and outputs the converted second input information, a pre-learned second artificial neural network that uses third input information, which is the sum of the first input information and the second input information, as input information, and an estimated echo signal obtained by estimating the echo signal from the second input information as output information, a pre-learned third artificial neural network having the third input information as input information and an estimated noise signal obtained by estimating the noise signal from the second input information as output information, and a voice signal estimator configured to output an estimated voice signal obtained by estimating the voice information based on the estimated echo signal, the estimated noise echo signal, and the second input information.
The multi-channel based noise and echo signal integrated cancellation device using deep neural network further comprises an attention unit outputting weight information obtained by applying an attention mechanism to the first input information and the second input information, and wherein the third input information further includes the weight information.
The second artificial neural network includes a plurality of artificial neural networks connected in series, and the third artificial neural network includes a plurality of artificial neural networks connected in series on a par with the second artificial neural network, wherein the plurality of artificial neural networks of the second artificial neural network re-estimates the echo signal based on information output from the artificial neural network in the previous step and wherein the plurality of artificial neural networks of the third artificial neural network re-estimates the noise signal based on the information output from the artificial neural network in the previous step.
The second artificial neural network re-estimates the echo signal using second input information, the estimated echo signal, and the noise signal as input information, and wherein the third artificial neural network re-estimates the noise signal by using second input information, the estimated echo signal, and the noise signal as input information.
The second artificial neural network includes a 2-A artificial neural network and a 2-B artificial neural network, and the third artificial neural network includes a 3-A artificial neural network and a 3-B artificial neural network, wherein the 2-A artificial neural network includes a pre-learned artificial neural network which takes third input information as input information and second output information including information obtained by estimating the echo signal based on the third input information as output information, wherein the 3-A artificial neural network includes a pre-learned artificial neural network which takes third input information as input information and third output information including information obtained by estimating the noise signal based on the third input information as output information.
The 2-B artificial neural network includes a pre-learned artificial neural network which mixes the second output information from the second input information and uses fourth input information obtained by subtracting the third output information as input information, and based on the fourth input information, uses fourth output information including information obtained by estimating an echo signal as output information, wherein the 3-B artificial neural network mixes the third output information from the third input information and uses fifth input information obtained by subtracting the second output information as input information, and based on the fifth input information, uses the fifth output information including information estimating the noise signal as output information.
The microphone encoder converts the microphone input signal in the time-domain into a signal in the latent-domain, and the multi-channel based noise and echo signal integrated cancellation device using deep neural network further comprises a decoder for converting the estimated voice signal in the latent domain into an estimated voice signal in the time domain.
A multi-channel based noise and echo signal integrated cancellation method using deep neural network comprises receiving a plurality of microphone input signal including an echo signal, and a speaker's voice signal through a plurality of microphone encoder, converting the plurality of microphone input signal into a plurality of pieces of conversion information and outputting the converted information, compressing the plurality of pieces of conversion information into first input information having a size of a single channel and outputting the converted first input information, receiving a far-end signal using a far-end signal encoder, converting the far-end signal into second input information, and outputting the converted second input information, outputting an estimated echo signal through a pre-learned second artificial neural network having third input information, which is the sum of the first input information and the second input information, as input information, and the estimated echo signal obtained by estimating the echo signal as output information, outputting an estimated noise signal through a pre-learned third artificial neural network using the third input information as input information the estimated noise signal obtained by estimating the noise signal as output information and outputting an estimated speech signal obtained by estimating the speech information based on the estimated echo signal, the estimated noise echo signal, and the second input information.
The third input information includes weight information generated by applying an attention mechanism to the first input information and the second input information.
The second artificial neural network includes a plurality of artificial neural networks connected in series, and the third artificial neural network includes a plurality of artificial neural networks connected in series on a par with the second artificial neural network, wherein the step of outputting an estimated echo signal obtained by estimating the echo signal includes re-estimating the echo signal by the plurality of artificial neural networks of the second artificial neural network based on information output from the artificial neural network in the previous step and wherein the step of outputting an estimated echo signal obtained by estimating the noise signal includes re-estimating the noise signal by the plurality of artificial neural networks of the third artificial neural based on the information output from the artificial neural network in the previous step.
A multi-channel based noise and echo signal integrated cancellation device using deep neural network according to an embodiment can increase the accuracy of estimation of the echo signal and the noise signal by iteratively and separately estimating the echo signal and the noise signal, thus more accurately than the prior art. Among the signals input to the microphone, there is an advantage in that an echo signal and a noise signal can be removed.
In addition, there is an advantage in that increasing the accuracy of the estimated echo signal and noise signal by applying an attention mechanism to the estimated echo signal and noise signal information, thereby providing an apparatus for estimating a voice signal capable of more accurately extracting only the user's voice information.
Accordingly, it is possible to more efficiently cancel the echo signal, and there is an effect of improving voice quality and intelligibility in the case of collecting and processing the speaker's voice through a microphone in an environment where echo signals exist, such as artificial intelligence speakers used in home environments, robots used in airports, voice recognition and PC voice communication systems.
Hereinafter, embodiments according to the present invention will be described with reference to the accompanying drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components have the same numerals as much as possible even if they are displayed on different drawings. In addition, in describing an embodiment of the present invention, if it is determined that a detailed description of a related known configuration or function hinders understanding of the embodiment of the present invention, the detailed description thereof will be omitted. In addition, embodiments of the present invention will be described below, but the technical idea of the present invention is not limited or limited thereto and can be modified and implemented in various ways by those skilled in the art.
In addition, terms used in this specification are used to describe embodiments and are not intended to limit and/or limit the disclosed invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.
In this specification, terms such as “include”, “comprise” or “have” are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or the existence or addition of more other features, numbers, steps, operations, components, parts, or combinations thereof is not excluded in advance.
In addition, throughout the specification, when a part is said to be “connected” to another part, this is not only the case where it is “directly connected”, but also the case where it is “indirectly connected” with another element in the middle. Terms including ordinal numbers, such as “first” and “second” used herein, may be used to describe various components, but the components are not limited by the terms.
Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted.
The voice enhancement technology is a technology for estimating a clear voice by removing an echo signal input by a microphone and is essential for voice applications such as voice recognition and voice communication. For example, in voice recognition, if a speech recognition model is trained with a clear signal without echo and then tested with a signal with noise, performance is reduced. Therefore, to solve this problem, the performance of voice recognition can be improved by introducing a voice enhancement technology that removes noise and echo before performing voice recognition. In addition, voice enhancement technology can be used to improve call quality by removing echoes from voice communication to deliver clear voice.
Hereinafter, a technique for efficiently estimating a speaker's voice signal included in a microphone input signal using a deep neural network will be described in more detail.
Referring to
y(t)=s(t)+d(t)+n(t) Equation (1):
The voice signal estimation apparatus 100 according to the present invention may output a final voice signal 30 obtained by estimating the speaker's voice signal 50 using the microphone input signal 20 and the far-end signal 10. Here, the microphone input signal including noise and echo may mean a microphone input signal in which both noise and echo exist.
The voice signal estimation apparatus 100 according to the first embodiment of the present invention reflects the characteristics of the first embodiment and may be referred to as a voice signal estimation apparatus using an attention mechanism.
Referring to
The encoders 110 and 130 serve to convert input signals in the time domain into signals in other domains, and the far-end signal encoder 110 converts the far-end signal 10, which is a signal output from the speaker 200, and the microphone encoder 130 serves to convert the microphone input signal 20 input to the microphone 300.
Specifically, the far-end signal encoder 110 uses the signal output to the speaker 200 as an input signal, and outputs first input information 11 by converting the far-end signal 10 including information in the time domain into a far-end signal in the latent domain. In the case of the latent region, it is not defined as a specific region, for example, a time domain or a frequency domain, and may be defined as a domain of a region generated according to a learning result of an artificial neural network. Therefore, the domain of the latent domain has a characteristic that the domain defined according to the learning environment and results is variable.
The first input information 11 output by the far-end signal encoder 110 is used to extract information about the echo signal 40 in the second input information 12 of the attention unit 120 and the first artificial neural network 140, which will be described later. Specifically, the echo signal 40 is a signal generated by reverberating the far-end signal 10 output from the speaker 200 and has the most similar character to the far-end signal 10 among various types of signals input to the microphone 300. Accordingly, when the information on the echo signal 40 is extracted based on the information on the far-end signal 10, the user's voice signal 50 can be more accurately extracted. A detailed description of this will be described later.
The microphone encoder 130 receives the microphone input signal 20 including the echo signal 40, the voice signal 50, and the noise signal 60 in the time domain from the microphone 300, and outputs the second input information 12 obtained by converting the microphone input signal 20 including information in the time domain into a microphone input signal in the latent domain. The description of the latent region is as described above, but since the first input information 11 and the second input information 12 are added together or used as input information of the same artificial neural network, the domain of the first input information 11 and the domain of the second input information 12 must match each other.
When learning is performed in the domain according to the prior art, information in the input time domain is used for learning using feature information extracted using Short Time Fourier Transform (STFT), whereas, in the case of the present invention, learning is performed using latent features extracted by learning in the latent-domain through processes such as 1D-convolution and ReLu.
Therefore, the far-end signal 10 information in the time domain input to the far-end signal encoder 110 is converted into first input information 11 including information in the latent domain by the far-end signal encoder 110, and the microphone input information 20 in the time domain input through the microphone 300 is converted into second input information 12 in the latent domain by the microphone encoder 130. The first input information 11 and the second input information 12 thus converted are used as input information of the attention unit 120, the first artificial neural network 140, and the decoder 150. And the voice signal 20 input to the microphone encoder 130 may be converted as shown in Equation (2) below.
w=H(yU) Equation (2):
The information output by the microphone encoder 130 is output as vector information due to the nature of the encoder. Specifically, in equation (2), y means the microphone input signal 20, and U means a positive value of N×L length, having N vectors according the size of the input information, and H(⋅) means a nonlinear function.
Among the information input to the first artificial neural network 140, the far-end signal 10 used to remove the echo signal is input to the far-end signal encoder 110 and output to information having vector information as shown in Equation (3) below.
w
f
=H(xQ) Equation (3):
In Equation (3), x denotes the far-end signal 10, Q denotes a positive value having N vectors and a length of N×L, and H(⋅) denotes a nonlinear function.
The first input information 11 and the second input information 12 output in this format may be input to the attention unit 120, converted into weight information 13, and then output. Hereinafter, the mechanism of the attention unit 120 will be described through
Referring to
The attention mechanism has the advantage of a simple structure in the case of the conventional Seq2seq model for estimating the speaker's voice, but since all information is compressed into one fixed-size vector, information loss occurs, and vanishing gradient, a chronic problem of RNNs, and there was a problem that led to a phenomenon in which performance deteriorated significantly when the input data was long.
Therefore, the technology introduced to solve this problem is the attention mechanism. The basic idea of the attention mechanism is that at every time step that predicts the output result from the decoder, refer to the hidden state of the encoder once again to determine the output. That is, which of the input information is more important is not always fixed, but the type of important information changes depending on the time. So, there is an advantage of being able to output information more accurately and quickly by interpreting by giving more weight to important information after figuring out the order of information.
Therefore, the attention unit 120 according to the present invention compares the far-end signal 10 and the microphone input signal 20 input to the attention unit 120, assigns weights to signals having a high correlation. And then information including information about weights is output as output information, and a processor as shown in
Expressing this as an equation, the first input information 11 and the second input information 12 may be converted as shown in Equations (4) and (5) below.
Here, σ(⋅) means the sigmoid function, w means the latent features of the microphone input signal, Wf is the latent features of the far-end signal, and Lw and Lwf mean the information passed through the 1×1 convolution (111, 112) in
Referring to
Referring to
The first artificial neural network 140 takes the third input information 14 as input information and second output information 15 including mask information for estimating the speaker's voice signal 50 as output information. The first artificial neural network 140 may include a learning session (not shown) for learning a speaker's voice signal based on the input information and reference information, and an inference session (not shown) estimating the speaker's voice signal based on the input information.
Any neural network that can be borrowed to the first artificial neural network 140 can be included in any neural network that outputs mask information that enables efficient estimation of the speaker's voice, and representatively, as shown in
The TCN artificial neural network sequentially uses 1*1 Conv 141, PReLU 142, LN 143, D-Conv 144, PReLU 145, LN 146 and 1*1 Conv 147 for the third input information 14 input to the neural network, finally, outputs the second output information 15 including mask information for estimating the speaker's voice signal 50 as output information.
The first artificial neural network 140 may perform learning in a direction of reducing loss using the estimated output information and actual reference information. Specifically, learning can be performed in the direction in which the value of the loss function becomes smaller based on the loss function as shown in Equation (6) below.
In Equation (6), Starget denotes the speaker's voice signal, and s{circumflex over ( )} denotes information output by the first artificial neural network 140.
Returning to
Specifically, as information output from the first artificial neural network 140, second output information 15 including mask information capable of extracting only the speaker's voice signal from the second input information 12 is output. Accordingly, the voice signal estimator 150 may estimate only the speaker's voice signal from the second input information 12 by using the mask information, and then extract and transmit the estimated voice signal to the decoder 160.
The decoder 160 may output the final voice signal 30 including time domain information based on the estimated voice signal 16 output from the voice signal estimator 150. Specifically, the third output information 15 output by the first artificial neural network 140, the second input information 12 output by the microphone encoder 130, and the estimated voice signal 16 estimated by the voice signal estimator 150 is information about signals estimated in the latent domain, not information in the time domain. Accordingly, the decoder 160 may convert the estimated voice signal 16 of the latent domain finally estimated in the latent domain into the final voice signal 30 in the time domain so that the speaker can recognize the speech.
To explain this using equation, the estimated latent region estimated voice signal 16 can be converted into a form containing information in the time domain with the transposed convolutional layer of Equation (2) described above, like the relationship between the short-time Fourier transform (STFT) and the inverse STFT. For example, it can be expressed as Equation (7) below.
{tilde over (s)}=DV Equation (7):
Here, {tilde over (s)} means a voice signal estimated in the time domain, and V means a matrix that converts N vectors into L length.
In the case of a voice estimation method according to the prior art, voice information of a speaker is estimated by estimating mask information based only on a microphone input signal input to a microphone. Therefore, in the case of the prior art, among the information included in the input signal, there was a problem of not distinguishing between information to be weighted and information not to be weighted. Therefore, there is a problem in that it is not possible to efficiently determine the speaker's voice among the signals input to the microphone.
However, the voice signal estimation apparatus 100 according to an embodiment extracts information on the echo signal 40 based on the far-end signal 10 information, and then the extracted information is input information of the first artificial neural network 140, there is an advantage that the first artificial neural network 140 can output mask information obtained by extracting only the user's voice signal 50 more accurately. Furthermore, since information to be weighted using the attention mechanism can be used as input information of the first artificial neural network 130, mask information with higher accuracy can be output.
Experiments on the experimental result data described in this specification were all conducted using TIMIT, Musan, and MS-SNSD databases (DBs), and all DBs consisted of signals sampled at 16 kHz. For the experiment, the learning dataset consisted of 7,000 utterances and the evaluation dataset consisted of 800 utterances using the noise DB and the DB obtained by convolving the echo signal with the voice signal.
And, to generate voice signals contaminated by noise and echo, RIR was generated by simulating various types of room environments using the RIR generator toolkit that generates RIR in a specific room through simulation.
Specifically, 500 RIRs to be applied to the training dataset and 100 RIRs to be applied to the evaluation dataset were prepared, and the room environment has been set for generating the RIR was randomly selected as shown in
As the noise signal, ITU-T recommendation P. 501 and MS-SNSD DB were used, noise was randomly added to the speech dataset for evaluation, and the signal-to-echo ratio (SER) at the time of addition was for learning, one of [−6 dB, −3 dB, 0 dB, 3 dB, 6 dB] was selected and randomly added, and the signal-to-noise ratio (SNR) was [0 dB, 4 dB, 8 dB, 12 dB] was selected and randomly added, and the evaluation was one of SER [−4 dB, −2 dB, 0 dB, 2 dB, 4 dB], and the SNR was [3 dB, 6 dB, 9 dB] was selected and randomly added, and
For evaluation, the results of 800 utterances were prepared using the utterances included in the evaluation dataset. For more accurate evaluation, perceptual evaluation of speech quality (PESQ), short-time objective intelligibility (STOI), signal to distortion ratio (SDR) and echo return loss enhancement (ERLE) were used, and the score was measured by dividing the section in which voice and echo exist simultaneously and the section in which only echo exists.
PESQ has a score between −0.5 and 4.5, STOI has a score between 0 and 1, SDR and ERLE have no specified value range, and in the case of ERLE, a higher score means better echo removal.
First, comparing the PESQ and STOI scores that evaluate the degree of voice quality, it can be understood that the algorithm using all deep neural networks improves the voice quality compared to the unprocessed case. In addition, when comparing the scores with the prior art, the method proposed by the present invention shows the highest score, and in all four objective evaluation indicators, the proposed invention according to the present invention has significantly improved scores compared to the prior art.
The voice signal estimation apparatus 100 according to the second embodiment of the present invention reflects the characteristics of the second embodiment and may be referred to as an integrated echo and noise cancellation apparatus using a plurality of deep neural networks sequentially.
Referring to
Among the audio signal apparatus 100 according to the second embodiment, the far-end signal encoder 110, the attention unit 120, the microphone encoder 130, the voice signal estimator 150, and the decoder 160 are same with the far-end signal encoder 110, the attention unit 120, the microphone encoder 130, the first artificial neural network 140, the voice signal estimator 150, and the decoder 160 described in
The second artificial neural network 170 and the third artificial neural network 180 according to
Therefore, each artificial neural network of the second artificial neural network 170 and each artificial neural network of the third artificial neural network 180 are neural networks that estimate the echo signal and the noise signal. So, if the artificial neural network can perform this, it may be included in the second artificial neural network 170 and the third artificial neural network 180. And the second artificial neural network 170 and the third artificial neural network 180 may include a Temporal Convolutional Network (TCN) artificial neural network as shown in
For convenience of explanation, it is assumed that the second artificial neural network 170 is an artificial neural network for estimating an echo signal and the third artificial neural network 180 is an artificial neural network for estimating a noise signal.
As shown in
In
For example, when the second artificial neural network 170 is an artificial neural network estimating an echo signal, each of the 2-A artificial neural network 171 and the 2-B artificial neural network 172 is an artificial neural network estimating an echo signal. And when the third artificial neural network 180 is an artificial neural network for estimating a noise signal, each of the 3-A artificial neural network 181 and the 3-B artificial neural network 182 is an artificial neural network estimating a noise signal.
The second artificial neural network 170 shown in
The third artificial neural network 180 according to
The voice signal estimator 150 according to
Referring to
Similarly, the 3-A artificial neural network 181, which is the first artificial neural network in the third artificial neural network 180, may include a pre-learned artificial neural network using the third input information 13 as input information, and outputting information obtained by primarily estimating the noise signal included in the third input information 13 as the third output information 22.
The 2-B artificial neural network 172 may include a pre-learned artificial neural network that uses the fourth input information 23 generated based on second output information 21 output from the 2-A artificial neural network 171, third output information 22 output from the 3-A artificial neural network 181, and the third input information 14 as input information, and outputs the estimated information by estimating only the echo signal from the fourth input information 23 as the fourth output information 25.
Looking at the information input to the 2-B artificial neural network 172, the second output information 21 output to the 2-A artificial neural network 171 corresponds to the echo signal included in the third input information 14. So, when the second output information 21 is mixed with the third input information 14 at the third point 3, an emphasized signal for the echo signal portion can be generated at the third point 3. Thereafter, the noise signal is removed at the fourth point 4 using the third output information 22 including information about the noise signal with respect to the generated signal to generate fourth input information 23. And the generated the fourth input information 23 is used as input information input to the 2-B artificial neural network 172.
Accordingly, the fourth input information 23 has noise removed from the third input information 14 and the information about the echo signal has more accurate information than the third input information 14. There is an effect that the information on the echo signal output from the 2-B artificial neural network 172 can be output more accurately from the 2-A artificial neural network 171.
Similarly, the 3-B artificial neural network 182 may include a pre-learned artificial neural network that uses the fifth input information 24 generated based on the third output information 22 output by the 3-A artificial neural network 181, the second output information 21 output by the 2-A artificial neural network 171, and the third input information 14 as input information and outputs the information estimated by estimating only the noise signal from the fifth input information 24 as fifth output information 26.
Looking at the information input to the 3-B artificial neural network 182, the third output information 22 output to the 3-A artificial neural network 181 corresponds to the noise signal included in the third input information 14. So, when the third output information 22 is mixed with the third input information 14 at the fifth point 5, an enhanced signal for the noise signal portion can be generated at the fifth point 5.
Thereafter, when the echo signal is removed from the sixth point 6 using the second output information 21 including information about the echo signal, fifth input information 24 is generated, and the fifth input information 24 generated in this way is used as input information input to the 2-C artificial neural network 182.
Accordingly, in the fifth input information 24, the echo is removed from the third input information 14, and information about the noise signal has information that has more accurate information than the third input information 14. Therefore, the fifth input information 24 can be used as input information of the 3-B artificial neural network 182, there is an effect that information on the noise signal output from the 3-B artificial neural network 182 can be output more accurately.
When the number of neural networks of the second artificial neural network 170 and the third artificial neural network 180 is three or more, as shown in
The sixth input information 27 thus generated is input as input information to the 2-C artificial neural network 173, and the 2-C artificial neural network 173 generates the sixth output information 29 including estimated information about an echo signal based on the sixth input information 27 as output information.
Similarly, the 3-C artificial neural network 183 can generate the seventh input information 28 by the principle described above based on the fourth output information 25, the fifth output information 26, and the third input information 14. The seventh input information 28 thus generated is input as input information to the 3-C artificial neural network 183, and the 3-C artificial neural network 183 outputs the seventh output information 30 including estimated information about a noise signal based on the seventh input information 28 as output information.
As described above, the number of neural networks of the second artificial neural network 170 and the third artificial neural network 180 can be implemented differently depending on the environment, and the number of the second artificial neural network 170 and the third artificial neural network 180 is one, the second output information 21 becomes the final estimated echo signal 31 of the second artificial neural network 170, and the third output information 22 becomes the third output information 22 becomes final estimated noise signal 32 of the artificial neural network 180 as in
In
Referring to
The decoder 160 may output a voice signal in the time domain based on the estimated audio signal 16 output from the voice signal estimator 150.
Specifically, the decoder 160 may serve to convert the latent region estimated voice signal 16 finally estimated in the latent region into the final voice signal 30 in the time domain so that the speaker can recognize the voice since the final estimated echo signal 31 output to the second artificial neural network 170, the final estimated noise signal 31 output to the third artificial neural network 180, and the third input information 14 output from the microphone encoder 130, and the estimated voice signal 16 estimated by the voice signal estimator 150 is information about signals estimated in the latent domain rather than information in the time domain.
Also, the voice signal estimation apparatus 100 according to the second embodiment may perform learning based on two loss functions. In detail, the voice signal estimation apparatus 100 performs learning by reducing the error of the final voice signal 30 estimated in the time domain or outputs information in the latent domain. And, the voice signal estimation apparatus 100 performs learning by reducing errors in information output by each the second artificial neural network 170 and the third artificial neural network of the artificial neural network 180.
Regarding the first learning method, the voice signal estimation apparatus 100 according to the second embodiment calculates the difference between the final voice signal 30 output from the decoder 160 and the voice signal 50 of the actual speaker and, the difference is defined a first loss function. Thereafter, the voice signal estimation apparatus 100 performs learning by updating at least one of the parameters, in the way the attention unit 120, the second artificial neural network 170, and the third artificial neural network 180 of the voice signal apparatus 100 in a direction in which the value of the first loss function decreases.
Specifically, the voice signal estimation apparatus 100 may perform learning using a loss function such as Equation (8) below.
In Equation (8), ∥⋅∥ denotes l2-norm, s{circumflex over ( )} denotes the estimated final voice signal, and Starget denotes the actual speaker's voice signal.
In the invention disclosed in the first learning method, the voice signal estimation apparatus 100 is viewed as one structure in the time domain and learning is performed. However, the second learning method of the disclosed invention performs learning on each artificial neural network of the second artificial neural network 170 and the third artificial neural network 180 in the latent region.
Specifically, the voice signal estimating apparatus 100 calculates the difference between the information estimated and outputted by each artificial neural network of the second artificial neural network 170 and the third artificial neural network 180 and the actual reference information as a second loss function. And learning can be performed by updating the parameters of each of the artificial neural networks of the second artificial neural network 170 and the third artificial neural network 180 in a direction in which the difference between the values of the second loss function decreases.
Therefore, the second loss function can be defined as the sum of difference between the output information of the n-th artificial neural network of the second artificial neural network 170, output information of the n-th artificial neural network of the third artificial neural network 180, and reference information, therefore. And it can be expressed as Equation (9) below.
10 log10(|dr−{circumflex over (d)}r,R|2)+10 log10(|nr−{circumflex over (n)}r,R|2) Equation (11):
Here, R means the total number of artificial neural networks constituting the second artificial neural network 170 and the third artificial neural network 180, and dr and nr mean reference information for echo signals in the latent domain and reference information for noise signals.
In performing learning, the voice signal estimation apparatus 100 according to an embodiment may perform learning using only the first loss function described above or may perform learning using only the second loss function and may perform learning using the third loss function which is sum of the first loss function and the second loss function. Therefore, Learning may be performed by updating at least one parameter of the attention unit 120 of the audio signal apparatus 100, the second artificial neural network 170, and the third of the artificial neural network 180 in a direction in which the value of the third loss function decreases. In addition, when learning is performed using the third loss function, learning may be performed using an Equation (12) below as a loss function equation.
In Equation 12, even if the number of each artificial neural network in the second artificial neural network 170 and the third artificial neural network 180 increases infinitely, learning can be performed by setting q=1/2, αs=0.7 so as not to exceed the weight of the loss function.
Since the basic conditions for the experimental environment for deriving the experimental results in
Referring to
First, comparing the PESQ and STOI scores, which evaluate the degree of voice quality, it appears that the algorithm using all deep neural networks improves voice quality compared to unprocessed. First, comparing the PESQ and STOI scores, which evaluate the degree of voice quality, it appears that the algorithm using all deep neural networks improves voice quality compared to unprocessed. In addition, when comparing the scores with the prior art, the method proposed in the present invention shows the highest score. In all four objective evaluation indicators, it can be seen that the scores are greatly improved compared to the prior art.
In
Referring to
y
i(t)=si(t)+di(t)+ni(t) Equation (13):
And d(t) means the echo signal input to the microphones 310 and 320 after the far-end signal is transformed by the nonlinearity in the speaker 200 and the room impulse response (RIR) between the speaker and the microphone, and s(t) means the speaker's voice signal, n means the noise signal, t means the time index, and i means the ith microphone input.
The voice signal estimation apparatus 100 according to the third embodiment of the present invention reflects the characteristics of the third embodiment and may be referred to as a multi-channel-based noise and echo signal integrated cancellation apparatus using a deep neural network.
Referring to
Since the far-end signal encoder 110, the attention unit 120, the first artificial neural network 140, the voice signal estimator 150, and the decoder 160 described above are the same with the far-end signal encoder 110, the attention unit 120, the first artificial neural network 140, the voice signal estimator 150, and the decoder 160 of the voice signal estimation apparatus 100 according to the third embodiment shown in
The encoder 100 according to the third embodiment is a component that serves to convert signals in the time domain input through the plurality of microphones 300 into signals in the latent domain, respectively. And the number of the encoder 100 may be provided corresponding to the number of the microphones 300. Therefore, the first microphone input signal 20-1 input through the first microphone 310 is input to the first microphone encoder 131, and the second microphone input signal 20-2 input through the second microphone 320 is input to the second microphone encoder 132, and the third microphone input signal 20-2 input through a third microphone (not shown) may be input to the second microphone encoder 132. Although
The plurality of microphone encoders 131, 132, and 133 may output converted signals 12-1, 12-2, and 12-3 obtained by converting input signals in the time domain into signals in other domains.
Specifically, the plurality of microphone encoders 131, 132, and 133 respectively receive a plurality of microphone input signals 20-1, 20-2, 20-3 including echo signals, voice signals, and noise signals in the time domain from the microphone 300, and the microphone input signals 20-1, 20-2 and 20-3 including information in the time domain are converted into signals in the latent domain. And the plurality of microphone encoders 131, 132, and 133 output the converted conversion signals 12-1, 12-2, and 12-3.
As described in
w=H(yU) Equation (2):
w
i
=H(yi,kUi) Equation (14):
In Equation (14), UI means a positive value of N×L length and has N vectors according to the size of input information, and H( ) means a non-linear function.
However, compared to single-channel, multi-channel microphone input has a larger dimension by the number of microphones, so a component that converts signals output through the microphone encoder 130 to a single-channel level is required to maintain parameters like those of a single channel network and synthesize information at the same dimension as information output through the far-end signal encoder 110. Therefore, in the case of the present invention, the conversion calls 12-1, 12-2, and 12-3 input to the channel conversion unit 190 by the channel conversion unit 190 compress information between channels to form a single channel level after being converted into information, and it can be output as the second input information 12. This process performed by the channel conversion unit 190 can be performed through 1D convolution operation on the input signals and can be expressed as Equation (15) below.
w
x
=H(concat[w1,w2, . . . ,wm]Ux) Equation (15):
In Equation (15), Ux means a positive value with a length of N*m×L having N*m vectors.
The second input information 12 output in this format is input to the attention unit 120 together with the first input information 11 output by the far-end signal encoder 110 and is converted into weight information 13 and output. In addition, the weight information 13 is mixed with the second input information 12 and converted into third input information 14, and then input to the first artificial neural network 140 to output mask information for estimating a voice signal. Since this process has been described in detail with reference to
Since the basic conditions for the experimental environment for deriving the experimental results in
Referring to
First, comparing the PESQ and STOI scores, which evaluate the degree of voice quality, the algorithm using all deep neural networks improved the voice quality compared to the unprocessed case, and the STOI and SDR are also better than the conventional technologies. In addition, comparing items 4 to 6, it can be seen that the quality of all items increases when a channel conversion unit is added according to the present invention while being extended to multi-channel.
Referring to
Among the voice signal estimation apparatus 100 according to the fourth embodiment, the far-end signal encoder 110, the first microphone encoder 131, the second microphone encoder 132, the third microphone encoder 133, and the channel converter 190 is the same as the far-end signal encoder 110, the first microphone encoder 131, the second microphone encoder 132, the third microphone encoder 133, and the channel converter 190 described in
The voice signal estimation apparatus 100 according to the fourth embodiment is based on the audio signal device 100 according to the second embodiment utilizing a plurality of artificial neural networks 120 and 130 and the multi-channel-based audio signal device 100. When compared with other embodiments, the difference is that the second attention unit 122 and the third attention unit 123 are used for the information output to the second artificial neural network 170 and the third artificial neural network 180.
If the final estimated echo signal 31 and the final estimated noise signal 32 respectively estimated by the second artificial neural network 170 and the third artificial neural network 180 are simply removed from the compressed mixture, there is a possibility that the possibility of voice distortion increases. Therefore, voice signal estimation apparatus 100 according to the fourth embodiment applies an attention mechanism between the final estimated echo signal 31 and the second input information 12 to prevent such speech distortion, and at the same time, a voice signal can be more accurately extracted by applying an attention mechanism between the final estimated noise signal 32 and the second input information 12.
That is, like the principle described in the attention unit 120 in
Looking at this through equations and
B
w
,{circumflex over (d)}
=σ(Lw
As shown in
B
w
,{circumflex over (n)}
=σ(Lw
In Equations (16) and (17), σ(⋅) denotes the sigmoid function, wx denotes the latent features of the second input signal 12, and {circumflex over (d)}r,R, {circumflex over (n)}r,R denotes the output information of the Rth artificial neural network of the second artificial neural network 170 and the third artificial neural network 180, and L denotes that the 1×1 convolution function is applied.
The information output according to Equation (16) is output after converted into first weight information 33 related to the echo signal by applying the 1D-Conv 227 and the sigmoid function 228, as shown in FIG. and can be expressed as Equation (18) below.
M
mix-echo=σ(Ld) Equation (18):
The information output according to Equation (17) is output after converted into first weight information 34 related to the noise signal by applying the 1D-Conv 237 and the sigmoid function 238 again as shown in
M
mix-noise=σ(Ln) Equation (19):
The first weight information 33 is mixed with the second input information 12 at the seventh point 7 and converted into first mixed information 31, and the second weight information 34 is mixed with the second input information 12 at the eighth point 8 and converted into second mixed information 32. Thereafter, at the ninth point 9, the first mixed information 31 and the second mixed information 32 are removed from the second input information 12, and only the remaining information is input to the voice signal estimator 150 and the estimated voice signal 16 is output and can be expressed as Equation (20) below.
D=w
x
−w
x
M
mix-echo
−w
x
M
mix-noise Equation (20):
The estimated latent domain estimated voice signal 16 can be converted into a form containing information in the time domain with the transposed convolutional layer of Equation (2) described above, like the relationship between the short-time Fourier transform (STFT) and inverse STFT, and it can be expressed as Equation (7) below.
{tilde over (s)}=DV Equation (7):
Here, {tilde over (s)} denotes a voice signal estimated in the time domain, and V denotes a matrix that converts N vectors into L length.
In addition, the voice signal estimation apparatus 100 according to the third embodiment may perform learning based on two loss functions, and specifically, learning is performed by reducing the error of the final voice signal 30 estimated in the time domain. Also, learning can be performed in the way for reducing an error in information output by each of the second artificial neural network 170 and the third artificial neural network 180 outputting information estimated in the latent region for the echo signal and the noise signal.
In the case of the first learning method, learning may be performed by updating at least one parameter of the attention unit 120, the second artificial neural network 170, and the third artificial neural network 180 of the audio signal device 100 in the direction in which the value of the first loss function decreases, by defining the difference between the final voice signal 30 output from the decoder 160 and the voice signal 50 of the actual speaker as the first loss function.
The second learning method is a method of performing learning on each of the artificial neural networks of the second artificial neural network 170 and the third artificial neural network 180 in the latent domain. Specifically, the second learning method defines a difference between information estimated and outputted by each artificial neural network of the second artificial neural network 170 and the third artificial neural network 180 and actual reference information as a second loss function. Thereafter, the second learning method is a method of updating the parameters of each of the artificial neural networks of the second artificial neural network 170 and the third artificial neural network 180 in a direction in which the difference in the value of the second loss function becomes small. Therefore, the second loss function can be defined as the sum of the differences between output information of the n-th artificial neural network of the second artificial neural network 170 and reference information therefor, and output information of the n-th artificial neural network of the third artificial neural network 180 and reference information therefor and it can be expressed as Equation (9) below.
In addition, in performing learning, the voice signal estimation apparatus 100 according to the fourth embodiment may perform learning using only the first loss function described above, only the second loss function, or the third loss function obtained by adding the first loss function and the second loss function. In addition, the voice signal estimation apparatus 100 according to the fourth embodiment uses a third loss function obtained by adding the first loss function and the second loss function to a direction in which the value of the third loss function decreases. And learning may be performed by updating at least one parameter of the attention unit 120, the second artificial neural network 170, and the third artificial neural network 180 of the voice signal estimation apparatus 100.
In the case of the method for learning the artificial neural network using the first loss function, the second loss function, and the third loss function, the voice signal estimation apparatus 100 according to the second embodiment has been described in detail while explaining, a detailed description thereof will be omitted.
Referring to
First, comparing the PESQ and STOI scores, which evaluate the degree of voice quality, the algorithm using all deep neural networks improved the voice quality compared to the unprocessed case, and the STOI and SDR are also better than the conventional technologies. In addition, comparing items 5 to 7, the quality of all items increases as the number of channels increases when the attention part has up to the third attention according to the fourth embodiment.
An apparatus for integrating and canceling noise and echo signals based on multi-channels using a deep neural network according to an embodiment can increase the accuracy of estimation of the echo signal and the noise signal by iteratively and separately estimating the echo signal and the noise signal. Among the signals accurately input to the microphone, there is an advantage in that echo signals and noise signals can be removed.
In addition, the present invention provides an apparatus for estimating a voice signal capable of more accurately extracting only user's voice information because the accuracy of the estimated echo signal and noise signal by applying an attention mechanism to the estimated echo signal and noise signal information.
Therefore, in the case of collecting and processing the speaker's voice through a microphone in an environment where echo signals exist, such as artificial intelligence speakers used in home environments, robots used in airports, voice recognition and PC voice communication systems, etc., echo signals are more efficiently processed, there is an effect of improving voice quality and intelligibility.
As described above, the embodiments can derive better performance by removing noise and echo before performing voice recognition and voice communication technology as a voice enhancement technology and can be applied to improve voice call quality in a mobile phone terminal or voice talk. In addition, recently, voice recognition is performed in various Internet of Things (IoT) devices, which can be performed not only in a quiet environment but also in an environment with ambient noise. Therefore, sound can re-enter and create an echo. Therefore, the performance of voice recognition performed by IoT devices can be improved by removing noise and echo before performing voice recognition. In addition, since the present embodiments provide a voice enhancement signal of excellent quality, they can be applied to various voice communication technologies to provide a clear voice quality.
The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.
Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and software may independently or collectively configure a processing device to operate as desired and can command the processing unit. Software and/or data may be embodied in any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.
The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks.—includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.
As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved. Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2021-0009000 | Jan 2021 | KR | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/KR2022/001164 | 1/21/2022 | WO |
