Patent Application
20220406325
The present disclosure relates to audio processing.
Sound source separation techniques have been proposed for separating a mix of multiple sounds generated by different sound sources into isolated sounds for individual sound sources. For example, “Determined Blind Source Separation Unifying Independent Vector Analysis and Nonnegative Matrix Factorization,” (Kitamura, Ono, Sawada, Kameoka, and Saruwatari, IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no. 9, pp. 1626-1641, September 2016) (hereafter, Kitamura et al.), discloses Independent Low-Rank Matrix Analysis (ILRMA), which achieves highly accurate sound source separation by taking into account independence of signals and low-ranking of sound sources.
“Singing Voice Separation with Deep U-Net Convolutional Networks,” (Jansson, Humphrey, Montecchio, Bittner, Kumar, and Weyde, Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR), 2017) (hereafter, Jansson et al.), discloses a technique for generating time-frequency domain masks for sound source separation by inputting amplitude spectrograms to a neural network.
However, in the techniques disclosed in Kitamura et al. and Jansson et al., a problem exists in that a processing load for sound source separation is excessive.
Considering the foregoing, one aspect of the present disclosure aims to reduce a processing load for sound source separation.
In order to solve the above problem, an audio processing method according to one aspect of the present disclosure includes: obtaining input data including first sound data, second sound data, and mix sound data, the first sound data included in the input data representing first components of a first frequency band, included in a first sound corresponding to a first sound source, the second sound data included in the input data representing second components of the first frequency band, included in a second sound corresponding to a second sound source that differs from the first sound source, and the mix sound data included in the input data representing mix components of an input frequency band including a second frequency band that differs from the first frequency band, the mix components being included in a mix sound of the first sound and the second sound; and generating, by inputting the obtained input data to a trained estimation model, at least one of: first output data representing first estimated components of an output frequency band including the second frequency band, included in the first sound, or second output data representing second estimated components of the output frequency band, included in the second sound.
A method of training an estimation model according to one aspect of the present disclosure includes: preparing a tentative estimation model; obtaining a plurality of training data, each training data including training input data and corresponding training output data; and establishing a trained estimation model that has learned a relationship between the training input data and the training output data by machine learning in which the tentative estimation model is trained using the plurality of training data. The training input data includes first sound data, second sound data, and mix sound data, the first sound data representing first components of a first frequency band, included in a first sound corresponding to a first sound source, the second sound data representing second components of the first frequency band, included in a second sound corresponding to a second sound source that differs from the first sound source, and the mix sound data representing mix components of an input frequency band including a second frequency band that differs from the first frequency band, the mix components being included in a mix sound of the first sound and the second sound. The training output data includes at least one of: first output data representing first output components of an output frequency band including the second frequency band, included in the first sound, or second output data representing second output components of the output frequency band, included in the second sound.
An audio processing system according to one aspect of the present disclosure includes: one or more memories for storing instructions; and one or more processors communicatively connected to the one or more memories and that execute the instructions to: obtain input data including first sound data, second sound data, and mix sound data. The first sound data included in the input data represents first components of a first frequency band, included in a first sound corresponding to a first sound source, the second sound data included in the input data represents second components of the first frequency band, included in a second sound corresponding to a second sound source that differs from the first sound source, and the mix sound data included in the input data represents mix components of an input frequency band including a second frequency band that differs from the first frequency band, the mix components being included in a mix sound of the first sound and the second sound. The one or more processors then execute the instructions to generate, by inputting the obtained input data to a trained estimation model, at least one of: first output data representing first estimated components of an output frequency band including the second frequency band, included in the first sound, or second output data representing second estimated components of the output frequency band, included in the second sound.
The storage device 12 comprises either a single memory or multiple memories that store programs executable by the controller 11, and a variety of data used by the controller 11. The storage device 12 is constituted of a known storage medium, such as a magnetic or semiconductor storage medium, or a combination of several types of storage media. The storage device 12 may be provided separate from the audio processing system 100 (e.g., cloud storage), and the controller 11 may perform writing to and reading from the storage device 12 via a communication network, such as a mobile communication network or the Internet. In other words, the storage device 12 need not be included in the audio processing system 100.
The storage device 12 stores a time-domain audio signal Sx representative of a sound waveform. The audio signal Sx represents a sound (hereafter, “mix sound”), which is a mix of a sound produced by a first sound source (hereafter, “first sound”) and a sound produced by a second sound source (hereafter, “second sound”). The first sound source and the second sound source are separate sound sources. The first sound source and the second sound source each may be a sound source such as a singer or a musical instrument. For example, the first sound is a singing voice of a singer (first sound source), and the second sound is an instrumental sound produced by a percussion instrument or other musical instrument (second sound source). The audio signal Sx is recorded in an environment in which the first sound source and the second sound source are produced in parallel, and by use of a sound receiver, such as a microphone array. However, signals synthesized by known synthesis techniques may be used as audio signals Sx. In other words, each of the first and second sound sources may be a virtual sound source.
The first sound source may comprise a single sound source or a set of multiple sound sources. Likewise, the second sound source may comprise a single sound source or a set of multiple sound sources. The first sound source and the second sound source are generally of different types, and the acoustic characteristics of the first sound differ from those of the second sound due to differences in the types of sound sources. However, the first sound source and the second sound source may be of the same type if the first sound source and the second source are separable by spatially positioning the first and the second sound sources. For example, the first sound source and the second sound source may be of the same type if the first and second sound sources are installed at different respective positions. In other words, the acoustic characteristics of the first sound and the second sound may approximate or match each other.
The controller 11 comprises either a single processor or multiple processors that control each element of the audio processing system 100. Specifically, the controller 11 is constituted of one or more types of processors, such as CPU (Central Processing Unit), SPU (Sound Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or Application Specific Integrated Circuit (ASIC) and so on. The controller 11 generates the audio signal Sz from the audio signal Sx stored in the storage device 12. The audio signal Sz is a time-domain signal representative of a sound in which one of the first sound and the second sound is emphasized relative to the other. In other words, the audio processing system 100 performs sound source separation to separate the audio signal Sx into respective sound source.
The sound outputter 13 outputs a sound represented by the audio signal Sz generated by the controller 11. The sound outputter 13 is, for example, a speaker or headphones. For convenience of explanation, a D/A converter that converts the audio signal Sz from digital to analog format, and an amplifier that amplifies the audio signal Sz are omitted from the figure.
The frequency analyzer 21 generates an intensity spectrum X(m) of the audio signal Sx sequentially for each unit period (frame) on the time axis. The symbol in denotes one unit period on the time axis. The intensity spectrum X(m) is, for example, an amplitude spectrum or a power spectrum. To generate the intensity spectrum X(m), any known frequency analysis, such as for example, short-time Fourier transform or wavelet transform can be employed. A complex spectrum calculated from the audio signal Sx can be used as the intensity spectrum X(m).
The mix sound represented by the audio signal Sx includes components in a frequency band BL and components in a frequency band BH. The frequency band BL and the frequency band BH differ from each other within the whole band BF. The frequency band BL is lower than the frequency band BH. Specifically, the frequency band BL is below a given frequency on the frequency axis within the whole band BF, and the frequency band BH is above the given frequency within the whole band BF. Accordingly, the frequency band BL and the frequency band BH do not overlap. For example, the frequency band BL ranges from 0 kHz to less than 4 kHz, and the frequency band BH ranges from 4 kHz to 8 kHz. It is of note, however, that the bandwidth of the frequency band BL and the bandwidth of the frequency band BH may be the same as or different from each other. Each of the first and second sounds constituting the mix sound contains components of the frequency band BL and components of the frequency band BH. The frequency band BL is an example of a “first frequency band” and the frequency band BH is an example of a “second frequency band.”
The sound source separator 22 in
For processing of the intensity spectrum X(m) by the sound source separator 22, any known sound source separation method can be employed. For sound source separation performed by the sound source separator 22, there may be used independent component analysis (ICA), independent vector analysis (IVA), non-negative matrix factorization (NMF), multichannel NMF (MNMF), independent low-rank matrix analysis (ILRMA), independent low-rank tensor analysis (ILRTA), or independent deeply-learned matrix analysis (IDLMA), for example. In the above description, an example is given of sound source separation in the frequency domain. However, the sound source separator 22 may also perform sound source separation in the time domain on the audio signal Sx.
The sound source separator 22 generates the intensity spectrum Y1(m) and the intensity spectrum Y2(m) by sound source separation of the components of the frequency band BL, included in the intensity spectrum X(m). Intensity spectra Y1(m) and Y2(m) are generated for each unit period. As illustrated in
As described above, in the first embodiment, in a case in which the components of the frequency band BH, included in the mix sound represented by the audio signal Sx, are not processed in the sound source separation, the processing load for the sound source separator 22 is reduced, compared with a configuration in which sound source separation is performed for the whole band BF including both the frequency band BL and the frequency band BH.
The band extender 23 in
The intensity spectrum Z1(m) represented by the first output data O1(m) is, as illustrated in
As illustrated in
As illustrated in
The first sound data D1(m) represents the intensity spectrum Y1(m) of the first sound. Specifically, the first sound data D1(m) generated for any one target period includes an intensity spectrum Y1(m) of that target period and intensity spectra Y1(Y1(m−4), Y1(m−2), Y1(m+2), Y1(m+4)) of other unit periods around the target period. More specifically, the first sound data D1(m) includes the intensity spectrum Y1(m) of the target period, the intensity spectrum Y1(m−2) of the unit period two units before the target period, the intensity spectrum Y1(m−4) of the unit period four units before the target period, the intensity spectrum Y1(m+2) of the unit period two units after the target period, and the intensity spectrum Y1(m+4) of the unit period four units after the target period. As will be understood from the above description, the first sound data D1(m) represents the first components of the first sound of the frequency band BL.
The second sound data D2(m) represents the intensity spectrum Y2(m) of the second sound. Specifically, the second sound data D2(m) generated for any one target period includes an intensity spectrum Y2(m) of that target period and intensity spectra Y2 (Y2(m−4), Y2(m−2), Y2(m+2), Y2(m+4)) of other unit periods around the target period. More specifically, the second sound data D2(m) includes the intensity spectrum Y2(m) of the target period, the intensity spectrum Y2(m−2) of the unit period two units before the target period, and the intensity spectrum Y2(m−4) of the unit period four units before the target period, the intensity spectrum Y2(m+2) of the unit period two units after the target period, and the intensity spectrum Y2(m+4) of the unit period four units after the target period. As will be understood from the above description, the second sound data D2(m) represents the second components of the frequency band BL, included in the second sound.
Each element of a vector V represented by the whole input data D(m) is normalized such that the magnitude of the vector V is 1 (i.e., unit vector). For example, with respect to the input data D(m) before normalization, it is assumed that the first sound data D1(m), the second sound data D2(m), and the mix sound data Dx(m) constitute an N-dimensional vector V with N elements el to eN. Each of the N elements e1 to eN constituting the input data D(m) after normalization is expressed by the following Equation (1), where n=1 to N,
The symbol ∥ ∥2 in Equation (1) means the L2 norm expressed in Equation (2) below, and the L2 norm corresponds to an index representative of the magnitude of the vector V (hereafter, “intensity index α”),
The generator 232 in
The estimation model M is constituted of, for example, a neural network.
The estimation model M is realized by a combination of (i) an estimation program that causes the controller 11 to perform operations for generating output data O(m) from input data D(m) and (ii) multiple variables K (specifically, weighted values and biases) applied to the operations. The estimation program and the variables K are stored in the storage device 12. The numerical values of the respective variables K are set in advance by machine learning.
The waveform synthesizer 24 in
As mentioned earlier, each of the elements En that constitute the input data D(m) is a value normalized using the intensity index α. Therefore, the volume of the audio signal Sz0 may differ from that of the audio signal Sx. The volume adjuster 25 generates an audio signal Sz by adjusting (i.e., scaling) the volume of the audio signal Sz0 to a volume equivalent to that of the audio signal Sx. The audio signal Sz is then supplied to the sound outputter 13, thereby being emitted as a sound wave. Specifically, the volume adjuster 25 multiplies the audio signal Sz0 by an adjustment value G that depends on a difference between the volume of the audio signal Sx and the volume of the audio signal Sz0, to generate the audio signal Sz. The adjustment value G is set such that a difference in volume between the audio signal Sx and the audio signal Sz is minimized.
When the audio processing Sa is started, the controller 11 (frequency analyzer 21) generates for each of the multiple unit periods an intensity spectrum X(m) of the audio signal Sx (Sa1). The controller 11 (sound source separator 22) generates intensity spectra Y1(m) and Y2(m) for each unit period by sound source separation of the components of the frequency band BL, included in the intensity spectrum X(m) (Sa2).
The controller 11 (obtainer 231) generates input data D(m) for each unit period from the intensity spectrum X(m), the intensity spectrum Y1(m), and the intensity spectrum Y2(m) (Sa3). The controller 11 (generator 232) generates output data O(m) for each unit period by inputting the generated input data D(m) to the estimation model M (Sa4). The controller 11 (waveform synthesizer 24) generates an audio signal Sz0 from a series of the first output data O1(m) or a series of the second output data O2(m) (Sa5). The controller 11 (volume adjuster 25) generates an audio signal Sz by multiplying the audio signal Sz0 by the adjustment value G (Sa6).
As described above, in the first embodiment, output data O(m) representative of the components of the whole band BF that includes the frequency band BL is generated from input data D(m) including first sound data D1(m) and second sound data D2(m) each representative of the components of the frequency band BL. In other words, output data O(m) that includes the components of the whole band BF is generated even in a configuration by which sound source separation is performed only for the frequency band BL included in the mix sound represented by the audio signal Sx. Therefore, the processing load for sound source separation can be reduced.
As illustrated in
The storage device 12 stores a plurality of training data T used for machine learning of the estimation model M.
The first signal Sr1 includes components of the frequency band BL and components of the frequency band BH. An intensity spectrum R1(m) of the first signal Sr1 is constituted of an intensity spectrum Y1(m) of the frequency band BL and an intensity spectrum H1(m) of the frequency band BH. The first sound data D1(m) of the training input data Dt(m) represents, of the intensity spectrum R1(m), an intensity spectrum Y1(m) of the frequency band BL. Specifically, the first sound data D1(m) of the target period is constituted of an intensity spectrum Y1(m) of the target period and intensity spectra Y1(Y1(m−4), Y1(m−2), Y1(m+2), Y1(m+4)) of other unit periods around the target period.
Similarly to the first signal Sr1, the second signal Sr2 includes components of the frequency band BL and components of the frequency band BH. An intensity spectrum R2(m) of the second signal Sr2 is constituted of an intensity spectrum Y2(m) of the frequency band BL and an intensity spectrum H2(m) of the frequency band BH. The second sound data Dt2(m) of the training input data Dt(m) represents, of the intensity spectrum R2(m), an intensity spectrum Y2(m) of the frequency band BL. Specifically, the second sound data Dt2(m) of the target period is constituted of an intensity spectrum Y2(m) of that target period and intensity spectra Y2(Y2(m−4), Y2(m−2), Y2(m+2), Y2(m+4)) of other unit periods around the target period.
The training output data Ot(m) of the training data T is a ground truth. The ground truth is constituted of first output data Ot1(m) and second output data Ot2(m). The first output data Ot1(m) represents the intensity spectrum R1(m) of the first signal Sr1. In other words, the first output data Ot1(m) is, of the mix sound represented by the reference signal Sr, a spectrum of the first sound throughout the whole band BF. The second output data Ot2(m) represents the intensity spectrum R2(m) of the second signal Sr2. In other words, the second output data Ot2(m) is, of the mix sound represented by the reference signal Sr, a spectrum of the second sound throughout the whole band BF.
Each element of a vector V, which is represented by the whole training input data Dt(m), is normalized such that the magnitude of the vector V is 1, similarly to the input data Dt(m) described above. Similarly, each element of a vector V represented by the whole training output data Ot(m) is normalized such that the magnitude of the vector V is 1.
The obtainer 31 of
The trainer 32 establishes an estimation model M by way of processing using the plurality of training data T (hereafter, “learning processing Sb”). The learning processing Sb is supervised machine learning that uses the training data T. Specifically, the trainer 32 repeatedly updates the multiple variables K that define the estimation model M such that a loss function L representative of an error between (i) output data O(m) generated by a tentative estimation model M in response to a supply of the input data Dt(m) of the training data T and (ii) the output data Ot(m) contained in the same training data T is reduced (ideally minimized). Thus, the estimation model M learns a potential relationship between the input data Dt(m) and the output data Ot(m) in the training data T. In other words, after training by the trainer 32, the estimation model M outputs, for unknown input data D(m), statistically valid output data O(m) based on that relationship.
The loss function L is expressed, for example, by the following Equation (3),
L=ε[O1(m),Ot1(m)]ε[O2(m),Ot2(m)] . . . (3)
The symbol ε[a,b] in Equation (3) is an error between element a and element b (e.g., a mean square error or cross entropy function).
The controller 11 (obtainer 31) obtains training data T from the storage device 12 (Sb1). The controller 11 (trainer 32) performs machine learning in which the tentative estimation model M is trained using the obtained training data T (Sb2). That is, the multiple variables K of the tentative estimation model M are repeatedly updated such that the loss function L is reduced between (i) the output data O(m) generated by the estimation model M from the input data Dt(m) of the training data T and (ii) the output data Ot(m) (i.e., the ground truth) of the training data T. To update the multiple variables K in accordance with the loss function L, an error back propagation method is used, for example.
The controller 11 determines whether a condition for ending the learning processing Sb is met (Sb3). The end condition is, for example, that the loss function L falls below a predetermined threshold or that the amount of change in the loss function L falls below a predetermined threshold. If the end condition is not met (Sb3: NO), the controller 11 (obtainer 31) obtains from the storage device 12 training data T that the controller 11 has not obtained (Sb1). In other words, the process of obtaining training data T (Sb1) and the process of updating the multiple variables K using the obtained training data T (Sb2) are repeated until the end condition is met. When the end condition is met (Sb3: YES), the controller 11 ends the learning processing Sb.
As explained above, in the first embodiment, the estimation model M is established such that the output data O(m) representative of components of both the frequency band BL and the frequency band BH are generated from the input data D(m) including the first sound data D1(m) and the second sound data D2(m) each representative of components of the frequency band BL. In other words, even in a configuration where sound source separation is performed only for the frequency band BL included in the mix sound represented by the audio signal Sx, the output data O(m) including the components of the frequency band BH is generated by using the trained estimation model M. Therefore, the processing load for sound source separation can be reduced.
The second embodiment is described below. For elements whose functions are similar to those of the first embodiment in each of the following embodiments and modifications, the reference signs used in the description of the first embodiment are used and detailed descriptions of such elements are omitted as appropriate.
In the first embodiment, an example is given of a configuration in which the mix sound data Dx(m) includes both mix components of the frequency band BL and mix components of the frequency band BH. However, since the components of the first sound of the frequency band BL are included in the first sound data D1(m), and the components of the frequency band BL in the second sound are included in the second sound data D2(m), it is not essential that the mix sound data Dx(m) includes the mix components of the frequency band BL. Taking the foregoing into account, in the second embodiment, the mix sound data Dx(m) does not include the mix components of the frequency band BL, included in the mix sound.
In the above description, the input data D(m) used for audio processing Sa is shown as an example. Similarly, the training input data Dt(m) used for the learning processing Sb includes the mix sound data Dx(m) representative of the components of the frequency band BH included in the mix sound represented by the reference signal Sr. In other words, the mix sound data Dx(m) for training represents, of the intensity spectrum X(m) of the reference signal Sr, the intensity spectrum XH(m) of the frequency band BH, and the intensity spectrum XL(m) of the frequency band BL is not reflected in the mix sound data Dx(m).
In the second embodiment, the same effects as those in the first embodiment are realized. Further, in the second embodiment, the mix sound data Dx(m) does not include the components within the frequency band BL, included in the mix sound. Therefore, compared with a configuration in which the mix sound data Dx(m) includes components of the whole band BF, there is an advantage that the processing load of the learning processing Sb and the size of the estimation model M are reduced.
In the first embodiment, an example is given of the mix sound data Dx(m) representative of mix components of the whole band BF, included in the mix sound. In the second embodiment, an example is given of the mix sound data Dx(m) representative of mix components of the frequency band BH, included in the mix sound. As will be understood from the above examples, the mix sound data Dx(m) is comprehensively expressed as data representative of mix components of an input frequency band including the frequency band BH, included in the mix sound. In other words, the mix sound data Dx(m) represents the mix components of the input frequency band that does not include the frequency band BL, included in the mix sound.
In the third embodiment, the same effects as those in the first embodiment are realized. In the third embodiment, since the intensity index α is included in the input data D(m), output data O(m) representative of the components of the loudness corresponding to the mix sound is generated. Therefore, the process of adjusting the intensity of the sound represented by the first output data O1(m) and the second output data O2(m) (volume adjuster 25) unnecessary.
From Result A in
As will be apparent from
In a learning processing Sb of the fourth embodiment, the loss function L expressed by Equation (3) is replaced by a loss function L expressed by the following Equation (4),
L=ε[O1(m),Ot1(m)]+ε[O2(m),Ot1(m)]+ε[XH(m),O1H(m)+O2H(m)] (4)
The symbol O1H(m) in Equation (4) is an intensity spectrum of the frequency band BH, included in the intensity spectrum Z1(m) represented by the first output data O1(m). The symbol O2H(m) is an intensity spectrum of the frequency band BH, included in the intensity spectrum Z2(m) represented by the second output data O2(m). Thus, the third term in the right-hand side of Equation (4) means an error between (i) the intensity spectrum XH(m) of the frequency band BH, included in the intensity spectrum X(m) of the reference signal Sr and (ii) the sum (H1(m)+H2(m)) of the intensity spectrum H1(m) and the intensity spectrum H2(m). As will be understood from the above description, the trainer 32 of the fourth embodiment trains the estimation model M under a condition (hereafter, “additional condition”) that a mix of the components of the frequency band BH, included in the intensity spectrum Z1(m) and the components of the frequency band BH included the intensity spectrum Z2(m) approximates or matches the components of the frequency band BH (intensity spectrum XH(m)), included in the intensity spectrum X(m) of the mix sound.
In the fourth embodiment, the same effects as those in the first embodiment are realized. In addition, according to the fourth embodiment, compared with the configuration that uses the trained estimation model M without the additional condition, it is possible to estimate with high accuracy the components of the frequency band BH, included in the first sound (first output data O1(m)), and the components of the frequency band BH, included in the second sound (second output data O2(m)). The configuration of the fourth embodiment is similarly applicable to the second and third embodiments.
Further, in the fifth embodiment, the second output data O2(m) representative of the intensity spectrum H2(m) of the frequency band BH, included in the second sound, is supplied to the waveform synthesizer 24 from the audio processor 20. In addition, the intensity spectrum Y2(m) of the frequency band BL, included in the second sound, is supplied to the waveform synthesizer 24 from the sound source separator 22. When a user instruction to emphasize the second sound is provided, the waveform synthesizer 24 synthesizes the intensity spectrum H2(m) and the intensity spectrum Y2(m), thereby to generate the intensity spectrum Z2(m) of the whole band BF, and generates an audio signal Sz0 from a series of the intensity spectra Z2(m). The intensity spectrum Z2(m) is, for example, a spectrum with an intensity spectrum Y2(m) in the frequency band BL and an intensity spectrum H2(m) in the frequency band BH.
In the fifth embodiment, the same effects as those in the first embodiment are realized. In the fifth embodiment, the output data O(m) does not include the components of the frequency band BL. Therefore, compared with a configuration (e.g., the first embodiment) in which the output data O(m) includes the components of the whole band BF, the processing load of the learning processing Sb and the size of the estimation model MT are reduced. On the other hand, according to the first embodiment in which the output data O(m) includes the components throughout the whole band BF, it is possible to easily generate sound throughout the whole band BF compared with the fifth embodiment.
In the first embodiment, an example is given of the first output data O1(m) representative of the components of the whole band BF, including the frequency band BL and the frequency band BH, included in the first sound. In the fifth embodiment, an example is given of the first output data O1(m) representative of the components of the frequency band BH, included in the first sound. As will be understood from the above examples, the first output data O1(m) is comprehensively expressed as data representative of first estimated components of an output frequency band including the frequency band BH included in the first sound. In other words, the first output data O1(m) may or may not include the components of the frequency band BL. Similarly, the second output data O2(m) is comprehensively expressed as data representative of second estimated components of the output frequency band including the frequency band BH, included in the second sound. In other words, the second output data O2(m) may or may not include the components of the frequency band BL.
The following are examples of specific modifications that can be added to each of the above examples. Two or more aspects freely selected from the following examples may be combined as appropriate to the extent that they do not contradict each other.
(1) In each of the above embodiments, an example is given of mix sound data Dx(m) including an intensity spectrum X(m) of a target period and intensity spectra X of other unit periods, but the content of the mix sound data Dx(m) is not limited thereto. For example, a configuration may be assumed in which the mix sound data Dx(m) of the target period includes only the intensity spectrum X(m) of the target period. The mix sound data Dx(m) of the target period may include an intensity spectrum X of a unit period either prior or subsequent to that target period. For example, the mix sound data Dx(m) may be configured to include only the prior intensity spectrum X (e.g., X(m−1)), or the mix sound data Dx(m) may be configured to include only the subsequent intensity spectrum X (e.g., X(m+1)). In each of the above embodiments, the mix sound data Dx(m) of the target period includes the intensity spectra X (X(m−4), X(m−2), X(m+2), X(m+4) for other unit periods spaced before and after the target period. However, the mix sound data Dx(m) may include an intensity spectrum X(m−1) of a unit period immediately before the target period or an intensity spectrum X(m+1) of a unit period immediately after the target period. Three or more consecutive intensity spectra X timewise (e.g., X(m−2), X(m−1), X(m), X(m+1), X(m+2)) may be included by the mix sound data Dx(m).
In the above description, the focus is on the mix sound data Dx(m), but the same applies to the first sound data D1(m) and the second sound data D2(m). For example, the first sound data D1(m) of the target period may include only the intensity spectrum Y1(m) of that target period, or may include the intensity spectrum Y1 of a unit period either prior or subsequent to the target period. The first sound data D1(m) of the target period may include an intensity spectrum Y1(m−1) of a unit period immediately before the target period, or an intensity spectrum Y1(m+1) of the unit period immediately after the target period. The same applies to the second sound data D2(m).
(2) In each of the above embodiments, the focus is on the frequency band BL that is below a predetermined frequency and the frequency band BH that is above the predetermined frequency. However, the relationship between the frequency band BL and the frequency band BH is not limited thereto. For example, a configuration may be assumed in which the frequency band BL is above the predetermined frequency and the frequency band BH is below the predetermined frequency. The frequency band BL and the frequency band BH are not limited to a continuous frequency band on the frequency axis. For example, the frequency band BF may be divided into sub frequency bands. One or more frequency bands selected from among the sub frequency bands may be considered to be the frequency band BL, and the remaining one or more frequency bands may be considered to be the frequency band BH. For example, among the sub frequency bands, a set of two or more odd-numbered frequency bands may be the frequency band BL, and a set of two or more even-numbered frequency bands may be the frequency band BH. Alternatively, among the sub frequency bands, a set of two or more even-numbered frequency bands may be the frequency band BL, and a set of two or more odd-numbered frequency bands may be the frequency band BH.
(3) In each of the above embodiments, an example is given of a case in which an audio signal Sx prepared in advance is processed. However, the audio processor 20 may perform the audio processing Sa on the audio signal Sx in real time in parallel with the recording of the audio signal Sx. It is of note that there will be a time delay of a length corresponding to the four unit periods in a configuration in which the mix sound data Dx(m) includes the intensity spectrum X(m+4), which is after the target period, as in the examples in the above embodiments.
(4) In each of the above embodiments, the band extender 23 generates both the first output data O1(m) representative of the intensity spectrum Z1(m) in which the first sound is emphasized and the second output data O2(m) representative of the intensity spectrum Z2(m) in which the second sound is emphasized. However, the band extender 23 may generate either the first output data O1(m) or the second output data O2(m) as the output data O(m). For example, a case can be assumed in which an audio processing system 100 is used for reducing a singing voice by applying the audio processing Sa to the mix of the singing voice (first sound) and an instrumental sound (second sound). In such a case, it suffices if the band extender 23 generates output data O(m) (second output data O2(m)) that represents the intensity spectrum Z2(m) in which the second sound is emphasized. Thus, the generation of the intensity spectrum Z1(m) in which the first sound is emphasized is omitted. As will be understood from the above description, the generator 232 is expressed as an element that generates at least one of the first output data O1(m) or the second output data O2(m).
(5) In each of the above embodiments, the audio signal Sz is generated in which one of the first and second sounds is emphasized. However, the processing performed by the audio processor 20 is not limited thereto. For example, the audio processor 20 may output as the audio signal Sz a weighted sum of a first audio signal generated from a series of the first output data O1(m) and a second audio signal generated from a series of the second output data O2(m). The first audio signal is a signal in which the first sound is emphasized, and the second audio signal is a signal in which the second sound is emphasized. Further, the audio processor 20 may perform audio processing, such as effects impartation, for each of the first audio signal and the second audio signal independently of each other, and add together the first audio signal and the second audio signal after the audio processing, thereby to generate the audio signal Sz.
(6) The audio processing system 100 may be realized by a server device communicating with a terminal device, such as a portable phone or a smartphone. For example, the audio processing system 100 generates an audio signal Sz by performing audio processing Sa on an audio signal Sx received from a terminal device, and transmits the audio signal Sz to the terminal device. In a configuration in which the audio processing system 100 receives an intensity spectrum X(m) generated by a frequency analyzer 21 mounted to a terminal device, the frequency analyzer 21 is omitted from the audio processing system 100. In a configuration in which the waveform synthesizer 24 (and the volume controller 25) are mounted to a terminal device, the output data O(m) generated by the band extender 23 is transmitted from the audio processing system 100 to the terminal device. Accordingly, the waveform synthesizer 24 and the volume adjuster 25 are omitted from the audio processing system 100.
The frequency analyzer 21 and the sound source separator 22 may be mounted to a terminal device. The audio processing system 100 may receive from the terminal device an intensity spectrum X(m) generated by the frequency analyzer 21, and an intensity spectrum Y1(m) and an intensity spectrum Y2(m) generated by the sound source separator 22. As will be understood from the above description, the frequency analyzer 21 and the sound source separator 22 may be omitted from the audio processing system 100. Even if the audio processing system 100 does not include the sound source separator 22, it is still possible to achieve the desired effect of reducing the processing load for sound source separation performed by external devices such as terminal devices.
(7) In each of the above embodiments, an example is given of the audio processing system 100 having the audio processor 20 and the learning processor 30. However, one of the audio processor 20 and the learning processor 30 may be omitted. A computer system with the learning processor 30 can also be described as an estimation model training system (machine learning system). The audio processor 20 may or may not be provided in the estimation model training system.
(8) The functions of the audio processing system 100 are realized, as described above, by cooperation of one or more processors constituting the controller 11 and the programs (P1, P2) stored in the storage device 12. The programs according to the present disclosure may be provided in a form stored in a computer-readable recording medium and installed on a computer. The recording medium is a non-transitory recording medium, for example, and an optical recording medium (optical disk), such as CD-ROM, is a good example. However, any known type of recording media such as semiconductor recording media or magnetic recording media are also included. Non-transitory recording media include any recording media except for transitory, propagating signals, and volatile recording media are not excluded. In a configuration in which a delivery device delivers a program via a communication network, a storage device 12 that stores the program in the delivery device corresponds to the above non-transitory recording medium.
For example, the following configurations are derivable from the above embodiments and modification.
An audio processing method according to one aspect (Aspect 1) of the present disclosure includes obtaining input data including first sound data, second sound data, and mix sound data, the first sound data included in the input data representing first components of a first frequency band, included in a first sound corresponding to a first sound source, the second sound data included in the input data representing second components of the first frequency band, included in a second sound corresponding to a second sound source that differs from the first sound source, and the mix sound data included in the input data representing mix components of an input frequency band including a second frequency band that differs from the first frequency band, the mix components being included in a mix sound of the first sound and the second sound; and generating, by inputting the obtained input data to a trained estimation model, at least one of: first output data representing first estimated components of an output frequency band including the second frequency band, included in the first sound, or second output data representing second estimated components of the output frequency band, included in the second sound.
According to the above configuration, at least one of (a1) the first output data representative of the first estimated components of the output frequency band including the second frequency band included in the first sound or (a2) the second output data representative of the second estimated components of the output frequency band including the second frequency band included in the second sound are generated from input data including (b1) the first sound data representative of the first components of the first frequency band, included in the first sound, and (b2) the second sound data representative of the second components of the first frequency band, included in the second sound. Thus, it suffices if sound represented by the first sound data comprises the first components of the first frequency band, included in the first sound, and if sound represented by the second sound data comprises the second components of the first frequency band, included in the second sound. According to the above configuration, it is only necessary to perform sound source separation on the first frequency band when sound separation is to be performed for a mix sound consisting of a first sound corresponding to a first sound source and a second sound corresponding to a second sound source to isolate the first and the second sounds. Therefore, the processing load for sound source separation is reduced.
The “first sound corresponding to the first sound source” means a sound that predominantly includes sound produced by the first sound source. In other words, the concept of “the first sound corresponding to the first sound source” covers not only the sound produced by the first sound source alone, but also a sound that includes, for example, the first sound produced by the first sound source plus a small amount of the second sound from the second sound source (namely, the second sound is not completely removed by the sound source separation). Similarly, “the second sound corresponding to the second sound source” means a sound that predominantly includes the sound produced by the second sound source. In other words, the concept of “the second sound corresponding to the second sound source” covers not only the sound produced by the second sound source alone, but also, a sound that includes, for example, the second sound produced by the second sound source plus a small amount of the first sound from the first sound source (namely, the first sound is not completely removed by the sound source separation).
The mix components represented by the mix sound data may comprise (i) mix components of both the first and second frequency bands (e.g., mix components throughout the whole band) included in the mix sound or (ii) mix components of the second frequency band (but not the mix components of the first frequency band) included in the mix sound.
The first frequency band and the second frequency band are two different frequency bands on the frequency axis. Typically, the first frequency band and the second frequency band do not overlap. However, the first frequency band and the second frequency band may partially overlap. The relationship between the position on the frequency axis of the first frequency band and the position on the frequency axis of the second frequency band may be freely selected. The bandwidth of the first frequency band and the bandwidth of the second frequency band may or may not be the same.
The first estimated components represented by the first output data may comprise components of the second frequency band only, included in the first sound, or the components of a frequency band including the first and second frequency bands, included in the first sound. Similarly, the second estimated components represented by the second output data may comprise the components of the second frequency band only, included in the second sound, or the components of a frequency band including the first and second frequency bands, included in the second sound.
The trained estimation model is a statistical model that has learned a relationship between input data and output data (first output data and second output data). The estimation model is typically a neural network, but the estimation model type is not limited thereto.
In an example (Aspect 2) of Aspect 1, the mix sound data included in the input data represents the mix components of the input frequency band that does not include the first frequency band, included in the mix sound of the first sound and the second sound. According to the above configuration, since the mix sound data does not include the mix components of the first frequency band, it is possible to reduce both the processing load required for machine learning of the estimation model and the size of the estimation model compared with a configuration in which the sound represented by the mix sound data includes the mix components of the first frequency band and the mix components of the second frequency band.
In an example (Aspect 3) of Aspect 1 or Aspect 2, the first sound data represents intensity spectra of the first components included in the first sound, the second sound data represents intensity spectra of the second components included in the second sound, the mix sound data represents intensity spectra of the mix components included in the mix sound of the first sound and the second sound. Preferably the input data may include: a normalized vector that includes the first sound data, the second sound data, and the mix sound data, and an intensity index representing a magnitude of the vector. According to the above configuration, since the intensity index is included in the input data, the first output data and the second output data representing the components of the loudness corresponding to the mix sound are generated. Therefore, advantageously, the process of adjusting (scaling) the intensity of the sound represented by the first output data and second output data is not necessary.
In an example (Aspect 4) of any one of Aspect 1 to Aspect 3, the estimation model is trained so that a mix of (i) components of the second frequency band included in the first estimated components represented by the first output data and (ii) components of the second frequency band included in the second estimated components represented by the second output data approximates components of the second frequency band included in the mix sound of the first sound and the second sound. According to the above configuration, the estimation model is trained so that a mix of the components of the second frequency band included in the first estimated components represented by the first output data and the components of the second frequency band included in the second estimated components represented by the second output data approximates the components of the second frequency band included in the mix sound. Therefore, compared with a configuration that uses an estimation model trained without taking the above condition into account, it is possible to estimate with higher accuracy the components of the second frequency band in the first sound (first output data) and the components of the second frequency band in the second sound (second output data).
In an example (Aspect 5) of any one of Aspect 1 to Aspect 4, the method further enables the first components and the second components to be generated by performing sound source separation of the mix sound of the first sound and the second sound, regarding the first frequency band. According to the above configuration, the processing load for sound source separation is reduced compared with a configuration in which sound source separation is performed for the whole band of the mix sound, because sound source separation is performed for the components of the first frequency band included in the mix sound.
In an example (Aspect 6) of any one of Aspect 1 to Aspect 5, the first output data represents the first estimated components including (a1) the first components of the first frequency band and (a2) components of the second frequency band, included in the first sound, and the second output data represents the second estimated components including (b1) the second components of the first frequency band and (b2) components of the second frequency band, included in the second sound. According to the above configuration, the first output data and the second output data are generated including components of both the first frequency band and the second frequency band. Therefore, compared with a configuration in which the first output data represents only the second frequency band components of the first sound and the second output data represents only the second frequency band components of the second sound, sound over both the first and second frequency bands can be generated in a simplified manner.
In a method of training an estimation model according to one aspect (Aspect 7) of the present disclosure, a tentative estimation model is prepared, and a plurality of training data, each training data including training input data and corresponding training output data is obtained. A trained estimation model is established that has learned a relationship between the training input data and the training output data by machine learning in which the tentative estimation model is trained using the plurality of training data. The training input data includes first sound data, second sound data, and mix sound data, the first sound data representing first components of a first frequency band, included in a first sound corresponding to a first sound source, the second sound data representing second components of the first frequency band, included in a second sound corresponding to a second sound source that differs from the first sound source, and the mix sound data representing mix components of an input frequency band including a second frequency band that differs from the first frequency band, the mix components being included in a mix sound of the first sound and the second sound. The training output data includes at least one of: first output data representing first output components of an output frequency band including the second frequency band, included in the first sound, or second output data representing second output components of the output frequency band, included in the second sound.
According to the above configuration, a trained estimation model is established that generates at least one of the first output data representing the first estimated components of an output frequency band including the second frequency band, included in the first sound, or the second output data representing the second estimated components of the output frequency band, included in the second sound, from the first sound data representing the first components of the first frequency band, included in the first sound, and the second sound data representing the second components of the first frequency band, included in the second sound. According to the above configuration, in a case in which sound separation is to be performed of a mix sound consisting of a first sound corresponding to a first sound source and a second sound corresponding to a second sound source to isolate the first and the second sounds, it is only necessary to perform sound source separation of the mix sound, regarding the first frequency band. Therefore, the processing load for sound source separation is reduced.
The present disclosure is also realized as an audio processing system that implements the audio processing method according to each of the above examples (Aspect 1 to Aspect 6), or a program that causes a computer to execute the audio processing method. The present disclosure is also realized as an estimation model training system that realizes the training method according to Aspect 7, or a program that causes a computer to execute the training method.
100 . . . audio processing system, 11 . . . controller, 12 . . . storage device, 13 . . . sound outputter, 20 . . . audio processor, 21 . . . frequency analyzer, 22 . . . sound source separator, 23 . . . band extender, 231 . . . obtainer, 232 . . . generator, 24 . . . waveform synthesizer, 25 . . . volume adjuster, 30 . . . learning processor, 31 . . . obtainer, 32 . . . trainer, M . . . estimation model.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-033347 | Feb 2020 | JP | national |
This application is a Continuation application of PCT Application No. PCT/JP2021/006263, filed on Feb. 19, 2021, and is based on and claims priority from Japanese Patent Application No. 2020-033347, filed on Feb. 28, 2020, the entire contents of each of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2021/006263 | Feb 2021 | US |
| Child | 17896671 | US |
