Patent Application
20210366455
The present invention relates to sound source technology for synthesizing sound signals.
There have been proposed sound sources that use neural networks (hereafter, “NNs”) to generate sound waveforms in accordance with input conditions (hereafter, “Deep Neural Network (DNN) sound sources”), such as an NSynth described in U.S. Pat. No. 10,068,557 (hereafter, “Patent Document 1”) or a Neural Parametric Singing Synthesizer (NPSS) described in Merlijn Blaauw, Jordi Bonada, “A Neural Parametric Singing Synthesizer Modeling Timbre and Expression from Natural Songs,” Appl. Sci. 2017, 7, 1313 (hereafter, “Non-Patent Document 1”).
The NSynth generates a sample of a sound signal for each sample cycle in accordance with embedding (embedding vector). The Timbre model of the NPSS generates a spectrum of a sound signal for each frame, depending on pitch and timing information.
There has been proposed a one-hot representation as a form of pitch data representative of pitches. The one-hot representation is a method of representing pitches using n bits (n is a natural number equal to or greater than two) corresponding to different pitches. For example, in a one-hot vector representing a single pitch, one bit corresponding to that pitch is set as “1” among the n bits consisting of the pitch data, and each of the remainder (N−1) bits is set as “0.”
In the NSynth in Patent Document 1, pitch data in one-hot style are input to the WaveNet mode, and a series of samples is then generated in accordance with the pitch data. The NPSS in Non-patent Document 1 inputs pitch data in one-hot style into an F0 model, to generate a trajectory of the pitch F0. A series of spectral envelopes is then generated in accordance with the trajectory of F0. Pitch data in one-hot style, however, have a large number of dimensions, equivalent to a total number of scales in a pitch range of a sound signal to be generated, and thus has a drawback in that a scale of the DNN source is increased.
In sound production systems in nature, sounds in different octaves tend to be generated by a cohesive physical structure, such as that of human vocal organs or sound production mechanisms of wind instruments. Conventional DNN sound sources have been unsuccessful in utilizing common features inherent in sounds in different octaves.
Thus, an object of the present disclosure is to generate high-quality sound signals in a wide range of pitches at a relatively small scale, by utilizing common features of sounds in different octaves.
A sound signal synthesis method according to one aspect of the present disclosure generates control data including pitch notation data indicative of a pitch name of a pitch of a sound signal to be synthesized and octave data indicative of an octave of the pitch of the sound signal to be synthesized; and estimates output data indicative of the sound signal to be synthesized by inputting the generated control data into a generative model that has learned a relationship between (i) training control data including training pitch notation data indicative of a pitch name of a pitch of a reference signal and training octave data indicative of an octave of the pitch of the reference signal and (ii) training output data indicative of the reference signal.
A training method of a generative model according to one aspect of the present disclosure prepares a reference signal with a pitch, pitch notation data indicative of a pitch name of the pitch, and octave data indicative of an octave of the pitch; and trains the generative model such that the generative model can generate output data indicative of the reference signal based on control data including the pitch notation data and the octave data.
A sound signal synthesis system according to one aspect of the present disclosure includes one or more processors; and one or more memories. The one or more memories are configured to store a generative model that has learned a relationship between training control data and training output data. The training control data includes training pitch notation data indicative of a pitch name of a pitch of a reference signal and training octave data indicative of an octave of the pitch of the reference signal, and the training output data indicates the reference signal. The one or more processors are communicatively connected to the one or more memories and configured to: generate control data that include pitch notation data indicative of a pitch name of a pitch of a sound signal to be synthesized and octave data indicative of an octave of the pitch of the sound signal to be synthesized; and estimate output data indicative of the sound signal to be synthesized by inputting the generated control data into the generative model.
A non-transitory computer-readable recording medium according to one aspect of the present disclosure stores a program executable by a computer to perform a sound signal synthesis method of: generating control data including pitch notation data indicative of a pitch name of a pitch of a sound signal to be synthesized and octave data indicative of an octave of the pitch of the sound signal to be synthesized; and estimating output data indicative of the sound signal to be synthesized by inputting the generated control data into a generative model that has learned a relationship between (i) training control data including training pitch notation data indicative of a pitch name of a pitch of a reference signal and training octave data indicative of an octave of the pitch of the reference signal and (ii) training output data indicative of the reference signal.
The control device 11 comprises one or more processors that control each of the elements that constitute the sound signal synthesis system 100. Specifically, the control device 11 is constituted of one or more of different types of processors, such as a Central Processing Unit (CPU), Sound Processing Unit (SPU), Digital Signal Processor (DSP), Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), or the like. The control device 11 generates a time-domain sound signal V that represents a waveform of the synthesis sound.
The storage device 12 comprises one or more memories that store programs executed by the control device 11 and various data used by the control device 11. The storage device 12 comprises a known recording medium, such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. It is of note that a storage device 12 may be provided separate from the sound signal synthesis system 100 (e.g., cloud storage), and the control device 11 may write and read data to and from the storage device 12 via a communication network, such as a mobile communication network or the Internet. In other words, the storage device 12 may be omitted from the sound signal synthesis system 100.
The display device 13 displays calculation results of a program executed by the control device 11. The display device 13 is, for example, a display. The display device 13 may be omitted from the sound signal synthesis system 100.
The input device 14 accepts a user input. The input device 14 is, for example, a touch panel. The input device 14 may be omitted from the sound signal synthesis system 100.
The sound output device 15 plays sound represented by a sound signal V generated by the control device 11. The sound output device 15 is, for example, a speaker or headphones. For convenience, a D/A converter, which converts the sound signal V generated by the control device 11 from digital to analog format, and an amplifier, which amplifies the sound signal V, are not shown. In addition, although
Description will first be given of pitch name and octave data; a generative model that generates output data in accordance with pitch notation data and octave data; and reference signals R used to train the generative model.
A pair of the pitch data and the octave data represent the pitch of a sound signal V. The pitch notation data (hereafter, “PN data”) X1 indicate a name of the pitch (“C,” “C #,” “D,” . . . “A #,” “B”) of a sound signal V from among the 12 sounds of different pitches corresponding to respective 12 chromatic scales within one octave. The octave data (hereafter, “Oct data”) X2 indicate an octave (indicate a number of octaves that exist relative to the reference octave) to which the pitch of the sound signal V belongs among different octaves. As illustrated in
The PN data X1 consist of 12 bits corresponding to different pitch names. Of the 12 bits constituting the PN data X1, one bit corresponding to the pitch name of a sound signal V is set as “1” and the other 11 bits are set as “0.” The Oct data X2 consist of five bits corresponding to different octaves (O1 to O5). Of the five bits constituting the Oct data X2, one bit corresponding to an octave that includes the pitch of the sound signal V is set as “1” and the other four bits are set as “0.” The Oct data X2 of the first embodiment are 5-bit data corresponding to 5 octaves; however, a number of octaves that can be represented by the Oct data X2 may be freely selected. The Oct data X2 representing any of n octaves (n is a natural number equal to or greater than 1) consist of n bits.
The generative model is a statistical model for generating a series of waveform spectra (e.g., mel spectrogram) of a sound signal V in accordance with the control data X, which include the PN data X1 and the Oct data X2. The control data X specify conditions of a sound signal V to be synthesized. The characteristics of the generative model are defined by more than one variable (coefficients, biases, etc.) stored in the storage device 12. The statistical model is a neural network used for estimating waveform spectra. The neural network may be of a regression type, such as WaveNet™, which estimates a probability density distribution of a current sample based on previous samples of the sound signal V. The algorithm may be freely selected. For example, the algorithm may be a Convolutional-Neural-Network (CNN) type, a Recurrent-Neural-Network (RNN) type, or a combination of the two. Furthermore, the algorithm may be of a type that includes an additional element, such as Long Short-Term Memory (LSTM) or ATTENTION. The variables of the generative model are established by training based on training data prepared by the preparation function (described later). The generative model in which the variables are established is used to generate the sound signal V in the generative function (described later).
To train the generative model, there are stored in the storage device 12 multiple pairs of a sound signal (hereafter, “reference signal”) R and score data, the reference signal R being indicative of a time-domain waveform of a score played by a player, and the score data being representative of the score. The score data in one pair include a series of notes. The reference signal R corresponding to the score data in the same pair contains a series of waveform segments corresponding to the series of notes of the score represented by the score data. The reference signal R comprises a series of samples of sample cycles (e.g., at a sample rate of 48 kHz) and is a time-domain signal representative of a sound waveform. The performance of the score may be realized by human instrumental playing, by singing by a singer, or by automated instrumental playing. Generation of a high-quality sound by machine learning generally requires a large volume of training data obtained by advance recording of a large number of sound signals of a target instrument or a target player, etc., for storage in the storage device 12 as reference signals R.
The preparation function illustrated in the upper section of
The time aligner 112 aligns, based on information such as waveform spectra obtained by the analyzer 111, start and end points of each of sound production units in score data for each reference signal R, with start and end points of a waveform segment corresponding to a sound production unit in the reference signal R. A sound production unit comprises, for example, a single note having a specified pitch and a specified sound duration. A single note may be divided into more than one sound production units by dividing the note at a point where waveform characteristics, such as those of tone, change.
The condition generator 113 generates, based on the information of the sound production units of the score data, timings of which are aligned with those in each reference signal R, control data X for each time t in each frame to output the generated control data X to the trainer 114, the control data X corresponding to the waveform segment of the time tin the reference signal R. The control data X specify the conditions of a sound signal V to be synthesized, as described above. The control data X include PN data X1, Oct data X2, start-stop data X3, and context data X4, as illustrated in
As a result of the processing by the analyzer 111 and the condition generator 113, pieces of sound production unit data for training a generative model that generates a sound signal V in a predetermined pitch range are prepared from pairs of a reference signal R and score data. Each piece of sound production unit data comprises a pair of control data X generated by the condition generator 113 and a waveform spectrum generated by the analyzer 111. The pieces of sound production unit data are divided, prior to training by the trainer 114, into a training dataset for training the generative model and a test dataset for testing the generative model. A majority of the sound production unit data are used as a training dataset with the remainder being used as a test dataset. Training with the training dataset is performed by dividing the pieces of sound production unit data into batches, with each batch consisting of a predetermined number of frames, and the training is performed on a per-batch-basis in order for all the batches.
As illustrated in the upper section of
When the preparation process is started, the control device 11 (analyzer 111) generates a waveform spectrum for each waveform segment from each of the reference signals R (Sa1). Next, the control device 11 (time aligner 112 and condition generator 113) generates, from score data that correspond to the waveform segment, control data X including the PN data X1 and the Oct data X2 of a sound production unit that corresponds to the waveform segment (Sa2). The control device 11 (trainer 114) trains a generative model using the control data X for each sound production unit and the waveform spectrum corresponding to the sound production unit, and establishes the variables of the generative model (Sa3).
In the embodiment described above, a configuration in which the pitch is represented by a set of the PN data X1 and the Oct data X2 is given as an example. However, there is also assumed a configuration in which pitch data in one-hot style representative of any of a plurality of pitches over a plurality of octaves (i.e., the pitches amount to a product of 12 chromatic scales and n octaves) are used (hereafter, “comparative example”). In contrast to the comparative example, in the first embodiment, the generative model is trained using as input the control data X including the PN data X1 and the Oct data X2. Therefore, the established generative model is a model that takes advantage of innate commonality of sounds in different octaves. This generative model is able to acquire an ability to generate sound signals V at a smaller scale than that required for a normal generative model trained using the pitch data of the comparative example, and yet attain a quality equivalent to that of the normal generative model. Alternatively, at the same scale as that required for the normal generative model, the generative model of the first embodiment is able to acquire an ability to generate sound signals V of a higher quality than those generated using the normal generative model. Furthermore, in the generative model of the first embodiment, even if training using the reference signal R is not performed for a pitch in a certain octave during training, specifying the pitch using the PN data X1 and the Oct data X2 at the time of generation increases a possibility of a sound signal V representative of the pitch being generated.
Description is next given of a generative function illustrated in the lower section of
The generator 122 generates a series of waveform spectra in accordance with the control data X′ by use of a generative model in which the variables are established, as illustrated in the lower section of
The synthesizer 123 receives a series of the waveform spectra in the frequency domain and synthesizes a sound signal V in the time domain in accordance with the series of the waveform spectra. The synthesizer 123 is a so-called vocoder. For example, the synthesizer 123 synthesizes the sound signal V by obtaining a minimum phase spectrum from a waveform spectrum and then performing an inverse Fourier transform on the waveform spectrum and the phase spectrum. Alternatively, a neural vocoder that has learned the relationship that potentially exists between the waveform spectra and sound signals V is used to directly synthesize the sound signal V from the waveform spectrum.
When the sound generation process for a certain sound production unit is started, the control device 11 (generation controller 121) generates control data X′ for that sound production unit based on the score data (Sb1). The control device 11 (generator 122) subsequently generates a waveform spectrum of the sound signal V of that sound production unit in accordance with the generated control data X′ by use of the generative model (Sb2). Then, the control device 11 (synthesizer 123) synthesizes the sound signal V of that sound production unit in accordance with the generated waveform spectrum (Sb3). The above process is sequentially performed for the sound production units of the score data, whereby a sound signal V corresponding to the score data is generated. It is of note that in a case that the sound signals V of two consecutive sound production units overlap, the signals are mixed together to calculate a sound signal V.
In the first embodiment, the pitch of a sound signal V to be synthesized is specified by the PN data X1′ and the Oct data X2′ contained in the control data X′. Consequently, it is possible to generate a high-quality sound signal V in accordance with the control data X′ by use of a generative model that is trained and established taking advantage of innate commonality of sounds in different octaves.
The generator 122 in the first embodiment generates a waveform spectrum. In contrast, in the second embodiment, the generator 122 generates a sound signal V by use of a generative model. The functional configuration of the second embodiment is basically the same as that shown in
The trainer 114 of the second embodiment receives the training dataset and trains the generative model by using in order: the control data X; and the waveform segments of the sound production units of each batch of the training dataset. The generative model estimates output data representative of a sample of the sound signal V at each sample cycle (time t). The trainer 114 calculates a loss function L (cumulative value for one batch) based on a series of the output data estimated from the control data X and the corresponding waveform segments of the training dataset, and optimizes the variables of the generative model so that the loss function L is minimized. The generative model thus established has learned relationships that potentially exist between the control data X in each of the pieces of sound production unit data and the waveform segments of the reference signal R.
The generator 122 of the second embodiment generates a sound signal V in accordance with control data X′ by use of the established generative model. Thus, the generator 122 estimates, at each sample cycle (time t), output data indicative of a sample of the sound signal V in accordance with the control data X′. In a case that the output data represent a probability density distribution for each of a plurality of samples, the generator 122 generates a random number that follows a probability density distribution of the component and outputs the random number as a sample of the sound signal V. In a case that the output data represent the value of a sample, the sample is output.
The first embodiment shown in
The PN data X1 and the Oct data X2 in the above embodiments are in the form of one-hot. However, they may be expressed in other formats. For example, either one or both of the PN data X1 and the Oct data X2 may be expressed in coarse representations.
The PN data X1 and the Oct data X2 in the above embodiments are described as having a fixed number of dimensions. However, any number of dimensions may be used. For example, PN data X1 that represent any of numerical values assigned to different pitches may be used, with a number of dimensions of the PN data X1 being smaller than 12 dimensions. The PN data X1 may be used to represent an intermediate pitch between two pitch names, with the number of dimensions of the PN data X1 being larger than 12 dimensions. An extra dimension may also be added to the Oct data X2. The number of dimensions of the Oct data X2 may be changed depending on the octave range of an instrument for which the sound signal V represents the played sound, or the number of dimensions of the Oct data X2 may be fixed to the number of dimensions required to represent the pitch of an instrument with the largest pitch range (compass) from among multiple types of instruments.
A sound signal V to be synthesized by the sound signal synthesis system 100 is not limited to instrumental sounds or voices. The present disclosure may be applied to dynamically control pitches even if a sound signal V to be synthesized is a vocalized animal sound or a natural sound such as that of wind in air or a wave in water.
The sound signal synthesis system 100 according to the embodiments described above are realized by coordination between a computer (specifically, the control device 11) and a computer program as described in the embodiments. The computer program according to each of the embodiments described above may be provided in a form readable by a computer and stored in a recording medium, and installed in the computer. The recording medium is, for example, a non-transitory recording medium. While an optical recording medium (an optical disk) such as a CD-ROM (Compact disk read-only memory) is a preferred example of a recording medium, the recording medium may also include a recording medium of any known form, such as a semiconductor recording medium or a magnetic recording medium. The non-transitory recording medium includes any recording medium except for a transitory, propagating signal, and does not exclude a volatile recording medium. The computer program may be provided to a computer in a form of distribution via a communication network. The subject that executes the computer program is not limited to a CPU and a processor for a neural network, such as a tensor processing unit and a neural engine, or a DSP (Digital Signal Processor) for signal processing may execute the computer program. Plural types of subjects selected from the above examples may cooperate to execute the computer program.
100 . . . sound signal synthesis system, 11 . . . controller, 12 . . . storage device, 13 . . . display device, 14 . . . input device, 15 . . . sound output device, 111 . . . analyzer, 112 . . . time aligner, 113 . . . condition generator, 114 . . . trainer, 121 . . . generation controller, 122 . . . generator, 123 . . . synthesizer.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2019-028683 | Feb 2019 | JP | national |
This application is a Continuation Application of PCT Application No. PCT/JP2020/006161, filed Feb. 18, 2020, and is based on and claims priority from Japanese Patent Application No. 2019-028683, filed Feb. 20, 2019, the entire contents of each of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2020/006161 | Feb 2020 | US |
| Child | 17398092 | US |
