Patent Application
20230135118
The present disclosure relates to an information processing method, an information processing device, and a program.
For example, Patent Literature 1 discloses a technique for generating sequence information for automatic generation of a music program or the like.
Patent Literature 1: JP 2002-207719 A
It is also conceivable to automatically generate music itself. For example, it is conceivable to use a track using a certain musical instrument as an input track and newly generate another track from the input track. In this case, the generated track is desirably a track whose consistency is increased so as to cooperate with the input track. The same applies to generation of various information other than music (for example, generation of a translation or the like).
An object of one aspect of the present disclosure is to provide an information processing method, an information processing device, and a program capable of generating a track having increased consistency with an input track.
An information processing method according to one aspect of the present disclosure includes generating an output track by using an input track including a plurality of first information elements provided over a certain period or a certain section and a learned model, wherein the output track includes a first track that is a same track as the input track or a changed track, and a second track including a plurality of second information elements provided over the certain period or the certain section, and the learned model is a learned model generated by using training data so as to output output data corresponding to the output track when input data corresponding to the first track is input.
An information processing device according to one aspect of the present disclosure includes a generation unit that generates an output track by using an input track including a plurality of first information elements provided over a certain period or a certain section and a learned model, wherein the output track includes a first track that is a same track as the input track or a track in which a part of the input track has been changed, and a second track including a plurality of second information elements provided over the certain period or the certain section, and the learned model is a learned model generated by using training data so as to output output data corresponding to the output track when input data corresponding to the first track is input.
A program according one aspect of the present disclosure causes a computer to function, the program causing the computer to execute generating an output track by using an input track including a plurality of first information elements provided over a certain period or a certain section and a learned model, wherein the output output track includes a first track that is a same track as the input track or a track in which a part of the input track has been changed, and a second track including a plurality of second information elements provided over the certain period or the certain section, and the learned model is a learned model generated by using training data so as to output output data corresponding to the output track when input data corresponding to the first track is input.
Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that in each of the following embodiments, the same parts are denoted by the same reference numerals, and redundant description will be omitted.
The present disclosure will be described according to the following order of items.
Hereinafter, an information processing device that can be used in an information processing method according to an embodiment will be mainly described as an example. The information processing device according to the embodiment is used, for example, as an information generation device that generates various types of information. Examples of the generated information include music, sentences, and the like. Information to be handled is referred to as a “track”. The track includes a plurality of information elements provided over a certain period or a certain section. An example of the information element in a case where the track is music is sound information of a musical instrument. Examples of the sound information include a pitch value of sound, a generation period of sound, and the like. In this case, the track may indicate the sound of the musical instrument at each time during the certain period. Examples of the information elements in a case where the track is a sentence include words, morphemes, and the like (hereinafter, it is simply referred to as a “word and the like”). In this case, the track may indicate a word or the like at each position in the certain section. Hereinafter, unless otherwise specified, a case where the track is music and the information elements is sound information will be described.
In the item “change input track”, the user U selects whether or not to make a change to the input track, and also selects the degree of change (change amount) when making a change. An example of the change amount is a ratio (%) of the sound information to be changed. The change amount may be selected from a plurality of numerical values prepared in advance, or may be directly input by the user U1. Specific contents of the change will be described later.
In the item “musical instrument selection”, the user U selects a musical instrument (second musical instrument) to be used for a newly generated track. The second musical instrument may be automatically selected or specified by the user U. Like the first musical instrument described above, any musical instrument may be the first musical instrument. The type of the second musical instrument may be the same as the type of the first musical instrument.
Note that
An input track is input to the input unit 10. For example, as described above with reference to
The storage unit 20 stores various types of information used in the information processing device 1. Among them,
The generation unit 30 generates an output track by using the input track input to the input unit 10 and the learned model 21. In
The track change unit 31 makes a change to the input track. The input track after the change is one mode of the first track. For example, the track change unit 31 changes a part of a plurality of pieces of sound information (a pitch value of sound, a generation period of sound, and the like of the first musical instrument) included in the input track. This will be described with reference to
The first track illustrated in
The first track illustrated in
The first track illustrated in
With the track change unit 31, a track partially different from the input track is obtained as the first track without being constrained by the input track input to the input unit 10. The degree of constraint (constraint strength) can be adjusted by the ratio of the sound to be changed. The adjustment amount is determined randomly, for example.
Returning to
A token <ON, M, 60> is a token (first token) indicating that the generation of a sound at the pitch value 60 of a musical instrument M starts at time 0. The following token <SHIFT, 1> is a token (second token corresponding) indicating that the state (musical instrument M and pitch value 60) indicated in the first token corresponding is maintained for a period of one unit of time. That is, SHIFT means that only the time moves (only time passes) while the state indicated by the immediately preceding token remains.
A token <ON, M, 64> is a token (first token) indicating that the generation of a sound at the pitch value 64 of the musical instrument M starts. The next token <SHIFT, 1> is a token (second token corresponding) indicating that the state (musical instrument M, pitch value 60, musical instrument M, and pitch value 64) indicated in the first token corresponding is maintained for a period of one unit of time.
A token <ON, M, 67> is a token (first token) indicating that the generation of a sound at the pitch value 67 of the musical instrument M starts. The following token <SHIFT, 2> is a token (second token corresponding) indicating that the state (musical instrument M, pitch value 60, musical instrument M, pitch value 64, musical instrument M, and pitch value 67) indicated in the first token corresponding is maintained for a period of two units of time.
The token <OFF, M, 60> is a token (first token) indicating that the generation of the sound at the pitch value 60 of the musical instrument M ends. The token <OFF, M, 64> is a token (first token) indicating that the generation of the sound at the pitch value 64 of the musical instrument M ends. The token <OFF, M, 67> is a token (first token) indicating that the generation of the sound at the pitch value 67 of the musical instrument M ends. The following token <SHIFT, 1> is a token (second token corresponding) indicating that the state indicated in the first token corresponding (no sound is generated by any musical instrument) is maintained for a period of one unit of time.
A token <ON, M, 65> is a token (first token) indicating that the generation of sound at the pitch value 65 of the musical instrument M starts. The following token <SHIFT, 1> is a token (second token corresponding) indicating that the state (musical instrument M and pitch value 65) indicated in the first token corresponding is maintained for a period of one unit of time.
The token <OFF, M, 65> indicates that the generation of the sound at the pitch value 65 of the musical instrument M ends (first token).
Note that, in the above description, an example has been described in which, when there is a plurality of sounds at the same time, tokens corresponding to low sounds are arranged in order. By determining the order in this manner, the learned model 21 can be easily learned.
Using the token sequence generated as described above (the token sequence illustrated on the lower side in
The first additional token is a token indicating a period that elapses until a time when each token appears in the token sequence. The token generation unit 32 may include (embed), in each token, a token indicating the total of the periods indicated by the second token until a time when each token appears in the token sequence. As described above, since the SHIFT of the second token means that only the time moves while the state indicated by the immediately preceding token remains, the embedding of the first additional token can also be referred to as time shift summarization embedding (TSE).
The second additional token is a token indicating a position of each token in the token sequence. The token generation unit 32 may include (may embed), in each token, a token indicating the position of the each token in the token sequence. The embedding of the second additional token can also be referred to as position embedding (PE).
An example of embedding of the above-described additional token (the first additional token and the second additional token) will be described with reference to
Examples of the first additional token corresponding to each of the above-described basic tokens include a token <SUM, 0>, a token <SUM, 6>, and a token <SUM, 6>. The token <SUM, 0> indicates that the period that elapses until a time when the token <ON, b, 24> appears is zero. The token <SUM, 6> indicates that the period that elapses until a time when the token <SHIFT, 6> and the token <OFF, b, 24> appear is six units of time.
Examples of the second additional token corresponding to each of the above-described basic tokens include a token <POS, 0>, a token <POS, 1>, and a token <POS, 2>. The token <POS, 0> indicates that the token <ON, b, 24> is at the zeroth position in the token sequence. The token <POS, 1> indicates that the token <SHIFT, 6> is at the first position in the token sequence. The token <POS, 2> indicates that the token <OFF, b, 24> is at the second position in the token sequence.
As described above, by including the additional token in addition to the basic token, a lot of information is given to the token sequence. The actual time information corresponding to the basic token can be included in the token sequence, in particular by embedding (TSE) the first additional token. Thus, it is possible to bypass the learning regarding the time in the generation of the learned model 21 and reduce the processing load regarding the learning.
Returning to
The encoder 21a extracts a feature amount from an input token sequence. The decoder 21b generates (reconfigures) an output token sequence from the feature amount extracted by the encoder 21a by using, for example, the token sequence with the highest probability. Learning of the encoder 21a may be performed by unsupervised learning such as variational auto encoder (VAE) or generative adversarial networks (GAN). The parameters of the encoder 21a and the decoder 21b are adjusted by comparing the input token sequence in the encoder 21a with the output token sequence generated by the decoder 21b. By repeating the adjustment, the learned model 21 in which the parameters of the encoder 21a and the decoder 21b are optimized is generated. An example of a generation flow of the learned model 21 will be described later again with reference to
By generating the token sequence corresponding to the track set of the first track and the second track as described above, for example, the token sequence of the second track having increased consistency with a first track set (that is, with the input track) is increased more than in a case where the token sequence corresponding only to the second track is generated. The music generation in consideration of the consistency with such a first track set has high affinity with the human music generation process, and a synergistic effect of creativity is easily exhibited. The music generation process of human with high affinity is, for example, a process of creating tracks one by one or creating music by being inspired by a certain track.
In the embodiment, the decoder 21b of the learned model 21 may generate each token in time order. In this case, in the process of generating the token sequence, the decoder 21b may generate the next token with reference to the generated token (attention function).
For example, as illustrated below the decoder 21b in the drawing, after the start token <START>, a token <ON, b, 24>, a token <ON, m, 60>, a token <SHIFT, 4>, a token <OFF, m, 60>, and a token <SHIFT, 2> are sequentially generated as basic tokens. At that time, the decoder 21b also generates (does not need to output) the additional token described above. In particular, by generating the first additional token, the decoder 21b becomes capable of generating the next token while also referring to the token at the corresponding time in the input token sequence. Consequently, in the output track, the consistency between the new track using the musical instrument m and the first track using the musical instrument b is further improved.
For example, the track generation unit 33 generates the output track by using the token sequence generated by the learned model 21 as described above. Some examples of output tracks will be described with reference to
Returning to
In Step S1, an input track is input. For example, the user U1 selects the input track as described above with reference to
In Step S2, it is determined whether or not to make a change. This determination is performed, for example, on the basis of an input result of the previous Step S1 (selection of whether or not to make a change to the input track, or the like). When a change is made (Step S2: Yes), the processing proceeds to Step S3. Otherwise (Step S2: No), the processing proceeds to Step S4.
In Step S3, a change is made to the input track. For example, the track change unit 31 makes a change to the input track input in the previous Step S1. The specific contents of the change have been described above with reference to
In Step S4, the input token sequence is generated. For example, the token generation unit 32 generates an input token sequence corresponding to the input track that has been input in the previous Step S1 and/or the input track (first track) to which a change has been made in the previous Step S3. The specific content of the generation has been described above with reference to
In Step S5, the output token sequence is acquired using the learned model. For example, the track generation unit 33 acquires the output token sequence corresponding to the output track by inputting the input token sequence generated in the previous Step S4 to the learned model 21. The specific content of the acquisition has been described above with reference to
In Step S6, the output track is generated. For example, the track generation unit 33 generates an output track corresponding to the output token sequence acquired in the previous Step S4.
In Step S7, the output track is output. For example, the output unit 40 outputs the output track generated in Step S6 as described above with reference to
After the processing of Step S7 is completed, the processing of the flowchart ends. For example, by such processing, an output track is generated and output from the input track.
In Step S11, a mini-batch sample set of a track set (corresponding to an output track) is prepared. Each mini-batch sample is configured by combining, for example, part of mini data of a plurality of pieces of music prepared in advance. By collecting a plurality (for example, 256) of such mini-batch samples, a mini-batch sample set is obtained. One mini-batch sample of the mini-batch sample set is used in one flow. Another mini-batch sample is used in another flow.
In Step S12, a change is made to the track. Since the change is as described above, the description thereof will not be repeated. The number of track sets may be increased by the change.
In Step S13, Forward calculation is performed. Specifically, the token sequence corresponding to part of tracks (corresponding to the first track) of the track set prepared in Step S12 described above is input to a neural network including the encoder and the decoder, and the token sequence corresponding to the new track set (corresponding to the first track set and the second track set) is output. An error function is obtained from the output track set and the previously prepared track set.
In Step S14, Backward calculation is performed. Specifically, a cross entropy error is calculated from the error function obtained in Step S13 described above. From the calculated cross entropy error, a parameter error of the neural network and a gradient of the error are obtained.
In Step S15, the parameter is updated. Specifically, the parameters of the neural network are updated according to the error obtained in Step S14 described above.
After the processing of Step S15 is completed, the processing returns to Step S11 again. In Step S11 in that case, a mini-batch sample different from the previously used mini-batch sample is used.
For example, the learned model can be generated as described above. The above is an example, and various known learning methods may be used in addition to the method using the mini-batch sample set as described above.
The CPU 1100 operates on the basis of a program stored in the ROM 1300 or the HDD 1400, and controls each unit. For example, the CPU 1100 develops a program stored in the ROM 1300 or the HDD 1400 in the RAM 1200, and executes processing corresponding to various programs.
The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is activated, a program depending on hardware of the computer 1000, and the like.
The HDD 1400 is a computer-readable recording medium that non-transiently records a program executed by the CPU 1100, data used by such a program, and the like. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the present disclosure as an example of program data 1450.
The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (for example, the Internet). For example, the CPU 1100 receives data from another device or transmits data generated by the CPU 1100 to another device via the communication interface 1500.
The input-output interface 1600 is an interface for connecting an input-output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard and a mouse via the input-output interface 1600. Further, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input-output interface 1600. Furthermore, the input-output interface 1600 may function as a media interface that reads a program or the like recorded in a predetermined recording medium. The medium is, for example, an optical recording medium such as a digital versatile disc (DVD) or a phase change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a tape medium, a magnetic recording medium, a semiconductor memory, or the like.
For example, in a case where the computer 1000 functions as the information processing device 1, the CPU 1100 of the computer 1000 implements the functions of the generation unit 30 and the like by executing the information processing program loaded on the RAM 1200. Further, the HDD 1400 stores a program according to the present disclosure (the program 22 of the storage unit 20) and data in the storage unit 20. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data 1450, but as another example, these programs may be acquired from another device via the external network 1550.
The embodiment of the present disclosure has been described above. The present disclosure is not limited to the above embodiment.
In the above embodiment, an example in which the input track includes one track and the output track includes two tracks has been described. However, the input track may include two or more tracks. The output track may include three or more tracks. An input-output mode (an input screen or the like in
In the above embodiment, the example in which the learned model is a model including an encoder and a decoder such as RNN and Seq2Seq has been described. However, not limited to these models, various learned models capable of reconfiguring a token sequence from the input token sequence may be used.
In the above embodiment, the case where the track is music and the information element is sound information has been described. However, various tracks including information elements other than sound information may be used. For example, the track may be a sentence, and the information element may be a word or the like. In this case, a plurality of first information elements is words and the like of a first language provided over a certain section, and the input track indicates a word or the like of the first language at each position in the certain section. A plurality of second information elements is words and the like of a second language provided over a certain section, and the second track indicates a word or the like of the second language at each position in the certain section. In terms of tokens, the first token indicates the occurrence and stop of each word or the like. The second token indicates a section (for example, a length of a word or the like) in which the state indicated by the first token corresponding is maintained. The token generation unit 32 generates a token sequence by arranging the first token and the second token in position order in the certain section. The first additional token is a token indicating a section that elapses until a time when each token appears in the token sequence. The second token is a token indicating a position of each token in the token sequence.
Part of the functions of the information processing device 1 may be implemented outside the information processing device 1 (for example, an external server). In that case, the information processing device 1 may include part or all of the functions of the storage unit 20 and the generation unit 30 in the external server. By the information processing device 1 communicating with the external server, the processing of the information processing device 1 described above is similarly implemented.
The information processing method described above is specified as follows, for example. As described with reference to
According to the information processing method described above, the track set of the first track and the second track is generated as the output track. Thus, for example, it is possible to generate the second track having increased consistency with the consistency with the first track set (that is, with the input track) is increased more than in a case where only the second track is output and generated. The music generation in consideration of the consistency with such a first track set has high affinity with the human music generation process, and the synergistic effect of creativity is easily exhibited.
In a case where the first track is a track in which a part of the input track has been changed as described with reference to
As described with reference to
As described with reference to
As described with reference to
The information processing device 1 described with reference to
The program 22 described with reference to
Note that the effects described in the present disclosure are merely examples and are not limited to the disclosed contents. There may be other effects.
Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments as it is, and various modifications can be made without departing from the gist of the present disclosure. Furthermore, components of different embodiments and modification examples may be appropriately combined.
Furthermore, the effects in the embodiments described in the present description are merely examples and are not limited, and other effects may be provided.
Note that the present technology can also have the following configurations.
(1)
An information processing method comprising
generating an output track by using an input track including a plurality of first information elements provided over a certain period or a certain section and a learned model, wherein
the output track includes a first track that is a same track as the input track or a changed track, and a second track including a plurality of second information elements provided over the certain period or the certain section, and
the learned model is a learned model generated by using training data so as to output output data corresponding to the output track when input data corresponding to the first track is input.
(2)
The information processing method according to (1), wherein
the first track is a track in which a part of the input track has been changed, and
the information processing method generates the first track by changing a part of the plurality of first information elements included in the input track.
(3)
The information processing method according to (1) or (2), wherein
the input data is an input token sequence corresponding to the first track,
the output data is an output token sequence corresponding to the output track, and
the information processing method acquires the output token sequence by inputting the input token sequence to the learned model.
(4)
The information processing method according to (3), wherein
the input token sequence is generated by arranging a first token indicating generation and stop of each of the plurality of first information elements and a second token indicating a period or section in which a state indicated by the first token corresponding is maintained in time order in the certain period or in position order in the certain section.
(5)
The information processing method according to (4), wherein
the input token sequence is generated by including, in the first token and the second token, an additional token indicating a time or a position when each of the first token and the second token appears in the input token sequence.
(6)
The information processing method according to (5), wherein
the additional token is a token indicating a sum of periods or sections indicated in the second token until a time when each of the first token and the second token appears in the input token sequence.
(7)
The information processing method according to any one of (1) to (6), wherein
the plurality of first information elements is sound information of a first musical instrument provided over the certain period, and the input track indicates sound information of the first musical instrument at each time during the certain period, and
the plurality of second information elements is sound information of a second musical instrument provided over the certain period, and the second track indicates sound information of the second musical instrument at each time during the certain period.
(8)
The information processing method according to (7), wherein
the sound information of the first musical instrument includes at least one of a pitch value of sound and a generation period of sound of the first musical instrument.
(9)
An information processing device comprising
a generation unit that generates an output track by using an input track including a plurality of first information elements provided over a certain period or a certain section and a learned model, wherein
the output track includes a first track that is a same track as the input track or a track in which a part of the input track has been changed, and a second track including a plurality of second information elements provided over the certain period or the certain section, and
the learned model is a learned model generated by using training data so as to output output data corresponding to the output track when input data corresponding to the first track is input.
(10)
A program for causing a computer to function, the program causing the computer to execute
generating an output track by using an input track including a plurality of first information elements provided over a certain period or a certain section and a learned model, wherein
the output track includes a first track that is a same track as the input track or a track in which a part of the input track has been changed, and a second track including a plurality of second information elements provided over the certain period or the certain section, and
the learned model is a learned model generated by using training data so as to output output data corresponding to the output track when input data corresponding to the first track is input.
1 INFORMATION PROCESSING DEVICE
1
a DISPLAY SCREEN
10 INPUT UNIT
20 STORAGE UNIT
21 LEARNED MODEL
21
a ENCODER
21
b DECODER
22 PROGRAM
30 GENERATION UNIT
31 TRACK CHANGE UNIT
32 TOKEN GENERATION UNIT
33 TRACK GENERATION UNIT
40 OUTPUT UNIT
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-081493 | May 2020 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2021/015318 | 4/13/2021 | WO |
