Patent Application
20230298547
The present disclosure relates to an information processing method, an information processing program, and an information processing device.
With the advancement of artificial intelligence (AI), utilization of computers in the field of art has been advanced.
For example, a technique is known in which machine learning is performed on existing music as learning data to generate a learning model for music generation, and a computer is used to compose new music (for example, Patent Literature 1). In such a technology, it is possible to imitate features of existing music or generate a more natural melody by using a Markov model.
Patent Literature 1: U.S. Pat. No. 9,110,817
According to the conventional technique, since the music data proposed (generated) by AI can be used in composition work, the user can compose music on the basis of more various viewpoints.
In the automatic composition function using AI, the user can receive the automatically created music information by setting an image such as bright or dark, but there is a demand for easily editing the music information and receiving provision of the music information matching the user's preference.
Accordingly, the present disclosure proposes an information processing method, an information processing program, and an information processing device capable of improving convenience of an automatic composition function by AI.
To solve the problems described above, an information processing method of an embodiment according to the present disclosure, the method including:
Hereinafter, embodiments of an information processing method, an information processing program, and an information processing device disclosed in the present application will be described in detail with reference to the drawings. Note that the present invention is not limited by this embodiment.
The present disclosure will be described according to the following order of items.
An example of information processing according to an embodiment will be described with reference to
In the present embodiment, a case where the information processing device 100 is an information processing device that provides a service related to creation of content as a work (also simply referred to as a “service”) will be described as an example. Note that, in the following, music (music content) will be described as an example of content, but the content is not limited to music, and may be various types of content such as video content of movies and the like and text content of books and the like (novels and the like). Furthermore, the music referred to herein is not limited to one piece of completed music (whole music), and is a concept including various music information such as a part of sound sources constituting one music (music) and a short sound used for sampling.
The information processing device 100 communicates with the producer terminal 200 of the user who uses the service provided by the information processing device 100 by using a network N (see
The information processing device 100 is a server device that executes information processing related to an automatic composition function by AI of the producer terminal 200. For example, the information processing device 100 is what is called a cloud server, executes automatic composition by AI according to information instructed from the producer terminal 200 via the network N, and provides the producer terminal 200 with generated music information.
The producer terminal 200 is an information processing terminal such as a smartphone, a personal computer (PC), or a tablet terminal. Various program applications are installed in the producer terminal 200. An application related to creation of music is installed in the producer terminal 200. For example, the producer terminal 200 has an automatic composition function by AI added by a plug-in (extension app) to an app such as a digital audio workstation (DAW) that achieves a comprehensive music production environment. For example, the plug-in can take the form of Steinberg's Virtual Studio Technology (VST) (registered trademark), AudioUnits, Avid Audio eXtension (AAX), or the like. Furthermore, the producer terminal 200 is not limited to the DAW, and for example, a mobile app such as iOS may be used.
A flow of information processing will be described with reference to
The producer terminal 200 activates the DAW and displays the interface provided from the information processing device 100 using a plug-in or the like. A producer C1 operates the producer terminal 200 to generate the style information and provides the style information to the information processing device 100 (step SS12).
Here, the user of the producer terminal 200 will be described as the producer C1, but the present invention is not limited thereto. For example, the user of the producer terminal 200 is any of an administrator who operates and manages the entire system, a composer who creates music, an arranger, a producer such as a studio engineer, and a general user who receives music information provided via the automatic composition function.
The information processing device 100 collects the style information from the producer terminal 200 (another terminal device not illustrated), and executes the machine learning of the composition model using the plurality of pieces of collected style information (step S13). For example, the composition model corresponds to a Markov model or the like.
The information processing device 100 generates new music information using the composition model and provides the music information to the producer terminal 200 (step S14).
Next, an example of the style information will be described. The style information includes melody information, chord information, base information, rhythm information, and the like.
The style information ID 710 is information for identifying the style information 700. The style type ID 711 is information for identifying the type to which the style information 700 belongs. The disclosure permission information 712 is information of whether or not to permit the corresponding style information 700 to be disclosed to a general user. The disclosure permission information 712 may be binary information.
The style palette sequence information 720 is information indicating the order of the style palette information 730. The style palette sequence information 720 includes a style palette ID uniquely indicating the style palette information 730 and a bar index as a set, and is information for managing the order of the style palette information 730 in music by including a plurality of the sets. For example, in the example illustrated in
The style palette information 730 is information in which a score ID of the score information 740 for the same bar is registered in a style palette ID that is identification information of the style palette information.
In the style palette information 730, score IDs of a plurality of pieces of score information 740 having similar chord progression of chord information may be bundled. Similar chord progressions are, for example, identical chord progressions. Alternatively, in the similar chord progression, each chord may be classified into Tonic (T), Sub-dominat (S), and Dominat (D), and sequences of T, S, and D may be the same. Note that in a case of C major A minor, T is C/Em/Am, S is F, Dm, and D is G, Dm7-5. Since both the chord progressions C-D-G-C and Em-Dm-Bm7-5 Am are T-S-D-T, they can be considered as the same chord progression. Furthermore, similar chord progressions can be classified on the basis of similar chord progressions by machine learning or deep learning, for example, instead of using music theory.
The score information 740 includes feature amounts of a plurality of types (elements) extracted from music. The score information 740 includes a score ID, melody information, chord progression information, base information, and rhythm information. The score ID is identification information of the score information. The melody information is a melody in a bar having a prescribed length. The chord progression information is information indicating a chord progression in a bar having a prescribed length. The base information is information indicating the progression of a base sound in a bar having a specified length. The rhythm information is information indicating the progression of a drum sound (the pattern or tempo of the drum) in a bar having a prescribed length.
The style palette meta information 750 includes key signature/scale information, tempo information, and bar number information. The key signature/scale information is information for identifying the key signature of the music corresponding to the style information 700, and whether it is a major key (C major, D major, A major, or the like) or a short key (A minor, E minor, B minor, or the like) is set. The tempo information is information indicating the tempo of music corresponding to the style information 700, and is set by Beats Per Minute (BPM) or the like. The bar number information indicates the number of bars corresponding to the style information 700. For example, four bars, eight bars, and the like are set as the bar number information.
The accompaniment audio information 760 includes a plurality of pieces of MP3 (MPEG-1 Audio Layer-3) information. The MP3 information is information of music uniquely recorded by the producer C1 in association with the melody information, the chord progression information, the base information, and the rhythm information set in the score information 740.
The information processing device 100 executes machine learning using the style information 700 described with reference to
For example, it is assumed that the information processing device 100 provides the style information serving as the learning data from the producer terminal 200. The information processing device 100 presents an interface for creating the style information to the producer terminal 200. The producer C1 generates new style information using the interface displayed on the producer terminal 200, and provides the information processing device 100 with the new style information.
Specifically, a process of creating music by a producer will be described.
The window 270 in
The window 271 in
In the rhythm information setting section 271a, four consecutive input buttons arranged side by side indicate the rhythm information of one bar. One input button corresponds to one beat. The highlighted input button is activated, and the non-highlighted input button is deactivated. Active and inactive information and a value of velocity for each beat corresponding to HH, SD, and BD are feature amounts of the rhythm information. The feature amounts of the rhythm information are associated with bar indexes for each bar. The producer C1 operates the producer terminal 200 to input rhythm information. The producer terminal 200 stores the input rhythm information as the rhythm information of the score information 740. Following the window 271 illustrated in
The window 272 in
The chords include C, Dm, Em, F, G, Am, Bm, and the like, and can be arranged for each bar. For example, in a case of four-bar style information, the chord progression information is |Em|Am|F|G|, |C|Dm|Em|F|, or the like. The chord can also be changed for each beat in the bar.
For example, when the producer C1 operates the producer terminal 200 to press the selection button 272a-21 of “Em” in a state where the input region 272a-11 is selected, the chord “Em” is set in the input region 272a-11. Thus, the producer C1 can easily select by a selection button having a basic chord, such as C, Dm, Em, F, G, Am, or Bm, but can also input a complex chord, such as CM7, C7, or Csus4.
The order of the chords included in the region 272a-1 is the feature amount of the chord progression information. The feature amount of the chord progression information is associated with the bar index for each bar. The producer terminal 200 stores the input chord progression information as the chord progression information of the score information 740. Following the window 272 illustrated in
The window 273 in
Note that the producer C1 can set a note different from the chord name as the base note. For example, for the C chord, a base note of F can be specified, for the Dm chord, a base note of B can be specified, and for the Em chord, a base note of F can be specified. For example, when the chord is different from the base note, the chord is expressed by a fraction, the denominator is expressed by the base note, and the numerator is expressed by the chord name. That is, when the base note of F is designated for the chord of C, the notation is “C/F”.
When the producer C1 operates the producer terminal 200 to press the selection button 273a-21 of “F” in a state where the input region 273a-11 is selected, the base note “F” is set in the input region 273a-11. In a case where the chord name C has already been set for the same input region, “C/F” is set. The order of the base notes included in the region 273a-1 is the feature amount of the base information. The feature amount of the base information is associated with the bar index for each bar. The producer terminal 200 stores the input base information as the base information of the score information 740. Following the window 273 illustrated in
The window 274 in
The producer C1 operates the producer terminal 200 while listening to the accompaniment, and presses the image of the keyboard of the piano of the melody information setting section 274a, thereby inputting (playing) the melody information corresponding to the pressed time and the pressed position of the keyboard. The input melody information is displayed in a partial region 274b. In the partial region 274b, the range of one bar is a range 274b-1. The melody information is displayed in a viewable manner by a bar, a position of the bar corresponds to a scale, and a length of the bar indicates a length of sound. The scale and the length of sound in each bar are feature amounts of the melody information. The feature amount of the melody information is associated with the bar index for each bar.
The producer terminal 200 stores the input melody information as the melody information of the score information 740. Note that the producer C1 can input the melody information again without registering the melody information in the score information 740 by making the retry request.
Here, in the window 274 of
Hereinafter, an example of processing in which the producer terminal 200 generates the style information 700 will be described. The producer terminal 200 sets a set of the chord progression information, the base information, and the rhythm information generated in
In a case where a plurality of pieces of score information 740 is generated, the producer terminal 200 bundles similar pieces of chord progression information into one piece of style palette information 730 on the basis of the chord progression information of the plurality of pieces of score information 740, and adds a unique style palette ID thereto.
The producer terminal 200 refers to a score information 470 corresponding to the score ID of the style palette information, specifies the bar index corresponding to the melody information, the chord progression information, the base information, and the rhythm information, specifies the relationship between the bar index and the style palette ID, and generates the style palette sequence information 720. The producer C1 may appropriately specify the relationship between the bar index and the style palette ID.
The producer terminal 200 sets the key signature/scale information, the tempo information, and the bar number information input in
In a case of generating the style information 700, the producer terminal 200 adds the unique style information ID 710. The producer terminal 200 sets the style type ID 711 and the disclosure permission information 712 according to the input from the producer C1.
As described above, the information processing device 100 provides the producer terminal 200 with an interface capable of generating music information by fixing a part of elements among a plurality of elements forming the music information and editing other elements. The information processing device 100 generates new music information on the basis of a plurality of pieces of music information created using the interface. For example, the music information corresponds to the score information 740 of the style information 700. The fixed elements are the chord progression information, the base information, and the rhythm information. The other elements are melody information, and when a plurality of pieces of melody information is generated, a plurality of pieces of score information 740 corresponding to the melody information is generated.
Consequently, the producer C1 can easily create the style information, and the information processing device 100 can easily receive provision of the style information. The information processing device 100 executes machine learning of the composition model by using the plurality of pieces of provided style information.
Subsequently, when the above-described processing ends, the producer C1 can perform composition processing by AI by executing data communication with the information processing device 100.
The window 280 in
The window 281 in
The producer C1 can change the chord progression set in the region 281a-1 to his/her preference. For example, the producer C1 can change the chord progression of |C|Dm|Em|F| to the chord progression of |C|Dm7|Em7|FM7| by operating the selection button in the region 281a-2 and the region 281a-1. The producer terminal 200 notifies the information processing device 100 of information of the changed chord progression.
In a case where the chord progression is changed as described above, the information processing device 100 can instantaneously create new style information. For example, in a case where the chord progression information (|C|Dm|Em|F|) is included in style information 700A selected in the window 280 in
The window 282 in
The producer C1 can change the base note set in the region 282a-1 to his/her preference. For example, the producer C1 changes the base information by operating the selection button of the region 282a-2 and the region 282a-1. When the base information is changed, new style information is generated and added to the style information list 280a as in the above-described case where the chord progression information is changed. When the change of the base information is completed in the window 282, the information processing device 100 is notified of the changed style information.
The information processing device 100 executes machine learning of the composition model on the basis of the updated style information. Upon receiving a composition request for the updated style information from the producer terminal 200, the information processing device 100 automatically creates new melody information of the updated style information using the composition model. The information processing device 100 notifies the producer terminal 200 of the new melody information and displays the new melody information in the window. For example, the information processing device 100 displays new melody information on the melody information display section 281b or the like included in the window 281 in
Here, new melody information generated by the composition model of the information processing device 100 can be arranged. The producer C1 operates the producer terminal 200 to designate a bar to be arranged, and makes an arrangement request to the information processing device 100. The bar to be arranged is referred to as an arrangement target bar. Referring to
The information processing device 100 arranges the melody information of the arrangement target bar among all bars of the melody information. For example, the information processing device 100 may have an arrangement table in which parameters and arrangement rules learned in advance are associated with each other, and may execute an arrangement according to a designated parameter when the parameter is designated by the producer C1. The information processing device 100 stores the score information 740 included in the arranged melody information as a history.
The above processing can be repeatedly executed, and the style information of the producer C1 is accumulated in the score information 740 preferred by the producer C1, so that the style information is optimized to the preference of the producer C1.
The information processing device 100 provides an interface for generating the style information to the producer terminal 200, and acquires the style information from the producer terminal 200. The information processing device 100 generates new music information on the basis of the acquired style information and provides the music information to the producer terminal 200.
The producer terminal 200 receives provision of the interface from the information processing device 100, and displays the window described in
The producer terminal 200 receives provision of the interface from the information processing device 100, and displays the style information list illustrated in
The service server 300 provides services related to various social networking services (SNSs). For example, the service server 300 receives provision of the score information 740 included in the style information from the information processing device 100, and generates MP3 on the basis of the score information 740 and Musical Instrument Digital Interface (MIDI) information. The service server 300 posts the generated MP3 to various SNS sites, and collects the evaluation information from the user of the SNS site.
Next, a configuration of the information processing device 100 illustrated in
The communication unit 110 is implemented by, for example, a network interface card (NIC) or the like. The communication unit 110 is connected to the network N in a wired or wireless manner, and transmits and receives information to and from the producer terminal 200 and the service server 300 via the network N.
The storage unit 120 is implemented by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 stores various data used for information processing. The storage unit 120 includes user information 121, a style information table 122, a style information management table 123, a produced music information table 124, and composition model data 125.
The user information 121 stores various types of information (user information) regarding the user.
The user information 121 includes a user ID, user meta information, and authority information. The user information 121 stores the user meta information and the authority information corresponding to each user ID in association with each user ID.
The user ID indicates identification information for uniquely specifying the user. For example, the user ID indicates identification information for uniquely specifying a user such as a producer, a general user, or a system administrator. The user meta information is, for example, additional information of the user such as a name and an address of the user. As the authority information, for example, values for identifying authority such as system administrator authority information, producer authority information, and general user authority information are stored. Note that the user information 121 is not limited to the above, and various types of information may be stored according to the purpose. Various pieces of information related to the user may be stored in the user meta information. For example, in a case where the user is a natural person, demographic attribute information such as gender and age of the user, psychographic attribute information, and the like may be stored in the user meta information.
The style information table 122 holds a plurality of pieces of style information provided from the producer terminal 200.
The style information management table 123 holds information that associates style information with a producer who created the style information.
The produced music information table 124 holds information that associates score information with a producer who produced the score information.
The composition model data 125 is data of a composition model generated by executing the machine learning using a plurality of pieces of style information included in the style information table 122 as inputs. For example, in such a composition model, the relationship between the time-series feature amounts of the chord progression information, the base information, and the rhythm information and the time-series feature amount of the melody information is learned, and thus, when the chord progression information, the base information, and the rhythm information are input to the composition model, the melody information estimated from the input chord progression information, the base information, and the rhythm information is predicted, thereby automatically generating the music information. The composition model corresponds to a Markov model or the like.
In the embodiment, a case where the melody information is predicted using the composition model will be described, but the present invention is not limited thereto, and composition model data 125 may be generated by executing the machine learning for predicting the chord progression information, the base information, and the rhythm information.
The description returns to
The control unit 130 includes an interface control unit 131, a machine learning unit 132, a composition unit 133, an arrangement unit 134, and a service server cooperation unit 135.
The interface control unit 131 is a processing unit that provides an interface capable of generating music information such as the style information to the producer terminal 200. First, processing of the interface control unit 131 regarding the windows 270 to 274 described with reference to
The interface control unit 131 transmits the information of the windows 270 to 274 described with reference to
The interface control unit 131 registers the style information received from the producer terminal 200 in the style information table 122. The interface control unit 131 registers the relationship between the style information ID included in the style information and the user ID in the style information management table 123.
The style information received by the interface control unit 131 from the producer terminal 200 includes a plurality of pieces of melody information with respect to fixed chord progression information, base information, and rhythm information.
In a case where any of the plurality of pieces of score information included in the style information is designated, the interface control unit 131 registers the score ID of the designated score information and the user ID in the produced music information table 124 in association with each other. For example, the producer operates the producer terminal 200 to specify the favorite score information.
Next, processing of the interface control unit 131 regarding the windows 280 to 282 described with reference to
When the producer selects one of the names of the style information displayed in the style information list 280a of the window 280, the producer terminal 200 notifies the interface control unit 131 of the style information ID corresponding to the selected name.
Upon receiving the style information ID, the interface control unit 131 acquires the style information corresponding to the style information ID from the style information table 122, and transmits the acquired style information to the producer terminal 200. By executing such processing, the interface control unit 131 sets the chord progression information of the style information in the region 281a-1 of the window 281 in
In a case where the chord progression information is changed in the window 281 in
Further, by the interface control unit 131 transmitting the style information to the producer terminal 200, the base information of the style information is set in the region 282a-2 of the window 282 in
Here, in a case of receiving a request for composition processing based on the changed style information from the producer terminal 200, the interface control unit 131 outputs the changed style information to the composition unit 153. The interface control unit 131 acquires the melody information corresponding to the changed style information from the composition unit 153, and transmits the acquired melody information to the producer terminal 200. Consequently, the new melody information is displayed on the melody information display section 281b or the like included in the window 281 in
In a case of receiving the designation of the parameter and the arrangement target bar, the interface control unit 131 outputs the melody information and the parameter and the information of the arrangement target bar to the arrangement unit 154. The interface control unit 131 acquires the arranged melody information from the arrangement unit 154, and transmits the acquired melody information to the producer terminal 200. In a case of receiving a request to add style information including the arranged melody information from the producer terminal 200, the interface control unit 131 registers the style information including the arranged melody information in the style information table 122.
By the interface control unit 131 executing the above-described processing, the new style information is accumulated in the style information table 122.
The machine learning unit 152 generates the composition model data 125 on the basis of the style information registered in the style information table 122. For example, the machine learning unit 152 generates the composition model data 125 by performing machine learning on the relationship between the time-series feature amounts of the chord progression information, the base information, and the rhythm information and the time-series feature amount of the melody information. The machine learning unit 152 may generate the composition model data 125 by using a music generation algorithm using a Markov chain, or may generate the composition model data 125 by using a music generation algorithm using deep learning. The machine learning unit 152 may acquire new style information from the interface control unit 131 and relearn the composition model.
The composition unit 133 generates melody information on the basis of the style information and the composition model data 125. For example, upon acquiring the changed style information from the interface control unit 131, the composition unit 133 inputs the time-series feature amounts of the chord progression information, the base information, and the rhythm information included in the style information to the composition model, and predicts (composes) the time-series feature amount of the melody information. The composition unit 133 outputs the predicted melody information (new music information) to the interface control unit 131.
Upon receiving the melody information, the parameter, and the information of the arrangement target bar, the arrangement unit 134 arranges the time-series feature amount in the arrangement target bar of the melody information. For example, the arrangement unit 134 has an arrangement table in which parameters and arrangement rules learned in advance are associated with each other, and executes an arrangement according to a parameter designated by the producer. The arrangement unit 134 outputs the arranged melody information to the interface control unit 131.
The service server cooperation unit 135 notifies the service server 300 of the style information and provides services related to various SNSs. The service server cooperation unit 135 provides the score information 740 included in the style information of the style information table 122 to the service server 300. The service server cooperation unit 135 receives, from the service server 300, the evaluation information regarding the score information 470 from the service server 300, and stores the received evaluation information in association with the style information. The interface control unit 131 may display the evaluation information in association with the name of the style information displayed in the style information list 280a of the window 280.
Next, a configuration of the producer terminal 200 illustrated in
The communication unit 210 is implemented by, for example, an NIC, a communication circuit, or the like. The communication unit 210 is connected to the network N in a wired or wireless manner, and transmits and receives information to and from other devices such as the information processing device 100 and other terminal devices via the network N.
Various operations are input to the input unit 220 from the user. The input unit 220 includes a keyboard and a mouse connected to a producer terminal 2000. The input unit 220 receives an input from the user. The input unit 220 receives the user's input using the keyboard or the mouse. The input unit 220 may have a function of detecting a voice. In this case, the input unit 220 may include a microphone that detects a voice.
Various types of information may be input to the input unit 220 via the display unit 260. In this case, the input unit 220 may have a touch panel capable of achieving functions equivalent to those of a keyboard and a mouse. In this case, the input unit 220 receives various operations from the user via the display screen by a function of a touch panel implemented by various sensors. Note that, as a method of detecting the user's operation by the input unit 220, a capacitance method is mainly employed in the tablet terminal, but any method may be employed as long as the user's operation can be detected and the function of the touch panel can be implemented, such as a resistive film method, a surface acoustic wave method, an infrared method, and an electromagnetic induction method, which are other detection methods. Furthermore, the producer terminal 200 may have an input unit that also receives an operation by a button or the like.
Furthermore, the producer may use input unit 220 to input the MP3 information to be set in the accompaniment audio information 760.
The output unit 230 outputs various types of information. The output unit 230 includes a speaker that outputs sound.
The storage unit 240 is implemented by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 240 stores various types of information used for displaying information. The storage unit 240 stores operation history information 241.
The operation history information 241 is information indicating a history of operations performed on the producer terminal 200 by the producer who creates the music when the application is activated. For example, the operation history information 241 may include various types of information regarding an operation of the producer, such as content of an operation performed by the producer and the date and time when the operation is performed. Examples of the operation include a selection of style information presented from the information processing device 100, a selection of a composition execution instruction button, and reproduction, editing, and production of music information received from the information processing device 100, and the like.
The control unit 250 is implemented by, for example, a CPU, an MPU, or the like executing a program stored inside the producer terminal 200 using a RAM or the like as a work area. Furthermore, the control unit 250 is a controller, and may be implemented by, for example, an integrated circuit such as an ASIC or an FPGA. The control unit 250 includes a display control unit 251, a transmission-reception unit 252, a request unit 253, and a reproduction unit 254.
The display control unit 251 controls various displays on the display unit 260. The display control unit 251 controls display on the display unit 260. The display control unit 251 controls display on the display unit 260 on the basis of the information received from the information processing device 100. The display control unit 251 controls display on the display unit 260 on the basis of information generated by processing by each component of the control unit 250. The display control unit 251 may control the display of the display unit 260 by an application that displays an image.
The display control unit 251 causes the display unit 260 to display the windows 270 to 274, and 280 to 283 (see
The transmission-reception unit 252 communicates with the information processing device 100, and transmits and receives various types of information. When the automatic composition function is activated, the transmission-reception unit 252 receives the style information transmitted from the information processing device 100. The transmission-reception unit 252 transmits the style information ID selected by the producer to the information processing device 100. Then, the transmission-reception unit 252 receives the style information from the information processing device 100. Furthermore, the transmission-reception unit 252 transmits music information (style information) of a melody arranged and produced by the producer, or the like to the information processing device 100.
The request unit 253 makes a composition request and an arrangement request to the information processing device 100. For example, when new style information is generated in the window 282 and an instruction of a composition request is received from the producer, information of the composition request including the chord progression information, the base information, and the rhythm information is transmitted to the information processing device 100. Consequently, the information processing device 100 generates new melody information and transmits the new melody information to the producer terminal 200.
Furthermore, upon receiving an instruction of an arrangement target bar and a parameter from the producer, the request unit 253 transmits an arrangement request including the melody information, the arrangement target bar, and the parameter to the information processing device 100. Consequently, the information processing device 100 arranges the arrangement target bar of the melody information and transmits the arranged new melody information to the producer terminal 200.
The reproduction unit 254 reproduces the music information generated by the information processing device 100. Specifically, the reproduction unit 254 sets arbitrary instrument information for each of the melody, the chord, and the base sound included in the music data, and reproduces each piece of data. Note that the reproduction unit 254 may reproduce a combination of each of a melody, a chord, and a base sound.
Further, in a case where the producer performs performance together with composition provided by the automatic composition function, the control unit 250 receives the performance by the producer. Furthermore, the control unit 250 also receives processing related to arrangement of composition provided by the automatic composition function and production of music by the producer.
The display unit 260 displays various types of information. The display unit 260 is implemented by, for example, a liquid crystal display, an organic electro-luminescence (EL) display, or the like. The display unit 260 displays various types of information under control of the display control unit 251. The display unit 260 can also display information such as an image (interface) provided from the information processing device 100.
Next, a procedure of various types of information processing according to the embodiment will be described with reference to
The interface control unit 131 receives the information of key signature/scale, tempo, and the number of bars selected by the producer from the producer terminal 200 in the window 270 in
The interface control unit 131 receives the base information input by the producer from the producer terminal 200 in the window 273 in
The interface control unit 131 executes data storage processing (step S108). In step S108, the interface control unit 131 generates style information on the basis of the information of the key signature/scale, the tempo, and the number of bars, the rhythm information, the chord progression information, the base information, the melody information, and the like received from the producer terminal 200, and registers the style information in the style information table 122. Note that the style information may be generated on the producer terminal 200 side, and the interface control unit 131 may receive the style information.
In a case where additional recording of the melody information is executed (step S109, Yes), the processing proceeds to step S106. On the other hand, in a case where the additional recording of the melody information is not executed (step S109, No), the interface control unit 131 ends the processing.
The interface control unit 131 receives the style information ID of the selected style name in the window 280 in
In a case of receiving a change of the chord progression information in the window 281 in
The composition unit 133 of the information processing device 100 executes the AI composition processing on the basis of the updated style information (step S205). The interface control unit 131 transmits AI-composed music information (melody information) to the producer terminal 200 (step S206).
In a case of not receiving the arrangement request from the producer terminal 200 (step S207, No), the interface control unit 131 proceeds to step S210. On the other hand, in a case of receiving the arrangement request from the producer terminal 200 (step S207, Yes), the interface control unit 131 proceeds to step S208.
The interface control unit 131 receives designation of the parameter and the arrangement target bar (step S208). The arrangement unit 134 of the information processing device 100 executes arrangement processing on the arrangement target bar of the melody information (step S209). The interface control unit 131 transmits the arranged music information (melody information) to the producer terminal 200 (step S210).
The interface control unit 131 registers the new style information in the style information table 122 (step S220). In a case of receiving a composition request for another melody information (step S212, Yes), the interface control unit 131 proceeds to step S205. On the other hand, in a case of not receiving a composition request for another melody information (step S212, No), the interface control unit 131 ends the processing.
As described above, the information processing device 100 provides the producer terminal 200 with an interface capable of generating music information by fixing a part of elements among a plurality of elements forming the music information and editing other elements. The information processing device 100 generates new music information on the basis of a plurality of pieces of music information created using the interface. For example, the music information corresponds to the score information 740 of the style information 700. The fixed elements are the chord progression information, the base information, and the rhythm information. The other elements are melody information, and when a plurality of pieces of melody information is generated, a plurality of pieces of score information 740 corresponding to the melody information is generated. Consequently, the producer can easily create the style information (a plurality of pieces of the melody information, the chord progression information, the base information, and the rhythm information), and the information processing device 100 can easily receive provision of the style information.
The information processing device 100 generates new music information by inputting the plurality of pieces of music information to the learning model. Consequently, new music information can be easily created on the basis of the plurality of pieces of music information. For example, new melody information can be generated on the basis of the chord progression information, the base information, and the rhythm information included in the style information.
The information processing device 100 relearns the learning model on the basis of the plurality of pieces of music information created by the producer. Consequently, new music information according to the preference of the producer can be generated.
The information processing device 100 makes a plurality of pieces of style information selectable, displays the plurality of pieces of style information on the producer terminal 200, and displays, in a viewable manner, each feature amount of the melody information, the chord progression information, the base information, and the rhythm information included in the selected style information. Thus, editing work can be executed based on the already created style information.
In a case where each feature amount of the melody information, the chord progression information, the base information, and the rhythm information of the style information is updated, the information processing device 100 performs registration as new style information. Consequently, the new style information can be easily created from a clone of the style information.
In a case where a range is specified for the melody information of the style information and an arrangement request is received, the information processing device 100 executes the arrangement processing on the specified range. Consequently, it is possible to easily create the melody information in which existing melody information is arranged. The information processing device 100 may execute the arrangement processing in a case where a range is designated for not only the melody information but also the chord progression information, the base information, and the rhythm information.
Every time new style information is generated, the information processing device 100 stores the style information in association with the user ID. Consequently, a plurality of pieces of style information generated by the producer can be accumulated by the AI composition work.
The plurality of pieces of style information stored in the information processing device 100 can be used in other services such as an SNS by cooperating with the service server 300. For example, the service server 300 can collect evaluation information of the style information by disclosing the style information, and display the evaluation information in association with the style information. Note that the accompaniment audio information 760 is set in the style information, and can give a variation of performance.
Here, each function, a hardware configuration, and data in the information processing system are conceptually illustrated using the drawing.
The server device illustrated in
A learning processing unit and a control unit of the server device illustrated in
A display operation unit, a control unit, a DAW app unit, a mobile display operation unit, a mobile control unit, a plug-in display operation unit, and a plug-in control unit of a producer app unit illustrated in
An SNS server illustrated in
As illustrated in
First, a configuration related to a server device will be described.
The server device includes the control unit, the learning processing unit, and the intra-server database unit. The control unit of the server device has a produced music information management function, a style information management function, and an arrangement information management function. The learning processing unit of the server device has a machine learning processing function and a deep learning processing function.
Next, a configuration related to the system administrator app unit will be described.
The system administrator app unit includes a display operation unit, a control unit, and a DAW app unit. The display operation unit has a produced music display function, a style information display edit function, and an arrangement information display function. The control unit has the style information management function and arrangement information management.
The DAW app unit has a DAW display edit function. The DAW app unit is, for example, music editing software (DAW or the like), and can display, for example, music information in the work information display function. If the DAW has, for example, an AI assisted music production function, it is possible to produce new music information while using the style information display editing function. Note that the system administrator app unit has a similar configuration, and has different authority with respect to the system of the user.
The producer mobile app unit will be described.
The producer app unit includes a mobile display operation unit and a mobile control unit. The mobile display operation unit has the produced music display function, the style information display edit function, and the arrangement information display function. The mobile control unit has the style information management function and the arrangement information management.
The producer plug-in app unit will be described.
The producer plug-in app unit includes a plug-in display operation unit, a plug-in control unit, and a DAW app unit. The plug-in display operation unit has the produced music display function, the style information display edit function, and the arrangement information display function. The plug-in control unit has the style information management function and the arrangement information management. The DAW app unit is similar to the description regarding the DAW display edit function described above.
The processing according to the above-described embodiments and modifications may be performed in various different modes (modifications) other than the above-described embodiments and modifications.
In the description of the above-described embodiment, the chord progression information, the base information, and the rhythm information are set, and when the input of the melody information is received, these pieces of information are recorded as the score information. However, a DAW display edit unit of the DAW app unit edits the chord progression information, the base information, the rhythm information, and the plurality of pieces of melody information, and the plug-in control unit reads the information of the DAW app unit and uploads the read information to the control unit 130 included in the information processing device 100, so that the style information can be collectively stored.
In the example described with reference to
In the above description of the embodiment, the melody information is arranged, but for example, the melody information can be fixed, and a change can be added to the chord progression information by the arrangement processing. For example, chord progression information of |C|Dm|Em|F| can be arranged into chord progression of |C|FM7|G7|F| by performing the arrangement processing.
In the description of the above-described embodiment, the information processing device 100 provides the score information 740 to the service server 300, but a composition contest can be held among a plurality of producers, and a general user can post the score information in the contest. The service server 300 can count the evaluation information, give a ranking, and determine the winner. Furthermore, the service server 300 may notify the information processing device 100 of the score information posted in the contest, and the information processing device 100 may generate and store new style information using the notified score information.
In the description of the embodiment described above, the information processing device 100 displays the window 280 in
In processing of mapping on three-dimensional graphics and allowing the producer to search, the information processing device 100 plots points corresponding to the style information in a space constituted of an X axis, a Y axis, and a Z axis on the basis of the score information (feature amounts of the melody information, the chord progression information, the base information, and the rhythm information) of the style information. The X axis is an axis indicating dark (minor) and bright (major), and indicates that it is brighter toward the positive direction. The Y axis is an axis indicating simple and tension, and indicates that it is more complicated toward the positive direction. The Z axis is an axis indicating that the tempo of the BPM is slow or fast, and indicates that the tempo is faster toward the positive direction.
In a case where a point plotted on the three-dimensional graphic is selected, the information processing device 100 acquires style information corresponding to the selected point from the style information table 122, transmits the acquired style information to the producer terminal 200, and causes the style information to be displayed. Consequently, features of the style information can be visually grasped, the style information preferred by the producer can be selected, and various types of editing can be performed.
The information device such as the information processing device 100 and the producer terminal 200 according to each of the above-described embodiments and modifications is implemented by the computer 1000 having a configuration as illustrated in
The CPU 1100 operates on the basis of a program stored in the ROM 1300 or the HDD 1400, and controls each unit. For example, the CPU 1100 develops a program stored in the ROM 1300 or the HDD 1400 in the RAM 1200, and executes processing corresponding to various programs.
The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is activated, a program depending on hardware of the computer 1000, and the like.
The HDD 1400 is a computer-readable recording medium that non-transiently records a program executed by the CPU 1100, data used by such a program, and the like. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the present disclosure as an example of program data 1450.
The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (for example, the Internet). For example, the CPU 1100 receives data from another device or transmits data generated by the CPU 1100 to another device via the communication interface 1500.
The input/output interface 1600 is an interface for connecting an input/output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard and a mouse via the input/output interface 1600. Further, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input/output interface 1600. Furthermore, the input/output interface 1600 may function as a media interface that reads a program or the like recorded in a predetermined recording medium. The medium is, for example, an optical recording medium such as a digital versatile disc (DVD) or a phase change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a tape medium, a magnetic recording medium, a semiconductor memory, or the like.
For example, in a case where the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 implements the functions of the control unit 130 and the like by executing the information processing program loaded on the RAM 1200. In addition, the HDD 1400 stores an information processing program according to the present disclosure and data in the storage unit 120. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data 1450, but as another example, these programs may be acquired from another device via the external network 1550.
An information processing device is an information processing method executed by a computer, including providing an interface capable of generating music information by fixing a part of elements among a plurality of elements forming the music information and editing other elements, and generating new music information on the basis of a plurality of pieces of music information created using the interface. Consequently, the producer can easily create the style information (a plurality of pieces of the melody information, the chord progression information, the base information, and the rhythm information), and the information processing device can easily receive provision of the style information.
The information processing device generates the new music information by inputting the plurality of pieces of music information to the learning model. Consequently, new music information can be easily created on the basis of the plurality of pieces of music information.
The information processing device relearns the learning model on the basis of the plurality of pieces of music information. Consequently, new music information according to the preference of the producer can be generated.
The information processing device provides the interface, and acquires fixed feature amounts for a part of elements and a plurality of feature amounts for other elements included in the plurality of pieces of music information. Consequently, the plurality of pieces of music information can be efficiently collected and used.
The information processing device selectively displays different pieces of music information by the interface, and in a case of receiving a selection of a piece of music information from the different pieces of music information, the information processing device displays feature amounts of a plurality of elements forming the selected piece of music information in a viewable manner. Thus, editing work can be executed based on the already created style information.
In a case where the feature amount of the element forming the selected piece of music information is updated, the information processing device generates new music information having an updated feature amount. Consequently, the new style information can be easily created from a clone of the style information.
In a case where a range is specified for a feature amount of an element forming the selected piece of music information, the information processing device arranges a feature amount included in the range and generates new music information having the arranged feature amount. Consequently, it is possible to easily create melody information in which existing music information (such as melody information) is arranged.
Every time new music information is generated, the information processing device stores the new music information in association with identification information of a user. Consequently, a plurality of pieces of style information generated by the producer can be accumulated by the AI composition work.
The information processing device maps a point corresponding to the music information in a multidimensional space on the basis of a feature amount included in the music information, and displays the mapped point. Consequently, features of the style information can be visually grasped, the style information preferred by the producer can be selected, and various types of editing can be performed.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-146425 | Aug 2020 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2021/027524 | 7/26/2021 | WO |
