The present disclosure relates to the fields of music composition, music orchestration and machine learning. Specifically, aspects of the present disclosure relate to automatic manipulation of compositional elements of a musical composition.
Currently, music is mostly created by some combination of a musician or musicians writing musical notes on paper or recording them and sometimes by several musicians collaborating on a piece of music over time as the creation evolves, sometimes in a studio where the composition process can take place over an indeterminate period.
In parallel Machine Learning and Artificial Intelligence have been making it possible to generate content based on training sets of existing content as labeled by human reviewers or musical convention.
The present disclosure describes a mechanism for changing music, on the fly (dynamically) based on written or artificially generated motifs, which are then modified using real or virtual faders that change the music based on the characteristics of its musical components such as time signature, melodic structure, modality, harmonic structure, harmonic density, rhythmic density and timbral density.
Music is made of many parameters including but not limited to time signature, melodic structure, modality, harmonic structure, harmonic density, rhythmic density and timbral density. Generally, these parameters are not applied by music generation software and are instead simply may be considerations the composer has when generating a new musical composition. When music is composed, a composer often begins with one or more motifs, uses them, and changes them throughout the piece. According to aspects of the present disclosure, a set of virtual or physical faders and switches may be used to make those changes automatically based on the above parameters (melodic structure, modality, etc.) as time continues. The time could be linear with the faders and switches being used to create a composition. Alternatively, faders and switches could be used to generate the music dynamically based on emotional elements or elements that appear in a game, movie, or video as described in patent application Ser. No. 16/677,303 filed Nov. 7, 2019, the entire contents of which are incorporated herein by reference. The present disclosure describes a system of faders and switches that are associated with various musical parameters that can be controlled by a human operator.
The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:
As used herein, the term musical input describes a musical motif such as a melody, harmony, rhythm or the like provided to the mixing console described below. Similarly, a musical output is a melody, harmony, rhythm or the like output by a mixing console after the musical input undergoes one or more of the operations described below. While some aspects of the disclosure may describe operation performed on a melody for simplicity, it should be understood that such operations may be performed on any type of musical input.
As can be seen in
The faders are used to vary compositional components along an axis. More faders may be used to vary other parameters of those components, as can switches. The assignment of the parameters to the faders and switches is not limited to a single preset and the composer can have broad control over their behavior. The composer may customize the behavior of each fader or switch individually using the dynamic parameters discussed herein as touchstones for slider behavior.
The first Variable Parameter is the selection of Scalar Elements. Even non-musicians are aware that major songs tend to be “happy” and minor songs tend to be “sad.” However, the scale from which a melody is composed has a many more nuanced choices. Even on the happy/sad scale, changes in the scale of the composition may have a more nuanced effect on the overall feel of the song than just changing the mood from happy to sad or vice versa. Additionally, there are more scales than just minor and major scales and transposition of an input melody to these scales may shift the overall mood of a piece and change the overall compositional nature of the melody. As can be seen if
The faders and/or switches may be coupled to a computer system or even a set of mechanical switches operated by humans and together or individually, the devices may be configured to manipulate the notes of a musical input based on the settings of the faders and switches. For example, and without limitation, a music composer might create a melodic phrase and have it encoded as data (e.g. using MIDI or MusicXML or voltages or any other naming or representational convention), and play that representation in real time on, for example, a digital keyboard or have recorded it previously. That representation then serves as the input to the faders and/or switches and a computer or other mechanism uses the algorithm described in this disclosure to Transform the notes, which are then rendered by an instrument module. One could use any instrument module from Analog Synthesizers to Frequency Modulation Synthesizers to Sampling Synthesizers to Physical Modeling Synthesizers to mechanical devices that make analog sounds like a piano roll or a Yamaha Disklavier. The computer may transform the representations of musical notes at the input to create by such transformation an output that is different from the input using the switches and faders as herein described. Alternatively, the faders and/or switches may be coupled to a computer system and together or individually, the devices may be configured to perform spectral analysis on an audio input to decompose the musical input's components into underlying tones, harmonies and timing and identify individual components that comprise the input. The devices may further be configured to manipulate the frequencies of the underlying spectral tones of the musical input to change the keys of the individual notes of the input. The devices may then reconstruct the decomposed musical elements and reconfigure as described here to generate a musical output that is different based on the positioning of the sliders and switches to effectuate the desired compositional changes. Alternatively, a Neural Network (NN) component may be trained with machine learning to generate a musical output in various different modes as discussed above based on the slider settings. The slider settings may adjust one or more inputs (controls) to the NN to determine the melodic mode of the output composition.
Looking at
It is noted that labels are somewhat arbitrary. Society agrees on a specific label Blue for the color blue but that is only by convention (or language). However, without labels, it would be difficult to remember and certainly harder to describe colors to others. Even emotions such as happy to sad on the modal continuum are subjective. Musicians (as evidenced by labeling on keyboard synthesizers) are very good at adapting to labels. A musician might label the Whole Tone scale as Ethereal (most would probably agree) and the Symmetrical Diminished as Spooky (more subjective). It really does not matter what labels are chosen and in fact individual composers can choose or change the labels as they see fit. What is important is that there is a mechanism for modifying compositions based on the changes proposed in this disclosure. First the Western Variants 303, Lydian ♭7, Altered Dominant and Melodic Minor are all different modes of the same scale (as the Greek modes are different modes of the major scale) and so these would naturally fit on a fader. The Blues Scale and the Harmonic Minor Scale are both well known to composers by those names and should probably go on switches under those names.
Looking at labeling for functions of a Fader Switch Matrix, such as that depicted in
Aspects of the present disclosure also address other elements of composition and orchestration or arranging. By way of example,
The Harmonic Density 401 may be mapped to one or more faders or to switches. In the broadest use for example and without limitation, the bottom of the fader would be unison. That is just the melody 402 and as you move the fader up the harmonization would go through Two Part Voicing 403, Structures in Fourths 404, Triadic Structures 405 in open voicing, and then in closed voicing, then adding upper structure harmonies like 9ths 11ths and 13ths 406. Finally, the most harmonically dense structures are clusters 407.
Alternatively, each of the Harmonic Density settings may be mapped to switches; again, “exclusive” meaning only one can be active at a time. However, you can have a Harmonic Density switch active while you have a Melodic Tonality switch active at the same time. These are Non-Exclusive—that is they can be used in combination with other parameters.
Another variant on Harmonic Density is Harmonic Substitution. Harmonic Substitution can be spread across two axes: from Consonance to Dissonance and the axis of Tonal Distance. Tonal Distance, as used herein and as understood by those skilled in the musical arts, means the distance from the notes within the key of the melody. Since Harmonic Density and Harmonic Substitution from Consonance to Dissonance and Tonal Distance are on a continuum, they would all be mapped to faders. As seen in
The next element for varying a musical input is called Melodic Structure. The elements of Melodic structure are Non-exclusionary and may be varied independently. As seen in
Retrograde and Inversion are mathematically based and can be defined as a function taking into account the shape and the key of the input. Since the techniques or Retrograde and Inversion are both binary functions, they are assigned to buttons 706 and 707. Note that unlike the melodic Scalar elements in
There are some other Areas of variability that can be controlled by faders as they span a continuum of values. As shown in
Rhythmic Density 803 is also variable that has a mappable range from Sparse 804—whole notes or longer to Dense 805—32nd or shorter. Rhythmic Density can be linear but would likely have unanticipated consequences. Using Machine Learning to contextualize Rhythmic density would likely yield more musical results. Rhythmic Complexity 806 is a bit more nuanced but rhythms across the beat lines are more complex than those on the beat lines and divisions like triplets, quintuplets and septuplets are even more complex. Generally, Rhythmic complexity goes from Simple 807 to Complex 808. Any mechanism from a simple switching algorithm to a complex NN may be used to change the rhythmic density of a musical input. In some implementations, a NN may be trained to recognize the Rhythm of the musical input and alter the rhythm of the input work to apply different note divisions to the musical input. For example, and without limitation, the NN may be trained to change whole notes to two half notes, half notes to two quarter notes, quarter notes to two eighth notes etc. The NN may also combine notes together to generate a faster beat for example two different half notes may become two different quarter notes. A NN trained on popular music from any era would naturally generate musical choices that could be fine-tuned using the faders.
The last continuum in this section is related to Timbre or Timbral Complexity 809. In traditional music flutes are close to a sine wave and are considered not complex timbrally while an oboe is more timbrally complex. Guitars have used varying degrees of distortion for years with traditional jazz guitars being very clean and Death Metal being very distorted. This continuum goes from Pure 810 to Distorted 811.
One last continuum is Tempo self-explanatory in this context—push the fader up and the song goes faster; pull it down and it goes slower.
Some other features of the system, while not unique on their own are unique within the context of a system like this one. Loop Length is adjustable and can be changed based on time, number of bars, etc. As shown, in
Also, as described in the referenced previous application, parameters fader and switch positions) can be controlled by events and actions in games and this can be done using emotional vectors and or Artificial Intelligence.
Matrixing it all Together
Settings of the faders and/or switches may be saved and used later or applied to other uses. The settings of the faders and/or switches may be saved in a data structure such as a table or three-dimensional matrices as shown in
These matrices may be saved for each musical composition generated to create further data for compositional analysis. The matrices may be provided to one or more neural networks with a machine learning algorithm along with other data such as emotional vectors, style data, context etc. The NN with machine learning algorithm may learn associations with slider settings that may be applicable to other musical compositions in a corpus of labeled musical compositions. Additionally, with sufficient training the NN with machine learning algorithm may eventually be able to assign slider settings for different moods, musical styles etc. based on the training data.
This application is a continuation-in-part of U.S. patent application Ser. No. 16/677,303 filed Nov. 7, 2019, the entire contents of which are incorporated herein by reference. U.S. patent application Ser. No. 16/677,303 claims the priority benefit of U.S. Provisional Patent Application No. 62/768,045, filed Nov. 11, 2018, the entire disclosures of which are incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
5451709 | Minamitaka | Sep 1995 | A |
6124543 | Aoki | Sep 2000 | A |
9583084 | Fagan | Feb 2017 | B1 |
9799312 | Cabral | Oct 2017 | B1 |
10964299 | Estes | Mar 2021 | B1 |
20030128825 | Loudermilk | Jul 2003 | A1 |
20030167907 | Annen | Sep 2003 | A1 |
20070044639 | Farbood et al. | Mar 2007 | A1 |
20080141850 | Cope | Jun 2008 | A1 |
20090304207 | Cooper | Dec 2009 | A1 |
20100307320 | Hoeberechts et al. | Dec 2010 | A1 |
20120047447 | Haq | Feb 2012 | A1 |
20130025435 | Rutledge et al. | Jan 2013 | A1 |
20140076126 | Terry | Mar 2014 | A1 |
20140180674 | Neuhauser | Jun 2014 | A1 |
20160253915 | Lee et al. | Sep 2016 | A1 |
20170228745 | Garcia et al. | Aug 2017 | A1 |
20170365277 | Park | Dec 2017 | A1 |
20190237051 | Silverstein | Aug 2019 | A1 |
20200005744 | Godunov | Jan 2020 | A1 |
20200074877 | Marradi | Mar 2020 | A1 |
20200105292 | Large et al. | Apr 2020 | A1 |
20200312287 | Galuten | Oct 2020 | A1 |
20200380940 | Abdallah | Dec 2020 | A1 |
20210043177 | Bar-Or | Feb 2021 | A1 |
20210201863 | Bosch Vicente | Jul 2021 | A1 |
Number | Date | Country |
---|---|---|
20180005277 | Jan 2018 | KR |
2007043679 | Apr 2007 | WO |
Entry |
---|
International Search Report and Written Opinion dated Jan. 29, 2020 for International Patent Application No. PCT/US2019/060306. |
Kim et al. “Music Emotion Recognition: A State of the Art Review”; 11th International Society for Music Information Retrieval converence; Publication [online]. 2010 [retrieved Jan. 13, 2020]. Retrieved from the Internet:; pp. 255-266. |
U.S. Appl. No. 16/677,603 to Albhy Galuten filed Nov. 7, 2019. |
International Search Report and Written Opinion dated Jun. 21, 2021 for International Patent Application No. PCT/US2021/025371. |
Non-final Office Action dated Dec. 27, 2021 for U.S. Appl. No. 16/677,303. |
Non-Final Office Action for U.S. Appl. No. 16/677,303, dated Dec. 27, 2021. |
Number | Date | Country | |
---|---|---|---|
20200312287 A1 | Oct 2020 | US |
Number | Date | Country | |
---|---|---|---|
62768045 | Nov 2018 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 16677303 | Nov 2019 | US |
Child | 16838775 | US |