1. Field of the Invention
The present invention generally relates to applying compositional grammar and rules to information retrieved or extracted from a musical selection. More specifically, the present invention relates to annotating feature data, applying instrumentation to the data, and rendering the same for playback, sharing, or further annotation.
2. Description of the Related Art
Music platforms that sell or handle label-owned or amateur-made songs are plentiful across the Internet, for example iTunes and Sound Cloud. Streaming solutions for label-owned and amateur-made content are likewise widely accessible, such as Pandora and Spotify. Music making sequencers or “virtual” musical instruments are also available from the Apple “App Store” and the Android “Marketplace.”
Notwithstanding the presence of these solutions, the music industry is lacking an accessible way for users to express and share thoughts musically in radio or studio quality without knowledge of music making or music production. For example, an amateur musician may not have the extensive skills necessary to produce a studio or radio quality track notwithstanding that musician otherwise having the ability to create musical content. Similarly, someone interested in post-processing may not have the underlying talent to generate musical content to be processed. Nor is there an easy way for musicians to collaborate in real-time or near real-time without being physically present in the same studio.
There is a need in the art for identifying the compositional elements of a music selection—music information retrieval or “MIR.” Through the use of machine learning and data science, hyper-customized user experiences could be created. For example, the aforementioned machine learning metrics may be applied to extracted music metrics to create new content. That content may be created without extensive musical or production training and without the need for expensive or complicated production equipment. Such a system could also allow for social co-creation of content in real-time or near real-time notwithstanding the physical proximity of contributors.
An embodiment of the present invention provides for composing music based on unprocessed audio. Through the method, melodic hums and rhythmic taps are received. Information is retrieved from the melodic hums and rhythmic taps to generate extracted musical features which are then used to generate an abstraction layer. A piece of musical content is composed using the abstraction layer and then rendered in accordance with the abstraction.
Embodiments of the present invention provide for the composition of new music based on analysis of unprocessed audio, which may be in the form of melodic hums and rhythmic taps. As a result of this analysis—music information retrieval or MIR—musical features such as pitch and tempo are output. These musical features are then used by a composition engine to generate a new and socially co-created piece of content represented as an abstraction. This abstraction is then used by a production engine to produce audio files that may be played back, shared, or further manipulated.
For example, hardware device 100 may be utilized to implement musical information retrieval. Hardware device 100 might also be used for composition and production. Composition, production, and rendering may occur on a separate hardware device 100 or could be implemented as a part of a single device 100.
Hardware device 100 as illustrated in
The aforementioned components of
Mass storage 130 may be implemented as tape libraries, RAID systems, hard disk drives, solid-state drives, magnetic tape drives, optical disk drives, and magneto-optical disc drives. Mass storage 130 is non-volatile in nature such that it does not lose its contents should power be discontinued. As noted above, mass storage 130 is non-transitory in nature although the data and information maintained in mass storage 130 may be received or transmitted utilizing various transitory methodologies. Information and data maintained in mass storage 130 may be utilized by processor 110 or generated as a result of a processing operation by processor 110. Mass storage 130 may store various software components necessary for implementing one or more embodiments of the present invention by loading various modules, instructions, or other data components into memory 120.
Portable storage 140 is inclusive of any non-volatile storage device that may be introduced to and removed from hardware device 100. Such introduction may occur through one or more communications ports, including but not limited to serial, USB, Fire Wire, Thunderbolt, or Lightning. While portable storage 140 serves a similar purpose as mass storage 130, mass storage device 130 is envisioned as being a permanent or near-permanent component of the device 100 and not intended for regular removal. Like mass storage device 130, portable storage device 140 may allow for the introduction of various modules, instructions, or other data components into memory 120.
Input devices 150 provide one or more portions of a user interface and are inclusive of keyboards, pointing devices such as a mouse, a trackball, stylus, or other directional control mechanism. Various virtual reality or augmented reality devices may likewise serve as input device 150. Input devices may be communicatively coupled to the hardware device 100 utilizing one or more the exemplary communications ports described above in the context of portable storage 140.
Display system 170 is any output device for presentation of information in visual or occasionally tactile form (e.g., for those with visual impairments). Display devices include but are not limited to plasma display panels (PDPs), liquid crystal displayus (LCDs), and organic light-emitting diode displays (OLEDs). Other displays systems 170 may include surface conduction electron emitters (SEDs), laser TV, carbon nanotubes, quantum dot displays, and interferometric modulator displays (MODs). Display system 170 may likewise encompass virtual or augmented reality devices.
Peripherals 180 are inclusive of the universe of computer support devices that might otherwise add additional functionality to hardware device 100 and not otherwise specifically addressed above. For example, peripheral device 180 may include a modem, wireless router, or otherwise network interface controller. Other types of peripherals 180 might include webcams, image scanners, or microphones although the foregoing might in some instances be considered an input device.
Prior to undertaking the steps discussed in
The aforementioned music retrieval operation involves receiving a melodic or rhythmic contribution at a microphone or other audio receiving device and transmitting that information to a computing device like hardware device 100 of
Upon receipt of the melodic musical contribution, hardware device 100 executes software to extract various elements of musical information from the melodic utterance. This information might include, but is not limited to, pitch, duration, velocity, volume, onsets and offsets, beat, and timbre. The extracted information is encoded into a symbolic layer.
Music information retrieval may operate in a similar fashion with respect to receipt of a tap or other rhythmic contribution at a microphone or audio receiving device operation in conjunction with a client application that provides for the transmission of information to a computing device like hardware device 100 of
Extracted musical information is reflected as a tuple in the symbolic layer. Tuples are ordered lists of elements with an n-tuple representing a sequence of n elements with n being a non-negative integer—as used in relation to the semantic web. Tuples are usually written by listing elements within parenthesis and separate by commas (e.g., (2, 7, 4, 1, 7)).
By encoding extracted musical information into the symbolic layer, audio information may be flexibly manipulated as it transitions from the audible analog domain to the digital data domain and back as a newly composed, produced, and rendered piece of musical content. The symbolic layer is MIDI-like in nature in that MIDI (Musical Instrument Digital Interface) allows for electronic musical instruments and computing devices to communicate with one another by using event messages to specify notation, pitch, and velocity; control parameters corresponding to volume and vibrato; and clock signals that synchronize tempo.
The symbolic layer operates as sheet music. Through use of this symbolic layer, other software modules and processing routines, including those operating as a part of a composition engine, are able to utilize retrieved musical information for the purpose of applying compositional grammar rules. These rules operate to filter and adjust the musical contributions and corresponding features to deduce intent in a manner similar to natural language processing. An end result of the execution of the composition engine against the extracted feature data is a musical blueprint.
Prior to validation, at step 215, an arrangement model may be referenced to correlate the symbolic layer to a dictionary of functions for various musical styles. This may include various aspects of chord progression, instrumentation, eastern versus western tonality, and other information that will drive, constrain, or otherwise influence the building of the musical blueprint, especially during the derivation of intent operation at step 230. Various fundamentals of music theory are introduced during this operation.
Abstraction layer information is validated at step 220 to determine if the context includes within a reasonable range or otherwise meets basic musical assertions. For example, melodic data or rhythmic data could be presented as pure white noise and might generate some extractable features. That small subset of features would not, however, likely meet a basic definition of a musical contribution. If validation evidences that the symbolic layer is not indicative of musical content, then composition engine will not attempt to further process and develop a musical blueprint for the same. If the symbolic layer meets some basic assertions associated with musical content, then the composition operation continues.
At step 230, an effort is made to derive the intent of the musical contribution and, more specifically, its extracted musical features as represented in the symbolic layer. Deriving the intent of the music generally means to derive the intended melodies and rhythms from extracted features in the MIR data and, potentially, data in a user profile (e.g., previously indicated preferences or affirmatively derived preferences). To identify the intent and prepare the symbolic layer for further production, a quantization process takes raw data and intelligently maps the same into a hierarchical structure of music. The preparation step further involves identification of empirical points in the extracted features, for example, those having the most metrical weight
At step 240, a seamless loop point is identified in the input file representing the symbolic layer. This loop point is used as a reference point for identifying the likes of chord progressions at step 250. The melody is, also at step 260, reduced to a fundamental skeletal melody based on the likes of harmonic tendencies and calculation of chord progressions. Skeletal melodies are representative of certain activity at, above, or below an emphasized point. The skeletal melody identification process is dynamic and based on runtime input
Rhythmic patterns are introduced at step 270 on the basis of extracted feature data for ‘taps’ or rhythmic musical contributions. Adjustments are made at step 280 to align hums and taps (melody and rhythm), which may involve various timing information including but not limited to the aforementioned loop point. Step 290 involves the application of supporting chords and bass as might be appropriate in light of a particular musical style or genre.
Corrections and normalization occur at step 295 before the completed blueprint is delivered for production and rendering as discussed in the context of
The production process of
The foregoing detailed description has been presented for purposes of illustration and description. The foregoing description is not intended to be exhaustive or to the present invention to the precise form disclosed. Many modifications and variations of the present invention are possible in light of the above description. The embodiments described were chosen in order to best explain the principles of the invention and its practical application to allow others of ordinary skill in the art to best make and use the same. The specific scope of the invention shall be limited by the claims appended hereto.
The present application is a continuation-in-part and claims the priority benefit of U.S. patent application Ser. No. 14/920,846 filed Oct. 22, 2015, which claims the priority benefit of U.S. provisional application No. 62/067,012 filed Oct. 22, 2014; the present application is also a continuation-in-part and claims the priority benefit of U.S. patent application Ser. No. 14/931,740 filed Nov. 3, 2015, which claims the priority benefit of U.S. provisional application No. 62/074,542 filed Nov. 3, 2014; the present application claims the priority benefit of U.S. provisional application No. 62/075,185 filed Nov. 4, 2014. The disclosure of each of the aforementioned applications is incorporated herein by reference.
Number | Date | Country | |
---|---|---|---|
62067012 | Oct 2014 | US | |
62074542 | Nov 2014 | US | |
62075185 | Nov 2014 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 14920846 | Oct 2015 | US |
Child | 14932911 | US | |
Parent | 14931740 | Nov 2015 | US |
Child | 14920846 | US |