Patent Grant
10964299
The present invention is directed to new and improved methods of and apparatus for producing libraries of sampled and/or synthesized virtual musical instruments that can be used to produce automated digital performances of music compositions having greater degree of uniqueness, expressiveness and realism, in diverse end-user applications.
Applicant's mission is to enable anyone to express themselves creatively through music regardless of their background, expertise, or access to resources. With this goal in mind, Applicant has been inventing and building tools powered by innovative technology designed to help people create and customize original music. As part of this process, Applicant has been bringing human know-how to automated music composition, performance, and production technology. This has involved creating sound sample libraries and datasets, for use in automatically composing, performing and producing high quality music through the fusion of advanced music theory and technological innovation. To date, Applicant's commercial AI-based music composition and production system, marketed under the brand name AMPER SCORE™, supports over one million individual samples and thousands of unique virtual musical instruments capable of producing a countless number of unique audio sounds to express and amplify human creative expression. Recorded by hand, every audio sound sample in Applicant's virtual musical instrument (VMI) sound sample libraries is sculpted with meticulous attention to detail and quality.
In view of the above, Applicant seeks to significantly improve upon and advance the art and technology of sampling sounds from diverse sources including (i) real musical instruments, (ii) natural sound sources found in nature, as well as (iii) artificial audio sources created by synthesis methods of one kind or another. Applicant also seeks to improve upon and advance the art of constructing and operating virtual musical instrument (VMI) libraries maintaining deeply audio-sampled and/or sound-synthesized virtual musical instruments that are designed for providing the notes and sounds required to perform virtual musical instruments and produce a digital performance of a music composition.
To appreciate the problems addressed and effectively solved by Applicant's inventions disclosed herein, it will be helpful to provide a brief overview on the art of sound sampling, and the surrounding music technology conventions that have supported this field and advanced music performance and production to its current state of the art, using virtual musical instrument libraries developed today around the world. At the same time, this overview will help set the stage for understanding why the same conventional music technology that has helped the industry reach its current state, also now hinders the industry in meeting the challenges of the present, moving into the future, and realizing the benefits this creative technology promises to bring to humanity.
Sound sampling (also known simply as “sampling”) is the process of recording small bits of audio sound for immediate playback via some form of a trigger. Historically, the sampling process has been around since the early days of Musique Concrete (in the 1940s) and came to commercial success with the invention of the Mellotron (1963). There are two main approaches to sampling, instrument sampling and loop sampling. Loop sampling is the art of recording slices of audio from pre-recorded music, such as a drum loop or other short audio samples (historically from vinyl). Amper Music uses the instrument sampling methodology in its SCORE™ AI Music Composition and Generation System. The instrument sampling process is to record and audio capture single note performances to replicate an instrument with any combination of notes.
In the early days (1960s-2000ish), the process of sampling was largely unchanged until the invention of computerized digital reproduction that enabled larger and deeper sampling methodologies, supporting highly complex sample instrument libraries in which each instrument is performed and recorded across its range of playable notes. As random-access memory (RAM) and hard drive storage sizes increased, libraries became more complex, and performance of these samples became extremely difficult to both perform and program via MIDI. Some companies developed solutions to help mitigate the time it takes to select these samples in real time.
Samplers differ from synthesizers in that the fundamental method of sound production begins with a sound sample or audio recording of an acoustic sound or instrument, electronic sound or instrument, ambient field recording, or virtually any other acoustical event. Each sample is typically realized as a separate sound file created in a suitable data file format, which is accessed and read when called during a performance. Typically, samples are triggered by some sort of MIDI input such as a note on a keyboard, an event produced by a MIDI-controlled instrument, or note generated by a computer software program running on a digital audio workstation.
In prior art sound sampling instruments, each sample is contained in a separate data file maintained in a sample library supported in a computer-based system. Most prior art sample libraries have several samples for the same note or event to create a more realistic sense of variation or humanization. Each time a note is triggered, the samples may cycle through the series before repeating or be played randomly.
Typically, the audio samples in a sample library system are organized and managed using relational database management technology (RDBMS). Modern sampling instruments require many Terrabytes of digital data storage for library data storage management capabilities, and large amounts of RAM for program memory support. In a prior art computer-based sound sample library system, the audio samples are typically stored in a zone (or other addressable region of memory) which is an indexed location in the sample library system, where a single sample is loaded and stored. In a sample library system, an audio sample can be mapped across a range of notes on a keyboard or other musical reference system. In general, there will be a Root key associated with each sample which, if triggered, will playback the sample at the same speed and pitch at which it was recorded. Playing other keys in the mapped range of a particular zone, will either speed up or slow down the sample, resulting in a change in pitch associated with the key.
Depending on the sample library system, zones may occupy just one or many keys, a could contain separate sample for each pitch. Some samplers allow the pitch or time/speed components to be maintained independent for a specific zone. For instance, if the sample has a rhythmic component that is synced to tempo, rhythmic part of the sound can be maintained fixed while playing other keys for pitch changes. Likewise, pitch can be fixed in certain circumstances.
In most conventional sound sample libraries, there will be an envelope section to control amplitude attack, decay, sustain and release (ADSR) parameters. This envelope may also be linked to other controls simultaneously such as, for example, the cutoff frequency of a low-pass filter used in sound production.
Typically, sound samples are either (i) One Shots, which play just once regardless of how long a key trigger is sustained, or (ii) Loops which can have several different loop settings, such as Forward, Backward, Bi-Directional, and Number of Repeats (where loops can be set to repeat as long as a note is sustained or for a specified number of times).
The effect of the Release stage on Loop playback can be to continue the repeat during the release or may cause a jump to a release portion of the sample. In more complex sampler instruments, there are often Release Samples specific to the type of sound and usually intended to create a better sense of realism. Like any synthesizer, most samplers will have controls for pitch bend range, polyphony, transposition and MIDI settings.
The energy spectrum as well as the amplitude of the sounds produced by sampled musical instruments will depend on the speed at which a piano key is hit, or the loudness of a horn note or a cymbal hit. Developers of virtual musical instrument libraries consider such factors and record each note at a variety of dynamics from pianissimo tofortissimo. These audio samples are then mapped to zones which are then be triggered by a certain range of MIDI note velocities. Some prior art sampling engines such as Kontakt from Native Instruments, allow for crossfading between velocity layers to make transitions smoother and less noticeable.
Grouping zones with common attributes expands the functionality of prior art sampling instruments. A common application of zone grouping is string articulations because there are numerous ways to play a note on a violin, for example: Legato bowing, spiccato, pizzicato, up/down bowing, sul tasto, sulponticello, or as a harmonic. In advanced prior art string libraries, zone groupings based on articulations have been superimposed over the same range on the keyboard. Also, a Key Trigger or a MIDI controller have been used to activate a certain group of samples.
Most prior art samplers have on-board effects processing such as filtering, EQ, dynamic processing, saturation and spatialization. This makes it possible to drastically change the sonic result and/or customize existing presets to meet the needs of a given application. Prior art sound sampling instruments have employed many of the same methods of modulation found in most synthesizers for the purpose of affecting parameters. These methods have included low frequency oscillators (LFOs) and envelopes. Also, signal processing methods and paths, automation, complex sequencing engines, etc. have been developed and deployed within prior art sampling instruments as well.
Beyond the prior art sampling instruments described above, there is a great volume of prior art technology relating to the field of sampled virtual musical instrument design. The following prior art map is provided to help clearly describe the various conventional technologies which are considered prior art to Applicant's present inventions:
Prior Art Methods of Capturing and Recording Sound Samples from Real Musical Instruments:
1. Capturing Audio Sample via Sound Recording
2. Triggering of Sound Samples (Playback)
3. Modulation of Sound Samples
Other than the MIDI Standard set in 1983, there are no real standards governing the instrument sampling industry, other than the assignment of MIDI Note Numbers to notes having a Note Name and a particular Pitch Frequency based on 12-EDO tuning, as illustrated in
The decisions of where to split a velocity of a musical instrument being sampled, how deeply should a musical instrument be sampled (e.g. how many round robins, how many microphones, how many velocities, which notes, etc.), and which MIDI data controls should be sent to select samples, have been choices left up to the sample-based musical instrument designer. While this provides an “art” to instrument sample design, it does not help when you need to know exactly how an instrument will perform and predict what it should do. This has provided “camps” of composers who prefer some approaches of sampled instruments (usually dictated by the company/person making the samples).
The MIDI data communications protocol was originally designed for hardware/physical instruments. MIDI is largely used/designed for musical devices to playback music in real-time. Because MIDI was a convention when software technology came into play, sending out data messages to outboard gear from the computer adapted the MIDI standard. Now that the music industry is largely software driven in most applications (and entirely software driven in others), the types of devices communicating are now much more sophisticated. Using MIDI, the industry is stuck in a 36 year-old technology.
MIDI's 127 data control point resolution is extremely limited. Much greater resolution is required to express things like “controller” data, “program change” (i.e. articulation switching). Consequently, MIDI has placed constraints on modern musical notation during both composition and performance stages.
Some performance constraints are known, and some are unknown. As programming logic is not inherent in the MIDI protocol, and instruments are not standardized across all the commercial parties involved, the “unknown” is more of a by-product of not having good work arounds, or having a system that is too antiquated to deliver the needed standards in a given application. A big issue with MIDI is that while the MIDI communication protocol is standardized, applications using MIDI are not. Thus, a device will know what value to send on a MIDI controller lane, but it is up to the manufacturer to specify what function it will actually perform. For example, CC1 (Continuous Controller #1) is set by MIDI as the “Modulation” controller. It is a standard physical “wheel” controller that goes from 0-127 in value that exists on nearly every physical musical keyboard. The initial intent was to add “vibrato” modulation to a sound and control how wide or fast that vibrato should happen. Nearly every modern software instrument developer uses CC1 to control dynamic expression or even filter a sustained sound, and some software synths still use it to control vibrato. For reference, CC11 is supposed to be used for Expression, and CC71 would typically be used to control filter.
Such conventional approaches provide a “wild west” approach to the challenge of how to implement MIDI based on “ease-of-access” on a physical controller. For example, physical MIDI controllers are typically Keyboards that range widely on what knobs, faders and wheels they were manufactured with, but 90% of keyboards always have a pitch and modulation wheel. Modulation wheel is set to CC1 so most software developers use this controller as the primary controller to manipulate samples. MIDI is only a communications protocol between musical devices.—the methods used, while initially designed to be standardized, were not.
Articulation switching (sample set switching) and Continuous Controller assignments are two areas that has not been standardized. Many software developers have hacked MIDI in a way to help switch articulations either by a key switch (using a MIDI note to change a set of sounds), or by program changes (less common, but was the designed controller to do instrument or sound set switching.)
With computerized score notation, these switches in articulation and controller data could be reflected, if the notation software had knowledge of the keyswitches or program changes, but if the score software did not have knowledge of what the software sampling company designed, how would you know when to write a staccato or marcato marking? For example, if the software knew that MIDI note=01, was switch to pizzacatto, then “pizz” could be written on the score. Same goes for MIDI controller data, if you had three different sources of controller data numbers, commonly used for dynamic (piano/forte) control, how do you determine a “dynamic” on a score based on velocity or CC1 (modulation), CC11 (expression) or CC7 (volume)?
As there is no standardization in the music industry on what articulations go where, what velocities happen, and how to trigger samples, etc.) conventional MIDI files are almost useless in the process of creating finalized audio music tracks.
The only music-theoretic states in a music composition that the MIDI Standard can reliably send to any notation software application is note placement (e.g. time and pitch) and duration, key, time signature, and tempo.
In response to the shortcomings and drawbacks of the MIDI Standard and its continuous controller codes (CC #s), there is a great need in the art to depart from conventions and create new methods and apparatus that will provide increased levels of control, quality, speed and performance desired in most musical production applications.
Also, there is a great need in the art to address and overcome the shortcomings and limitations of the outdated MIDI Standard while trying to meet the growing needs of an industry which is seeking to provide artificial intelligence (AI) based support in the field of musical composition, generation and performance, while overcoming the shortcomings and drawbacks of prior art methods and technologies.
Accordingly, a primary object of the present invention is to provide a new and improved automated method of and system for producing digital performances of musical compositions, however generated, using a new and improved virtual musical instrument (VMI) library management system that supports the automated playback of sampled notes and/or audio sounds produced by audio sampling, and/or synthesized sounds created by sound synthesis methods and not by audio sampling, and the automated selection of such notes and sounds for playback from such virtual musical instrument (VMI) libraries, using an automated selection and performance subsystem that employs ruled-based instrument performance logic to predict what samples should be performed based on the music-theoretic states of the music composition, while overcoming the shortcomings and drawbacks of prior art MIDI systems and methods.
Another object of the present invention is to provide a new level of artificial musical intelligence and awareness to automated music performance systems so that such machines demonstrate the capacity of appearing aware of (i) the virtual musical instrument types being used, (ii) the notes and sounds recorded or synthesized by each virtual musical instrument, and (iii) how to control those sampled and/or synthesized notes and audio sounds given all of the music-theoretic states contained in the music composition to be digitally performed by an ensembled of deeply-sampled virtual musical instruments automatically selected for music performance and production.
Another object of the present invention is to provide a new and improved method of producing a digital music performance comprising: (a) providing a music composition to an automated music performance system supporting virtual musical instrument (VMI) libraries provided with instrument performance logic; and (b) processing the music composition so as to automatically abstract music-theoretic state data for driving the automated music performance system and the instrument performance logic, including automated selection of instruments and sampled (and/or synthesized) notes and sounds from the VMI libraries so as to produce a digital music performance of the music composition.
Another object of the present invention is to provide a new and improved method of producing a digital music performance comprising: (a) providing a music sound recording to an automated music performance system supporting deeply-sampled virtual musical instrument (DS-VMI) libraries provided with instrument performance logic; and (b) processing the music sound recording so as to automatically abstract music-theoretic state data for driving the automated music performance system and the instrument performance logic, including automated selection of instruments and sampled and/or synthesized notes from the DS-VMI libraries so as to produce a digital music performance of the music performance recording.
Another object of the present invention is to provide a new and improved automated music performance system driven by music-theoretic state descriptors, including roles, notes and music metrics, automatically abstracted from a musical structure however composed or performed, for generating a unique digital performance of the musical structure, wherein the automated music performance system comprises: a plurality of deeply-sampled virtual musical instrument (DS-VSI) libraries, wherein each deeply-sampled virtual music instrument (DS-VMI) library supports a set of music-theoretic state (MTS) responsive performance rules automatically triggered by the music theoretic state descriptors, including roles, notes and music metrics, automatically abstracted from the music structure to be digitally performed by the automated music performance system; and an automated deeply-sampled virtual music instrument (DS-VMI) library selection and performance subsystem for managing the deeply-sampled virtual musical instrument (DS-VMI) libraries, including automated selection of virtual musical instruments and sampled and/or synthesized notes to be performed during a digital performance of said musical structure, in response to the abstracted music-theoretic state descriptors.
Another object of the present invention is to provide such an automated music performance system via the virtual musical instrument (VMI) libraries, which integrated with at least one of a digital audio workstation (DAW), a virtual studio technology (VST) plugin, a cloud-based information network, and an automated AI-driven music composition and generation system.
Another object of the present invention is to provide a new and improved automated music production system supporting a complete database of information on what sampled and/or synthesized notes and sounds are maintained and readily available in the system, and supported by an automated music performance system that is capable of automatically determining how the notes and sounds are accessed, tagged, and how they need to be triggered for final music assembly, based upon the full music-theoretic state of the music composition being digitally performed, characterized by the music-theoretic state data (i.e. music composition meta-data) transmitted with role, note, music metric and meta data to the automated music performance system, and by doing so, provide the system with the capacity to revival a human composer's ability to search, choose, and make artistic decisions on instrument articulations and sample libraries.
Another object of the present invention is to provide a new musical instrument sampling method and improved automated music performance system configured for audio sample playback using deeply-sampled virtual musical instruments (DS-VMIs), and/or digitally-synthesized virtual musical instruments (DS-VMI), that are controlled by performance logic responsive to the music-theoretic states of the music composition being digitally performed by the virtual musical instruments of the present invention, so as to produce musical sounds that are contextually-consistent with the actual music-theoretic states of music reflected in the music composition, and represented in the music-theoretic state descriptor data file automatically generated by the automated music performance system of the present invention to drive its operation on a music composition time-unit by time-unit basis.
Another object of the present invention is to provide a next generation automated music production system and method that supports a richer and more flexible system of music performance that enables better and higher-quality automated performances of virtual musical instrument libraries, not otherwise possible using conventional MIDI technologies.
Another object of the present invention is to provide a new method of producing a digital music performance based on a music composition or a music sound recording, processed to automatically abstract music-theoretic state data, and then provided to an automated music performance subsystem supporting libraries of deeply-sampled and/or digitally-synthesized virtual musical instrument (DS-VMI), capable of producing the notes and sounds for the digital music performance system.
Another object of the present invention is to provide an automated music performance system, wherein each deeply-sampled and/or digitally-synthesized virtual musical instrument libraries are maintained in a VMI library management subsystem that is provided with instrument performance logic (i.e. logical performance rules) based on a set of known standards for the corresponding (real) musical instrument, specifying what note performances are possible with each specific deeply-sampled and/or digitally-synthesized virtual musical instrument, so that the automated music performance subsystem can reliably notate the digital performance of a music composition prior to music production, and reliably perform the virtual musical instruments during the digital music performance of the music composition, with expression and vibrance beyond that achievable by conventional performance scripting technologies.
Another object of the present invention is to provide an automated music performance system, wherein for each deeply-sampled and/or digitally-synthesized virtual musical instrument library maintained in the system, its associated performance logic (i.e. performance rules), responsive to the music-theoretic state of the analyzed music composition, are programmed to fully capture what notes change with a dynamic shift, what articulation is intended, whether or not a specified note should be played/performed in a staccato or a pizzicato, and how the note samples should be triggered during final assembly given the music-theoretic state of the music composition being digitally performed by the deeply-sampled and/or digitally-synthesized virtual musical instruments.
Another object of the present invention is to provide a new and improved automated music production system, wherein a human being composes an orchestrated piece of music expressed in a music-theoretic (score) representation and provides the music composition to the automated musical performance system to digitally perform the music composition using an automated selection of one or more of the virtual musical instruments supported by the automated music performance system, controlled by the state-based performance logic created for each of the virtual musical instruments maintained in the automated music performance system, and responsive to role-organized note data abstracted from the music composition to be digitally performed.
Another object of the present invention is to provide a new and improved automated music performance system for generating digital performances of music compositions containing notes selected from virtual musical instrument (VMI) libraries based on the music-theoretic states of the music compositions being digitally performed.
Another object of the present invention is to provide a new and improved method of automatically selecting sampled notes from deeply-sampled and/or digitally-synthesized virtual musical instrument (DS-VMI) libraries using music theoretic-state descriptor data automatically abstracted from a music composition to be digitally performed, and processing selected notes using music-theoretic state responsive performance rules to produce the notes for the digital performance of the music composition.
Another object of the present invention is to provide a new and improved automated music performance system for producing a digital performance of a music composition using deeply-sampled virtual musical instrument (DS-VMI) libraries, from which sampled notes are predictively selected using timeline-indexed music-theoretic state descriptor data, including roles and music note metrics, automatically abstracted from the music composition.
Another object of the present invention is to provide a new and improved automated music composition and performance system and method employing deeply-sampled virtual musical instruments for producing digital music performances of music compositions using music-theoretic state descriptor data, including roles, notes and note metrics, automatically abstracted from the music compositions before automated generation of the digital performances.
Another object of the present invention is to provide a new and improved method of automatically generating digital music performances of music compositions using deeply-sampled and/or digitally-synthesized virtual musical instrument libraries supporting music-theoretic state responsive performance rules executed within an automated music performance and production system.
Another object of the present invention is to provide a new and improved predictive process for automatically selecting sampled notes from deeply-sampled virtual musical instrument (DS-VMI) libraries, and processing the selected sampled notes using performance logic, so as to produce sampled notes in a digital performance of a music composition that are musically consistent with the music-theoretic states of the music composition being digitally performed.
Another object of the present invention is to provide a new and improved system and process for automatically abstracting role, note, performance and other music-theoretic state data from along the timeline of a music composition to be digitally performed by an automated music performance system supported by deeply-sampled and/or digitally-synthesized virtual musical instrument (DS-VMI) libraries, and automatically producing music-theoretic state descriptor data characterizing the music composition for use in driving the automated music performance system.
Another object of the present invention is to provide new and improved methods of automatically processing music compositions in sheet music or MIDI-format and automatically producing digital music performances using an automated music performance system supporting deeply-sampled and/or digitally-synthesized virtual musical instrument (DS-VMI) libraries employing instrument performance logic triggered by music-theoretic state data abstracted from the music composition to be digitally performed, using abstracted roles as the logical linkage of such automated instrument performance.
Another object of the present invention is to provide a new and improved methods of automatically producing digital music performances based on music compositions, in either sheet music or MIDI-format, supplied to a cloud-based network via an application programming interface (API) to drive an automated music performance process.
Another object of the present invention is to provide a new and improved system for classifying and cataloging a group of real musical instruments, deeply sampling the real musical instrument, and naming and performing deeply-sampled virtual musical instrument (DS-VMI) libraries created for such deeply-sampled real musical instruments.
Another object of the present invention is to provide a new and improved an automated music performance system in the form of digital audio workstation (DAW) integrated with a deeply-sampled virtual musical instrument (DS-VMI) library management system for cataloging deeply-sampled virtual musical instrument (DS-VMI) libraries used to produce the sampled notes for a digital music performance of a music composition, and supporting logical performance rules for processing the sampled notes in a manner musically consistent with the music-theoretic states of the music composition being digitally performed.
Another object of the present invention is to provide a new and improved sound sampling and recording system employing sampling templates to produce a musical instrument data file for organizing and managing the sample notes recorded during an audio sampling and recording session involving the deep sampling and recording of a specified type of real musical instrument so as to produce a deeply-sampled virtual musical instrument (DS-VMI) library containing information items such as real instrument name, recording session, instrument type, and instrument behavior, and sampled notes performed with specified articulations and mapped to note/velocity/microphone/round-robin descriptors.
Another object of the present invention is to provide a new and improved deeply-sampled virtual music instrument (DS-VMI) library management system including data files storing sets of sampled notes performed by a specified type of real musical instrument deeply-sampled during an audio sampling session and mapped to note/velocity/microphone/round-robin descriptors, and supporting music-theoretic state responsive performance logic for processing the sampled notes that can be performed by the deeply-sampled virtual musical instrument.
Another object of the present invention is to provide a new and improved method of classifying deeply-sampled virtual musical instruments (DS-VMI) supported in a deeply-sampled virtual musical instrument (DS-VMI) library management subsystem using instrument definitions based on attributes including instrument types, instrument behaviors during performance, aspects (values), release types, offset values, microphone type, position and timbre tags used during recording.
Another object of the present invention is to provide a new and improved method of sampling, recording, and cataloging real musical instruments for use in developing corresponding deeply-sampled virtual musical instrument (DS-VMI) libraries for deployment in a deeply-sampled virtual musical instrument (DS-VMI) library management system.
Another object of the present invention is to provide a new and improved method of operating an automated music performance system employing a digital audio workstation (DAW) interfaced with a deeply-sampled virtual musical instrument (DS-VMI) library management subsystem controlled by an automated deeply-sampled virtual musical instrument (DS-VMI) library selection and performance subsystem.
Another object of the present invention is to provide a new and improved method of creating a deeply-sampled virtual musical instrument (DS-VMI) library using an instrument sampling template process.
Another object of the present invention is to provide a new and improved system for notating or documenting the digital performance of a music composition performed using a set of deeply-sampled virtual musical instrument (DS-VMI) libraries controlled using logical music performance rules operating upon sampled notes selected from the deeply-sampled virtual musical instrument (DS-VMI) libraries when the music-theoretic states determined in the music composition match conditions set in the logical music performance rules.
Another object of the present invention is to provide a new and improved automated music performance system, comprising: (i) a system user interface subsystem for a system user using a digital audio workstation (DAW) provided with music composition and notation software programs to produce a music composition to be digitally performed, and (ii) an automated music performance engine (AMPE) subsystem interfaced with the system user interface subsystem, for producing a digital performance based on the music composition, wherein the system user interface subsystem transfers a music composition to the automated music performance engine subsystem, wherein the automated music performance engine subsystem includes: (i) an automated music-theoretic state (MTS) data abstraction subsystem for automatically processing the music composition and abstracting all music-theoretic states contained in the music composition and producing a set of music-theoretic state descriptors data (i.e. notes, roles, metrics and meta-data) representative thereof; (ii) a deeply-sampled virtual musical instrument (DS-VMI) library management subsystem for managing the sample libraries supporting the deeply-sampled virtual musical instruments to be selected for performance of notes specified in the music composition; and (iii) an automated deeply-sampled virtual musical instrument (DS-VMI) selection and performance subsystem for selecting deeply-sampled virtual musical instruments in the DS-VMI library management subsystem and processing the sampled notes selected from selected deeply-sampled virtual musical instrument (DS-VMI) libraries using music-theoretic state (MTS) responsive performance rules (i.e. logic), to automatically produce the sampled notes selected for a digital performance of the music composition, and wherein the automated music performance engine (AMPE) subsystem transfers the digital performance to the system user interface subsystem for production, review and evaluation.
Another object of the present invention is to provide a new and improved automated music performance system supported by a hardware platform comprising various components including multi-core CPU, multi-core GPU, program memory (DRAM), video memory (VRAM), hard drive (SATA), LCD/touch-screen display panel, microphone/speaker, keyboard interface, WIFI/Bluetooth network adapters, and power supply and distribution circuitry, integrated around a system bus architecture.
Another object of the present invention is to provide a new and improved method of automated digital music performance generation using deeply-sampled virtual musical instrument (DS-VMI) libraries and contextually-aware (i.e. music state aware) performance logic supported in the automated music performance system.
Another object of the present invention is to provide a new and improved method of automated digital music performance generation using deeply-sampled virtual musical instrument (DS-VMI) libraries and contextually-aware (i.e. music state aware) performance logic supported in the automated music performance system, comprising the steps of: (a) selecting real musical instruments to be sampled, recorded, and catalogued for use in the deeply-sampled virtual musical instrument (DS-VMI) library management subsystem; (b) using an instrument type and behavior based schema (i.e. plan) for sampling, recording and cataloguing the selected real musical instruments in the virtual musical instrument sample library management system of present invention; (c) using the instrument type and behavior based schema to develop the action part of music-theoretic state (MTS) responsive performance rules for processing sampled notes in deeply-sampled virtual musical instrument (DS-VMI) libraries being managed in the library management system, during the automated music performance process; (d) loading the DS-VMI libraries and associated music-theoretic state (MTS) responsive performance rules into the automated performance system before the automated music performance generation process, (e) during a music composition process, producing and recording the musical notes in a composed piece of music; (f) providing the music composition to the automated music performance engine (AMPE) subsystem for automated processing and generating timeline-indexed music-theoretic state descriptor data (i.e. music composition meta-data) for the music composition, (g) providing the music-theoretic state descriptors (i.e. music composition meta-data) to the automated music performance engine (AMPE) subsystem for use in selecting sampled notes from deeply-sampled virtual musical instrument libraries maintained in DS-VMI library management system, and using music-theoretic state (MTS) responsive performance rules (i.e. logic) for processing the selected sampled notes to produce the notes of digital music performance of the music composition, (h) assembling and finalizing the processed sampled notes in the digital performance of the music composition, and (i) producing the performed notes of the digital performance of the music composition, for review and evaluation by human listeners.
Another object of the present invention is to provide a new and improved method of generating a digital performance of a composed piece of music (i.e. a musical composition) using the automated music composition and performance system, comprising the steps of (a) producing a digital representation of an automatically composed piece of music to be orchestrated and arranged for a digital performance using selected deeply-sampled and/or digitally-synthesized virtual musical instrument (DS-VMI) libraries performed using music-theoretic state (MTS) responsive performance rules, (b) automatically determining the music-theoretic states of music in a music composition along its timeline, and producing a set of timeline-indexed music-theoretic state descriptor data (i.e. roles, notes, metrics and meta-data) for use in the automated music performance system, (c) based on the roles abstracted from the music composition, selecting types of virtual musical instruments available for digital performance of the music composition in a deeply-sampled and/or digitally-synthesized virtual musical instrument (DS-VMI) library management system, (d) using the set of music theoretic-state meta-data descriptor data to automatically select notes from virtual musical instrument libraries, and using music-theoretic state responsive performance rules to process the selected notes to generate the processed notes for a digital performance of the music composition, (e) assembling and finalizing the processed selected notes in the generated digital performance of the music composition, and (f) producing the performed notes in the digital performance of the music composition, for review and evaluation by human listeners.
Another object of the present invention is to provide a new and improved process of automated selection of sampled notes in virtual musical instrument (VMI) libraries to produce the notes for the digital performance of a composed piece of music in accordance with the principles of the present invention, involving (a) the parsing and analyzing the music composition to abstract music-theoretic state descriptor data (i.e. role, notes, metrics and meta data), (b) formatting the music-theoretic state descriptor data (i.e. music composition meta-data) abstracted from the music composition, (c) using music-theoretic state descriptor data (i.e. music composition meta-data) to select notes from the virtual musical instrument (VMI) libraries and processing the selected notes using music-theoretic state (MTS) responsive performance logic maintained in the VMI library management subsystem, to produce the notes in the digital performance of the music composition, and (d) assembling and finalizing the processed notes for the digital performance of the music composition, for subsequent production, review and evaluation.
Another object of the present invention is to provide a new and improved method of automated selection and performance of notes in virtual instrument (VMI) libraries to generate a digital performance of a composed piece of music, comprising the steps of (a) capturing or producing a digital representation of a music composition to be orchestrated and arranged for a digital performance using a set of virtual musical instrument (VMI) libraries performed using music-theoretic state performance logic (i.e. rules) constructed and assigned to each virtual musical instrument (VMI), (b) automatically determining the music-theoretic states of music in a music composition along its timeline, and producing a set of timeline-indexed music-theoretic state descriptor data (i.e. roles, notes, metrics and meta-data) for use in the automated music performance system, (c) based on the roles abstracted from the music composition, selecting types of virtual musical instruments available for digital performance of the music composition in a virtual musical instrument (VMI) library management system, (d) using the set of music theoretic-state meta-data descriptor data to automatically select (e.g. filter, tag, and/or trigger) the notes from deeply-sampled virtual musical instrument libraries, and using music-theoretic state responsive performance rules to process the selected notes to generate the notes for a digital performance of the music composition, selected sampled notes to generate notes for a digital performance of the music composition, (e) assembling and finalizing the process notes in the digital performance of the music composition, and (f) producing the performed notes in the digital performance of the music composition, for review and evaluation by human listeners.
Another object of the present invention is to provide a new and improved automated music performance system comprising (i) a system user interface subsystem for a system user using digital audio workstation (DAW) supported by a keyboard and/or MIDI devices, to produce a music composition for digital performance, and (ii) an automated music performance engine (AMPE) subsystem interfaced with the system user interface subsystem, for producing a digital performance based on the music composition, wherein the system user interface subsystem transfers a music composition to the automated music performance engine, wherein the automated music performance engine includes (i) an automated music-theoretic state (MTS) data abstraction subsystem for automatically abstracting all music-theoretic states contained in the music composition and producing a set of music-theoretic state descriptors representative thereof, (ii) a deeply-sampled virtual musical instrument (DS-VMI) library management subsystem for managing deeply-sampled and/or digitally-synthesized virtual musical instruments to be selected for performance of notes specified in the music composition, and (iii) an automated virtual musical instrument selection and performance subsystem for selecting deeply-sampled and/or digitally-synthesized virtual musical instruments in the DS-VMI library management subsystem and performing notes from selected virtual musical instruments using music-theoretic state (MTS) responsive performance rules, to automatically produce a digital performance of the music composition, wherein the automated music performance engine (AMPE) subsystem transfers the digital performance to the system user interface subsystem for production, review and evaluation.
Another object of the present invention is to provide a new and improved method of automatically generating a digital performance of a music composition, comprising the steps of (a) selecting real musical instruments to be sampled, recorded, and catalogued for use in the deeply-sampled virtual musical instrument library management subsystem, (b) using an instrument type and behavior based schema (i.e. plan) for sampling, recording and cataloguing the selected real musical instruments in the virtual musical instrument sample library management system; (c) using the instrument-type and behavior based schema to develop the action part of music-theoretic state (MTS) responsive performance rules for processing sampled notes in virtual musical instrument sample libraries being managed in the library management system, during the automated music performance process, (d) loading the DS-VMI libraries and associated music-theoretic state (MTS) responsive performance rules into the automated performance system before the automated music performance generation process, (e) during a music composition process, producing and recording the musical notes in a music composition, (f) providing the music composition to the automated music performance engine (AMPE) subsystem and generating timeline-indexed music-theoretic state descriptor data (i.e. music composition meta-data) for the music composition, (g) providing the music-theoretic state descriptor data (i.e. music composition meta-data) to the automated music performance system to automatically select sampled notes from deeply-sampled virtual musical instrument libraries maintained in DS-VMI library management system, (h) using the music-theoretic state (MTS) responsive performance logic (i.e. rules) in the deeply-sampled virtual musical instrument libraries to process the selected sampled notes to produce the sampled notes of the digital music performance of the music composition, (i) assembling and finalizing the processed sampled notes in the digital performance of the composed piece of music, and (j) producing the performed notes of a digital performance of the composed piece of music for review and evaluation by human listeners.
Another object of the present invention is to provide a new and improved method of generating a digital performance of a composed piece of music (i.e. a musical composition) using the automated music composition and performance system, comprising the steps of (a) producing a digital representation of an automatically composed piece of music to be orchestrated and arranged for a digital performance using selected deeply-sampled virtual musical instruments performed using music-theoretic state (MTS) responsive performance rules, (b) automatically determining the music-theoretic states of music in a music composition along its timeline, and producing a set of timeline-indexed music-theoretic state descriptor data (i.e. roles, notes, metrics and meta-data) for use in the automated music performance system, (c) based on the roles abstracted from the music composition, selecting types of deeply-sampled virtual musical instruments available for digital performance of the music composition in a deeply-sampled virtual musical instrument (DS-VMI) library management system, (d) using the set of music theoretic-state meta-data descriptor data to automatically select sampled notes from deeply-sampled virtual musical instrument libraries, and using music-theoretic state responsive performance rules to process the selected sampled notes to generate the notes for a digital performance of the music composition, (e) assembling and finalizing the processed sampled notes in the generated digital performance of the music composition, and (f) producing the performed notes in the digital performance of the music composition, for review and evaluation by human listeners.
Another object of the present invention is to provide a new and improved process of automated selection of sampled notes in deeply-sampled virtual musical instrument (DS-VMI) libraries to produce the notes for the digital performance of a composed piece of music in accordance with the principles of the present invention, involving (a) the parsing and analyzing the music composition to abstract music-theoretic state descriptor data (i.e. notes, roles, metrics and meta data), (b) formatting the music-theoretic state descriptor data (i.e. music composition meta-data) abstracted from the music composition, (c) using music-theoretic state descriptor data to select sampled notes from deeply-sampled virtual musical instruments (DS-VMI) and processing the sampled notes using music-theoretic state (MTS) responsive performance logic maintained in the DS-VMI library management subsystem, to produce sampled notes in the digital performance of the music composition, and (d) assembling and finalizing the notes for the digital performance of the music composition, for subsequent production, review and evaluation.
Another object of the present invention is to provide a new and improved method of automated selection and performance of notes stored in deeply-sampled virtual music instrument (DS-VMI) libraries to generate a digital performance of a composed piece of music, comprising the steps of (a) capturing or producing a digital representation of a music composition to be orchestrated and arranged for a digital performance using a set of deeply-sampled virtual musical instruments performed using music-theoretic state performance logic (i.e. rules) constructed and assigned to each deeply-sampled virtual musical instrument (DS-VMI) library supporting its corresponding virtual musical instrument, (b) automatically determining the music-theoretic states of music in a music composition along its timeline, and producing a set of timeline-indexed music-theoretic state descriptor data (i.e. roles, notes, metrics and meta-data) for use in the automated music performance system, (c) based on the roles abstracted from the music composition, selecting types of deeply-sampled virtual musical instruments available for digital performance of the music composition in a deeply-sampled virtual musical instrument (DS-VMI) library management system, (d) using the set of music theoretic-state meta-data descriptor data to automatically select sampled notes from deeply-sampled virtual musical instrument libraries, and using music-theoretic state responsive performance rules to process the selected sampled notes to generate the notes for a digital performance of the music composition, (e) assembling and finalizing the notes in the digital performance of the music composition; and (f) producing the notes in the digital performance of the music composition, for review and evaluation by human listeners.
Another object of the present invention is to provide a new and improved automated music composition, performance and production system comprising (i) a system user interface subsystem for a system user to provide the emotion-type, style-type musical experience (MEX) descriptors and timing parameters for a piece of a music to be automatically composed, performed and produced, (ii) an automated music composition engine (AMCE) subsystem interfaced with the system user interface subsystem to receive MEX descriptors and timing parameters, and (ii) an automated music performance engine (AMPE) subsystem interfaced with the automated music composition engine subsystem and the system user interface subsystem, for automatically producing a digital performance based on the music composition produced by the automated music composition engine subsystem, wherein the automated music composition engine subsystem transfers a music composition to the automated music performance engine, wherein the automated music performance engine includes (i) an automated music-theoretic state (MTS) data abstraction subsystem for automatically abstracting all music-theoretic states contained in the music composition and producing a set of music-theoretic state descriptors representative thereof, (ii) a deeply-sampled virtual musical instrument (DS-VMI) library management subsystem for managing deeply-sampled virtual musical instruments to be selected for performance of notes specified in the music composition, and (iii) an automated deeply-sampled virtual musical instrument (DS-VMI) selection and performance subsystem for selecting deeply-sampled virtual musical instruments in the DS-VMI library management subsystem and performing notes from selected deeply-sampled virtual musical instruments using music-theoretic state (MTS) responsive performance rules, to automatically produce a digital performance of the music composition, and wherein the automated music performance engine (AMPE) subsystem ultimately transfers the digital performance to the system user interface subsystem for production, review and evaluation;
Another object of the present invention is to provide a new and improved enterprise-level internet-based music composition, performance and generation system supported by a data processing center with web servers, application servers and database (RDBMS) servers operably connected to the infrastructure of the Internet, and accessible by a network of web-enabled client machines, social network servers, and web-based communication servers, and allowing anyone with a web-based browser on a mobile computing device to access automated music composition, performance and generation services on websites to musically-score videos, images, slide-shows, podcasts, and other events with automatically composed, performed and produced music using deeply-sampled virtual musical instrument (DS-VMI) methods of the present invention as disclosed and taught herein.
Another object of the present invention is to provide a new and improved method of automated digital music performance generation using deeply-sampled virtual musical instrument (DS-VMI) libraries and contextually-aware (i.e. music state aware) driven performance principles practiced within an automated music composition, performance and production system, comprising the steps of: (a) selecting real musical instruments to be sampled, recorded, and catalogued for use in the deeply-sampled virtual musical instrument library management subsystem, (b) using an instrument type and behavior based schema (i.e. plan) for sampling, recording and cataloguing the selected real musical instruments in the virtual musical instrument sample library management system of present invention; (c) using the instrument-type and behavior based schema to develop the action part of music-theoretic state (MTS) responsive performance rules for processing sampled notes in the deeply-stored virtual musical instrument (DS-VMI) libraries being managed in the library management system, during the automated music performance process, (d) loading the DS-VML libraries and associated music-theoretic state (MTS) responsive performance rules into the automated performance system before the automated music performance generation process, (e) during an automated music composition process, the system user providing emotion and style type musical experience (MEX) descriptors and timing parameters to the system, then the system transforming MEX descriptors and timing parameters into a set of music-theoretic system operating parameters for use during the automated music composition and generation process, (f) providing the music-theoretic system operating parameters (MT-SOP descriptors) to the automated music composition engine (ACME) subsystem for use in automatically composing a music composition, (g) providing the music composition to the automated music performance engine (AMPE) subsystem and producing a timeline indexed music-theoretic state descriptors data (i.e. music composition meta-data), (h) the automated music performance engine (AMPE) subsystem using the music-theoretic state descriptor data to automatically select instrument types and sampled notes from deeply-sampled virtual musical instrument libraries, and using music-theoretic state descriptor responsive performance rules to process selected sampled notes, and generate the notes for the digital performance of the music composition, (i) assembling and finalizing the processed sampled notes in the digital performance of the music composition, and ( ) producing performed the notes of a digital performance of the music composition for review and evaluation by human listeners.
Another object of the present invention is to provide a new and improved method of generating a digital performance of a composed piece of music (i.e. a musical composition) using the automated music composition and performance system, comprising the steps of (a) producing a digital representation of an automatically composed piece of music to be orchestrated and arranged for a digital performance using selected deeply-sampled virtual musical instruments performed using music-theoretic state (MTS) responsive performance rules, (b) automatically determining the music-theoretic states of music in a music composition along its timeline, and producing a set of timeline-indexed music-theoretic state descriptor data (i.e. roles, notes, metrics and meta-data) for use in the automated music performance system, (c) based on the roles abstracted from the music composition, selecting types of deeply-sampled virtual musical instruments available for digital performance of the music composition in a deeply-sampled virtual musical instrument (DS-VMI) library management system, (d) using the set of music theoretic-state meta-data descriptor data to automatically select sampled notes from deeply-sampled virtual musical instrument libraries, and using music-theoretic state responsive performance rules to process the selected sampled notes to generate the notes for a digital performance of the music composition, (e) assembling and finalizing the processed sampled notes in the generated digital performance of the music composition, and (f) producing the performed notes in the digital performance of the music composition, for review and evaluation by human listeners.
Another object of the present invention is to provide a new and improved process of automated selection of sampled notes in deeply-sampled virtual musical instrument (DS-VMI) libraries to produce the notes for the digital performance of a music composition, comprising (a) the parsing and analyzing the music composition to abstract music-theoretic state descriptor data (i.e. music composition meta data), (b) formatting the music-theoretic state descriptor data (i.e. music composition meta-data) abstracted from the music composition, (c) using music-theoretic state descriptor data (i.e. music composition meta-data) to select sampled notes from deeply-sampled virtual musical instrument (DS-VMI) libraries and processing sampled notes using music-theoretic state (MTS) responsive performance logic maintained in the DS-VMI library management subsystem, to produce processed sampled notes in the digital performance of the music composition, and (d) assembling and finalizing the processed sampled notes for the digital performance of the music composition, for subsequent production, review and evaluation.
Another object of the present invention is to provide a new and improved method of automated selection and performance of notes in deeply-sampled virtual instrument libraries to generate a digital performance of a composed piece of music, comprising the steps of (a) capturing or producing a digital representation of a music composition to be orchestrated and arranged for a digital performance using a set of libraries of deeply-sampled and/or digitally-synthesized virtual musical instruments (DS-VMI) selected and performed using music-theoretic state performance logic (i.e. rules) constructed and assigned to each virtual musical instrument (VMI), (b) automatically determining the music-theoretic states of music in a music composition along its timeline, and producing a set of timeline-indexed music-theoretic state descriptor data (i.e. roles, notes, metrics and meta-data) for use in the automated music performance system, (c) based on the roles abstracted from the music composition, selecting types virtual musical instruments available for digital performance of the music composition in a virtual musical instrument (VMI) library management system, (d) using the set of music theoretic-state meta-data descriptor data to automatically select notes from virtual musical instrument libraries, and using music-theoretic state responsive performance rules to process the selected notes to generate the notes for a digital performance of the music composition, (e) assembling and finalizing the processed notes in the digital performance of the music composition; and (f) producing the performed notes in the digital performance of the music composition, for review and evaluation by human listeners.
Another object of the present invention is to provide a new and improved process of automatically abstracting the music-theoretic states as well as note data from a music composition to be digitally performed by an automated music performance system, and automatically producing music-theoretic state descriptor data (i.e. music composition meta-data) along the timeline of the music composition, for driving the automated music performance system to produce music that is contextually consistent with the music-theoretic states contained in the music composition.
Another object of the present invention is to provide a new and improved method of generating a set of music-theoretic state descriptors for a music composition, during the preprocessing state of an automated music performance process, wherein the exemplary set of music-theoretic state descriptors include, but are not limited to, MIDI Note Value (A1, B2, etc.), Duration of Notes, Position of Notes in a Measure, Position of Notes in a Phrase, Position of Notes in a Section, Position of Notes in a Chord, Note Modifiers (Accents), Dynamics, MIDI Note Value Precedence and Antecedence, What Instruments are Available, What Instruments are Playing, and What Instruments Should or Might Be Played, Position of Notes from Other Instruments, Relation of Sections to Each Other, Meter and Position of Downbeats and Beats, Tempo Based Rhythms, What Instruments are assigned to a role (e.g. play in background, play as a bed, play bass, etc.), and how many instruments are available.
Another object of the present invention is to provide a new and improved framework for classifying and cataloging a group of real musical instruments, and standardizing how such musical instruments are sampled, named and performed as virtual musical instruments during a digital performance of a piece of composed music, wherein musical instruments are classified by their performance behaviors, and musical instruments with common performance behaviors are classified under the same or common instrument type, thereby allowing like musical instruments to be organized and catalogued in the same class and be readily available for selection and use when the instrumentation and performance of a composed piece of music in being determined.
Another object of the present invention is to provide a new and improved catalog of deeply-sampled virtual musical instruments maintained in the deeply-sampled virtual musical instrument (DS-VMI) library management subsystem of the present invention.
Another object of the present invention is to provide a new and improved sampling template for organizing and managing an audio sampling and recording session involving the deep sampling of a specified type of real musical instrument to produce a deeply-sampled virtual musical instrument (DS-VMI) library, including information items such as real instrument name, instrument type, recording session—place, date, time, and people, categorizing essential attributes of each note sample to be captured from the real instrument or sample sound to be captured from an audio sound source during the sampling session, etc.
Another object of the present invention is to provide a new and improved musical instrument data file, structured using the sampling template of the present invention, and organizing and managing sample data recorded during an audio sampling and recording session involving the deep sampling of a specified type of real musical instrument to produce musical instrument data file for a deeply-sampled virtual musical instrument (DS-VMI) library.
Another object of the present invention is to provide a new and improved definition of a deeply-sampled virtual music instrument (DS-VMI) library according to the principles of the present invention, showing a virtual musical instrument data set containing (i) all data files for the sets of sampled notes performed by a specified type of real musical instrument deeply-sampled during an audio sampling session and mapped to note/velocity/microphone/round-robin descriptors, and (ii) MTS-responsive performance logic (i.e. performance rules) for use with samples in the deeply-sampled virtual musical instrument.
Another object of the present invention is to provide a new and improved music-theoretic state (MTS) responsive performance logic (i.e. set of logical performance rules) written to a specific deeply-sampled or digitally-synthesized virtual musical instrument (DS-VMI) library, for controlling specific types of performance for the virtual musical instruments supported in the deeply-sampled and/or digitally-synthesized virtual musical instrument (DS-VMI) library management subsystem of the present invention.
Another object of the present invention is to provide a new and improved classification scheme for deeply-sampled virtual musical instruments (DS-VMI) that are cataloged in the DS-VMI library management subsystem, using Instrument Definitions based on one or more of the following attributes: instrument behaviors during performance, aspects (Values), release types, offset values, microphone type, microphone position and timbre tags used during recording, and MTS responsive performance rules created for a given DS-VMI library.
Another object of the present invention is to provide a new and improved method of sampling, recording, and cataloging real musical instruments for use in developing corresponding deeply-sampled virtual musical instrument (DS-VMI) libraries for deployment in the deeply-sampled virtual musical instrument (DS-VMI) library management system of present invention, comprising (a) classifying the type of real musical instrument to be sampled and added to the sample virtual musical instrument library, (b) based on the instrument type, assigning a behavior and note range to the real musical instrument to be sampled, (c) based on behavior and note range, creating a sample instrument template for the real musical instrument to be sampled, indicating what notes to sample on the instrument based on its type, as well as a note range that is associated with the real instrument, (d) using the sample instrument template, sampling the real musical instrument and record all samples (e.g. sampled notes) and assign file names to each sample according to a naming structure, (e) cataloging the deeply-sampled virtual musical instrument in the DS-VMI library management system, (f) writing logical instrument contractor rules for each virtual musical instrument and groups of virtual musical instruments, specifying conditions under which the specified virtual musical instrument will be automatically selected and contracted to perform in the digital performance of a music composition, and (g) writing performance logic (i.e. performance rules) for each deeply-sampled virtual musical instrument, specifying the conditions under which specified sampled notes will be automatically and predictively selected from the deeply-sampled virtual musical instrument and used in the digital performance of a music composition.
Another object of the present invention is to provide a new and improved method of operation of the automated music performance system, comprising (a) the music composition meta-data abstraction subsystem automatically parsing and analyzing a music composition to be digitally performed so as to automatically abstract and produce a set of timeline indexed music-theoretic state descriptor data (i.e. music composition meta-data) specifying the music-theoretic states of the music composition, (b) automated deeply-sampled virtual musical instrument (DS-VMI) selection and performance subsystem uses the set of music-theoretic state descriptors (i.e. music composition meta-data) to (i) select sampled notes from deeply-sampled virtual musical instruments in the library subsystem, (ii) use the music-theoretic state (MTS) responsive performance logic to process sampled notes selected from DS-VMI libraries, and (iii) assemble and finalize the processed sampled notes selected for a digital performance of the music composition, and (c) the automated music performance system producing the performed notes selected for the digital performance of the music composition, for review and evaluation by human listeners.
Another object of the present invention is to teach a new method of creating new deeply-sampled virtual musical instrument (DS-VMI) libraries using a new instrument template process, wherein what articulations to record and how to tag and represent those recorded articulations are specified in great detail, better supporting the recording, cataloging, developing and defining the deeply-sampled virtual musical instruments according to the present Invention.
Another object of the present invention is to provide a novel system of virtual musical instrument performance logic supported by an automated performance system employing a set of deeply-sampled virtual musical instruments (DS-VMIs) developed to capture and express in the music performance logic (e.g. a set of logical music performance rules) which a e is used to operate the deeply-sampled virtual musical instruments to provide instrument performances that are contextually-aware and consistent with all or certain music-theoretic states contained in the music composition that is driving the musical instrumentation, orchestration and performance process.
Another object of the present invention is to provide a new and improved method of and system for automatically transforming the instrumental arrangement and/or performance style of a music composition during automated generation of digital performances of the music composition, using virtual musical instruments and sampled notes selected from deeply-sampled virtual musical instrument (DS-VMI) libraries, based on the music-theoretic states of the music composition being digitally performed.
Another object of the present invention is to provide a new and improved method of and system for automatically transforming the instrumental arrangement and/or performance style of a music composition to be digitally performed by providing instrumental arrangement and performance style descriptors to an automated music performance system supporting deeply-sampled virtual musical instrument (DS-VMI) libraries that produce sampled notes in a digital performance of the music composition.
Another object of the present invention is to provide a Web-based system and method that supports (i) Automated Musical (Re)Arrangement and (ii) Musical Instrument Performance Style Transformation of a music composition to be digitally performed, by way of (i) selecting Musical Arrangement Descriptors and Musical Instrument Performance Style Descriptors from a GUI-bases system user interface, (ii) providing the user-selected Musical Arrangement Descriptors and Musical Instrument Performance Style Descriptors to the automated music performance system, (iii) then remapping/editing the Musical Roles abstracted from the given music composition, and (iv) modifying the Musical Instrument Performance Logic supported in the DS-VMI Libraries, that is indexed/tagged with the Music Instrument Performance Style Descriptors selected by the system user.
Another object of the present invention is to provide a new and improved method of and system for automatically generating digital performances of music compositions or digital music recordings using deeply-sampled virtual musical instrument (DS-VMI) libraries driven by data automatically abstracted from the music compositions or digital music recordings.
Another object of the present invention is to provide a new and improved method of and system for automatically generating deeply-sampled virtual musical instrument (DS-VMI) libraries having artificial intelligence (AI) driven instrument selection and performance capabilities.
Another object of the present invention is to provide a new and improved deeply-sampled virtual musical instrument (DS-MI) library management system having artificial intelligence (AI) driven instrument performance capabilities and adapted for use with digital audio workstations (DAWs) and cloud-based information services.
These and other benefits and advantages to be gained by using the features of the present invention will become more apparent hereinafter and in the appended Claims to Invention.
The following Objects of the Present Invention will become more fully understood when read in conjunction of the Detailed Description of the Illustrative Embodiments, and the appended Drawings, wherein:
FIGS. 28B1 through 28B8 provide a set of exemplary rules for use during automated role assignment processes carried out by the system (i) when processing and evaluating a music composition (or recognized music recording), (ii) when selecting instrument types and sample instrument libraries, and (iii) when selecting and processing samples during instrument performances within the DS-VMI library subsystem, in accordance with the principles of the present invention;
FIGS. 33B1 through 33B8 provide tables describing a set of exemplary rules for use during automated role assignment processes carried out by the system (i) when processing and evaluating a music composition (or recognized music recording), (ii) when selecting instrument types and sample instrument libraries, and (iii) when selecting and processing samples during instrument performances within the DS-VMI library subsystem, in accordance with the principles of the present invention;
FIGS. 38B1 through 38B8 provide tables describing a set of exemplary rules for use during automated role assignment processes carried out by the system (i) when processing and evaluating a music composition (or recognized music recording), (ii) when selecting instrument types and sample instrument libraries, and (iii) when selecting and processing samples during instrument performances within the DS-VMI library subsystem, in accordance with the principles of the present invention;
Referring to the accompanying Drawings, like structures and elements shown throughout the figures thereof shall be indicated with like reference numerals.
The present Application relates to and improves upon Applicant's inventions disclosed prior US Patent Applications and US Granted Patents, specifically: co-pending patent application Ser. No. 16/253,854 filed Jan. 22, 2019; U.S. Pat. No. 10,163,429; and U.S. patent application Ser. No. 14/869,911 filed Sep. 29, 2015, now U.S. Pat. No. 9,721,551 granted on Apr. 1, 2017. Each of these US Patent Applications and US Patents are commonly and owned by Amper Music, Inc., and are incorporated herein by reference in their entirety as if fully set forth herein.
Articulations: Variants of ways of playing a note on an instrument, for example: violin sustained (played with a bow) vs violin pizzicato (played with fingers as a pluck) Descriptor: A Style and Mood pairing to reflect a specific type of music (Happy Classic Rock).
MIDI: Musical Instrument Digital Interface—universally accepted Standards format developed in 1983 to facilitate the communication between many different manufacturers of digital music instruments.
Mix: This is the processing of selecting and balancing microphones through various digital signal processes. This can include microphone position in a room and proximity to an instrument, microphone pickup patterns, outboard equipment (reverbs, compressors, etc.) and brand-type of microphones used.
Performance Notation System: The method of describing how musical notes are performed.
Round Robin: A set of samples that are recorded all at the same dynamic and same note. This provides some slight alterations to the sound so that in fast repetition the sound does not sound static and can provide a more realistic performance.
Sampling: The method of recording single performances (often single notes or strikes) from any instrument for the purposes of reconstructing that instrument for realistic playback.
Sample Instrument Library: A collection of samples assembled into virtual musical instrument(s) for organization and playback.
Sample Release Type: After a sample is triggered by a note-on event, a note-off event can trigger a sample to provide a more realistic “end” to a note. For example: Hitting a cymbal and then immediately muting it with the hand (also known as “choking”). There are three categories of Sample Releases: Short, a sample that triggers if a note-off event occurs before a given threshold; Long, a sample that triggers if a note-off event occurs after a given threshold (or no threshold).
Performance, an alternate performance of a Long or Short sample.
Sample Trigger Style: This is the type of sample that is to be played. One-Shot: A Sample that does not require a note-off event and will play its full amount whenever triggered (example: snare drum hit). Sustain: A sample that is looped and will play indefinitely until a note-off is given.
Legato: A special type of sample that contains a small performance from a starting note to a destination note.
Overview on the Automated Music Performance System of the Present Invention, and the Employment of its Automated Music Performance Engine in Diverse Applications
As shown and described through the present Patent Specification, a music composition, provided in either sheet music or MIDI-music format or by other means, is supplied by the system user as input through the system user input output (IO) interface, and used by the Automated Music Performance Engine Subsystem (AMPE) of the present invention, illustrated and described in great technical detail in
While the illustrative embodiments shown and described herein employ deeply-sampled virtual musical instruments (DS-VMI) containing data files representing notes and sounds produced by audio-sampling techniques described herein, it is understood that such notes and sounds can also be produced or created using digital sound synthesis and modeling methods supported by commercially available software tools including, but not limited to, MOTU® MX4 Synthesis Engine and/or MACHFIVE 3 software products, both by MOTU, Inc. of Cambridge, Mass.
In general, and preferably, the automated music performance system of the various illustrative embodiments of the present invention disclosed herein will be realized as an industrial-strength, carrier-class Internet-based network of object-oriented system design, deployed over a global data packet-switched communication network comprising numerous computing systems and networking components, as shown. The system user interface may be supported by a portable, mobile or desktop Web-based client computing system, while the other system components of the network are realized using a global information network architecture. Alternatively, the entire automated music performance system may be realized on a single portable or desktop computing system, as the application may require. In the case of using a global information network to deploy the automated music performance system, the information network of the present invention can be referred to as an Internet-based system network. The Internet-based system network can be implemented using any object-oriented integrated development environment (IDE) such as for example: the Java Platform, Enterprise Edition, or Java EE (formerly J2EE); IBM Websphere; Oracle Weblogic; a non-Java IDE such as Microsoft's NET IDE; or other suitably configured development and deployment environments well known in the art. Preferably, although not necessary, the entire system of the present invention would be designed according to object-oriented systems engineering (OOSE) methods using UML-based modeling tools such as ROSE by Rational Software, Inc. using an industry-standard Rational Unified Process (RUP) or Enterprise Unified Process (EUP), both well known in the art. Implementation programming languages can include C, Objective C, C, Java, PHP, Python, Haskell, and other computer programming languages known in the art. Preferably, the system network is deployed as a three-tier server architecture with a double-firewall, and appropriate network switching and routing technologies well known in the art. In some deployments, private/public/hybrid cloud service providers, such Amazon Web Services (AWS), may be used to deploy Kubernetes, an open-source software container/cluster management/orchestration system, for automating deployment, scaling, and management of containerized software applications, such as the enterprise-level applications, as described herein.
The innovative system architecture of the automated music performance system of the present invention is inspired by the co-inventors' real-world experience (i) composing musical scores for diverse kinds of media including movies, video-games and the like, (ii) performing music using real and virtual musical instruments of all kinds from around the world, and (iii) developing virtual musical instruments by sampling the sounds produced by real instruments, as well as natural and synthetic audio sound sources identified above, and also synthesizing digital notes and sounds using digital synthesis methods, to create the note/sound sample libraries that support such virtual musical instruments (VMIs) maintained in the automated music performance systems of the present invention.
As used herein, the term “virtual musical instrument (VMI)” refers to any sound producing instrument that is capable of producing a musical piece (i.e. a music composition) on a note-by-note and chord-by-chord basis, using (i) a sound sample library of digital audio sampled notes, chords and sequences of notes, recorded from real musical instruments or synthesized using digital sound synthesis methods described above, and/or (ii) a sound sample library of digital audio sounds generated from natural sources (e.g. wind, ocean waves, thunder, babbling brook, etc.) as well as human voices (singing or speaking) and animals producing natural sounds, and sampled and recorded using the sound/audio sampling techniques disclosed herein. Alternatively, such notes and sounds in a virtual musical instrument (VMI) can also be designed, created and produced using digital sound synthesis methods supported using modern sound synthesis software products including, but not limited to, MOTU MX4 and MACHFIVE software products, and the Synclavier® synthesizer systems from Synclavier Digital, and other note/sound design tools, well known in the art.
Notably, the methods of music note/sound sampling and synthesis used to building virtual musical instruments (VMIs) for use with the automated music performance system of the present invention, are fundamentally different from prior art loop synthesis methods where many loops, and tracks, of music are pre-recorded and stored in a memory storage device (e.g. a database) and subsequently accessed and combined together, to create a piece of music, and where there is no underlying music theoretic characterization/specification of the notes and chords in the components of music used in such prior art Loop synthesis methods.
In marked contrast, strict musical-theoretic specification of each musical event (e.g. note, chord, phrase, sub-phrase, rhythm, beat, measure, melody, and pitch) within a piece of music being automatically composed and generated by the system/machine of the present invention, must be maintained by the system during the entire music composition/generation process in order to practice the virtual music instrument (VMI) synthesis methods in accordance with the principles of the present invention.
The automated music performance system of the present invention is a complex system comprised of many subsystems, wherein advanced computational machinery is used to support highly specialized generative processes that support the automated music performance and production process of the present invention. Each of these components serves a vital role in a specific part of the automated music performance engine (AMPE) system of the present invention, and the combination of each component into the automated music composition and generation engine creates a value that is truly greater than the sum of any or all of its parts. A concise and detailed technical description of the structure and functional purpose of each of these subsystem components is provided hereinafter.
Regarding the overall timing and control of the subsystems within the system, reference should be made to flow chart set forth in
As shown in
However, it is to be pointed out at this juncture that various three alternative system architectures have been disclosed and taught herein to illustrate various ways of and means for supplying “music compositions” to the automated music performance engine subsystem of the present invention, for the purpose of automatically generating a nearly infinite variety of possible digital music performances, for each music composition supplied as input to the system via its system interface (e.g. API, GUI-based interface, XML, etc.).
The first illustrative embodiment teaches providing sheet-music type music compositions to the automated music performance system of the present invention, and supporting OCR/OMR software techniques to read graphically expressed music performance notation. The second illustrative embodiment teaches providing MIDI-type music compositions to the automated music performance system of the present invention. The third illustrative embodiment teaches providing music experience (MEX) descriptors to an automated music composition engine, and automatically processing the generated music composition to the automatically generate a digital music performance of the music composition. These three illustrative embodiments will be described in great technical detail hereinafter.
However, it should be pointed out that there are other sources for providing “music composition” input to the automated music performance engine system of the present invention, accessible of a local area network (LAN) or over a cloud-based wide area network (WAN) as the application may require. For example, a sound recording of a music composition performance can be supplied to an audio-processor programmed for automatically recognizing the notes performed in the performance and generating a music notation of the musical performance recording. Commercially available automatic music transcription software, such as AnthemScore by Lunsversus, Inc., can be adapted to support this illustrative embodiment of the present invention. The output of the automatic music transcription software system can be provided to the music composition pre-processor supported by the first illustrative embodiment of the present invention, to generate music-theoretic state descriptor data (including roles, notes, music metrics and meta data) that is then supplied to the automated music performance system of the present invention.
Alternatively, the music composition input can be a sound recording of a tune sung vocally, and this song can be audio-processed and transcribed into a music composition with notes and other performance notation. This music composition can be provided to the music composition pre-processor supported by the first illustrative embodiment of the present invention, to generate music-theoretic state descriptor data (including roles, notes, music metrics and meta data) that is then supplied to the automated music performance system of the present invention.
These and other methods of providing a piece of composed music, or performed music, to the automated music performance system of the present invention, will become more apparent hereinafter in view of the present invention disclosure and Claims to Invention appended hereto.
First Illustrative Embodiment of the Automated Music Performance System of the Present Invention, where a Human Composer Composes an Orchestrated “Music Composition” Expressed in a Sheet-Music Format Kind of Music-Theoretic Representation and Wherein the Music Composition is Provided to the Automated Musical Performance System of the Present Invention so that this System can Select Deeply-Sampled Virtual Musical Instruments Supported by the Automated Music Performance System Based on Roles Abstracted During Music Composition Processing, and Digitally Perform the Music Composition Using Automated Selection of Notes from Deeply-Sampled Virtual Musical Instrument Libraries
As shown in
As shown in
Specification of the First Illustrative Embodiment of the Automated Music Performance System of the Present Invention
In general, the automatic or automated music performance system shown in
For purpose of illustration, the digital circuitry implementation of the system is shown as an architecture of components configured around SOC or like digital integrated circuits. As shown, the system comprises the various components, comprising: SOC sub-architecture including a multi-core CPU, a multi-core GPU, program memory (DRAM), and a video memory (VRAM); a hard drive (SATA); a LCD/touch-screen display panel; a microphone/speaker; a keyboard; WIFI/Bluetooth network adapters; pitch recognition module/board; and power supply and distribution circuitry; all being integrated around a system bus architecture and supporting controller chips, as shown.
The primary function of the multi-core CPU is to carry out program instructions loaded into program memory (e.g. micro-code), while the multi-core GPU will typically receive and execute graphics instructions from the multi-core CPU, although it is possible for both the multi-core CPU and GPU to be realized as a hybrid multi-core CPU/GPU chip where both program and graphics instructions can be implemented within a single IC device, wherein both computing and graphics pipelines are supported, as well as interface circuitry for the LCD/touch-screen display panel, microphone/speaker, keyboard or keypad device, as well as WIFI/Bluetooth (BT) network adapters and the pitch recognition module/circuitry. The purpose of the LCD/touch-screen display panel, microphone/speaker, keyboard or keypad device, as well as WIFI/Bluetooth (BT) network adapters and the pitch recognition module/circuitry will be to support and implement the functions supported by the system interface subsystem, as well as other subsystems employed in the system.
Specification of the Automated Music Performance System of the Present Invention, and its Supporting Subsystems Including the Automated Music Performance Engine (AMPE) Subsystem
Specification of the Pitch Octave Generation Subsystem
A melody note octave table can be used in connection with the loaded set of notes to determines the frequency of each note based on its relationship to the other melodic notes and/or harmonic structures in a musical piece. In general, there can be anywhere from 0 to just-short-of infinite number of melody notes in a piece. The system automatically determines this number each music composition and generation cycle.
For example, for a note “C,” there might be a one third probability that the C is equivalent to the fourth C on a piano keyboard, a one third probability that the C is equivalent to the fifth C on a piano keyboard, or a one third probability that the C is equivalent to the fifth C on a piano keyboard.
The resulting frequencies of the pitches of notes and chords in the musical piece are used during the automated music performance process so as to generate a part of the piece of music being composed.
Specification of the Instrumentation Subsystem
This subsystem is supported by instrument tables indicating all possibilities of instruments, typically not probabilistic-based, but rather plain tables, providing an inventory of instrument options that may be selected by the system).
The parameter programming tables employed in the subsystem will used during the automated music performance process of the present invention. For example, if the music composition state data reflects a “Pop” style, the subsystem might load data sets including Piano, Acoustic Guitar, Electric Guitar, Drum Kit, Electric Bass, and/or Female Vocals.
The instruments and other musical sounds selected for the musical piece are used during the automated music performance process of the present invention so as to generate a part of the music composition being digitally performed.
Specification of the Instrument Selector Subsystem
The Instrument Selector Subsystem is supported by an instrument selection table, and parameter selection mechanisms (e.g. random number generator, or another parameter based parameter selector). Using the Instrument Selector Subsystem, instruments may be selected for each piece of music being composed, as follows. Each Instrument group in the instrument selection table has a specific probability of being selected to participate in the piece of music being composed, and these probabilities are independent from the other instrument groups. Within each instrument group, each style of instrument and each instrument has a specific probability of being selected to participate in the piece and these probabilities are independent from the other probabilities. As described herein, other methods of instrument selection may be used during the automated music composition performance process.
The instruments and other musical sounds selected by Instrument Selector Subsystem for the musical piece are used during the automated music performance process of the present invention so as to generate a part of the music composition being digitally performed using the DS-VMI library.
Specification of the Continuous Controller Processing Subsystem
In general, the Continuous Controller Processing Subsystem automatically determines the controller code and/or similar information of each note to be performed in the digital performance of a music composition, and the automated music performance engine will automatically process selected samples of note to carry out the processing instructions associated with the controller code data reflected in the music-theoretic state data file of the music composition. During operation, the controller code processing subsystem processes the “controller code” information for the notes and chords of the music composition being digitally performed by the DS-VMIs selected from the DS-VMI library management system. This information is based on either music composition inputs, computationally-determined value(s), or a combination of both.
The Continuous Controller Processing Subsystem is supported by controller code parameter tables, and parameter selection mechanisms (e.g. random number generator). The form of controller code data is typically given on a scale of 0-127, following the MIDI Standard. Volume (CC 7) of 0 means that there is minimum volume, whereas volume of 127 means that there is maximum volume. Pan (CC 10) of 0 means that the signal is panned hard left, 64 means center, and 127 means hard right.
Each instrument, instrument group, and music performance has specific instructions for different processing effects, controller code data, and/or other audio/MIDI manipulating tools being selected for use. With each of the selected manipulating tools, the controller code processing subsystem automatically determines: (i) how detected controller codes expressed in the input music composition will be performed on sampled notes selected from the DS-VMI libraries to affect and/or change the performance of notes in the musical piece, section, phrase, or other structure(s); and how to specifically process selected note samples from the DS-VMI libraries to carry out the controller code performance instructions reflected in the music composition, or more specifically, reflected in the music-theoretic state data file automatically generated for the music composition being digitally performed.
The Continuous Controller Processing Subsystem uses the instrument, instrument group and piece-wide controller code parameter tables and data sets loaded into the system. For example, instrument and piece-wise continuous controller code (CC) tables (i.e. containing performance rules) for the violin instrument has processing rules for controlling parameters such as: reverb; delay; panning; tremolo, etc. As described herein, other processing methods may be employed during the automated music composition performance process.
In general, controller code information expressed in any music composition informs how the music composition is intended to be performed or played during the digital music performance. For example, a piece of composed music orchestrated in a Rock style might have a heavy dose of delay and reverb, whereas a Vocalist might incorporate tremolo into the performance. However, the controller code information expressed in the music composition may be unrelated to the emotion and style characteristics of the music performance, and provided solely to effect timing requests. For example, if a music composition needs to accent a certain moment, regardless of the controller code information thus far, a change in the controller code information, such as moving from a consistent delay to no delay at all, might successfully accomplish this timing request, lending itself to a more musical orchestration in line with the user requests. As it is expected that Controller code will be used frequently in a MIDI-music representation of a music composition to be digitally performed, the Continuous Controller Processing Subsystem will be very useful in many digital music performances using the automated music performance system of the present invention. During operation of the Continuous Controller Processing Subsystem, any continuous controller (CC) code expressed in a music composition for instrumentation purposes will be automatically detected and processed on selected samples from the DS-VMI libraries during the automated music performance process, as described in greater detail hereinbelow.
Specification of the Deeply-Sampled Virtual Musical Instrument (DS-VMI) Library Management Subsystem (i.e. Digital Audio Sample Producing Subsystem) and its Use in the Automated Music Performance System
As shown in
In general, to generate music from any piece of composed music, the system needs musical instrument libraries for acoustically realizing the musical events (e.g. pitch events such as notes, rhythm events, and audio sounds) played by virtual instruments and audio sound sources specified in the musical score representation of the piece of composed music. There are many different techniques available for creating, designing and maintaining virtual music instrument libraries, and musical sound libraries, for use with the automated music composition and generation system of the present invention, namely: Digital Audio Sampling Synthesis Methods; Partial Timbre Synthesis Methods, Frequency Modulation (FM) Synthesis Methods; Methods of Sonic Reproduction; and other forms and techniques of Virtual Instrument Synthesis.
The preferred method, though not exclusive method, is the Digital Audio Sampling Synthesis Method which involves recording a sound source (such as a real instrument or other audio event) and organizing these samples in an intelligent manner for use in the system of the present invention. In particular, each audio sample contains a single note, or a chord, or a predefined set of notes. Each note, chord and/or predefined set of notes is recorded at a wide range of different volumes, different velocities, different articulations, and different effects, etc. so that a natural recording of every possible use case is captured and available in the sampled instrument library. Each recording is manipulated into a specific audio file format and named and tagged with meta-data with identifying information. Each recording is then saved and stored, preferably, in a database system maintained within or accessible by the automatic music composition and generation system. For example, on an acoustical piano with 88 keys (i.e. notes), it is not unexpected to have over 10,000 separate digital audio samples which, taken together, constitute the fully digitally-sampled piano instrument. During music production, these digitally sampled notes are accessed in real-time to generate the music composed by the system. Within the system of the present invention, these digital audio samples function as the digital audio files that are retrieved and organized by subsystems B33 and B34, as described in detail below.
Using the Partial Timbre Synthesis Method, popularized by New England Digital's SYNCLAVIER Partial-Timbre Music Synthesizer System in the 1980's, each note along the musical scale that might be played by any given instrument being model (for partial timbre synthesis library) is sampled, and its partial timbre components are stored in digital memory. Then during music production/generation, when the note is played along in a given octave, each partial timbre component is automatically read out from its partial timbre channel and added together, in an analog circuit, with all other channels to synthesize the musical note. The rate at which the partial timbre channels are read out and combined determines the pitch of the produced note. Partial timbre-synthesis techniques are taught in U.S. Pat. Nos. 4,554,855; 4,345,500; and 4,726,067, incorporated by reference.
Using state-of-the-art Virtual Instrument Synthesis Methods, such as supported by MOTU's MachFive 3 Universal Sampler and Virtual Music Instrument Design Tools, musicians can also use digital synthesis methods to design and create custom audio sound libraries for almost any virtual instrument, or sound source, real or imaginable, to support music performance and production in the systems of the present invention.
There are other techniques that have been developed for musical note and instrument synthesis, such as FM synthesis, and these technologies can be found employed in various commercial products for virtual instrument design and music production.
Specification of the Digital Audio Sample Retriever Subsystem
Specification of the Digital Audio Sample Organizer Subsystem
Specification of the Piece Consolidator Subsystem
Specification of the Piece Format Translator Subsystem
Specification of the Piece Deliver Subsystem
Specification of the Feedback Subsystem
In general, during system operation, the Feedback Subsystem allows for inputs ranging from very specific to very vague, and acts on this feedback accordingly. For example, a user might provide information, or the system might determine on its own accord, that the digital music performance should, for example: (i) include a specific musical instrument or instruments or audio sound sources supported in the DS-VMI libraries; (ii) use a particular performance style or method controlled by performance logic supported in the system; and/or (iii) reflect performance features desired by the or music producer or end-listener. This feedback can be provided through a previously populated list of feedback requests, or an open-ended feedback form, and can be accepted as any word, image, or other representation of the feedback.
As shown, the Feedback Subsystem receives various kinds of data which is autonomously analyzed by a Piece Feedback Analyzer supported within Subsystem. In general, the Piece Feedback Analyzer considers all available input, including, but not limited to, autonomous or artificially intelligent measures of quality and accuracy and human or human-assisted measures of quality and accuracy, and determines a suitable response to an analyzed music performance of a music composition. Data outputs from the Piece Feedback Analyzer can be limited to simple binary responses and can be complex, such as dynamic multi-variable and multi-state responses. The analyzer then determines how best to modify a music performance's rhythmic, harmonic, and other values based on these inputs and analyses. Using the system-feedback architecture of the present invention, the data in any music performance can be transformed after the creation of the music performance.
Preferably, the Feedback Subsystem is capable of performing Autonomous Confirmation Analysis, which is a quality assurance (QA)/self-checking process, whereby the system examines the digital performance of a music composition that was generated, compares the music performance against the original system inputs (i.e. input music composition and abstracted music-theoretic state data), and confirms that all attributes of the digital performance that were requested, have been successfully created and delivered in the music performance, and that the resultant digital performance is unique. This process is important to ensure that all music performances that are sent to a user are of sufficient quality and will match or surpass any user's performance expectations.
As shown, the Feedback Subsystem analyzes the digital audio file and additional performance formats to determine and confirm (i) that all attributes of the requested music performance are accurately delivered, (ii) that digital audio file and additional performance formats are analyzed to determine and confirm “uniqueness” of the musical performance, and (iii) the system user analyzes the audio file and/or additional performance formats, during the automated music performance process of the present invention. A unique music performance of a particular music composition is one that is different from all other music performance of the particular music composition. Uniqueness can be measured by comparing all attributes of a music performance to all attributes of all other music performances in search of an existing musical performance that nullifies the new performance's uniqueness.
If music performance uniqueness is not successfully confirmed, then the feedback subsystem modifies the inputted musical experience descriptors and/or subsystem music-theoretic parameters, and then restarts the automated music performance process to recreate the digital music performance. If musical performance uniqueness is successfully confirmed, then the feedback subsystem performs a User Confirmation Analysis, which is a feedback and editing process, whereby a user receives the music performance produced by the system and determines what to do next, for example: accept the current music performance; request a new music performance based on the same inputs; or request a new or modified music performance based on modified inputs. This is the point in the system's operation that allows for editability of a created music performance, equal to providing feedback to a human performer (or music conductor) and setting him/her off to enact the change requests.
Thereafter, the system user (e.g. human listener or automated machine analyzer) analyzes the audio file and/or additional performance formats and determines whether or not feedback is necessary. To perform this analysis, the system user can (i) listen to the music performance in part or in whole, (ii) view the music composition score file (represented with standard MIDI conventions) supporting the music performance, and/or (iii) interact with the music performance so that the user can fully experience the music performance and decide on how it might be changed in particular ways during the music performance regeneration process.
In the event that feedback is not determined to be necessary for a particular music performance, then the system user either (i) continues with the current music performance, or (ii) uses the exact same user-supplied music composition and associated parameters to create a new music performance for the music composition using the system. In the event that feedback is determined to be necessary, then the system user provides/supplied desired feedback to the system, and regenerates the music performance using the automated music performance system.
In the event the system users desires to provide feedback to the system via the GUI of the system interface subsystem, then a number of feedback options will be typically made available to the system user through a system menu supporting, for example, a set of pull-down menus designed to solicit user input in a simple and intuitive manner.
Specification of the Music Editability Subsystem
Specification of the Preference Saver Subsystem
Specification of the Method of Automated Digital Music Performance Generation Using Deeply-Sampled Virtual Musical Instrument Libraries and Contextually-Aware (I.E. Music State Aware) Performance Logic Supported in the Automated Music Performance System
Specification of the Method of Generating a Digital Performance of a Musical Composition Using the Automated Music Composition and Performance System
Specification of the Process of Automated Selection of Sampled Notes in Deeply-Sampled Virtual Musical Instrument (DS-VMI) Libraries to Produce the Sampled Notes for the Digital Performance of a Composed Piece of Music
Specification of the Method of Automated Selection and Performance of Notes in Deeply-Sampled Virtual Musical Instrument Libraries to Generate a Digital Performance of a Composed Piece of Music
As shown in Block A of
Suitable OCR/OMR-enabled commercial music score composition programs such as Sibelius Scorewriter Program by Sibelius Software Limited; Finale Music Notation and Scorewriter Software by MakeMusic, Inc.; MuseScore Composition and Notation Program by MuseScore BVBA www.musescore.org; and Capella Music Notation or Scorewriter Program by Capella Software AG; can be used to scan and read sheet music and generate an electronic file format that can be subsequently processed by the automated music performance system in accordance with the principles of the present invention disclosed and taught herein.
As shown in Block B of
As shown in Block C of
As shown in Block D of
As shown in Block E of
As shown in Block F of
As shown in Block G of
As shown in Block H of
As shown in Block I of
Music-Theoretic State Descriptors Automatically Evaluated by the Automated Music Performance System of the First Illustrative Embodiment During Automated Selection of Musical Instruments and Sampled Notes During Each Digital Performance
The function of DS-VMI behavior-sample selection/choosing supported by the automated DS-VMI Selection and Performance Subsystem shown in
When carrying out this automated data evaluation process, for the purpose of automatically selecting/choosing instrument types and sampled notes and appropriate sample note processing, the music-theoretic state data descriptor file schematically depicted in
As shown, the system comprises: (i) a system user interface subsystem for a system user using digital audio workstation (DAW) supported by a keyboard and/or MIDI devices, to produce a music composition for digital performance, and (ii) an automated music performance engine (AMPE) subsystem interfaced with the system user interface subsystem, for producing a digital performance based on the music composition. As shown, the system user interface subsystem transfers a music composition to the automated music performance engine. Also, the automated music performance engine includes: (i) an automated music-theoretic state (MTS) data abstraction subsystem for automatically abstracting all music-theoretic states contained in the music composition and producing a set of music-theoretic state descriptors representative thereof; (ii) a deeply-sampled virtual musical instrument (DS-VMI) library management subsystem for managing deeply-sampled virtual musical instruments to be selected for performance of notes specified for each Role in the music composition; and (iii) an automated deeply-sampled virtual musical instrument (DS-VMI) selection and performance subsystem for selecting deeply-sampled virtual musical instruments in the DS-VMI library management subsystem and performing for the Roles, notes from selected deeply-sampled virtual musical instruments using music-theoretic state (MTS) responsive performance rules, to automatically produce a digital performance of the music composition. As shown, the automated music performance engine (AMPE) subsystem transfers the digital performance to the system user interface subsystem for production, review and evaluation.
As shown in
Specification of the Second Illustrative Embodiment of the Automated Music Performance System of the Present Invention
In general, the automatic or automated music performance system shown in
For purpose of illustration, the automated music performance system comprises the various components, comprising: SOC sub-architecture including a multi-core CPU, a multi-core GPU, program memory (DRAM), and a video memory (VRAM); a hard drive (SATA); a LCD/touch-screen display panel; a microphone/speaker; a keyboard; WIFI/Bluetooth network adapters; pitch recognition module/board; and power supply and distribution circuitry; all being integrated around a system bus architecture and supporting controller chips, as shown.
The primary function of the multi-core CPU is to carry out program instructions loaded into program memory (e.g. micro-code), while the multi-core GPU will typically receive and execute graphics instructions from the multi-core CPU, although it is possible for both the multi-core CPU and GPU to be realized as a hybrid multi-core CPU/GPU chip where both program and graphics instructions can be implemented within a single IC device, wherein both computing and graphics pipelines are supported, as well as interface circuitry for the LCD/touch-screen display panel, microphone/speaker, keyboard or keypad device, as well as WIFI/Bluetooth (BT) network adapters and the pitch recognition module/circuitry. The purpose of the LCD/touch-screen display panel, microphone/speaker, keyboard or keypad device, as well as WIFI/Bluetooth (BT) network adapters and the pitch recognition module/circuitry will be to support and implement the functions supported by the system interface subsystem, as well as other subsystems employed in the system.
Specification of the Method of Automatically Generating a Digital Performance of a Music Composition
Specification of the Digital Performance of a Composed Piece of Music (I.E. A Musical Composition) Using the Automated Music Composition and Performance System
Specification of the Method of Automated Selection and Performance of Notes in Deeply-Sampled Virtual Instrument Libraries to Generate a Digital Performance of a Composed Piece of Music
As shown in Block A of
As shown in Block B of
As shown in Block C of
As shown in Block D of
As shown in Block E of
As shown in Block F of
As shown in Block G of
As shown in Block H of
As shown in Block I of
The function of DS-VMI behavior-sample choosing supported by the automated DS-VMI Selection and Performance Subsystem shown in
When carrying out this automated data evaluation process, for the purpose of automatically selecting/choosing instrument types and sampled notes and appropriate sample note processing, the music-theoretic state data descriptor file schematically depicted in
As shown in
The automated music composition engine subsystem transfers a music composition to the automated music performance engine. The automated music performance engine includes: (i) an automated music-theoretic state (MTS) data abstraction subsystem for automatically abstracting all music-theoretic states contained in the music composition and producing a set of music-theoretic state descriptors representative thereof; (ii) a deeply-sampled virtual musical instrument (DS-VMI) library management subsystem for managing deeply-sampled virtual musical instruments to be selected for performance of notes specified in the music composition; and (iii) an automated deeply-sampled virtual musical instrument (DS-VMI) selection and performance subsystem for selecting deeply-sampled virtual musical instruments in the DS-VMI library management subsystem and performing notes from selected deeply-sampled virtual musical instruments using music-theoretic state (MTS) responsive performance rules, to automatically produce a digital performance of the music composition. The automated music performance engine (AMPE) subsystem ultimately transfers the digital performance to the system user interface subsystem for production, review and evaluation.
In
Specification of the Third Illustrative Embodiment of the Automated Music Production System of the Present Invention
Specification of the Method of Automated Digital Music Performance Generation Using Deeply-Sampled Virtual Musical Instrument Libraries and Contextually-Aware (I.E. Music State Aware) Driven Performance Principles
Specification for the Method of Generating a Digital Performance of a Musical Composition Using the Automated Music Composition and Performance System
Specification of the Process of Automated Selection of Sampled Notes in Deeply-Sampled Virtual Musical Instrument (DS-VMI) Libraries to Produce the Notes for the Digital Performance of a Music Composition
Specification of the Method of Automated Selection and Performance of Notes in Deeply-Sampled Virtual Instrument Libraries to Generate a Digital Performance of a Music Composition
As shown at Block A in
As shown at Block B in
As shown at Block C in
As shown at Block D in
As shown at Block E in
As shown at Block F in
As shown at Block G in
As shown at Block H in
Music-Theoretic State Descriptors Automatically Evaluated by the Automated Music Performance System of the Third Illustrative Embodiment During Automated Selection of Musical Instruments and Sampled Notes During Each Digital Performance
The function of DS-VMI behavior-sample choosing supported by the automated DS-VMI Selection and Performance Subsystem shown in
When carrying out this automated data evaluation process, for the purpose of automatically selecting/choosing instrument types and sampled notes and appropriate sample note processing, the music-theoretic state data descriptor file schematically depicted in
The steps involved in this process will depend on the particular format of the music composition requiring automated music-theoretic note and state analysis, as taught herein and illustrated throughout the Drawings and Specification. In the three illustrative system embodiments, slightly different methods will be employed to accommodate the different formats of music composition under automated analysis. However, each music composition under automated analysis will typically employ similar methods to automatically abstract time-indexed note data, music metrics, and meta-data contained in the music composition, all of which is preferably organized under abstracted Musical Roles (or Parts) to be performed by selected Virtual Musical Instruments (or MIDI-controlled Real Musical Instruments MIDI-RMI) during an automated digital music performance of the analyzed music composition. The details of each of these music composition analysis methods, constructed in accordance with the illustrative embodiments of the present invention, will be described in detail below.
Method of Generating a Music-Theoretic State Descriptor Representation for a Sheet-Type Music Composition to be Used in the Automated Music Performance System During Selection, Assembling and Performance of Sampled Notes from Deeply-Sampled Virtual Musical Instruments Supported by the Automated Music Performance System of the Present Invention
As shown at Block A in
At Block B in
In
As shown at Block C in
In accordance with the principles of the present invention, all deeply-sampled virtual musical instruments in the DS-VMI library management subsystem are provided with some level of intelligent performance control via coding (aka sample selection/playback) written as performance logic for them, whether it's a simple “Hit” of a Snare or the complex “Strum” of a Guitar. The performance dictates what sample to trigger and how to trigger with note, velocity, manual, automation, and articulation information. To reiterate: the composer writes out the notes in the music composition, and when to play those notes, and then the automated music performance system adapts those notes on how to playback those samples.
In order to make a deeply-sampled virtual musical instrument (DS-VMI) sound realistic, the automated music performance system does not need to interpret a direct note-for-note playback, but rather is capable of calling many instruments and extrapolating sampled note information to choose a correct sample playback at any instant in time. For instance, if the composer says play a G chord on a downbeat, then a possible performance written for the guitar might be to build the chord as “E1 String: 3rd fret (G), A1 String: 2nd Fret (B), D2 String: Open (D), G2 String: Open (G), B2 String: Open (B), E3 String: 3rd fret (G)”. Then select the samples based on each separately recorded string, set a velocity range to trigger in, modifier type: Open (choices: Mute, Dead or Open) which strum direction: Down (choices: up or down). The samples would be requested and played in the audio engine, and then a timing differential would be added between each string to make the performance of the chord occur on the downbeat. If there was a G on the next beat, then the automated music performance system would play the same configuration, but may choose the up sample sets and start from the top E3 string, down through to the E1 string.
Each virtual musical instrument in the automated music performance system has a specific instrument performance logic (i.e. “a Performance) based on its parent template. Notably, Performances are actually set to Instruments specifically, and can be applied in batches based on their template/instrument_type association.
Some musical instruments could have one performance assigned, or have hundreds of performances assigned. Each performance adds dimensionality to an instrument's capabilities. The automated music performance system of the present invention interprets what the music composition contains in terms of its full music-theoretic states of music, along its entire timeline. The music composition contains chords and/or notes with timing information in them. To support its contextual-awareness capacities, the automated music performance system includes automated music-theoretic state abstraction processing algorithm(s) which automatically analyze what those notes are in the music composition to be digitally performed, and formulas can be used around these notes to help determine a playback scheme for triggering the samples through an audio mixing engine supported within the automated music performance system.
As will be described in greater detail hereinafter, a human or machine composer transmits a music composition to be digitally performed to the automated music performance system of the present invention. The music composition containing note data is automatically analyzed by the system to generate music-theoretic state data (i.e. music composition meta-data) such as: roles, note data, music metric data such as the position of notes within a song structure (chorus, verse, etc.), mode/key, chords, notes and their position within a measure, how long the notes are held (note duration) and when they are performed, and other forms of music composition meta-data.
By analyzing the music composition, the automated music performance system automatically abstracts and organizes collected note data, music metric data, music composition meta-data within an enveloped assigned an abstracted Musical Role (or Part), so as to inform the automated music performance system of the notes and possible music-theoretic states contained in the music composition such as, for example:
Duration of Notes:
The performance tool can isolate where items are within 3 levels of granularity. Measure, Phrase, and Section. The composer creates music measure by measure, assembles those measures into phrases and then the phrases belong to sections. The performance system uses the positions of notes to determine a velocity, articulation choice, or a manual switch. These are chosen through deterministic, stochastic, or purely random methods/algorithms.
Position of Notes in a Chord:
When a composer sends out a chord to a specific deeply-sampled virtual musical instrument, the automated music performance system can isolate a note performance based on what notes can be assigned to the deeply-sampled virtual musical instrument. Understanding the note relationship within the chord allows the automated music performance system, with its music-theoretic state responsive performance rules, to automatically process and change specific tuning to a sample, a velocity change, how the chord should be voiced, which string to play, or even what note in the chord to play (if it's a monophonic virtual musical instrument). Assigned Instrument Roles can help orchestration decisions.
Note Modifiers:
Accents: This is an extra layer of data that is written from the composer to unify a layer of accents (or strong-beat) control that allow for sample selection on quick dynamic changes (on single beats). For example switching from a regular stick-hit on a snare to a rim-shot, or changing the velocity of a piano from mf to ff.
Dynamics:
Dynamics, with regards to sample selection, can request playback, where to blend the two sample sets together, as well as different recorded sampled manuals. For example: Violins at ppp may select a “con sordino” (or with-mute) sample set. Or when moving from pp to p on a piano —start blending two samples together to create the timbral shift. Dynamics can also inform control data explained further below.
Note Value Precedence and Antecedence:
Equipped with music-theoretic state descriptor data streams and logical performance rules assigned to deeply-sampled virtual musical instruments libraries, the automated music performance system of the present invention is provided with an artificial intelligence and awareness of notes that come before and come after any given note along the timeline of a music composition being digitally performed using the DS-VMI libraries. This capacity helps inform the automated music performance system when to switch between articulations of sampled notes, as well as when to use legato, perform a note-off release, then a note on (repeated round robin), or when to choose a transition effect. For example, moving from a higher hand-shape on a guitar to a lower hand shape, the automated music performance system can then insert the transition effect of “finger noise-down by middle distance.”
Instrumentation Awareness:
Role: When an instrument is assigned into a Role, this allows for other instruments to know that instruments importance, where it fits within the structure of note assignments, performance assignment, and what sample sets that should be chosen. For example, if a string part is assigned a fast, articulated performance, the sample set chosen would be short note recordings. Examples of Roles are specified in
Availability: Knowing what instruments are available as to assign instrument performances to more valuable and important roles. For example, when two guitars are assigned, one takes a lead, mono role, while the other supports rhythm. When only one guitar is selected, which role is more important and move to that role (or move between the two based on material type (song structure location).
What is playing: This is being aware of which other instruments are playing which Roles helps to determine range, volume and activity level assigned to an instrument performance.
Physical Limitations: “The four handed drummer” problem—Limiting the voices based on physical constraints of the instrument, while allowing the users to select more than one-type of item. For example, if there are 4 cymbals, 1 hi-hat, 1 snare, and 4 toms and a kick: if there is a fill in a drum part, don't play more than 2 “hand hits” and 2 “foot hits” at one time.
Position of Notes from other Notes: This allows for complicated and orchestration decisions based on available notes, what other instruments are playing those notes, position in Role type and importance of that Role.
Structural Awareness:
Relation of Sections: Similar to how notes within a section are selected, knowing which sections have happened and what permutation of a section you are in can inform sample changes, such as dynamic shifts, moving from a type of articulation to another type. For example, switching from the first verse to the second, you would have the piano play “pedal” on the first verse, but maybe be drier or heavier on the second verse and play “regular” or “without pedal”.
Meter and position of Downbeats and Beats: Similar to how samples are selected from an accent lane, knowing what meter, where the strong vs weak beats are and the relation with in a part of a phrase will determine what sample could be selected.
Tempo:
Having knowledge of the music-theoretic parameter, Tempo, of the music composition can enable the automated music performance system of the present invention to automatically switch sample sets that are based on length or agility. Knowledge of Tempo can also help determine note cut-off and secondary note cut-off performances.
Each instrument assigned to a Role abstracted from the music composition to be digitally performed becomes an “instrument assignment.” This assignment is then given a mixing algorithm with a set of controllable DSPs (from volume to filters, reverb, etc.). These algorithms are written with the same parameters as the sample selections—but happen on an “instrument assignment” (also known as a “instrument_type”) level, not on the specific sample set or instruments. The instrument assignment becomes an audio bus, which allows for any specific instrument, within the assignment constraints, to be swapped out with a similar instrument type. For example, when a grand piano is being used and the user wants to swap it out with an upright piano, that assignment would stay the same—using all the same DSP and mixing algorithms. Finally all these assignments (that have become busses) are assigned to a master mixing bus and are delivered to users as either stems (each buss individually) or a master track.
As shown in
Specification of Role Assignment Rules/Principles of the Present Invention
FIGS. 28B1 through 28B8 provide a set of exemplary Rules for use during the automated role assignment processes carried out by within the automated music performance system of the first illustrative embodiment of the present invention.
The following describes an exemplary way of assigning Roles to Notes, roles to Instrument Types and roles to Instrument Performance logic (i.e. Role Assignments) across the various stages of the automated music performance system of the present invention.
Roles are a way of organizing notes along a timeline that are sent to assigned Instrument Types to be handled by the Instrument Performance Logic which will select the correct samples for playback in the production of a musical piece.
Instruments and Performance Logic (Rules) are all labeled (tagged) with data that allow for rulesets to choose the appropriate Instrument/Performance combination.
With all three types of input (e.g. OCM, MIDI, AMPER), the following rules can be applied to an assigned Instrument Type, and then a specific Sample Instrument would be assigned to the Role Assignment. Each of these Roles can have a many variants of a role, if multiple roles of similar type are needed (e.g. accent.a, accent.b, or accent.1, accent.2, etc.).
Accent Role:
1. Role-to-Note Assignment Rule: If the density of notes are fairly sparse and follow along a consistent strong beat to weak beat periodicity, or/and if several instrument parts have regular periodicity in strong beat groupings, then assign to the Accent Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Accent Role to instrument_types reserved for accents, which are typically percussive, (e.g. “.hit( )” aspect value of: aux_perc, big_hit, cymbal, etc.).
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign the Accent Role to Instrument Performance Logic of other roles by through change in velocity or not play/play notes in current assigned role (ex: augmenting role).
Back Beat Role:
1. Role-to-Note Assignment Rule: If notes have a periodicity of primarily weak beat and that are tonal, then assign to the Back Beat Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role Instrument Types that provide a more rhythmic and percussive tonal performance (mono or polyphonic)
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign this Role to Instrument Performances (i.e. Instrument Performance Logic/Rules) such as, e.g. acoustic_piano with “triadic chords closed voicing”, acoustic_guitar with “up-strum top three strings”, etc.
Background Role:
1. Role-to-Note Assignment Rule: If notes have a medium-low density (playing slightly more than once per chord, polytonal), then assign the Background Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Background Role to instrument types that can support polyphonic (note) performances.
3. Role-to-Performance-Logic Assignment Rule: Play polyphonic chords or parts of chords in instrument types (e.g. keyboard, acoustic_piano, synth_strings, etc.)
Big Hit Role:
1. Role-to-Note Assignment Rule: If notes happen with extreme irregularity and are very sparse, and/or either fall with a note in the accent lane or outside of any time signature, then assign the Big Hit Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this role instrument type primarily to single hit, non-tonal, percussive instruments
3. Role-to-Instrument-Performance-Logic Assignment Rule: Performance Logic is play a “hit( )” in the assigned instrument type (e.g. big_hit, bass_drum, etc.)
Color Role:
1. Role-to-Note Assignment Rule: If notes happen in small clusters, with rests between each set of clusters, and have some regular periodicity less than once per phrase, then assign the Color role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Color Role to instrument types that are either percussive (if notes are unpitched) or tonal (if notes follow pitches within the key) and can be assigned to typically monophonic or instruments with harmonic/rhythmic tags (e.g. instruments with delay tag, tickies, synth_lead, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: May assign performances that are softer in velocity or lighter in articulation attack.
Consistent Role:
1. Role-to-Note Assignment Rule: If notes are relatively dense, have some periodicity, and change in either note pattern organization, rhythmic pattern organization more than once per a few bars, then assign the Consistent Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Color Role to instrument types that have typically monophonic performances (e.g. synth_lead, guitar_lead).
3. Role-to-Instrument-Performance-Logic Assignment Rule: May assign performances that utilize various arpeggiation patterns (e.g. line up/down, sawtooth, etc.)
Constant Role:
1. Role-to-Note Assignment Rule: If notes that are relatively dense, and have very static rhythmic information with periodicity, then assign the Constant Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Constant Role to either tonal (monophonic) or percussive instrument types.
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign the Constant Role to Instrument Performances that are tempo dependent (e.g. shaker with “front( )” only, synth_lead with “arpeggiation up”, etc.).
Decoration Role:
1. Role-to-Note Assignment Rule: If notes happen in small clusters, with rests between each set of clusters, and occur one per phrase or longer, then assign the Decoration role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Decoration Role to instrument types that are either percussive (if notes are unpitched) or tonal (if notes follow pitches within the key) and can be assigned to typically monophonic or instruments with harmonic/rhythmic tags (e.g. instruments with delay tag, tickies, synth_lead, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: May assign performances that are softer in velocity or lighter in articulation attack.
High Lane Role:
1. Role-to-Note Assignment Rule: If notes that are unpitched that happen in usually rapid succession or high density, then assign the High Lane Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Unless otherwise directed by an external input, usually assign the High Lane Role to high-pitched in timbre percussion instruments (e.g. tickies, shakers, aux_drum (“rim”), etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: Choose performances that activate articulations that are tagged with “high” and/or “short”
High-Mid Lane Role:
1. Role-to-Note Assignment Rule: If notes that are unpitched that happen in medium-high density, then assign the High-Mid Lane Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Unless otherwise directed by an external input, usually assign this Role to high-pitched or medium in timbre percussion instruments (e.g. tickies, aux_drum, hand_drum, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule:
Choose performances that activate articulations that are tagged with “middle” and/or “short/medium”
Low-Mid Lane Role:
1. Role-to-Note Assignment Rule: If notes that are unpitched that happen in medium-low density, then assign the Low-Mid Lane Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Unless otherwise directed by an external input, usually assign this Role to medium to medium-low in timbre percussion instruments (e.g. aux_drum, hand_drum, taiko, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule:
Choose performances that activate articulations that are tagged with “middle,” “low,” and/or “medium/long”
Low Lane Role:
1. Role-to-Note Assignment Rule: Assign the Low Lane Role to notes that are unpitched that happen in low density.
2. Role-to-Instrument-Type Assignment Rule: Unless otherwise directed by an external input, assign this Role usually to instrument types that are low in timbre percussion (e.g. bass_drum, surdo, taiko, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule:
Choose performances that activate articulations that are tagged with “low,” and/or “long”
Middle Role:
1. Role-to-Note Assignment Rule: If notes have a medium density (playing more than once per chord, polytonal, with occasional running lines), then assign the Middle Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Typically assign this Role to instrument types that can support polyphonic playback and performance. (e.g. keyboard, acoustic_piano, synth_strings, violins, etc.).
3. Role-to-Instrument-Performance-Logic Assignment Rule: Typically assign this Role to Instrument Performances support polyphonic performance. (e.g. keyboard, acoustic_piano, synth_strings, violins, etc.).
On Beat Role:
1. Role-to-Note Assignment Rule: If notes have a periodicity of primarily strong beat and that are tonal, then assign the On Beat Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role to instrument types that produce more rhythmic and percussive tonal performances (mono or polyphonic)
Role-to-Instrument-Performance-Logic Assignment Rule: Assign this Role to Instrument
3. Performances that have “strong” performance tag association (eg: acoustic_bass “roots with 5ths”, acoustic_guitar with “down-strum power chord”, etc.)
Pad Role:
1. Role-to-Note Assignment Rule: If notes are sustained through the duration of a chord, then assign the Pad Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Typically assign this Role to polyphonic instrument types that sustain notes (e.g. mid_pad, synth_string, synth_bass)
3. Role-to-Instrument-Performance-Logic Assignment Rule: Typically assign this Role to Instrument Performances involving polyphonic performances that sustain notes during a chord, and change notes on chord change (e.g. mid_pad, synth_string, synth_bass)
Pedal Role:
1. Role-to-Note Assignment Rule: If notes sustain through chords and stay on one pitch (often the root) of an entire phrase, then assign the Pedal Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role typically to monophonic instrument types such as, e.g. low_pad, synth_bass, pulse, etc.
Role-to-Instrument-Performance-Logic Assignment Rule: Assign this Role Typically to Instrument
3. Performances supporting monophonic performances that either sustain indefinitely, or can quickly reattack consecutively to create a pulse-like pedal tone (e.g. low_pad, synth_bass, pulse, etc.) Primary Role:
1. Role-to-Note Assignment Rule: If notes are mostly monophonic, played with more density, rhythmic structure variances, often with some repetition and periodicity, and are accompanied with either great dynamic markings, high velocities, and/or other indications depending on the medium read, then assign the Primary Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role typically to instrument types often used to perform as lead instruments (e.g. violin, lead_synth, lead_guitar, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: May choose limited polyphonic or monophonic performance, that may utilize a great amount of articulation control and switching.
Secondary Role:
1. Role-to-Note Assignment Rule: If notes are mostly monophonic, play with more density, rhythmic structure variances, often with some repetition and periodicity, and are accompanied with either great dynamic markings, high velocities, are either lower in pitch or play less dense then another part, and/or other indications depending on the medium read, then assign the Secondary Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role to instrument types that often perform as lead instruments (e.g. violin, lead_synth, lead_guitar, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: May choose limited polyphonic or monophonic performance, that may utilizes a great amount of articulation control and switching. Drum Set Roles: These are the roles listed below that are given specific rhythmic parts (non-tonal) that should be assigned to one role-performer, but have to be broken out because the instruments used are naturally separated. Notes will need to be parsed into different roles, and often can be determined by MIDI note pitch, staff position, or rhythmic density.
Hi-Hat Drum Set Role:
1. Role-to-Note Assignment Rule: Assign this Role to often repeated consecutive notes, usually a quarter note or faster.
2. Role-to-Instrument-Type Assignment Rule: This Role may be assigned to instrument types such related to hi-hats.
3. Role-to-Instrument-Performance-Logic Assignment Rule: Depending on density of part, and perceived style, this role will be assigned a specific performance that can determine how to switch all the articulations contained within a hi-hat. (e.g. closed hit with open on 4 and)
Snare Drum Set Role:
1. Role-to-Note Assignment Rule: This Role is often assigned to notes close to or around the weak beats.
2. Role-to-Instrument-Type Assignment Rule: This Role may be assigned to be instrument types related to Snare (stick_snare, brush_snare, synth_snare, etc.).
3. Role-to-Instrument-Performance-Logic Assignment Rule: Depending on density of part, and perceived style, this Role will be assigned a specific performance that can determine how to switch all the articulations contained within a snare drum.
Cymbal Drum Set Role:
1. Role-to-Note Assignment Rule: This Role may be assigned to either repeated consecutive notes (ride) or single notes on downbeats of measures or phrases (crash).
2. Role-to-Instrument-Type Assignment Rule: This Role may be assigned to be instrument types related to Cymbal.
3. Role-to-Instrument-Performance-Logic Assignment Rule: Depending on density of part, and perceived style, the Cymbal Drum set role will be assigned to a specific performance that can determine how to switch all the articulations contained within a Cymbal.
Tom Drum Set Role:
1. Role-to-Note Assignment Rule: This Role may be assigned to clusters of notes that happen at the end of measures, or that are denser, but that are less consistent than Hi-Hat or Cymbal(ride).
2. Role-to-Instrument-Type Assignment Rule: This Role may be assigned to Instrument Types related to a Tom Drum Set.
3. Role-to-Instrument-Performance-Logic Assignment Rule: This role will be assigned to instrument performances based on density and position in measure that will determine which toms play which pitches and when the pitches switch. (e.g. Tom “low pitch only”, Tom “low tom with low-mid tom accent”)
Kick Drum Set Role:
1. Role-to-Note Assignment Rule: This Role is assigned often to notes close to or around the strong beats.
2. Role-to-Instrument-Type Assignment Rule: The Kick Drum Set Role may be assigned instrument types related to Kick.
3. Role-to-Instrument-Performance-Logic Assignment Rule: The Kick Drum set role may be assigned to instrument performances related to Kick. Depending on density of part, and perceived style, this Role will be assigned a specific performance that can determine how to switch all the articulations contained within a Kick.
As described above, these Role Assignment Rules/Principles are illustrative in nature and will vary from illustrative embodiment to illustrative embodiment, when practicing the present invention.
In some ways, this output data file for the pre-analyzed music composition is an augmented music performance notation file that is Role-organized and timeline indexed and contains all of the music state data required for the automated music performance system of the present invention to make intelligent and contextually-aware instrument and note selections and processing operations in real-time, to digitally perform the music composition in a high-quality, deeply expressive and contextually relevant manner, using the instrument performance logic deployed in the innovative DS-VMI Libraries of the present invention. As described hereinabove, this performance logic will typically be expressed in the form of “If X, then Y” performance rules, driven by the music-theoretic states that are captured and reflected in the structure of the music-theoretic state data descriptor file generated for each music composition to be digitally performed. However, this performance logic may be implemented in other ways which will occur to those with ordinary skill in the art.
Using the music-theoretic music composition analysis method of the present invention, each pre-analyzed music composition state descriptor file generated by the process, will embody Role-based note data, music (note) metric data and music-meta data automatically abstracted from the music composition to be digitally performed. Also, each music-theoretic state data descriptor file output from this pre-processor should be capable of driving the automated music performance engine of the present invention, by virtue of the fact that the music-theoretic state descriptor data will logically trigger (and cause to execute) relevant musical instrument performance rules that have been created and assigned to groups of sampled notes/sounds managed in each deeply-sampled virtual musical instrument (DS-VMI) Libraries maintained by the automated music performance system.
When the music-theoretic state data descriptor file, functioning as an augmented music performance notation file, is supplied to the Automated Music Performance Engine of the present invention, the Automated Music Performance Engine automatically analyzes and processes the data file for Roles, Notes, Music Metrics, and Meta-Data contained in the music-theoretic state data descriptor file. If the Automated Music Performance Engine determines that certain Music-Theoretic State Data Descriptors are present in the input music composition/performance file (representative of certain music conditions present in the music composition to be digitally performed), then certain Musical Instrument Performance Rules will be automatically triggered and executed to process and handle particular sampled notes, and corresponding Music Instrument Performance Rules will operate on the notes and generate the processed notes required by the input music composition/performance file being processed by the Automated Music Performance Engine, to produce a unique and expressive musical experience, with a sense of realism hitherto unachievable when using conventional machine-driven music performance engines.
Method of Generating a Music-Theoretic State Descriptor Representation of a MIDI-Type Music Composition for Use in Applying Performance Logic in the Automated Music Performance System and Selecting, Performing and Assembling Sampled Notes from Deeply-Sampled Virtual Musical Instruments Supported by the Automated Music Performance System of the Present Invention
As shown at Block A in
At Block B in
As shown at Block C in
As shown at Block D in
As shown in
Specification of Role Assignment Rules/Principles of the Present Invention
FIGS. 33B1 through 33B8 provide a set of exemplary Rules for use during the automated role assignment processes carried out by within the automated music performance system of the second illustrative embodiment of the present invention.
The following describes an exemplary way of assigning Roles to Notes, roles to Instrument Types and roles to Instrument Performance logic (i.e. Role Assignments) across the various stages of the automated music performance system of the present invention.
Roles are a way of organizing notes along a timeline that are sent to assigned Instrument Types to be handled by the Instrument Performance Logic which will select the correct samples for playback in the production of a musical piece.
Instruments and Performance Logic (Rules) are all labeled (tagged) with data that allow for rulesets to choose the appropriate Instrument/Performance combination.
With all three types of input (e.g. OCM, MIDI, AMPER), the following rules can be applied to an assigned Instrument Type, and then a specific Sample Instrument would be assigned to the Role Assignment. Each of these Roles can have a many variants of a role, if multiple roles of similar type are needed (e.g. accent.a, accent.b, or accent.1, accent.2, etc.).
Accent Role:
1. Role-to-Note Assignment Rule: If the density of notes are fairly sparse and follow along a consistent strong beat to weak beat periodicity, or/and if several instrument parts have regular periodicity in strong beat groupings, then assign to the Accent Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Accent Role to instrument_types reserved for accents, which are typically percussive, (e.g. “.hit( )” aspect value of: aux_perc, big_hit, cymbal, etc.).
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign the Accent Role to Instrument Performance Logic of other roles by through change in velocity or not play/play notes in current assigned role (ex: augmenting role).
Back Beat Role:
1. Role-to-Note Assignment Rule: If notes have a periodicity of primarily weak beat and that are tonal, then assign to the Back Beat Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role Instrument Types that provide a more rhythmic and percussive tonal performance (mono or polyphonic)
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign this Role to Instrument Performances (i.e. Instrument Performance Logic/Rules) such as, e.g. acoustic_piano with “triadic chords closed voicing”, acoustic_guitar with “up-strum top three strings”, etc.
Background Role:
1. Role-to-Note Assignment Rule: If notes have a medium-low density (playing slightly more than once per chord, polytonal), then assign the Background Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Background Role to instrument types that can support polyphonic (note) performances.
3. Role-to-Performance-Logic Assignment Rule: Play polyphonic chords or parts of chords in instrument types (e.g. keyboard, acoustic_piano, synth_strings, etc.)
Big Hit Role:
1. Role-to-Note Assignment Rule: If notes happen with extreme irregularity and are very sparse, and/or either fall with a note in the accent lane or outside of any time signature, then assign the Big Hit Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this role instrument type primarily to single hit, non-tonal, percussive instruments
3. Role-to-Instrument-Performance-Logic Assignment Rule: Performance Logic is play a “hit( )” in the assigned instrument type (e.g. big_hit, bass_drum, etc.) Color Role:
1. Role-to-Note Assignment Rule: If notes happen in small clusters, with rests between each set of clusters, and have some regular periodicity less than once per phrase, then assign the Color role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Color Role to instrument types that are either percussive (if notes are unpitched) or tonal (if notes follow pitches within the key) and can be assigned to typically monophonic or instruments with harmonic/rhythmic tags (e.g. instruments with delay tag, tickies, synth_lead, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: May assign performances that are softer in velocity or lighter in articulation attack.
Consistent Role:
1. Role-to-Note Assignment Rule: If notes are relatively dense, have some periodicity, and change in either note pattern organization, rhythmic pattern organization more than once per a few bars, then assign the Consistent Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Color Role to instrument types that have typically monophonic performances (e.g. synth_lead, guitar_lead).
3. Role-to-Instrument-Performance-Logic Assignment Rule: May assign performances that utilize various arpeggiation patterns (eg: line up/down, sawtooth, etc.)
Constant Role:
1. Role-to-Note Assignment Rule: If notes that are relatively dense, and have very static rhythmic information with periodicity, then assign the Constant Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Constant Role to either tonal (monophonic) or percussive instrument types.
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign the Constant Role to Instrument Performances that are tempo dependent (e.g. shaker with “front( )” only, synth_lead with “arpeggiation up”, etc.).
Decoration Role:
1. Role-to-Note Assignment Rule: If notes happen in small clusters, with rests between each set of clusters, and occur one per phrase or longer, then assign the Decoration role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Decoration Role to instrument types that are either percussive (if notes are unpitched) or tonal (if notes follow pitches within the key) and can be assigned to typically monophonic or instruments with harmonic/rhythmic tags (e.g. instruments with delay tag, tickies, synth_lead, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: May assign performances that are softer in velocity or lighter in articulation attack.
High Lane Role:
1. Role-to-Note Assignment Rule: If notes that are unpitched that happen in usually rapid succession or high density, then assign the High Lane Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Unless otherwise directed by an external input, usually assign the High Lane Role to high-pitched in timbre percussion instruments (e.g. tickies, shakers, aux_drum (“rim”), etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: Choose performances that activate articulations that are tagged with “high” and/or “short”
High-Mid Lane Role:
1. Role-to-Note Assignment Rule: If notes that are unpitched that happen in medium-high density, then assign the High-Mid Lane Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Unless otherwise directed by an external input, usually assign this Role to high-pitched or medium in timbre percussion instruments (e.g. tickies, aux_drum, hand_drum, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule:
Choose performances that activate articulations that are tagged with “middle” and/or “short/medium”
Low-Mid Lane Role:
1. Role-to-Note Assignment Rule: If notes that are unpitched that happen in medium-low density, then assign the Low-Mid Lane Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Unless otherwise directed by an external input, usually assign this Role to medium to medium-low in timbre percussion instruments (e.g. aux_drum, hand_drum, taiko, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule:
Choose performances that activate articulations that are tagged with “middle,” “low,” and/or “medium/long” Low Lane Role:
1. Role-to-Note Assignment Rule: Assign the Low Lane Role to notes that are unpitched that happen in low density.
2. Role-to-Instrument-Type Assignment Rule: Unless otherwise directed by an external input, assign this Role usually to instrument types that are low in timbre percussion (e.g. bass_drum, surdo, taiko, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule:
Choose performances that activate articulations that are tagged with “low,” and/or “long” Middle Role:
1. Role-to-Note Assignment Rule: If notes have a medium density (playing more than once per chord, polytonal, with occasional running lines), then assign the Middle Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Typically assign this Role to instrument types that can support polyphonic playback and performance. (e.g. keyboard, acoustic_piano, synth_strings, violins, etc.).
3. Role-to-Instrument-Performance-Logic Assignment Rule: Typically assign this Role to Instrument Performances support polyphonic performance. (e.g. keyboard, acoustic_piano, synth_strings, violins, etc.).
On Beat Role:
1. Role-to-Note Assignment Rule: If notes have a periodicity of primarily strong beat and that are tonal, then assign the On Beat Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role to instrument types that produce more rhythmic and percussive tonal performances (mono or polyphonic)
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign this Role to Instrument Performances that have “strong” performance tag association (eg: acoustic_bass “roots with 5ths”, acoustic_guitar with “down-strum power chord”, etc.)
Pad Role:
1. Role-to-Note Assignment Rule: If notes are sustained through the duration of a chord, then assign the Pad Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Typically assign this Role to polyphonic instrument types that sustain notes (e.g. mid_pad, synth_string, synth_bass)
3. Role-to-Instrument-Performance-Logic Assignment Rule: Typically assign this Role to Instrument Performances involving polyphonic performances that sustain notes during a chord, and change notes on chord change (e.g. mid_pad, synth_string, synth_bass) Pedal Role:
1. Role-to-Note Assignment Rule: If notes sustain through chords and stay on one pitch (often the root) of an entire phrase, then assign the Pedal Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role typically to monophonic instrument types such as, e.g. low_pad, synth_bass, pulse, etc.
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign this Role typically to Instrument Performances supporting monophonic performances that either sustain indefinitely, or can quickly reattack consecutively to create a pulse-like pedal tone (e.g. low_pad, synth_bass, pulse, etc.)
Primary Role:
1. Role-to-Note Assignment Rule: If notes are mostly monophonic, played with more density, rhythmic structure variances, often with some repetition and periodicity, and are accompanied with either great dynamic markings, high velocities, and/or other indications depending on the medium read, then assign the Primary Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role typically to instrument types often used to perform as lead instruments (e.g. violin, lead_synth, lead_guitar, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: May choose limited polyphonic or monophonic performance, that may utilizes a great amount of articulation control and switching.
Secondary Role:
1. Role-to-Note Assignment Rule: If notes are mostly monophonic, play with more density, rhythmic structure variances, often with some repetition and periodicity, and are accompanied with either great dynamic markings, high velocities, are either lower in pitch or play less dense then another part, and/or other indications depending on the medium read, then assign the Secondary Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role to instrument types that often perform as lead instruments (e.g. violin, lead_synth, lead_guitar, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: May choose limited polyphonic or monophonic performance, that may utilizes a great amount of articulation control and switching.
Drum Set Roles: These are the roles listed below that are given specific rhythmic parts (non-tonal) that should be assigned to one role-performer, but have to be broken out because the instruments used are naturally separated. Notes will need to be parsed into different roles, and often can be determined by MIDI note pitch, staff position, or rhythmic density.
Hi-Hat Drum Set Role:
1. Role-to-Note Assignment Rule: Assign this Role to often repeated consecutive notes, usually a quarter note or faster.
2. Role-to-Instrument-Type Assignment Rule: This Role may be assigned to instrument types such related to hi-hats.
3. Role-to-Instrument-Performance-Logic Assignment Rule: Depending on density of part, and perceived style, this role will be assigned a specific performance that can determine how to switch all the articulations contained within a hi-hat. (e.g. closed hit with open on 4 and)
Snare Drum Set Role:
1. Role-to-Note Assignment Rule: This Role is often assigned to notes close to or around the weak beats.
2. Role-to-Instrument-Type Assignment Rule: This Role may be assigned to be instrument types related to Snare (stick_snare, brush_snare, synth_snare, etc.).
3. Role-to-Instrument-Performance-Logic Assignment Rule: Depending on density of part, and perceived style, this Role will be assigned a specific performance that can determine how to switch all the articulations contained within a snare drum.
Cymbal Drum Set Role:
1. Role-to-Note Assignment Rule: This Role may be assigned to either repeated consecutive notes (ride) or single notes on downbeats of measures or phrases (crash).
2. Role-to-Instrument-Type Assignment Rule: This Role may be assigned to be instrument types related to Cymbal.
3. Role-to-Instrument-Performance-Logic Assignment Rule: Depending on density of part, and perceived style, the Cymbal Drum set role will be assigned to a specific performance that can determine how to switch all the articulations contained within a Cymbal.
Tom Drum Set Role:
1. Role-to-Note Assignment Rule: This Role may be assigned to clusters of notes that happen at the end of measures, or that are denser, but that are less consistent than Hi-Hat or Cymbal(ride).
2. Role-to-Instrument-Type Assignment Rule: This Role may be assigned to Instrument Types related to a Tom Drum Set.
3. Role-to-Instrument-Performance-Logic Assignment Rule: This role will be assigned to instrument performances based on density and position in measure that will determine which toms play which pitches and when the pitches switch. (e.g. Tom “low pitch only”, Tom “low tom with low-mid tom accent”) Kick Drum Set Role:
1. Role-to-Note Assignment Rule: This Role is assigned often to notes close to or around the strong beats.
2. Role-to-Instrument-Type Assignment Rule: The Kick Drum Set Role may be assigned instrument types related to Kick.
3. Role-to-Instrument-Performance-Logic Assignment Rule: The Kick Drum set role may be assigned to instrument performances related to Kick. Depending on density of part, and perceived style, this Role will be assigned a specific performance that can determine how to switch all the articulations contained within a Kick.
As described above, these Role Assignment Rules/Principles are illustrative in nature and will vary from illustrative embodiment to illustrative embodiment, when practicing the present invention.
Method of Generating a Music-Theoretic State Descriptor Representation of Automatically-Generated Music Composition for Use in Applying Performance Logic in the Automated Music Performance System and Selecting, Performing and Assembling Sampled Notes from Deeply-Sampled Virtual Musical Instruments Supported by the Automated Music Performance System of the Present Invention
As shown at Block A in
As shown at Block B in
As shown at Block C in
As shown in
Specification of Role Assignment Rules/Principles of the Present Invention
FIGS. 38B1 through 38B8 provide a set of exemplary Rules for use during the automated role assignment processes carried out by within the automated music performance system of the first illustrative embodiment of the present invention.
The following describes an exemplary way of assigning Roles to Notes, roles to Instrument Types and roles to Instrument Performance logic (i.e. Role Assignments) across the various stages of the automated music performance system of the present invention.
Roles are a way of organizing notes along a timeline that are sent to assigned Instrument Types to be handled by the Instrument Performance Logic which will select the correct samples for playback in the production of a musical piece.
Instruments and Performance Logic (Rules) are all labeled (tagged) with data that allow for rulesets to choose the appropriate Instrument/Performance combination.
With all three types of input (e.g. OCM, MIDI, AMPER), the following rules can be applied to an assigned Instrument Type, and then a specific Sample Instrument would be assigned to the Role Assignment. Each of these Roles can have a many variants of a role, if multiple roles of similar type are needed (e.g. accent.a, accent.b, or accent.1, accent.2, etc.).
Accent Role:
1. Role-to-Note Assignment Rule: If the density of notes are fairly sparse and follow along a consistent strong beat to weak beat periodicity, or/and if several instrument parts have regular periodicity in strong beat groupings, then assign to the Accent Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Accent Role to instrument_types reserved for accents, which are typically percussive, (e.g. “.hit( )” aspect value of: aux_perc, big_hit, cymbal, etc.).
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign the Accent Role to Instrument Performance Logic of other roles by through change in velocity or not play/play notes in current assigned role (ex: augmenting role).
Back Beat Role:
1. Role-to-Note Assignment Rule: If notes have a periodicity of primarily weak beat and that are tonal, then assign to the Back Beat Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role Instrument Types that provide a more rhythmic and percussive tonal performance (mono or polyphonic)
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign this Role to Instrument Performances (i.e. Instrument Performance Logic/Rules) such as, e.g. acoustic_piano with “triadic chords closed voicing”, acoustic_guitar with “up-strum top three strings”, etc.
Background Role:
1. Role-to-Note Assignment Rule: If notes have a medium-low density (playing slightly more than once per chord, polytonal), then assign the Background Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Background Role to instrument types that can support polyphonic (note) performances.
3. Role-to-Performance-Logic Assignment Rule: Play polyphonic chords or parts of chords in instrument types (e.g. keyboard, acoustic_piano, synth_strings, etc.)
Big Hit Role:
1. Role-to-Note Assignment Rule: If notes happen with extreme irregularity and are very sparse, and/or either fall with a note in the accent lane or outside of any time signature, then assign the Big Hit Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this role instrument type primarily to single hit, non-tonal, percussive instruments
3. Role-to-Instrument-Performance-Logic Assignment Rule: Performance Logic is play a “hit( )” in the assigned instrument type (e.g. big_hit, bass_drum, etc.)
Color Role:
1. Role-to-Note Assignment Rule: If notes happen in small clusters, with rests between each set of clusters, and have some regular periodicity less than once per phrase, then assign the Color role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Color Role to instrument types that are either percussive (if notes are unpitched) or tonal (if notes follow pitches within the key) and can be assigned to typically monophonic or instruments with harmonic/rhythmic tags (e.g. instruments with delay tag, tickies, synth_lead, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: May assign performances that are softer in velocity or lighter in articulation attack.
Consistent Role:
1. Role-to-Note Assignment Rule: If notes are relatively dense, have some periodicity, and change in either note pattern organization, rhythmic pattern organization more than once per a few bars, then assign the Consistent Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Color Role to instrument types that have typically monophonic performances (e.g. synth_lead, guitar_lead).
3. Role-to-Instrument-Performance-Logic Assignment Rule: May assign performances that utilize various arpeggiation patterns (eg: line up/down, sawtooth, etc.)
Constant Role:
1. Role-to-Note Assignment Rule: If notes that are relatively dense, and have very static rhythmic information with periodicity, then assign the Constant Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Constant Role to either tonal (monophonic) or percussive instrument types.
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign the Constant Role to Instrument Performances that are tempo dependent (e.g. shaker with “front( )” only, synth_lead with “arpeggiation up”, etc.).
Decoration Role:
1. Role-to-Note Assignment Rule: If notes happen in small clusters, with rests between each set of clusters, and occur one per phrase or longer, then assign the Decoration role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign the Decoration Role to instrument types that are either percussive (if notes are unpitched) or tonal (if notes follow pitches within the key) and can be assigned to typically monophonic or instruments with harmonic/rhythmic tags (e.g. instruments with delay tag, tickies, synth_lead, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: May assign performances that are softer in velocity or lighter in articulation attack.
High Lane Role:
1. Role-to-Note Assignment Rule: If notes that are unpitched that happen in usually rapid succession or high density, then assign the High Lane Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Unless otherwise directed by an external input, usually assign the High Lane Role to high-pitched in timbre percussion instruments (e.g. tickies, shakers, aux_drum (“rim”), etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: Choose performances that activate articulations that are tagged with “high” and/or “short”
High-Mid Lane Role:
1. Role-to-Note Assignment Rule: If notes that are unpitched that happen in medium-high density, then assign the High-Mid Lane Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Unless otherwise directed by an external input, usually assign this Role to high-pitched or medium in timbre percussion instruments (e.g. tickies, aux_drum, hand_drum, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: Choose performances that activate articulations that are tagged with “middle” and/or “short/medium”
Low-Mid Lane Role:
1. Role-to-Note Assignment Rule: If notes that are unpitched that happen in medium-low density, then assign the Low-Mid Lane Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Unless otherwise directed by an external input, usually assign this Role to medium to medium-low in timbre percussion instruments (e.g. aux_drum, hand_drum, taiko, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule:
Choose performances that activate articulations that are tagged with “middle,” “low,” and/or “medium/long”
Low Lane Role:
1. Role-to-Note Assignment Rule: Assign the Low Lane Role to notes that are unpitched that happen in low density.
2. Role-to-Instrument-Type Assignment Rule: Unless otherwise directed by an external input, assign this Role usually to instrument types that are low in timbre percussion (e.g. bass_drum, surdo, taiko, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule:
Choose performances that activate articulations that are tagged with “low,” and/or “long” Middle Role:
1. Role-to-Note Assignment Rule: If notes have a medium density (playing more than once per chord, polytonal, with occasional running lines), then assign the Middle Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Typically assign this Role to instrument types that can support polyphonic playback and performance. (e.g. keyboard, acoustic_piano, synth_strings, violins, etc.).
3. Role-to-Instrument-Performance-Logic Assignment Rule: Typically assign this Role to Instrument Performances support polyphonic performance. (e.g. keyboard, acoustic_piano, synth_strings, violins, etc.).
On Beat Role:
1. Role-to-Note Assignment Rule: If notes have a periodicity of primarily strong beat and that are tonal, then assign the On Beat Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role to instrument types that produce more rhythmic and percussive tonal performances (mono or polyphonic)
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign this Role to Instrument Performances that have “strong” performance tag association (eg: acoustic_bass “roots with 5ths”, acoustic_guitar with “down-strum power chord”, etc.)
Pad Role:
1. Role-to-Note Assignment Rule: If notes are sustained through the duration of a chord, then assign the Pad Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Typically assign this Role to polyphonic instrument types that sustain notes (e.g. mid_pad, synth_string, synth_bass)
3. Role-to-Instrument-Performance-Logic Assignment Rule: Typically assign this Role to Instrument Performances involving polyphonic performances that sustain notes during a chord, and change notes on chord change (e.g. mid_pad, synth_string, synth_bass)
Pedal Role:
1. Role-to-Note Assignment Rule: If notes sustain through chords and stay on one pitch (often the root) of an entire phrase, then assign the Pedal Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role typically to monophonic instrument types such as, e.g. low_pad, synth_bass, pulse, etc.
3. Role-to-Instrument-Performance-Logic Assignment Rule: Assign this Role typically to Instrument Performances supporting monophonic performances that either sustain indefinitely, or can quickly reattack consecutively to create a pulse-like pedal tone (e.g. low_pad, synth_bass, pulse, etc.) Primary Role:
1. Role-to-Note Assignment Rule: If notes are mostly monophonic, played with more density, rhythmic structure variances, often with some repetition and periodicity, and are accompanied with either great dynamic markings, high velocities, and/or other indications depending on the medium read, then assign the Primary Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role typically to instrument types often used to perform as lead instruments (e.g. violin, lead_synth, lead_guitar, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: May choose limited polyphonic or monophonic performance, that may utilizes a great amount of articulation control and switching.
Secondary Role:
1. Role-to-Note Assignment Rule: If notes are mostly monophonic, play with more density, rhythmic structure variances, often with some repetition and periodicity, and are accompanied with either great dynamic markings, high velocities, are either lower in pitch or play less dense then another part, and/or other indications depending on the medium read, then assign the Secondary Role to these notes.
2. Role-to-Instrument-Type Assignment Rule: Assign this Role to instrument types that often perform as lead instruments (e.g. violin, lead_synth, lead_guitar, etc.)
3. Role-to-Instrument-Performance-Logic Assignment Rule: May choose limited polyphonic or monophonic performance, that may utilizes a great amount of articulation control and switching.
Drum Set Roles: These are the roles listed below that are given specific rhythmic parts (non-tonal) that should be assigned to one role-performer, but have to be broken out because the instruments used are naturally separated. Notes will need to be parsed into different roles, and often can be determined by MIDI note pitch, staff position, or rhythmic density.
Hi-Hat Drum Set Role:
1. Role-to-Note Assignment Rule: Assign this Role to often repeated consecutive notes, usually a quarter note or faster.
2. Role-to-Instrument-Type Assignment Rule: This Role may be assigned to instrument types such related to hi-hats.
3. Role-to-Instrument-Performance-Logic Assignment Rule: Depending on density of part, and perceived style, this role will be assigned a specific performance that can determine how to switch all the articulations contained within a hi-hat. (e.g. closed hit with open on 4 and) Snare Drum Set Role:
1. Role-to-Note Assignment Rule: This Role is often assigned to notes close to or around the weak beats.
2. Role-to-Instrument-Type Assignment Rule: This Role may be assigned to be instrument types related to Snare (stick_snare, brush_snare, synth_snare, etc.).
3. Role-to-Instrument-Performance-Logic Assignment Rule: Depending on density of part, and perceived style, this Role will be assigned a specific performance that can determine how to switch all the articulations contained within a snare drum.
Cymbal Drum Set Role:
1. Role-to-Note Assignment Rule: This Role may be assigned to either repeated consecutive notes (ride) or single notes on downbeats of measures or phrases (crash).
2. Role-to-Instrument-Type Assignment Rule: This Role may be assigned to be instrument types related to Cymbal.
3. Role-to-Instrument-Performance-Logic Assignment Rule: Depending on density of part, and perceived style, the Cymbal Drum set role will be assigned to a specific performance that can determine how to switch all the articulations contained within a Cymbal.
Tom Drum Set Role:
1. Role-to-Note Assignment Rule: This Role may be assigned to clusters of notes that happen at the end of measures, or that are denser, but that are less consistent than Hi-Hat or Cymbal(ride).
2. Role-to-Instrument-Type Assignment Rule: This Role may be assigned to Instrument Types related to a Tom Drum Set.
3. Role-to-Instrument-Performance-Logic Assignment Rule: This role will be assigned to instrument performances based on density and position in measure that will determine which toms play which pitches and when the pitches switch. (e.g. Tom “low pitch only”, Tom “low tom with low-mid tom accent”)
Kick Drum Set Role:
1. Role-to-Note Assignment Rule: This Role is assigned often to notes close to or around the strong beats.
2. Role-to-Instrument-Type Assignment Rule: The Kick Drum Set Role may be assigned instrument types related to Kick.
3. Role-to-Instrument-Performance-Logic Assignment Rule: The Kick Drum set role may be assigned to instrument performances related to Kick. Depending on density of part, and perceived style, this Role will be assigned a specific performance that can determine how to switch all the articulations contained within a Kick.
As described above, these Role Assignment Rules/Principles are illustrative in nature and will vary from illustrative embodiment to illustrative embodiment, when practicing the present invention.
Method of Sampling, Recording and Cataloging Real Musical Instruments and Producing a Deeply-Sampled Virtual Musical Instrument (DS-VMI) Library Management System for Use in the Automated Music Performance System of the Present Invention
Specification of Instrument Names, Instrument Types, Instrument Behaviors Classified in the Deeply-Sampled Virtual Musical Instrument DS-VMI Library Management System of the Present Invention
Lists of Instruments
In
List of Instrument Types
In the illustrative embodiment, the classifier called “Type” is used to denote how a usable template is created and how the Instrument should be assigned in the automated music performance system, and thus the Instrument should be recorded during the sampling session.
List of Behaviors
In
Preferably, the system is designed so that by selecting a Type from the Type List will result in the automated generation of a sampling template specifying what Notes to sample on the real instrument (to be sampled) based on its Type, as well as a Note Range that is associated with it. If there is no note range, then it's not a tonal behavior/aspect, and does not have a “range”. When a user wants to add an instrument into the automated music performance system, the instrument list is referenced to determine if the requested instrument relates to an Instrument Type. However, the Instrument List does not dictate a number of sample attributes, namely: how many round robins, velocities and other granular level sample things that need to be addressed. Often, these decisions are made on the day of the sampling session and is based on time and financial constraints. Also, a file naming structure for sound samples should be developed and used that helps parse out the names to be read by the Type and Instrument Lists.
During system operation, the automated music performance system of the present invention automatically (i) classifies each deeply-sampled virtual musical instrument (DS-VMI) entered into its instrument catalog, (ii) informs the system of the type of the instrument and what range of note it performs, (iii) sets a foundation for the automated music performance logic subsystem to be generated for the instrument, enabling automatic selection of appropriate sample articulations that dramatically alter the sound produced from each deeply-sampled virtual musical instrument, based on the music-theoretic states of an input music composition being digitally performed.
Specification of the Method of Sampling and Recording Samples from Real Musical Instruments and Other Non-Musical Audio Sources of the Present Invention
Specification of the Music-Theoretic State Responsive Instrument Contracting Logic for the Deeply-Sampled Virtual Musical Instruments of the Present Invention
Specification of the Music-Theoretic State (MTS) Responsive Performance Logic for the Deeply-Sampled Virtual Musical Instruments of the Present Invention
These two rule-based subsystems described above and schematically depicted in
Classification of Virtual Musical Instruments in the DS-VMS Library Management Subsystem
Method of Sampling, Recording, and Cataloging Real Musical Instruments for Use in Developing Corresponding Deeply-Sampled Virtual Musical Instruments (DS-VMI) for Deployment in the Deeply-Sampled Virtual Musical Instrument (DS-VMI) Library Management System of Present Invention
In order to be able to predictively select sampled notes from a deeply-sampled virtual musical instrument library, that plays very well with the music-theoretic states of the music composition being digitally performed, the present invention teaches to sample the real instrument based on its Instrument Type, Behavior and how it is performed. Also, the present invention also teaches to catalogue each sampled note using a naming convention that is expressed in a performance logic (i.e. set of performance rules) created for the Type of the deeply-sampled virtual musical instrument, executed upon the detection of conditions in the music-theoretic state of the music composition that matches the condition expressed in the conditional part of the performance rules.
Using this technique, it is possible for the automated music performance system to be provided a degree of artificial intelligence and predictive insight on what sampled notes in the DS-VMI library management subsystem should be selected and processed for assembly and finalization in the digital performance being produced for the music composition provided to the system.
As indicated at Step A in
In general, in accordance with the spirit of the present invention, sampled audio sounds can be synthesized sampled notes, AI produced samples, Sample Modeling, or sampled audio sounds, and therefore, sampled audio can represent (i) a sample note produced by a real (tonal) musical instrument typically tuned to produce tonal sounds or notes (e.g. piano, string instruments, drums, horns, (ii) a sampled sound produced by an atonal sound source (e.g. ocean breeze, thunder, airstream, babbling brook, doors closing, and electronic sound synthesizers, etc.) or (iii) a sampled voice singing or speaking, etc.
Also, the term “virtual musical instrument (VMI)” as used throughout the Patent Specification is any virtual musical instrument is made from (i) a library of sampled audio sound files representative of musical notes and/or other sounds, and/or (ii) a library of digitally synthesized sounds representative of musical notes and/or other sounds. When using an audio-sound sampling method, the notes and/or sounds do not have to be sampled and recorded from a real musical instrument (e.g. piano, drums, string instrument, etc.), but may be produced from non-musical instrument audio source, including sources of nature, human voices, animal sounds, etc.
When using a digital sound synthesis method, the notes and/or sounds may be digitally designed, created and produced using sound synthesis software tools such as, for example, MOTU's MACHFIVE and MX4 software tools, and Synclavier® sound synthesis software products, and the notes and sounds produced for these VMI libraries may have any set of sonic characteristics and/or attributes that can be imagined by the sound designer and engineered into a digital file for loading and storage in, and playback from the virtual musical instrument (VMI) library being developed in accordance with the principles of the present invention.
When using a digital audio/sound synthesis method to produce the notes and sound files for a particular virtual musical instrument (VMI) library, the users may readily adapt the sampling template, instrument definitions, and cataloging principles used for sound sampling methods disclosed and taught herein for digitally-synthesized virtual musical instruments (DS-VMI) having notes and sounds created using digital sound synthesis methods known in the art.
It is appropriate at this junction to describe in greater detail how such tools and devices may be readily adapted and used when producing notes and sounds for VMIs using the digital sound synthesis (DS) method.
A synthesis sound module can be defined as a set of synthesis parameters (FM, Spectral, Additive, etc.) that could contain a sound generating oscillator(s) that is assigned a waveform(s), manipulated by amplitude, frequency and filters, with control of each manipulation via other oscillators, generated envelopes, gates, and external controllers. In the VMI sound synthesis space, each designed synthesis module with specified static or ranged parameters can be assigned the same Behavior and Aspect value schema as when developing a deeply-sampled virtual musical instrument (DS-VMI) library. A single digitally-synthesized VMI could contain multiple sound modules to support a robust deep synthesis of a single instrument type. For example, a sound module could be created to mimic the sustain of a violin, a pizzicato of a violin, or a tremolo of a violin, each are separate modules, but could exist as a single VMI so that the role/performance algorithm that is assigned to the violin instrument could use either the sampled version or the synthesis version agnostically.
These classifications of these sound modules for digitally synthesized VMIs is done in the same way that a single sound sample would be classified, but instead of a bank of individual note samples, a sound module would provide open handles for data to be submitted, for example: Instrument Definition: Synthesized Harp: 2 Sound modules: Sound Module 1 consists of 2 Oscillators (sine and noise), Sine oscillator has an envelope applied that controls amplitude over time (decay), Noise oscillator has a filter and amplitude envelope applied that has a hard attack and a very fast decay. Second Sound Module has 3 Oscillators, (Sine+0(semitones), Sine+12(semitones), Noise). Both sine oscillators have an envelope applied that controls amplitude over time (decay) with the first sine oscillator at −30 db gain and the second at 0 db gain. Noise oscillator has a filter and amplitude envelope applied that has a hard attack and a very fast decay. The instrument definition has open handles for manipulation by the engine: Pitch Selection (oscillator pitch change, based on MIDI note), Velocity Selection (oscillator filter and volume change based on MIDI velocity), and Gate (trigger of note on/off, based on MIDI note start and end times).
Each synthesized instrument definition can be cataloged (with the exception of the cataloging of the single sample note recorded audio) against the same template instrument definition as used when developing a deeply-sampled virtual musical instrument (DS-VMI) library. Using the prior example, the Synthesized Harp would fall under the instrument type “Harp” template which states the Behavior is a “single_note” and can change Aspects with the values of “regular” or “harmonic”. The first sound module would be cataloged as the “regular” aspect and the second would be the “harmonic” aspect. If the system had available the Synthesized harp and set a harp performance, the instrument would perform the same way as the sampled harp would, allowing for switching of regular and harmonics, and pitch/velocity controlled data, but instead of playing back samples, the engine would render the synthesized reproduction through the sound modules.
Returning now to the operations flow of the system, as indicated at Step B in
As indicated at Step C in
As indicated at Step D in
Sound Sampling Process According to the Present Invention:
As indicated at Step F in
Instrument Contractor (i.e. Instrumentation) Logic:
As indicated at Step G in
In general, all instruments in the automated music performance system will get a specific type of performance (or logical instructions) written for them, and executable when specific music-theoretic states are detected along the timeline of a music composition being digitally performed. These performances can range from “play a simple hit at {x} velocity” to a “strum a guitar with 6 strings, muting the first two, playing an up stroke on all 6, assembling this position of a chord”.
Preferably, each logical performance rule will have an “IF X, THEN Y” format, where X specifies a particular state or condition detected in the music composition and characterized in the music composition meta-data file (i.e. music-theoretic state descriptor data), and Y specifies the specific performance instruction to be performed by the virtual musical instrument on the sampled note selected from a deeply-sampled virtual musical instrument, that has been selected by the logical contractor rules performed by the automated instrument contracting subsystem, employed within the automated music performance system.
Below are common examples of music-theoretic states (i.e. music composition meta-data) abstracted from the music composition being digitally performed:
When analyzing and detecting music-theoretic state data (i.e. music composition meta data), the automated music performance subsystem will identify the performance rule associated with the MIDI note values, and determine for what logical performance rule both the music composition state and the performance rule state (i.e. X) matches, and if for performance rule with a match, then the automated music performance system automatically executes the performance rule on the sampled note. Such performance rule execution will typically involve processing the sampled note in some way so that the virtual musical instrument will reasonably perform the sampled note at a specified trigger point, and thereby adapt to the musical notes that are being played around the sampled note. By assigning logical performance rules to certain groups of sampled notes in a (contractor-selected) deeply-sampled virtual musical instrument library, based on instrument type, the automated music performance system is provided with both artificial musical intelligence and contextual awareness, so that it has the capacity to select, process and playback various sampled notes in any given digital performance of the music composition.
Values (specially velocity/dynamics) for sampled note processing can be deterministic or random.
As indicated at Step H in
For example, each instrument group in the DS-VMI library management system is placed into a family of like instruments called “Types.” This means that each Instrument Type will have exactly the same expected Behavior/Aspect values associated with them.
Many DS-VMI performances will have logical performance rules written for each Type, depending on how an instrument is desired to operate within a given descriptor. Example of the Shaker: Forward, Back and Double, also has 3 velocities associated with it. A soft shake, a sharper “louder” shake, and then a very short, hard “accent” forward shake. These velocities are divided from midi velocity values 1-100, 101-126, and 127-127. One logical performance rule might state: IF the composer sends a series of 8th notes, THEN play Forward @ 127, Back at @ 100, Forward @ 110, Back @ 100. Another logical performance rule might state: IF the composer sends a series of 8th notes, THEN Play forward, but choose between 101-126 with a 30% chance of playing 127, play back between 90-100, etc. Another logical performance rule might state: IF the composer gives a note on a downbeat, and had a series of notes before it, THEN play a Double @
127. Note: because the shaker has a lot of sound that precedes it (the pre-transient)—all shakers will be asked to play 250 milliseconds before the actual notes are sent by the composer to “play”—this allows all the shakers to perform in time, without sounding chopped, or late.
While the above examples of logical performance rules are rudimentary, they clearly highlight the fact that even the simplest instrument (e.g. shaker) can have multiple instrument performances just given that the instrument has 3 different articulations which it can play.
Performance logic created for and used in the DS-VMI libraries of the present invention is not only used for intelligent selection of musical instruments and sampled notes, but also for DSP control involving modifying sampled note selections based on dynamic choice, role assignment, role priority, and other virtual musical instruments available in the library management system. Logical performance rules can be written for executing algorithmic automation and intelligent selection of how to send control to note behavior and sample selection. Logical performance rules can be written to create algorithms that modulate parameters to affect the sound, which may include dynamic blending, filter control, volume level, or a host of other parameters.
Matrixing and Using Instrument Types to Create Circular Awareness
Allowing instruments to be aware of each other provides some unique and untested waters within performance automation. Consideration might also be given to timing, volume control and iteration and part copy/mutation, as discussed below.
Regarding timing, one use case could be if one instrument slows down, what do the other instruments do. If one instrument is assigned a slightly shuffled beat pattern, then can the others respond.
Regarding volume control, allowing instruments to self-adjust their overall volume based on other instruments playing around will drastically help in the automation of volume control based on user selectivity and instrument role assignments.
Not with regard to specific compositional note assignment, but with regards to how instruments perform, such performance mutation based on other instruments playing types of performances would allow users to select performers and mutate those performers within a given instrument family. Want a mix between Hendrix and Santana? Mutate the performance to select different types of guitar articulations and when to choose various types.
Generating a Digital Music Performance of a Music Composition Using the Sampled Notes Selected from the Deeply-Sampled Virtual Musical Instruments Supported by the Automated Music Performance System of the Present Invention
By virtue of this method of the presence invention described above, it is now possible to make better use of deeply-sampled virtual musical instruments used in digital music performances and productions, with increased music performance uniqueness and differentiation. Pre-existing deeply-sampled virtual musical instrument (DS-VMI libraries can be readily transformed into virtual musical instruments with artificial intelligence and awareness of how to perform its sampled notes and sounds in response to the actual music-theoretic states reflected in the music composition being digitally performed. As a result of the present invention, the value and utility of preexisting deeply-sampled virtual musical instrument libraries can be quickly expanded to meet the growing needs in the global marketplace for acoustically rich and contextually-relevant digital performances of music compositions in many diverse applications, while reducing the costs of licensing musical loops required in conventional music performance and production practices. Consequentially, the present invention creates new value in both current and new music performance and production applications.
Fourth Illustrative Embodiment of the Automated Music Performance System of the Present Invention, where a Human Composer Composes an Orchestrated “Music Composition” Expressed in a Sheet-Music Format Kind of Music-Theoretic Representation and Wherein the Music Composition is Provided to the Automated Musical Performance System of the Present Invention so that this System can Select Deeply-Sampled Virtual Musical Instruments Supported by the Automated Music Performance System Based on Roles Abstracted During Music Composition Processing, and Digitally Perform the Music Composition Using Automated Selection of Notes from Deeply-Sampled Virtual Musical Instrument Libraries
Having described various illustrative embodiments of the automated music performance system of the present invention, it is understood that there will be applications where added functionality will be desired or required, and the system architecture of the present invention is uniquely positioned to support such musical functionalities as will be described below.
For example, consider the function of “musical arrangement”, wherein a previously composed work is musically reconceptualized to produce new and different pieces of music, containing elements of the prior music composition. A musical arrangement of a prior music composition may differ from the original work by means of reharmonization, melodic paraphrasing, orchestration, or development of the formal structure. Sometimes, musical arrangement of a musical composition involves a reworking of a piece of music so that it can be played by a different instrument or different combination of instruments, based on the original music composition. However imagined, musical arrangement is an important function when composing and producing music.
Also, consider the function of “musical instrument performance style” used when performing a particular musical instrument. Often, the technique employed in practicing a particular musical instrument performance style will significantly change the musical performance by the instrument playing the same group of notes, and therefore is also considered an important function when composing and producing music.
Therefore, another object of the present invention is to provide a fourth illustrative embodiment automated music performance system and method of the present invention that supports (i) Automated Musical (Re)Arrangement and (ii) Musical Instrument Performance Style Transformation of a music composition to be digitally performed by the automated music performance system.
As will be described in great technical detail below, these two creative musical functions described above can be implemented in the automated music performance system of the present invention as follows: (i) selecting Musical Arrangement Descriptors and Musical Instrument Performance Style Descriptors described in
These additional features of the present invention will be described in greater detail hereinbelow in the context of the automated music performance system of the fourth illustrative embodiment shown in
As shown in
The Role Assignment Rules shown and described herein in great detail for the first, second and third illustrative embodiments of the present invention also can be used to practice the automated music performance system of the fourth illustrative embodiment of the present invention, and carry out each of its stages of data processing described hereinabove.
As shown in
As indicated at Block A in
As indicated at Block B in
As shown at Block C, the Role Analysis Method involves performing the following data processing operations: (a) determining the Position of notes in a measure, phrase, section, piece; (b) determining the Relation of notes of precedence and antecedence; (c) assigning MIDI note values (A1, B2, etc.); (d) reading the duration of notes; (e) evaluating position of notes in relation to strong vs weak beats; (f) reading historical standard notation practices for possible articulation usages; (g) reading historical standard notation practices for dynamics (via automation); and (h) determining the position of notes in a chord for optionally determining voice-part extraction.
As indicated at Block D in
As indicated at Block E in
As indicated at Block F in
As indicated at Block G in
As indicated at Block H in
As indicated at Block I in
As indicated at Block J in
At any time, the system user can return to the system user interface shown in
By virtue of the present invention, automated music (re)arranging and performance style transform functionalities are now available to the automated music performance system of the present invention, along with other custom modes, wherein the music-theoretic state data —automatically abstracted and collected from any music composition—is automatically transformed in a specified manner to generate a suitable and different musical-theoretic state descriptor file that is then used (as system input) by the automated music performance system of the present invention.
The advantage of such functionalities will be to enable others to (i) provide a musical composition as system input (via an API, sheet music, audio, MIDI or any other file), and (ii) then make a few simple selections from an arrangement/style menu or have these and/or any selections be made automatically by the system, to then automatically generate new kinds of digital music performances having different instrument arrangements and performed according to different performance styles.
In a regular or normal mode of operation, abstracted and collected music theoretic state data parameters (e.g. Roles, Notes, Metrics and Meta-Data) will be transmitted to the automated music performance system without modification or transformation. However, in other alternative musical arrangement/style-transformation modes supported by the system, the abstracted music-theoretic state data parameters (including the Roles, Notes, Metrics and Meta-Data) will be transformed to change the musical instrumental arrangement (in one way or another) and/or performance style thereof in an automated and creative manner to meet the creative desires of users around the world.
The innovative functionalities and technological advancements enabled by the present invention promise to create enormous new value in the market allowing billions of ordinary users with minimal music experience or education to automatically rearrange millions of music compositions (and music recordings) to perform, create and deliver new musical experiences by the users selecting (from a menu) or having the system automatically create and/or select system input parameters under descriptors such as: Music Performance Arrangement Descriptors; Music Instrument Performance Style Descriptors; to name just a few.
Employing the Automated Music Performance Engine Subsystem of the Present Invention in Other Applications
The Automated Music Performance Engine of the present invention will have use in many application beyond those described this invention disclosure.
For example, consider the use case where the system is used to provide indefinitely lasting music or hold music (i.e. streaming music). In this application, the system will be used to create unique music of definite or indefinite length. The system can be configured to convey a set of musical experiences and styles and can react to real-time audio, visual, or textual inputs to modify the music and, by changing the music, work to bring the audio, visual, or textual inputs in line with the desired programmed musical experiences and styles. For example, the system might be used in Hold Music to calm a customer, in a retail store to induce feelings of urgency and need (to further drive sales), or in contextual advertising to better align the music of the advertising with each individual consumer of the content.
Another use case would be where the system is used to provide live scored music in virtual reality or other social environments, real or imaginary. Here, the system can be configured to convey a set of musical experiences and styles and can react to real-time audio, visual, or textual inputs. In this manner, the system will be able to “live score” content experiences that do well with a certain level of flexibility in the experience constraints. For example, in a video game, where there are often many different manners in which to play the game and courses by which to advance, the system would be able to accurately create music for the game as it is played, instead of (the traditional method of) relying on pre-created music that loops until certain trigger points are met. The system would also serve well in virtual reality and mixed reality simulations and experiences.
The present invention has been described in great detail with reference to the above illustrative embodiments. It is understood, however, that numerous modifications will readily occur to those with ordinary skill in the art having had the benefit of reading the present invention disclosure.
As described in great detail herein, the automatic music performance and production system of the present invention supports the input of conventionally-notated musical information of music compositions of any length or complexity, containing musical events such as, for example, notes, chords, pitch, melodies, rhythm, tempo and other qualifies of music. However, it is understood that the system can also be readily adapted to support non-conventionally notated musical information, based on conventions and standards that may be developed in the future, but can be used as a source of musical information input to the automated music performance and production system of the present invention. Understandably, such alternative embodiments will involve developing music composition processing algorithms that can process, handle and interpret the musical information, including notes and states expressed along the timeline of the music composition
While the automated music performance and generation system of the present invention has been disclosed for use in automatically generating digital music performances for music compositions that have been completed, and represented in either music score format or MIDI-music format, it is understood that the automated music performance system of the present invention can be readily adapted to digitally perform music being composed in a “live” or “on-the-fly” manner for the enjoyment of others, using the deeply-sampled virtual musical instruments (DS-VMI) selected from the DS-VMI library management subsystem of the system. In such alternative embodiments, music being composed is either digitally represented in small time-blocks of music score (i.e. sheet music) representation as illustrated in
Also, in alternative embodiments of the present invention described hereinabove, the automated music performance and production system can be realized a stand-alone appliance, instrument, embedded system, enterprise-level system, distributed system, as well as an application embedded within a social communication network, email communication network, SMS messaging network, telecommunication system, and the like. Such alternative system configurations will depend on particular end-user applications and target markets for products and services using the principles and technologies of the present invention.
Alternate Methods of Sound Sample Representation and Sound Sample Synthesis when Developing Virtual Musical Instrument (VMI) Libraries According to Principles of the Present Invention
As disclosed herein, when using the sound/audio sampling method to produce notes and sounds for a virtual musical instrument (VMI) library system according to the present invention, storage of each audio sample in the .wav audio file format is just one form of storing a digital representation of each audio samples within the automated music performance system of the present invention, whether representing a musical note or an audible sound event. The system described in the present invention should not be limited to sampled audio in .wav format, and should include other forms of audio file format including, but not limited to, the three major groups of audio file formats, namely:
Also, when practicing a digital sound/audio synthesis method to synthesize notes and sounds for a virtual musical instrument (VMI) library system according to the present invention, MOTU's MACHFIVE and/or MX4 software tools, and Synclavier® software tools, are just a few software tools for producing a digital representation of each synthesized audio sample within the automated music performance system of the present invention. Other software tools can be used to create or synthesize digital sounds representative of notes and sounds of various natures.
The cataloging of Behaviors and Aspect values can also be applied to other forms of audio replication/synthesis specifically with regards to Role and Instrument Performance Assignment. For example, a synthesis module can be provided within the automated music performance engine, to support various controls to Attack and Release that could mimic the same kinds of Behaviors that a violin can perform. These Instrument Performance settings can be stored and sent to the synthesis module for the purpose of mimicking the same instrument_type template as violin, and assigned to this instrument_type for use within the automated music performance system.
These and all other such modifications and variations are deemed to be within the scope and spirit of the present invention as defined by the accompanying Claims to Invention.
Modifications to the Present Invention which Readily Come to Mind
The illustrative embodiments disclose the use of a novel method of developing and deploying deeply-sampled virtual musical instruments (DS-VMIs) provided with performance logic rules based on the behavior of its real corresponding musical instrument designed to predict and control the performance of the deeply-sampled virtual musical instrument in response to real-time detection of the music-theoretic states including notes of the music composition to be digitally performed using the deeply-sampled virtual musical instruments. Using this novel virtual musical instrument (VMI) design, it is now possible for libraries of deeply-sampled virtual musical instruments to produce more expressive, more intelligent and richer performances when driven by any source of composed music, however composed. However, it is understood that alternative products and technologies may be used to practice the various methods and apparatus of the present invention disclosed herein. For example, machine learning may be used within the automated music performance system to support deterministic or stochastic based music performances. The use of machine learning would analyze music compositions to abstract music-theoretic state data on each input music composition. Machine learning (ML) may also be used to analyze digital performances, either currently existing in the system, or through a training against real-world performances, through sample matching and recognition against audio. Then, with this analyzation, the automated music composition would come up with predictive models on how the automated music performance system would choose the modifications to sampled notes from a particular instrument, when the modification are placement specific (i.e. are called for by the logical performance rules).
These and other variations and modifications will come to mind in view of the present invention disclosure. While several modifications to the illustrative embodiments have been described above, it is understood that various other modifications to the illustrative embodiment of the present invention will readily occur to persons with ordinary skill in the art. All such modifications and variations are deemed to be within the scope and spirit of the present invention as defined by the accompanying Claims to Invention.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4108035 | Alonso | Aug 1978 | A |
| 4178822 | Alonso | Dec 1979 | A |
| 4279185 | Alonso | Jul 1981 | A |
| 4345500 | Alonso | Aug 1982 | A |
| 4356752 | Suzuki | Nov 1982 | A |
| 4399731 | Aoki | Aug 1983 | A |
| 4554855 | Alonso | Nov 1985 | A |
| 4680479 | Alonso | Jul 1987 | A |
| 4704933 | Kurakake | Nov 1987 | A |
| 4731847 | Lybrook | Mar 1988 | A |
| 4745836 | Dannenberg | May 1988 | A |
| 4771671 | Hoff | Sep 1988 | A |
| 4926737 | Minamitaka | May 1990 | A |
| 4982643 | Minamitaka | Jan 1991 | A |
| 5099740 | Minamitaka | Mar 1992 | A |
| 5208416 | Hayakawa | May 1993 | A |
| 5315057 | Land | May 1994 | A |
| 5375501 | Okuda | Dec 1994 | A |
| 5393926 | Johnson | Feb 1995 | A |
| 5451709 | Minamitaka | Sep 1995 | A |
| 5453569 | Saito | Sep 1995 | A |
| 5492049 | Aoki | Feb 1996 | A |
| 5496962 | Meier | Mar 1996 | A |
| 5510573 | Cho | Apr 1996 | A |
| 5521324 | Dannenberg | May 1996 | A |
| 5679913 | Bruti | Oct 1997 | A |
| 5696343 | Nakata | Dec 1997 | A |
| 5723802 | Johnson | Mar 1998 | A |
| 5736663 | Aoki | Apr 1998 | A |
| 5736666 | Goodman | Apr 1998 | A |
| 5753843 | Fay | May 1998 | A |
| 5877445 | Hufford | Mar 1999 | A |
| 5883326 | Goodman | Mar 1999 | A |
| 5913259 | Grubb | Jun 1999 | A |
| 5958005 | Thorne | Sep 1999 | A |
| 6006018 | Burnett | Dec 1999 | A |
| 6012088 | Li | Jan 2000 | A |
| 6028262 | Minamitaka | Feb 2000 | A |
| 6051770 | Milburn | Apr 2000 | A |
| 6072480 | Gorbet | Jun 2000 | A |
| 6075193 | Aoki | Jun 2000 | A |
| 6084169 | Hasegawa | Jul 2000 | A |
| 6103964 | Kay | Aug 2000 | A |
| 6122666 | Beurket | Sep 2000 | A |
| 6162982 | Aoki | Dec 2000 | A |
| 6175072 | Aoki | Jan 2001 | B1 |
| 6252152 | Aoki | Jun 2001 | B1 |
| 6291756 | Urbanek | Sep 2001 | B1 |
| 6297439 | Browne | Oct 2001 | B1 |
| 6319130 | Ooseki | Nov 2001 | B1 |
| 6337433 | Nishimoto | Jan 2002 | B1 |
| 6363350 | Lafe | Mar 2002 | B1 |
| 6385581 | Stephenson | May 2002 | B1 |
| 6388183 | Leh | May 2002 | B1 |
| 6392133 | Georges | May 2002 | B1 |
| 6395970 | Aoki | May 2002 | B2 |
| 6462264 | Elam | Oct 2002 | B1 |
| 6506969 | Baron | Jan 2003 | B1 |
| 6545209 | Flannery | Apr 2003 | B1 |
| 6576828 | Aoki | Jun 2003 | B2 |
| 6606596 | Zirngibl | Aug 2003 | B1 |
| 6633908 | Leymann | Oct 2003 | B1 |
| 6636247 | Hamzy | Oct 2003 | B1 |
| 6636855 | Holloway | Oct 2003 | B2 |
| 6637020 | Hammond | Oct 2003 | B1 |
| 6654794 | French | Nov 2003 | B1 |
| 6684238 | Dutta | Jan 2004 | B1 |
| 6700048 | Terada | Mar 2004 | B1 |
| 6746246 | Cliff | Jun 2004 | B2 |
| 6747201 | Birmingham | Jun 2004 | B2 |
| 6756533 | Aoki | Jun 2004 | B2 |
| 6765997 | Zirngibl | Jul 2004 | B1 |
| 6822153 | Comair | Nov 2004 | B2 |
| 6865533 | Addison | Mar 2005 | B2 |
| 6888999 | Herberger | May 2005 | B2 |
| 6897367 | Leach | May 2005 | B2 |
| 6963839 | Ostermann | Nov 2005 | B1 |
| 6969796 | Sasaki | Nov 2005 | B2 |
| 7003515 | Glaser | Feb 2006 | B1 |
| 7022907 | Lu | Apr 2006 | B2 |
| 7058428 | Yamaki | Jun 2006 | B2 |
| 7075000 | Gang | Jul 2006 | B2 |
| 7102067 | Gang | Sep 2006 | B2 |
| 7105734 | Tucmandl | Sep 2006 | B2 |
| 7115808 | Lu | Oct 2006 | B2 |
| 7133900 | Szeto | Nov 2006 | B1 |
| 7188143 | Szeto | Mar 2007 | B2 |
| 7250567 | Gayama | Jul 2007 | B2 |
| 7268791 | Jannink | Sep 2007 | B1 |
| 7310629 | Mendelson | Dec 2007 | B1 |
| 7356556 | Brydon | Apr 2008 | B2 |
| 7356572 | Jiang | Apr 2008 | B2 |
| 7396990 | Lu | Jul 2008 | B2 |
| 7424682 | Pupius | Sep 2008 | B1 |
| RE40543 | Aoki | Oct 2008 | E |
| 7454472 | Szeto | Nov 2008 | B2 |
| 7454480 | Labio | Nov 2008 | B2 |
| 7478132 | Chen | Jan 2009 | B2 |
| 7491878 | Orr | Feb 2009 | B2 |
| 7498504 | Bourgeois | Mar 2009 | B2 |
| 7542996 | Fanning | Jun 2009 | B2 |
| 7552183 | Coletrane | Jun 2009 | B2 |
| 7568010 | Lyle | Jul 2009 | B2 |
| 7582823 | Kim | Sep 2009 | B2 |
| 7583793 | Jacovi | Sep 2009 | B2 |
| 7605323 | Ishii | Oct 2009 | B2 |
| 7672873 | Kindig | Mar 2010 | B2 |
| 7693746 | Wachi | Apr 2010 | B2 |
| 7696426 | Cope | Apr 2010 | B2 |
| 7703022 | Arthurs | Apr 2010 | B2 |
| 7705231 | Morris | Apr 2010 | B2 |
| 7711838 | Boulter | May 2010 | B1 |
| 7720910 | Goodman | May 2010 | B2 |
| 7720914 | Goodman | May 2010 | B2 |
| 7720934 | Yanase | May 2010 | B2 |
| 7729481 | Thompson | Jun 2010 | B2 |
| 7730178 | Labio | Jun 2010 | B2 |
| 7737853 | Moskowitz | Jun 2010 | B2 |
| 7754955 | Egan | Jul 2010 | B2 |
| 7754959 | Herberger | Jul 2010 | B2 |
| 7774420 | Keohane | Aug 2010 | B2 |
| 7790974 | Sherwani | Sep 2010 | B2 |
| 7792782 | Yun | Sep 2010 | B2 |
| 7792834 | Sorensen | Sep 2010 | B2 |
| 7818397 | Jiang | Oct 2010 | B2 |
| 7822830 | Junghuber | Oct 2010 | B2 |
| 7831670 | Goodman | Nov 2010 | B2 |
| 7840838 | Stewart | Nov 2010 | B2 |
| 7844673 | Bostick | Nov 2010 | B2 |
| 7884274 | Wieder | Feb 2011 | B1 |
| 7902447 | Abrego | Mar 2011 | B1 |
| 7917148 | Rosenberg | Mar 2011 | B2 |
| 7919707 | Harvey | Apr 2011 | B2 |
| 7948357 | Bodin | May 2011 | B2 |
| 7949649 | Whitman | May 2011 | B2 |
| 7958156 | Shriwas | Jun 2011 | B2 |
| 7962558 | Nelson | Jun 2011 | B2 |
| 7974838 | Lukin | Jul 2011 | B1 |
| 7985917 | Morris | Jul 2011 | B2 |
| 8022287 | Yamashita | Sep 2011 | B2 |
| 8026436 | Hufford | Sep 2011 | B2 |
| 8035490 | Hunt | Oct 2011 | B2 |
| 8042118 | Gilfix | Oct 2011 | B2 |
| 8053659 | Ricard | Nov 2011 | B2 |
| 8073854 | Whitman | Dec 2011 | B2 |
| 8143509 | Robertson | Mar 2012 | B1 |
| 8161115 | Yamamoto | Apr 2012 | B2 |
| 8166111 | Bhakta | Apr 2012 | B2 |
| 8184783 | Thompson | May 2012 | B2 |
| 8229935 | Lee | Jul 2012 | B2 |
| 8259192 | Nair | Sep 2012 | B2 |
| 8271354 | Deguchi | Sep 2012 | B2 |
| 8280889 | Whitman | Oct 2012 | B2 |
| 8316146 | Ehn | Nov 2012 | B2 |
| 8352331 | Dunning | Jan 2013 | B2 |
| 8354579 | Park | Jan 2013 | B2 |
| 8357847 | Huet | Jan 2013 | B2 |
| 8359382 | Gailloux | Jan 2013 | B1 |
| 8402097 | Szeto | Mar 2013 | B2 |
| 8428453 | Spiegel | Apr 2013 | B1 |
| 8475173 | Mears | Jul 2013 | B2 |
| 8489606 | Lee | Jul 2013 | B2 |
| 8527905 | Chen | Sep 2013 | B2 |
| 8554868 | Skyrm | Oct 2013 | B2 |
| 8583615 | White | Nov 2013 | B2 |
| 8586847 | Ellis | Nov 2013 | B2 |
| 8595301 | Banks | Nov 2013 | B2 |
| 8627308 | Deluca | Jan 2014 | B2 |
| 8631358 | Louch | Jan 2014 | B2 |
| 8644971 | Weinstein | Feb 2014 | B2 |
| 8660849 | Gruber | Feb 2014 | B2 |
| 8670979 | Gruber | Mar 2014 | B2 |
| 8706503 | Cheyer | Apr 2014 | B2 |
| 8710343 | Kellett | Apr 2014 | B2 |
| 8726167 | Arthurs | May 2014 | B2 |
| 8762435 | Rosenberg | Jun 2014 | B1 |
| 8775972 | Spiegel | Jul 2014 | B2 |
| 8798438 | Yagnik | Aug 2014 | B1 |
| 8799000 | Guzzoni | Aug 2014 | B2 |
| 8826453 | Arrelid | Sep 2014 | B2 |
| 8866846 | Kim | Oct 2014 | B2 |
| 8868744 | Levell | Oct 2014 | B2 |
| 8874026 | Anstandig | Oct 2014 | B2 |
| 8874147 | Chu | Oct 2014 | B2 |
| 8880615 | Tummalapenta | Nov 2014 | B2 |
| 8892446 | Cheyer | Nov 2014 | B2 |
| 8892660 | Chen | Nov 2014 | B2 |
| 8898766 | Garmark | Nov 2014 | B2 |
| 8903716 | Chen | Dec 2014 | B2 |
| 8909725 | Sehn | Dec 2014 | B1 |
| 8914752 | Spiegel | Dec 2014 | B1 |
| 8921677 | Severino | Dec 2014 | B1 |
| 8927846 | Matusiak | Jan 2015 | B2 |
| 8930191 | Gruber | Jan 2015 | B2 |
| 8938507 | Bostick | Jan 2015 | B2 |
| 8942986 | Cheyer | Jan 2015 | B2 |
| 8949525 | Niemela | Feb 2015 | B2 |
| 8969699 | Tabata | Mar 2015 | B2 |
| 8996538 | Cremer | Mar 2015 | B1 |
| 9015285 | Ebsen | Apr 2015 | B1 |
| 9018505 | Okuda | Apr 2015 | B2 |
| 9021038 | Banks | Apr 2015 | B2 |
| 9026943 | Spiegel | May 2015 | B1 |
| 9031243 | Leboeuf | May 2015 | B2 |
| 9032543 | Arrelid | May 2015 | B2 |
| 9042921 | Karmarkar | May 2015 | B2 |
| 9043412 | Chen | May 2015 | B2 |
| 9043850 | Hoffert | May 2015 | B2 |
| 9063640 | Hoffert | Jun 2015 | B2 |
| 9066048 | Hoffert | Jun 2015 | B2 |
| 9069868 | Chen | Jun 2015 | B2 |
| 9071562 | Chung | Jun 2015 | B2 |
| 9071798 | Hoffert | Jun 2015 | B2 |
| 9076264 | Gillespie | Jul 2015 | B1 |
| 9076423 | Matusiak | Jul 2015 | B2 |
| 9083770 | Dröse | Jul 2015 | B1 |
| 9092759 | Hammer | Jul 2015 | B2 |
| 9094137 | Sehn | Jul 2015 | B1 |
| 9094806 | Chu | Jul 2015 | B2 |
| 9099064 | Sheffer | Aug 2015 | B2 |
| 9100618 | Hoffert | Aug 2015 | B2 |
| 9110955 | Bernhardsson | Aug 2015 | B1 |
| 9111164 | Anderton | Aug 2015 | B1 |
| 9112849 | Werkelin Ahlin | Aug 2015 | B1 |
| 9113301 | Spiegel | Aug 2015 | B1 |
| 9117447 | Gruber | Aug 2015 | B2 |
| 9143681 | Ebsen | Sep 2015 | B1 |
| 9148424 | Yang | Sep 2015 | B1 |
| 9158754 | Whitman | Oct 2015 | B2 |
| 9165255 | Shetty | Oct 2015 | B1 |
| 9195383 | Garmark | Nov 2015 | B2 |
| 9213747 | Cremer | Dec 2015 | B2 |
| 9225310 | Lukin | Dec 2015 | B1 |
| 9225674 | Fioretti | Dec 2015 | B2 |
| 9225897 | Sehn | Dec 2015 | B1 |
| 9230218 | González | Jan 2016 | B2 |
| 9237202 | Sehn | Jan 2016 | B1 |
| 9246967 | Garmark | Jan 2016 | B2 |
| 9276886 | Samaranayake | Mar 2016 | B1 |
| 9277126 | Marlin | Mar 2016 | B2 |
| 9288200 | Werkelin Ahlin | Mar 2016 | B1 |
| 9294425 | Son | Mar 2016 | B1 |
| 9299104 | Boulter | Mar 2016 | B2 |
| 9313154 | Son | Apr 2016 | B1 |
| 9318108 | Gruber | Apr 2016 | B2 |
| 9319445 | Garmark | Apr 2016 | B2 |
| 9342613 | Arthurs | May 2016 | B2 |
| 9350312 | Wishnick | May 2016 | B1 |
| 9354868 | Deluca | May 2016 | B2 |
| 9361645 | Boulter | Jun 2016 | B2 |
| 9361869 | Rex | Jun 2016 | B2 |
| 9367587 | Bieschke | Jun 2016 | B2 |
| 9369514 | Bernhardsson | Jun 2016 | B2 |
| 9380059 | Aslund | Jun 2016 | B2 |
| 9385983 | Sehn | Jul 2016 | B1 |
| 9390696 | Kiely | Jul 2016 | B2 |
| 9396354 | Murphy | Jul 2016 | B1 |
| 9402093 | Lieu | Jul 2016 | B2 |
| 9406072 | Whitman | Aug 2016 | B2 |
| 9407712 | Sehn | Aug 2016 | B1 |
| 9407816 | Sehn | Aug 2016 | B1 |
| 9423998 | Dziuk | Aug 2016 | B2 |
| 9424604 | Boulter | Aug 2016 | B2 |
| 9430783 | Sehn | Aug 2016 | B1 |
| 9432428 | Werkelin Ahlin | Aug 2016 | B2 |
| D766967 | Giovannini | Sep 2016 | S |
| 9436962 | Boulter | Sep 2016 | B2 |
| 9438582 | Garmark | Sep 2016 | B2 |
| 9443266 | Boulter | Sep 2016 | B2 |
| 9448763 | Dziuk | Sep 2016 | B1 |
| 9449341 | Boulter | Sep 2016 | B2 |
| 9451329 | Whitman | Sep 2016 | B2 |
| D768674 | Hanover | Oct 2016 | S |
| 9473432 | Chung | Oct 2016 | B2 |
| 9482882 | Hanover | Nov 2016 | B1 |
| 9482883 | Meisenholder | Nov 2016 | B1 |
| 9483166 | Dziuk | Nov 2016 | B2 |
| 9489113 | Dziuk | Nov 2016 | B2 |
| 9489527 | Arrelid | Nov 2016 | B2 |
| 9503500 | Bernhardsson | Nov 2016 | B2 |
| 9503780 | Hoffert | Nov 2016 | B2 |
| 9509269 | Rosenberg | Nov 2016 | B1 |
| 9510024 | Öman | Nov 2016 | B2 |
| 9510131 | Zhu | Nov 2016 | B2 |
| 9514476 | Kay | Dec 2016 | B2 |
| 9515630 | Wallander | Dec 2016 | B2 |
| 9516082 | Hoffert | Dec 2016 | B2 |
| 9529888 | Hoffert | Dec 2016 | B2 |
| 9531989 | Jehan | Dec 2016 | B1 |
| 9532171 | Allen | Dec 2016 | B2 |
| 9536560 | Jehan | Jan 2017 | B2 |
| 9542917 | Sheffer | Jan 2017 | B2 |
| 9542918 | Matusiak | Jan 2017 | B2 |
| 9547679 | Whitman | Jan 2017 | B2 |
| 9554186 | Tsiridis | Jan 2017 | B2 |
| 9563268 | Smith | Feb 2017 | B2 |
| 9563700 | Garmark | Feb 2017 | B2 |
| 9565456 | Helferty | Feb 2017 | B2 |
| 9568994 | Jehan | Feb 2017 | B2 |
| 9570059 | Garmark | Feb 2017 | B2 |
| D781906 | Yu | Mar 2017 | S |
| D782520 | Yu | Mar 2017 | S |
| D782533 | Yu | Mar 2017 | S |
| 9589237 | Qamar | Mar 2017 | B1 |
| 9600466 | Whitman | Mar 2017 | B2 |
| 9606620 | Dziuk | Mar 2017 | B2 |
| 9607594 | Chon | Mar 2017 | B2 |
| 9609448 | Bentley | Mar 2017 | B2 |
| 9613118 | Whitman | Apr 2017 | B2 |
| 9613654 | Cameron | Apr 2017 | B2 |
| 9626436 | Rodger | Apr 2017 | B2 |
| 9635068 | Garmark | Apr 2017 | B2 |
| 9635416 | Hoffert | Apr 2017 | B2 |
| 9635556 | Afzelius | Apr 2017 | B2 |
| 9641891 | Hoffert | May 2017 | B2 |
| 9654531 | Hoffert | May 2017 | B2 |
| 9654532 | Strigeus | May 2017 | B2 |
| 9654822 | Hoffert | May 2017 | B2 |
| 9659068 | Mattsson | May 2017 | B1 |
| 9661379 | Hoffert | May 2017 | B2 |
| 9668217 | Bamberger | May 2017 | B1 |
| 9679305 | Bhat | Jun 2017 | B1 |
| 9716733 | Strigeus | Jul 2017 | B2 |
| 9728173 | Watanabe | Aug 2017 | B2 |
| 9729816 | Jehan | Aug 2017 | B1 |
| 9740023 | Ashwood | Aug 2017 | B1 |
| 9741327 | Rutledge | Aug 2017 | B2 |
| 9742871 | Gibson | Aug 2017 | B1 |
| 9746692 | Streets | Aug 2017 | B1 |
| 9749378 | Olenfalk | Aug 2017 | B2 |
| 9753925 | Cremer | Sep 2017 | B2 |
| 9766854 | Medaghri | Sep 2017 | B2 |
| 9773483 | Rutledge | Sep 2017 | B2 |
| 9787687 | Lof | Oct 2017 | B2 |
| 9792010 | Hoffert | Oct 2017 | B2 |
| 9794309 | Persson | Oct 2017 | B2 |
| 9794827 | Zhu | Oct 2017 | B2 |
| 9798514 | Silva | Oct 2017 | B2 |
| 9798823 | Cody | Oct 2017 | B2 |
| 9799312 | Cabral | Oct 2017 | B1 |
| 9800631 | Persson | Oct 2017 | B2 |
| 9825801 | Bakken | Nov 2017 | B1 |
| 9843764 | Jehan | Dec 2017 | B1 |
| 9875010 | Persson | Jan 2018 | B2 |
| 9881596 | Matusiak | Jan 2018 | B2 |
| 9883284 | Bohrarper | Jan 2018 | B2 |
| 9904506 | Jehan | Feb 2018 | B1 |
| 9917869 | Hoffert | Mar 2018 | B2 |
| D814186 | Spiegel | Apr 2018 | S |
| D814493 | Brody | Apr 2018 | S |
| D815127 | Phillips | Apr 2018 | S |
| D815128 | Phillips | Apr 2018 | S |
| D815129 | Phillips | Apr 2018 | S |
| D815130 | Phillips | Apr 2018 | S |
| 9933993 | Garmark | Apr 2018 | B2 |
| 9934467 | Jacobson | Apr 2018 | B2 |
| 9934785 | Hulaud | Apr 2018 | B1 |
| 9935943 | Werkelin Ahlin | Apr 2018 | B2 |
| 9935944 | Garmark | Apr 2018 | B2 |
| 9942283 | Garmark | Apr 2018 | B2 |
| 9942356 | Gibson | Apr 2018 | B1 |
| 9948736 | Mengistu | Apr 2018 | B1 |
| 9973635 | Edling | May 2018 | B1 |
| 9973806 | Tsiridis | May 2018 | B2 |
| 9978426 | Smith | May 2018 | B2 |
| 9979768 | Hoffert | May 2018 | B2 |
| D820298 | Yu | Jun 2018 | S |
| 10002123 | Whitman | Jun 2018 | B2 |
| 10003840 | Richman | Jun 2018 | B2 |
| 10021156 | Conway | Jul 2018 | B2 |
| 10025786 | Jehan | Jul 2018 | B2 |
| 10033474 | Gibson | Jul 2018 | B1 |
| 10034064 | Hoffert | Jul 2018 | B2 |
| 10038962 | Bentley | Jul 2018 | B2 |
| D824924 | Phillips | Aug 2018 | S |
| D825581 | Phillips | Aug 2018 | S |
| D825582 | Phillips | Aug 2018 | S |
| 10055413 | Jehan | Aug 2018 | B2 |
| 10063600 | Marsh | Aug 2018 | B1 |
| 10063608 | Oskarsson | Aug 2018 | B2 |
| 10066954 | Swanson | Sep 2018 | B1 |
| 10075496 | Garmark | Sep 2018 | B2 |
| 10082939 | Medaghri Alaoui | Sep 2018 | B2 |
| D829742 | Phillips | Oct 2018 | S |
| D829743 | Phillips | Oct 2018 | S |
| D829750 | Phillips | Oct 2018 | S |
| D830375 | Phillips | Oct 2018 | S |
| D830395 | Phillips | Oct 2018 | S |
| D831691 | Brody | Oct 2018 | S |
| 10089309 | Polacek | Oct 2018 | B2 |
| 10089578 | Jehan | Oct 2018 | B2 |
| 10095467 | Balassanian | Oct 2018 | B2 |
| 10097604 | Hoffert | Oct 2018 | B2 |
| 10101960 | Jehan | Oct 2018 | B2 |
| 10102680 | Jurgenson | Oct 2018 | B2 |
| 10108708 | O'Driscoll | Oct 2018 | B2 |
| 10109264 | Cabral | Oct 2018 | B2 |
| 10110649 | Hoffert | Oct 2018 | B2 |
| 10110947 | Hoffert | Oct 2018 | B2 |
| 10115435 | Lee | Oct 2018 | B2 |
| 10133545 | Gibson | Nov 2018 | B2 |
| 10133918 | Chang | Nov 2018 | B1 |
| 10133974 | Cassidy | Nov 2018 | B2 |
| 10134059 | Mishra | Nov 2018 | B2 |
| 10148789 | Gibson | Dec 2018 | B2 |
| 10165357 | Bohrarper | Dec 2018 | B2 |
| 10165402 | Davis | Dec 2018 | B1 |
| 10171055 | Bailey | Jan 2019 | B2 |
| 10185538 | Oskarsson | Jan 2019 | B2 |
| 10185539 | Bailey | Jan 2019 | B2 |
| 10185767 | Bernhardsson | Jan 2019 | B2 |
| 10187676 | Lieu | Jan 2019 | B2 |
| 10209950 | Smith | Feb 2019 | B2 |
| 10223063 | Gibson | Mar 2019 | B2 |
| 10235127 | Jehan | Mar 2019 | B2 |
| 10248380 | Bailey | Apr 2019 | B2 |
| 10250933 | Whitman | Apr 2019 | B2 |
| 10262641 | Silverstein | Apr 2019 | B2 |
| D847788 | Baker | May 2019 | S |
| 10282163 | Jehan | May 2019 | B2 |
| 10298636 | Krawczyk | May 2019 | B2 |
| 10311842 | Silverstein | Jun 2019 | B2 |
| 10334073 | Gibson | Jun 2019 | B2 |
| 10360260 | Zhu | Jul 2019 | B2 |
| 10366334 | Jacobson | Jul 2019 | B2 |
| 10372757 | Jehan | Aug 2019 | B2 |
| 10380649 | Johnson | Aug 2019 | B2 |
| 10387478 | Cao | Aug 2019 | B2 |
| 10387481 | Jehan | Aug 2019 | B2 |
| 10387489 | Reiley, Jr. | Aug 2019 | B1 |
| 10396744 | Mcclellan | Aug 2019 | B2 |
| 10412183 | Gibson | Sep 2019 | B2 |
| 10423943 | Wood | Sep 2019 | B2 |
| 10445809 | Dunning | Oct 2019 | B2 |
| 10459904 | Whitman | Oct 2019 | B2 |
| 10460248 | Stowell | Oct 2019 | B2 |
| 10467998 | Silverstein | Nov 2019 | B2 |
| 10467999 | Lyske | Nov 2019 | B2 |
| 10482857 | Lyske | Nov 2019 | B2 |
| 10600398 | Pachet | Mar 2020 | B2 |
| 10657934 | Kolen | May 2020 | B1 |
| 10672371 | Silverstein | Jun 2020 | B2 |
| 10679596 | Balassanian | Jun 2020 | B2 |
| 10699684 | Wood | Jun 2020 | B2 |
| 20010007960 | Yoshihara | Jul 2001 | A1 |
| 20010025561 | Milbum | Oct 2001 | A1 |
| 20010037196 | Iwamoto | Nov 2001 | A1 |
| 20010047717 | Aoki | Dec 2001 | A1 |
| 20020000156 | Nishimoto | Jan 2002 | A1 |
| 20020002899 | Gjerdingen | Jan 2002 | A1 |
| 20020007720 | Aoki | Jan 2002 | A1 |
| 20020007721 | Aoki | Jan 2002 | A1 |
| 20020007722 | Aoki | Jan 2002 | A1 |
| 20020011145 | Aoki | Jan 2002 | A1 |
| 20020017188 | Aoki | Feb 2002 | A1 |
| 20020023529 | Kurakake | Feb 2002 | A1 |
| 20020029685 | Aoki | Mar 2002 | A1 |
| 20020033090 | Iwamoto | Mar 2002 | A1 |
| 20020035915 | Tolonen | Mar 2002 | A1 |
| 20020129023 | Holloway | Sep 2002 | A1 |
| 20020134219 | Aoki | Sep 2002 | A1 |
| 20020177186 | Sternheimer | Nov 2002 | A1 |
| 20020184128 | Holtsinger | Dec 2002 | A1 |
| 20020193996 | Squibbs | Dec 2002 | A1 |
| 20030013497 | Yamaki | Jan 2003 | A1 |
| 20030018727 | Yamamoto | Jan 2003 | A1 |
| 20030037664 | Comair | Feb 2003 | A1 |
| 20030089216 | Birmingham | May 2003 | A1 |
| 20030131715 | Georges | Jul 2003 | A1 |
| 20030160944 | Foote | Aug 2003 | A1 |
| 20030183065 | Leach | Oct 2003 | A1 |
| 20030200859 | Futamase | Oct 2003 | A1 |
| 20030205124 | Foote | Nov 2003 | A1 |
| 20030205125 | Futamase | Nov 2003 | A1 |
| 20040019645 | Goodman | Jan 2004 | A1 |
| 20040024822 | Werndorfer | Feb 2004 | A1 |
| 20040025668 | Jarrett | Feb 2004 | A1 |
| 20040027369 | Kellock | Feb 2004 | A1 |
| 20040089140 | Georges | May 2004 | A1 |
| 20040089141 | Georges | May 2004 | A1 |
| 20040159213 | Eruera | Aug 2004 | A1 |
| 20040215731 | Tzann-en Szeto | Oct 2004 | A1 |
| 20050051021 | Laakso | Mar 2005 | A1 |
| 20050076772 | Gartland-Jones | Apr 2005 | A1 |
| 20050086052 | Shih | Apr 2005 | A1 |
| 20050091278 | Wang | Apr 2005 | A1 |
| 20050102351 | Jiang | May 2005 | A1 |
| 20050109194 | Gayama | May 2005 | A1 |
| 20050180462 | Yi | Aug 2005 | A1 |
| 20050223071 | Hosono | Oct 2005 | A1 |
| 20050267896 | Goodman | Dec 2005 | A1 |
| 20050273499 | Goodman | Dec 2005 | A1 |
| 20060011044 | Chew | Jan 2006 | A1 |
| 20060015560 | Macauley | Jan 2006 | A1 |
| 20060018447 | Jacovi | Jan 2006 | A1 |
| 20060059236 | Sheppard | Mar 2006 | A1 |
| 20060065104 | Ball | Mar 2006 | A1 |
| 20060122840 | Anderson | Jun 2006 | A1 |
| 20060130635 | Rubang, Jr. | Jun 2006 | A1 |
| 20060168346 | Chen | Jul 2006 | A1 |
| 20060212818 | Lee | Sep 2006 | A1 |
| 20060230909 | Song | Oct 2006 | A1 |
| 20060230910 | Song | Oct 2006 | A1 |
| 20060236848 | Stone | Oct 2006 | A1 |
| 20060243119 | Rubang, Jr. | Nov 2006 | A1 |
| 20060258340 | Eronen | Nov 2006 | A1 |
| 20070005719 | Szeto | Jan 2007 | A1 |
| 20070006708 | Laakso | Jan 2007 | A1 |
| 20070022732 | Holloway | Feb 2007 | A1 |
| 20070044639 | Farbood | Mar 2007 | A1 |
| 20070094341 | Bostick | Apr 2007 | A1 |
| 20070106731 | Bhakta | May 2007 | A1 |
| 20070112919 | Lyle | May 2007 | A1 |
| 20070116195 | Thompson | May 2007 | A1 |
| 20070137463 | Lumsden | Jun 2007 | A1 |
| 20070174401 | Chu | Jul 2007 | A1 |
| 20070208990 | Kim | Sep 2007 | A1 |
| 20070209006 | Arthurs | Sep 2007 | A1 |
| 20070221044 | Orr | Sep 2007 | A1 |
| 20070227342 | Ide | Oct 2007 | A1 |
| 20070261535 | Sherwani | Nov 2007 | A1 |
| 20070285250 | Moskowitz | Dec 2007 | A1 |
| 20070288589 | Chen | Dec 2007 | A1 |
| 20070300101 | Stewart | Dec 2007 | A1 |
| 20080010372 | Khedouri | Jan 2008 | A1 |
| 20080053293 | Georges | Mar 2008 | A1 |
| 20080136605 | Hunt | Jun 2008 | A1 |
| 20080139177 | Jiang | Jun 2008 | A1 |
| 20080141850 | Cope | Jun 2008 | A1 |
| 20080147774 | Tummalapenta | Jun 2008 | A1 |
| 20080156178 | Georges | Jul 2008 | A1 |
| 20080168154 | Skyrm | Jul 2008 | A1 |
| 20080189171 | Wasserblat | Aug 2008 | A1 |
| 20080195742 | Gilfix | Aug 2008 | A1 |
| 20080212947 | Nesvadba | Sep 2008 | A1 |
| 20080222264 | Bostick | Sep 2008 | A1 |
| 20080230598 | Bodin | Sep 2008 | A1 |
| 20080235285 | Della Pasqua | Sep 2008 | A1 |
| 20080256208 | Keohane | Oct 2008 | A1 |
| 20080288095 | Miyajima | Nov 2008 | A1 |
| 20090019174 | Ehn | Jan 2009 | A1 |
| 20090031000 | Szeto | Jan 2009 | A1 |
| 20090064851 | Morris | Mar 2009 | A1 |
| 20090069914 | Kemp | Mar 2009 | A1 |
| 20090071315 | Fortuna | Mar 2009 | A1 |
| 20090089389 | Chen | Apr 2009 | A1 |
| 20090114079 | Egan | May 2009 | A1 |
| 20090119097 | Master | May 2009 | A1 |
| 20090132668 | Coletrane | May 2009 | A1 |
| 20090164598 | Nelson | Jun 2009 | A1 |
| 20090193090 | Banks | Jul 2009 | A1 |
| 20090216744 | Shriwas | Aug 2009 | A1 |
| 20090217805 | Lee | Sep 2009 | A1 |
| 20090222536 | Junghuber | Sep 2009 | A1 |
| 20090238538 | Fink | Sep 2009 | A1 |
| 20090244000 | Thompson | Oct 2009 | A1 |
| 20090249945 | Yamashita | Oct 2009 | A1 |
| 20090291707 | Choi | Nov 2009 | A1 |
| 20090316862 | Sugimoto | Dec 2009 | A1 |
| 20100018382 | Feeney | Jan 2010 | A1 |
| 20100031804 | Chevreau | Feb 2010 | A1 |
| 20100043625 | Van Geenen | Feb 2010 | A1 |
| 20100050854 | Huet | Mar 2010 | A1 |
| 20100115432 | Arthurs | May 2010 | A1 |
| 20100131895 | Wohlert | May 2010 | A1 |
| 20100192755 | Morris | Aug 2010 | A1 |
| 20100212478 | Taub | Aug 2010 | A1 |
| 20100224051 | Kurebayashi | Sep 2010 | A1 |
| 20100250510 | Herberger | Sep 2010 | A1 |
| 20100250585 | Hagg | Sep 2010 | A1 |
| 20100257995 | Kamiya | Oct 2010 | A1 |
| 20100288106 | Sherwani | Nov 2010 | A1 |
| 20100305732 | Serletic | Dec 2010 | A1 |
| 20100307320 | Hoeberechts | Dec 2010 | A1 |
| 20100307321 | Mann | Dec 2010 | A1 |
| 20110010321 | Pachet | Jan 2011 | A1 |
| 20110075851 | Leboeuf | Mar 2011 | A1 |
| 20110142420 | Singer | Jun 2011 | A1 |
| 20110184542 | Tsoneva | Jul 2011 | A1 |
| 20110224969 | Mulligan | Sep 2011 | A1 |
| 20110258383 | Niemela | Oct 2011 | A1 |
| 20110273455 | Powar | Nov 2011 | A1 |
| 20110276396 | Rathod | Nov 2011 | A1 |
| 20110316793 | Fushiki | Dec 2011 | A1 |
| 20110320545 | Hammer | Dec 2011 | A1 |
| 20120005667 | Deluca | Jan 2012 | A1 |
| 20120007605 | Benedikt | Jan 2012 | A1 |
| 20120007884 | Kim | Jan 2012 | A1 |
| 20120084373 | Chen | Apr 2012 | A1 |
| 20120131115 | Levell | May 2012 | A1 |
| 20120185778 | Arthurs | Jul 2012 | A1 |
| 20120210212 | Chen | Aug 2012 | A1 |
| 20120259240 | Llewellynn | Oct 2012 | A1 |
| 20120297958 | Rassool | Nov 2012 | A1 |
| 20120312145 | Kellett | Dec 2012 | A1 |
| 20130005346 | Chu | Jan 2013 | A1 |
| 20130006627 | Guthery | Jan 2013 | A1 |
| 20130110505 | Gruber | May 2013 | A1 |
| 20130110519 | Cheyer | May 2013 | A1 |
| 20130124658 | Fioretti | May 2013 | A1 |
| 20130139271 | Arrelid | May 2013 | A1 |
| 20130185081 | Cheyer | Jul 2013 | A1 |
| 20130283150 | Chen | Oct 2013 | A1 |
| 20130287227 | Wallander | Oct 2013 | A1 |
| 20130332400 | Gonzalez | Dec 2013 | A1 |
| 20130332532 | Bernhardsson | Dec 2013 | A1 |
| 20130332842 | Bernhardsson | Dec 2013 | A1 |
| 20140000440 | Georges | Jan 2014 | A1 |
| 20140006483 | Garmark | Jan 2014 | A1 |
| 20140006947 | Garmark | Jan 2014 | A1 |
| 20140040401 | Banks | Feb 2014 | A1 |
| 20140052282 | Balassanian | Feb 2014 | A1 |
| 20140053711 | Serletic, II | Feb 2014 | A1 |
| 20140055633 | Marlin | Feb 2014 | A1 |
| 20140058735 | Sharp | Feb 2014 | A1 |
| 20140069263 | Chen | Mar 2014 | A1 |
| 20140089897 | Deluca | Mar 2014 | A1 |
| 20140108929 | Garmark | Apr 2014 | A1 |
| 20140115114 | Garmark | Apr 2014 | A1 |
| 20140129953 | Spiegel | May 2014 | A1 |
| 20140139555 | Levy | May 2014 | A1 |
| 20140164361 | Chung | Jun 2014 | A1 |
| 20140164524 | Chung | Jun 2014 | A1 |
| 20140174279 | Wong | Jun 2014 | A1 |
| 20140214927 | Garmark | Jul 2014 | A1 |
| 20140215334 | Garmark | Jul 2014 | A1 |
| 20140230629 | Wieder | Aug 2014 | A1 |
| 20140230630 | Wieder | Aug 2014 | A1 |
| 20140230631 | Wieder | Aug 2014 | A1 |
| 20140260915 | Okuda | Sep 2014 | A1 |
| 20140279817 | Whitman | Sep 2014 | A1 |
| 20140289241 | Anderson | Sep 2014 | A1 |
| 20140301573 | Kiely | Oct 2014 | A1 |
| 20140310779 | Lof | Oct 2014 | A1 |
| 20140331332 | Arrelid | Nov 2014 | A1 |
| 20140337959 | Garmark | Nov 2014 | A1 |
| 20140344718 | Rapaport | Nov 2014 | A1 |
| 20140355789 | Bohrarper | Dec 2014 | A1 |
| 20140359024 | Spiegel | Dec 2014 | A1 |
| 20140359032 | Spiegel | Dec 2014 | A1 |
| 20140368734 | Hoffert | Dec 2014 | A1 |
| 20140368735 | Hoffert | Dec 2014 | A1 |
| 20140368737 | Hoffert | Dec 2014 | A1 |
| 20140368738 | Hoffert | Dec 2014 | A1 |
| 20140372888 | Hoffert | Dec 2014 | A1 |
| 20140373057 | Hoffert | Dec 2014 | A1 |
| 20150017915 | Hennequin | Jan 2015 | A1 |
| 20150026578 | Rav-Acha | Jan 2015 | A1 |
| 20150033932 | Balassanian | Feb 2015 | A1 |
| 20150039726 | Hoffert | Feb 2015 | A1 |
| 20150039780 | Hoffert | Feb 2015 | A1 |
| 20150039781 | Hoffert | Feb 2015 | A1 |
| 20150040169 | Hoffert | Feb 2015 | A1 |
| 20150058733 | Novikoff | Feb 2015 | A1 |
| 20150059558 | Morell | Mar 2015 | A1 |
| 20150088828 | Strigeus | Mar 2015 | A1 |
| 20150088890 | Hoffert | Mar 2015 | A1 |
| 20150088899 | Hoffert | Mar 2015 | A1 |
| 20150089075 | Strigeus | Mar 2015 | A1 |
| 20150106887 | Aslund | Apr 2015 | A1 |
| 20150113407 | Hoffert | Apr 2015 | A1 |
| 20150154979 | Uemura | Jun 2015 | A1 |
| 20150161908 | Ur | Jun 2015 | A1 |
| 20150179157 | Chon | Jun 2015 | A1 |
| 20150194185 | Eronen | Jul 2015 | A1 |
| 20150199122 | Garmark | Jul 2015 | A1 |
| 20150206523 | Song | Jul 2015 | A1 |
| 20150229684 | Olenfalk | Aug 2015 | A1 |
| 20150234833 | Cremer | Aug 2015 | A1 |
| 20150248618 | Johnson | Sep 2015 | A1 |
| 20150255052 | Rex | Sep 2015 | A1 |
| 20150277707 | Dziuk | Oct 2015 | A1 |
| 20150289023 | Richman | Oct 2015 | A1 |
| 20150289025 | Mcleod | Oct 2015 | A1 |
| 20150293925 | Greenzeiger | Oct 2015 | A1 |
| 20150317391 | Harrison | Nov 2015 | A1 |
| 20150317680 | Richman | Nov 2015 | A1 |
| 20150317690 | Mishra | Nov 2015 | A1 |
| 20150317691 | Mishra | Nov 2015 | A1 |
| 20150319479 | Mishra | Nov 2015 | A1 |
| 20150324594 | Arrelid | Nov 2015 | A1 |
| 20150331943 | Luo | Nov 2015 | A1 |
| 20150334455 | Hoffert | Nov 2015 | A1 |
| 20150340021 | Sheffer | Nov 2015 | A1 |
| 20150365719 | Hoffert | Dec 2015 | A1 |
| 20150365720 | Hoffert | Dec 2015 | A1 |
| 20150365795 | Allen | Dec 2015 | A1 |
| 20150370466 | Hoffert | Dec 2015 | A1 |
| 20160006927 | Sehn | Jan 2016 | A1 |
| 20160007077 | Hoffert | Jan 2016 | A1 |
| 20160055838 | Serletic, II | Feb 2016 | A1 |
| 20160066004 | Lieu | Mar 2016 | A1 |
| 20160071549 | Von Sneidern | Mar 2016 | A1 |
| 20160080780 | Öman | Mar 2016 | A1 |
| 20160080835 | Von Sneidern | Mar 2016 | A1 |
| 20160085773 | Chang | Mar 2016 | A1 |
| 20160085863 | Allen | Mar 2016 | A1 |
| 20160094863 | Helferty | Mar 2016 | A1 |
| 20160099901 | Allen | Apr 2016 | A1 |
| 20160103589 | Dziuk | Apr 2016 | A1 |
| 20160103595 | Dziuk | Apr 2016 | A1 |
| 20160103656 | Dziuk | Apr 2016 | A1 |
| 20160124953 | Cremer | May 2016 | A1 |
| 20160124969 | Rashad | May 2016 | A1 |
| 20160125078 | Rashad | May 2016 | A1 |
| 20160125860 | Rashad | May 2016 | A1 |
| 20160127772 | Tsiridis | May 2016 | A1 |
| 20160132594 | Rashad | May 2016 | A1 |
| 20160133241 | Rashad | May 2016 | A1 |
| 20160133242 | Morell | May 2016 | A1 |
| 20160147435 | Brody | May 2016 | A1 |
| 20160148605 | Minamitaka | May 2016 | A1 |
| 20160148606 | Minamitaka | May 2016 | A1 |
| 20160173763 | Marlin | Jun 2016 | A1 |
| 20160180887 | Sehn | Jun 2016 | A1 |
| 20160182422 | Sehn | Jun 2016 | A1 |
| 20160182590 | Afzelius | Jun 2016 | A1 |
| 20160182875 | Sehn | Jun 2016 | A1 |
| 20160189222 | Richman | Jun 2016 | A1 |
| 20160189223 | Mcleod | Jun 2016 | A1 |
| 20160189232 | Meyer | Jun 2016 | A1 |
| 20160189249 | Meyer | Jun 2016 | A1 |
| 20160191574 | Garmark | Jun 2016 | A1 |
| 20160191590 | Werkelin Ahlin | Jun 2016 | A1 |
| 20160191599 | Stridsman | Jun 2016 | A1 |
| 20160191997 | Eklund | Jun 2016 | A1 |
| 20160192096 | Bentley | Jun 2016 | A1 |
| 20160196812 | Rashad | Jul 2016 | A1 |
| 20160203586 | Chang | Jul 2016 | A1 |
| 20160210545 | Anderton | Jul 2016 | A1 |
| 20160210947 | Rutledge | Jul 2016 | A1 |
| 20160210951 | Rutledge | Jul 2016 | A1 |
| 20160226941 | Esún | Aug 2016 | A1 |
| 20160234151 | Son | Aug 2016 | A1 |
| 20160239248 | Sehn | Aug 2016 | A1 |
| 20160247189 | Shirley | Aug 2016 | A1 |
| 20160247496 | Pachet | Aug 2016 | A1 |
| 20160249091 | Lennon | Aug 2016 | A1 |
| 20160260123 | Mishra | Sep 2016 | A1 |
| 20160260140 | Shirley | Sep 2016 | A1 |
| 20160267944 | Lammers | Sep 2016 | A1 |
| 20160285937 | Whitman | Sep 2016 | A1 |
| 20160292269 | O'Driscoll | Oct 2016 | A1 |
| 20160292272 | O'Driscoll | Oct 2016 | A1 |
| 20160292771 | Afzelius | Oct 2016 | A1 |
| 20160294896 | O'Driscoll | Oct 2016 | A1 |
| 20160309209 | Lieu | Oct 2016 | A1 |
| 20160313872 | Garmark | Oct 2016 | A1 |
| 20160321708 | Sehn | Nov 2016 | A1 |
| 20160323691 | Zhu | Nov 2016 | A1 |
| 20160328360 | Pavlovskaia | Nov 2016 | A1 |
| 20160328409 | Ogle | Nov 2016 | A1 |
| 20160334945 | Medaghri Alaoui | Nov 2016 | A1 |
| 20160334978 | Persson | Nov 2016 | A1 |
| 20160334979 | Persson | Nov 2016 | A1 |
| 20160334980 | Persson | Nov 2016 | A1 |
| 20160335045 | Medaghri Alaoui | Nov 2016 | A1 |
| 20160335046 | Medaghri Alaoui | Nov 2016 | A1 |
| 20160335047 | Medaghri Alaoui | Nov 2016 | A1 |
| 20160335048 | Medaghri Alaoui | Nov 2016 | A1 |
| 20160335049 | Persson | Nov 2016 | A1 |
| 20160335266 | Ogle | Nov 2016 | A1 |
| 20160337260 | Persson | Nov 2016 | A1 |
| 20160337419 | Persson | Nov 2016 | A1 |
| 20160337425 | Medaghri Alaoui | Nov 2016 | A1 |
| 20160337429 | Persson | Nov 2016 | A1 |
| 20160337432 | Persson | Nov 2016 | A1 |
| 20160337434 | Bajraktari | Nov 2016 | A1 |
| 20160337854 | Afzelius | Nov 2016 | A1 |
| 20160342199 | Smith | Nov 2016 | A1 |
| 20160342200 | Dziuk | Nov 2016 | A1 |
| 20160342201 | Jehan | Nov 2016 | A1 |
| 20160342295 | Jehan | Nov 2016 | A1 |
| 20160342382 | Jehan | Nov 2016 | A1 |
| 20160342594 | Jehan | Nov 2016 | A1 |
| 20160342598 | Jehan | Nov 2016 | A1 |
| 20160342686 | Garmark | Nov 2016 | A1 |
| 20160342687 | Garmark | Nov 2016 | A1 |
| 20160343363 | Garmark | Nov 2016 | A1 |
| 20160343399 | Jehan | Nov 2016 | A1 |
| 20160343410 | Smith | Nov 2016 | A1 |
| 20160366458 | Whitman | Dec 2016 | A1 |
| 20160378269 | Conway | Dec 2016 | A1 |
| 20160379274 | Irwin | Dec 2016 | A1 |
| 20160381106 | Conway | Dec 2016 | A1 |
| 20170010796 | Dziuk | Jan 2017 | A1 |
| 20170017993 | Shirley | Jan 2017 | A1 |
| 20170019441 | Garmark | Jan 2017 | A1 |
| 20170019446 | Son | Jan 2017 | A1 |
| 20170024092 | Dziuk | Jan 2017 | A1 |
| 20170024093 | Dziuk | Jan 2017 | A1 |
| 20170024399 | Boyle | Jan 2017 | A1 |
| 20170024486 | Jacobson | Jan 2017 | A1 |
| 20170024650 | Jacobson | Jan 2017 | A1 |
| 20170024655 | Stowell | Jan 2017 | A1 |
| 20170039027 | Dziuk | Feb 2017 | A1 |
| 20170048563 | Öman | Feb 2017 | A1 |
| 20170048750 | Zhu | Feb 2017 | A1 |
| 20170075468 | Dziuk | Mar 2017 | A1 |
| 20170083505 | Whitman | Mar 2017 | A1 |
| 20170084261 | Watanabe | Mar 2017 | A1 |
| 20170085552 | Werkelin Ahlin | Mar 2017 | A1 |
| 20170085929 | Arpteg | Mar 2017 | A1 |
| 20170092247 | Silverstein | Mar 2017 | A1 |
| 20170092324 | Leonard | Mar 2017 | A1 |
| 20170102837 | Toumpelis | Apr 2017 | A1 |
| 20170103075 | Toumpelis | Apr 2017 | A1 |
| 20170103740 | Hwang | Apr 2017 | A1 |
| 20170116533 | Jehan | Apr 2017 | A1 |
| 20170118192 | Garmark | Apr 2017 | A1 |
| 20170124713 | Jurgenson | May 2017 | A1 |
| 20170134795 | Tsiridis | May 2017 | A1 |
| 20170139912 | Whitman | May 2017 | A1 |
| 20170140060 | Cody | May 2017 | A1 |
| 20170140261 | Qamar | May 2017 | A1 |
| 20170149717 | Sehn | May 2017 | A1 |
| 20170150211 | Helferty | May 2017 | A1 |
| 20170154109 | Lynch | Jun 2017 | A1 |
| 20170161119 | Boyle | Jun 2017 | A1 |
| 20170161382 | Ouimet | Jun 2017 | A1 |
| 20170169107 | Bernhardsson | Jun 2017 | A1 |
| 20170169858 | Lee | Jun 2017 | A1 |
| 20170177297 | Jehan | Jun 2017 | A1 |
| 20170177585 | Rodger | Jun 2017 | A1 |
| 20170177605 | Hoffert | Jun 2017 | A1 |
| 20170180438 | Persson | Jun 2017 | A1 |
| 20170180826 | Hoffert | Jun 2017 | A1 |
| 20170187672 | Banks | Jun 2017 | A1 |
| 20170187771 | Falcon | Jun 2017 | A1 |
| 20170188102 | Zhang | Jun 2017 | A1 |
| 20170192649 | Bakken | Jul 2017 | A1 |
| 20170195813 | Bentley | Jul 2017 | A1 |
| 20170220316 | Garmark | Aug 2017 | A1 |
| 20170229030 | Aguayo, Jr. | Aug 2017 | A1 |
| 20170230295 | Polacek | Aug 2017 | A1 |
| 20170230354 | Afzelius | Aug 2017 | A1 |
| 20170230429 | Garmark | Aug 2017 | A1 |
| 20170230438 | Turkoglu | Aug 2017 | A1 |
| 20170235540 | Jehan | Aug 2017 | A1 |
| 20170235541 | Smith | Aug 2017 | A1 |
| 20170235826 | Garmark | Aug 2017 | A1 |
| 20170244770 | Eckerdal | Aug 2017 | A1 |
| 20170248799 | Streets | Aug 2017 | A1 |
| 20170248801 | Ashwood | Aug 2017 | A1 |
| 20170249306 | Allen | Aug 2017 | A1 |
| 20170251039 | Hoffert | Aug 2017 | A1 |
| 20170262139 | Patel | Sep 2017 | A1 |
| 20170262253 | Silva | Sep 2017 | A1 |
| 20170262994 | Kudriashov | Sep 2017 | A1 |
| 20170263029 | Yan | Sep 2017 | A1 |
| 20170263030 | Allen | Sep 2017 | A1 |
| 20170263225 | Silverstein | Sep 2017 | A1 |
| 20170263226 | Silverstein | Sep 2017 | A1 |
| 20170263227 | Silverstein | Sep 2017 | A1 |
| 20170263228 | Silverstein | Sep 2017 | A1 |
| 20170264578 | Allen | Sep 2017 | A1 |
| 20170264660 | Eckerdal | Sep 2017 | A1 |
| 20170264817 | Yan | Sep 2017 | A1 |
| 20170270125 | Mattsson | Sep 2017 | A1 |
| 20170286536 | Rando | Oct 2017 | A1 |
| 20170286752 | Gusarov | Oct 2017 | A1 |
| 20170289234 | Andreou | Oct 2017 | A1 |
| 20170289489 | Hoffert | Oct 2017 | A1 |
| 20170295250 | Samaranayake | Oct 2017 | A1 |
| 20170300567 | Jehan | Oct 2017 | A1 |
| 20170301372 | Jehan | Oct 2017 | A1 |
| 20170308794 | Fischerström | Oct 2017 | A1 |
| 20170344246 | Burfitt | Nov 2017 | A1 |
| 20170344539 | Zvoncek | Nov 2017 | A1 |
| 20170346867 | Olenfalk | Nov 2017 | A1 |
| 20170353405 | O'Driscoll | Dec 2017 | A1 |
| 20170358285 | Cabral | Dec 2017 | A1 |
| 20170358320 | Cameron | Dec 2017 | A1 |
| 20170366780 | Jehan | Dec 2017 | A1 |
| 20170372364 | Andreou | Dec 2017 | A1 |
| 20170374003 | Allen | Dec 2017 | A1 |
| 20170374508 | Davis | Dec 2017 | A1 |
| 20180004480 | Medaghri Alaoui | Jan 2018 | A1 |
| 20180005026 | Shaburov | Jan 2018 | A1 |
| 20180005420 | Bondich | Jan 2018 | A1 |
| 20180007286 | Li | Jan 2018 | A1 |
| 20180007444 | Li | Jan 2018 | A1 |
| 20180018079 | Monastyrshyn | Jan 2018 | A1 |
| 20180018397 | Cody | Jan 2018 | A1 |
| 20180018948 | Silverstein | Jan 2018 | A1 |
| 20180025004 | Koenig | Jan 2018 | A1 |
| 20180025372 | Ahmed | Jan 2018 | A1 |
| 20180041517 | Lof | Feb 2018 | A1 |
| 20180052921 | Deglopper | Feb 2018 | A1 |
| 20180054592 | Jehan | Feb 2018 | A1 |
| 20180054704 | Toumpelis | Feb 2018 | A1 |
| 20180069743 | Bakken | Mar 2018 | A1 |
| 20180076913 | Kiely | Mar 2018 | A1 |
| 20180089904 | Jurgenson | Mar 2018 | A1 |
| 20180095715 | Jehan | Apr 2018 | A1 |
| 20180096064 | Lennon | Apr 2018 | A1 |
| 20180103002 | Sehn | Apr 2018 | A1 |
| 20180109820 | Pompa | Apr 2018 | A1 |
| 20180129659 | Jehan | May 2018 | A1 |
| 20180129745 | Jehan | May 2018 | A1 |
| 20180136612 | Zayets-Volshin | May 2018 | A1 |
| 20180137845 | Prokop | May 2018 | A1 |
| 20180139333 | Edling | May 2018 | A1 |
| 20180150276 | Vacek | May 2018 | A1 |
| 20180157746 | Zhu | Jun 2018 | A1 |
| 20180164986 | Al Majid | Jun 2018 | A1 |
| 20180167726 | Bohrarper | Jun 2018 | A1 |
| 20180181849 | Cassidy | Jun 2018 | A1 |
| 20180182394 | Hulaud | Jun 2018 | A1 |
| 20180188054 | Kennedy | Jul 2018 | A1 |
| 20180188945 | Garmark | Jul 2018 | A1 |
| 20180189020 | Oskarsson | Jul 2018 | A1 |
| 20180189021 | Oskarsson | Jul 2018 | A1 |
| 20180189023 | Garmark | Jul 2018 | A1 |
| 20180189226 | Hofverberg | Jul 2018 | A1 |
| 20180189278 | Garmark | Jul 2018 | A1 |
| 20180189306 | Lamere | Jul 2018 | A1 |
| 20180189408 | O'Driscoll | Jul 2018 | A1 |
| 20180190253 | O'Driscoll | Jul 2018 | A1 |
| 20180191654 | O'Driscoll | Jul 2018 | A1 |
| 20180191795 | Oskarsson | Jul 2018 | A1 |
| 20180192082 | O'Driscoll | Jul 2018 | A1 |
| 20180192108 | Lyons | Jul 2018 | A1 |
| 20180192239 | Liusaari | Jul 2018 | A1 |
| 20180192240 | Liusaari | Jul 2018 | A1 |
| 20180192285 | Schmidt | Jul 2018 | A1 |
| 20180226063 | Wood | Aug 2018 | A1 |
| 20180233119 | Patti | Aug 2018 | A1 |
| 20180239580 | Garmark | Aug 2018 | A1 |
| 20180246694 | Gibson | Aug 2018 | A1 |
| 20180246961 | Gibson | Aug 2018 | A1 |
| 20180248965 | Gibson | Aug 2018 | A1 |
| 20180248976 | Gibson | Aug 2018 | A1 |
| 20180248978 | Gibson | Aug 2018 | A1 |
| 20180300331 | Jehan | Oct 2018 | A1 |
| 20180321904 | Bailey | Nov 2018 | A1 |
| 20180321908 | Bailey | Nov 2018 | A1 |
| 20180323763 | Bailey | Nov 2018 | A1 |
| 20180332024 | Garmark | Nov 2018 | A1 |
| 20180351937 | Ahlin | Dec 2018 | A1 |
| 20180358053 | Smith | Dec 2018 | A1 |
| 20180367229 | Gibson | Dec 2018 | A1 |
| 20180367580 | Marsh | Dec 2018 | A1 |
| 20190018557 | O'Driscoll | Jan 2019 | A1 |
| 20190018645 | Mcclellan | Jan 2019 | A1 |
| 20190018702 | O'Driscoll | Jan 2019 | A1 |
| 20190023705 | Le Fur | Jan 2019 | A1 |
| 20190026817 | Helferty | Jan 2019 | A1 |
| 20190073191 | Bailey | Mar 2019 | A1 |
| 20190074807 | Bailey | Mar 2019 | A1 |
| 20190237051 | Silverstein | Aug 2019 | A1 |
| 20190279606 | Silverstein | Sep 2019 | A1 |
| 20190304418 | Silverstein | Oct 2019 | A1 |
| 20190340245 | Zhu | Nov 2019 | A1 |
| 20190341898 | Mcclellan | Nov 2019 | A1 |
| 20190362696 | Balassanian | Nov 2019 | A1 |
| 20200168187 | Silverstein | May 2020 | A1 |
| 20200168188 | Silverstein | May 2020 | A1 |
| 20200168189 | Silverstein | May 2020 | A1 |
| 20200168190 | Silverstein | May 2020 | A1 |
| 20200168191 | Silverstein | May 2020 | A1 |
| 20200168192 | Silverstein | May 2020 | A1 |
| 20200168193 | Silverstein | May 2020 | A1 |
| 20200168194 | Silverstein | May 2020 | A1 |
| 20200168195 | Silverstein | May 2020 | A1 |
| 20200168196 | Silverstein | May 2020 | A1 |
| 20200168197 | Silverstein | May 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 2002355066 | Mar 2007 | AU |
| 2894332 | Dec 2015 | CA |
| 2894332 | Dec 2015 | CA |
| 2895728 | Jan 2016 | CA |
| 2910158 | Apr 2016 | CA |
| 106663264 | May 2017 | CN |
| 106688031 | May 2017 | CN |
| 107004225 | Aug 2017 | CN |
| 107111430 | Aug 2017 | CN |
| 107111828 | Aug 2017 | CN |
| 107251006 | Oct 2017 | CN |
| 107430697 | Dec 2017 | CN |
| 107430767 | Dec 2017 | CN |
| 107431632 | Dec 2017 | CN |
| 107710188 | Feb 2018 | CN |
| 107924590 | Apr 2018 | CN |
| 108604378 | Sep 2018 | CN |
| 10047266 | Apr 2001 | DE |
| 10047266 | Apr 2001 | DE |
| 112011103172 | Jul 2013 | DE |
| 112011103081 | Sep 2013 | DE |
| 1683034 | Jul 2006 | EP |
| 2015542 | Jan 2009 | EP |
| 2015542 | Jan 2009 | EP |
| 2096324 | Sep 2009 | EP |
| 2248311 | Nov 2010 | EP |
| 2378435 | Oct 2011 | EP |
| 2388954 | Nov 2011 | EP |
| 2663899 | Nov 2013 | EP |
| 2808870 | Dec 2014 | EP |
| 2808870 | Dec 2014 | EP |
| 2868060 | May 2015 | EP |
| 2868061 | May 2015 | EP |
| 2925008 | Sep 2015 | EP |
| 2999191 | Mar 2016 | EP |
| 2999191 | Mar 2016 | EP |
| 3035273 | Jun 2016 | EP |
| 3035273 | Jun 2016 | EP |
| 3041245 | Jul 2016 | EP |
| 3041245 | Jul 2016 | EP |
| 3055790 | Aug 2016 | EP |
| 3059973 | Aug 2016 | EP |
| 3061245 | Aug 2016 | EP |
| 3076353 | Oct 2016 | EP |
| 3093786 | Nov 2016 | EP |
| 3093786 | Nov 2016 | EP |
| 3094098 | Nov 2016 | EP |
| 3094099 | Nov 2016 | EP |
| 3096323 | Nov 2016 | EP |
| 3151576 | Apr 2017 | EP |
| 3196782 | Jul 2017 | EP |
| 3215962 | Sep 2017 | EP |
| 3255862 | Dec 2017 | EP |
| 3255862 | Dec 2017 | EP |
| 3255889 | Dec 2017 | EP |
| 3255889 | Dec 2017 | EP |
| 3258394 | Dec 2017 | EP |
| 3258436 | Dec 2017 | EP |
| 3268876 | Jan 2018 | EP |
| 3285453 | Feb 2018 | EP |
| 3285453 | Feb 2018 | EP |
| 3287913 | Feb 2018 | EP |
| 3306892 | Apr 2018 | EP |
| 3306892 | Apr 2018 | EP |
| 3310066 | Apr 2018 | EP |
| 3321827 | May 2018 | EP |
| 3324356 | May 2018 | EP |
| 3328090 | May 2018 | EP |
| 3330872 | Jun 2018 | EP |
| 3343448 | Jul 2018 | EP |
| 3343448 | Jul 2018 | EP |
| 3343483 | Jul 2018 | EP |
| 3343484 | Jul 2018 | EP |
| 3343844 | Jul 2018 | EP |
| 3343880 | Jul 2018 | EP |
| 3367269 | Aug 2018 | EP |
| 3367639 | Aug 2018 | EP |
| 3404893 | Nov 2018 | EP |
| 3425919 | Jan 2019 | EP |
| 419KOLNP2006 | Sep 2007 | IN |
| 298031 | Jul 2011 | IN |
| 1369MUM2011 | Aug 2011 | IN |
| 3680749 | Aug 2005 | JP |
| 5941065 | Feb 2014 | JP |
| 1020160013213 | Feb 2016 | KR |
| 535612 | Oct 2012 | SE |
| 9324645 | Dec 1993 | WO |
| 1997021210 | Jun 1997 | WO |
| 0108134 | Feb 2001 | WO |
| 0135667 | May 2001 | WO |
| 0184353 | Nov 2001 | WO |
| 0186624 | Nov 2001 | WO |
| 0186624 | Nov 2001 | WO |
| 05057821 | Jun 2005 | WO |
| 2005057821 | Jun 2005 | WO |
| 2006071876 | Jul 2006 | WO |
| 2007106371 | Sep 2007 | WO |
| 12096617 | Jul 2012 | WO |
| 2012136599 | Oct 2012 | WO |
| 2012150602 | Nov 2012 | WO |
| 2013003854 | Jan 2013 | WO |
| 2013080048 | Jun 2013 | WO |
| 2013153449 | Oct 2013 | WO |
| 2013153449 | Oct 2013 | WO |
| 2013181662 | Dec 2013 | WO |
| 2013181662 | Dec 2013 | WO |
| 2013184957 | Dec 2013 | WO |
| 2013185107 | Dec 2013 | WO |
| 2012150602 | Jan 2014 | WO |
| 2014001912 | Jan 2014 | WO |
| 2014001912 | Jan 2014 | WO |
| 2014001913 | Jan 2014 | WO |
| 2014001913 | Jan 2014 | WO |
| 2014001914 | Jan 2014 | WO |
| 2014001914 | Jan 2014 | WO |
| 2014057356 | Apr 2014 | WO |
| 2014057356 | Apr 2014 | WO |
| 2013003854 | May 2014 | WO |
| 2014064531 | May 2014 | WO |
| 2014068309 | May 2014 | WO |
| 14144833 | Sep 2014 | WO |
| 14153133 | Sep 2014 | WO |
| 2014166953 | Oct 2014 | WO |
| 2014194262 | Dec 2014 | WO |
| 2014194262 | Dec 2014 | WO |
| 2014204863 | Dec 2014 | WO |
| 2014204863 | Dec 2014 | WO |
| 2015040494 | Mar 2015 | WO |
| 2015040494 | Mar 2015 | WO |
| 2015056099 | Apr 2015 | WO |
| 2015056102 | Apr 2015 | WO |
| 15170126 | Nov 2015 | WO |
| 2015192026 | Dec 2015 | WO |
| 2016007285 | Jan 2016 | WO |
| 2016044424 | Mar 2016 | WO |
| 2016054562 | Apr 2016 | WO |
| 2016065131 | Apr 2016 | WO |
| 2016085936 | Jun 2016 | WO |
| 2016100318 | Jun 2016 | WO |
| 2016100318 | Jun 2016 | WO |
| 2016100342 | Jun 2016 | WO |
| 2016107799 | Jul 2016 | WO |
| 2016108086 | Jul 2016 | WO |
| 2016108087 | Jul 2016 | WO |
| 2016112299 | Jul 2016 | WO |
| 2016118338 | Jul 2016 | WO |
| 2016156553 | Oct 2016 | WO |
| 2016156554 | Oct 2016 | WO |
| 2016156555 | Oct 2016 | WO |
| 2016156555 | Oct 2016 | WO |
| 2016179166 | Nov 2016 | WO |
| 2016179235 | Nov 2016 | WO |
| 2016184866 | Nov 2016 | WO |
| 2016184867 | Nov 2016 | WO |
| 2016184868 | Nov 2016 | WO |
| 2016184869 | Nov 2016 | WO |
| 2016184871 | Nov 2016 | WO |
| 2016186881 | Nov 2016 | WO |
| 16209685 | Dec 2016 | WO |
| 2017015218 | Jan 2017 | WO |
| 2017015224 | Jan 2017 | WO |
| 2017019457 | Feb 2017 | WO |
| 2017019458 | Feb 2017 | WO |
| 2017019460 | Feb 2017 | WO |
| 17048450 | Mar 2017 | WO |
| 2017040633 | Mar 2017 | WO |
| 2017048450 | Mar 2017 | WO |
| 17058844 | Apr 2017 | WO |
| 2017058844 | Apr 2017 | WO |
| 2017070427 | Apr 2017 | WO |
| 2017075476 | May 2017 | WO |
| 2017095800 | Jun 2017 | WO |
| 2017095807 | Jun 2017 | WO |
| 2017103675 | Jun 2017 | WO |
| 2017106529 | Jun 2017 | WO |
| 2017109570 | Jun 2017 | WO |
| 2017140786 | Aug 2017 | WO |
| 2017147305 | Aug 2017 | WO |
| 2017151519 | Sep 2017 | WO |
| 2017153435 | Sep 2017 | WO |
| 2017153437 | Sep 2017 | WO |
| 2017175061 | Oct 2017 | WO |
| 2017182304 | Oct 2017 | WO |
| 2017210129 | Dec 2017 | WO |
| 2017218033 | Dec 2017 | WO |
| 2018006053 | Jan 2018 | WO |
| 2018015122 | Jan 2018 | WO |
| 2018017592 | Jan 2018 | WO |
| 2018022626 | Feb 2018 | WO |
| 2018033789 | Feb 2018 | WO |
| 18226418 | Dec 2018 | WO |
| 18226419 | Dec 2018 | WO |
| Entry |
|---|
| US 10,126,932 B1, 11/2018, Trncic (withdrawn) |
| Richard Portelli, “ORB Composer Getting Started 1.0.0”, Hexachords Entertainment, updated Apr. 1, 2018, (33 Pages). |
| Richard Portelli, “ORB Composer Documentation 1.0.0”, Hexachords Entertainment, updated Apr. 2, 2018, (36 Pages). |
| Richard Portelli, “ORB Composer Dashboard—Screenshot”, Hexachords Entertainment, updated Aug. 17, 2019, (1 Page). |
| Richard Portelli, “Getting Started with ORB Composer S V 1.5”, Hexachords Entertainment, updated Dec. 8, 2019, (21 Pages). |
| Richard Portelli, “Getting Started with ORB Composer S V 1.0”, Hexachords Entertainment, updated Mar. 3, 2019, (15 Pages). |
| IBM, “IBM Watson Beat: Cutting a track for the Red Bull Racing with a music-making machine”, published and accessed at https://www.ibm.com/case-studies/ibm-watson-beat, on Feb. 4, 2019, (9 pages). |
| The Lilypond Development Team, “Wikipedia Summary of LilyPond Music Engraving Software”, published and accessed at https://en.wikipedia.org/wiki/LilyPond, on Dec. 8, 2019, (8 Pages). |
| The Lilypond Development Team, “LilyPond Notation Reference (2015) Version 2.19.83”, downloaded from http://www.lilypond.com, Dec. 2019, (882 Pages). |
| The Lilypond Development Team, “LilyPond Music Notation for Everyone: Text Input,” published and accessed at http://lilypond.org/text-input.html, on Dec. 8, 2019, (4 Pages). |
| The Lilypond Development Team, “LilyPond Learning Manual (2015) Version 2.19.83”, downloaded from http://www.lilypond.com on Dec. 8, 2019, (216 Pages). |
| The Lilypond Development Team, “LilyPond Music Glossary (2015) Version 2.19.83”, downloaded from http://www.lilypond.com on Dec. 8, 2019, (98 Pages). |
| The Lilypond Development Team, “LilyPond Usage (2015) Version 2.19.83”, downloaded from http://www.lilypond.com on Dec. 8, 2019, (69 Pages). |
| Avid Corporation, Screenshots from Avid Website entitled “Music Creation Solutions: Overview; Meeting the Challenge; Integrated Hardware & Software; and Notation and Scoring,” published and accessed from https://www.avid.com/solutions/music-creation on Dec. 8, 2019, (3 Pages). |
| Amazon.com, Inc., Webpages from Amazon Web Services, Inc., for the AWS Deepcomposer, published and accessed by https://aws.amazon.com/deepcomposer/. on Dec. 8, 2019, (9 Pages). |
| Boomy Corporation, “Boomy Talks AI Music: We Want to Make Music That's Meaningful”, published at https://musically.com/2019/07/31/boomy-talks-ai-music-we-want-to-make-music-thats-meaningful/ on Jul. 31, 2019, (13 Pages). |
| Musical.ly Inc., “2018 Music AI: The Music-Ally Guide”, published on Nov. 22, 2018, and downloaded from https://musically.com/wp-content/uploads/2018/11/Music-Ally-AI-Music-Guide.pdf , (24 Pages). |
| “NotePerformer 3 User Guide”, Wallander Instruments AB, updated Sep. 12, 2019, (64 Pages). |
| “NotePerformer 3.2 Version History”, Wallander Instruments AB, updated Sep. 2, 2019, (33 Pages). |
| “Affective Key Characteristics”, from Christian Schubart's “Ideen zu einer Aesthetik der Tonkunst” (1806), translated by Rita Steblin in A History of Key Characteristics in the 18th and Early 19th Centuries, UMI Research Press, 1983, and republished at http://www.wmich.edu/mus-theo/courses/keys.html, (3 Pages). |
| “Characteristics of Musical Keys,: a selection of information from the Internet about the emotion or moodassociated with musical keys”, published at http://biteyourownelbow.com/keychar.htm, on Oct. 14, 2009, (6 Page). |
| Ableton AG, “Ableton Reference manual Version 10”, Jan. 2018, (pp. 1-759). |
| Alex Rodriguez Lopez, Antonio Pedro Oliveira, and Amilcar Cardosa, “Real-Time Emotion-Driven Music Engine”, Centre for Informatics and Systems, University of Coimbra, Portugal, Conference Paper, Jan. 2010, published in ResearchGate on Jun. 2015, (6 Pages). |
| Alexis John Kirke, and Eduardo Reck Miranda, “Artificial Social Composition: A Multi-Agent System for Composing Music Performances by Emotional Communication”, 2009, Interdisciplinary Center for Computer Music Research, University of Plymouth, UK, (19 Pages). |
| Alison Mattek, “Computational Methods for Portraying Emotion in Generative Music Composition”, May 2010, Undergraduate Thesis, Department of Music Engineering, University of Miami, Miami, Florida, (62 Page). |
| Alper Gungormusler, Natasa Paterson-Paulberg, and Mads Haahr, “BarelyMusician: An Adaptive Music Engine For /Video Games”, AES 56th International Conference, London, UK, Feb. 11-13, 2015, published in ResearchGate, Feb. 2015, (9 Pages). |
| Anthony Prechtl, Robin Laney, Alistair Willis, Robert Samuels, Algorithmic Music as Intelligent Game Music, Apr. 2014, published in AISB50: The 50th Annual Convention of the AISB, Apr. 11, 2014, London, UK, (5 Pages). |
| Avid Technology Inc., “Pro Tools Reference Guide”, Dec. 2018, (pp. 1-1489). |
| Michael Levine, Behind the Audio, “Why Hans Zimmer got the Job You Wanted (And You Didn't)”, Jul. 2013, (pp. 1-3). |
| Bernard A. Hutchins Jr., Walter H. Ku, “A Simple Hardware Pitch Extractor”, JAES, Mar. 1, 1982,vol. 30 issue 3, pp. 135-139, Audio Engineering Society Inc., Ithaca, New York, (5 Page). |
| Bill Manaris, Dana Hughes, Yiorgos Vassilandonakis, “Monterey Mirror: Combining Markov Models, Genetic Algorithms, and Power Laws”, Computer Science Department, College of Charleston, SC, USA, appearred in Proceedings of 1st Workshop in Evolutionary Music, 2011 IEEE Congress on Evolutionary Computation (CEC 2011), New Orleans, LA, USA, Jun. 5, 2011, pp. 33-40, (8 Pages). |
| Bitwig, Dave Linnenbank, “Bitwig Studio User Guide”, Feb. 2017, (pp. 1-383). |
| Bongjun Kim, Woon Seung Yeo, “Probabilistic Prediction of Rhythmic Characteristics in Markov Chain-Based Melodic Sequences”, 2013 Graduate School of Culture Technology, Korea Republic, published in 2013 ICMC Idea, pp. 29-432, (4 Pages ). |
| Brit Cruise, “Real Time Control of Emotional Affect in Algorithmic Music”, May 31, 2010, britcruise.com, (20 Pages). |
| Cambridge Innovation Capital Press Release, “Cambridge Innovation Capital Leads Follow-On Funding Round For Digital Music Creator Jukedeck”, Dec. 7, 2015, Cambridge University, Cambridge England, (3 Pages). |
| Caroline Palmer, Sean Hutchins, “What is Musical Prosody, Psychology of Learning and Motivation”, 2005, vol. 46, Elsevier Press, Montreal, Canada, (63 Pages). |
| Cheng Long, Raymond Chi-Wing Wong, Raymond Ka Wai Sze, “A Melody Composer Based on Frequent Pattern Mining”, 2013, The Hong Kong University of Science and Technology, Hong Kong, (4 Pages). |
| Chih-Fang Huang, Wei-Gang Hong, Min-Hsuan Li, “A Research of Automatic Composition and Singing Voice Synthesis System for Taiwanese Popular Songs”, published in Proceedings ICMC, 2014, Sep. 4-20, 2014, Athens, Greece, (6 Page). |
| Chih-Fang Huang, En-Ju Lin, “An Emotion-Based Method to Perform Algorithmic Composition”, Jun. 2013, Department of Information Communications, Kainan University, Taiwan, (4 Pages). |
| Christopher Ariza, “An Open Design for Computer-Aided Algorithmic Music Composition: athenaCL”, 2005, New York University, NY, NY, published on Dissertation.com, Boca Raton, Florida, 2005 (ISBN 1-58112-292-6), (25 Pages). |
| Christopher Ariza, Navigating the Landscape of Computer Aided Algorithmic Composition Systems: a Definition, Seven Descriptors, and a Lexicon of Systems and Research, New York University, New York, New York, published as MIT Open Course Ware, 21M.380 Music and Technology: Algorithmic and Generative MusicSpring, 2010, (8 Pages). |
| Chunyang Song, Marcus Pearce, Christopher Harte, “Synpy: A Python Toolkit for Syncopation Modelling”, 2015, Queen Mary, University of London, London UK, (6 Page). |
| Claudio Galmonte, Dimitrij Hmeljak, “Study for a Real-Time Voice-to-Synthesized-Sound Converter”, 1996, University of Trieste, Italy, (6 Pages). |
| Cockos Inc, “Up and Running: A Reaper User Guide”, Apr. 2019, (pp. 1-464). |
| Dave Phillips, Finlay, Ohio, USA, Review of Henrich K. Taube: Notes from the Metalevel: Introduction to Algorithmic Music Composition (2004), published in Computer Music Journal (CMJ), vol. 26, Issue 3,2005 Fall, The MIT Press, Cambridge, MA, at http://www.computermusicjournal.org/reviews/29-3/phillips-taube.html, 3 Pages. |
| David Cope, “Experiments in Music Intelligence (EMI)”, University of California, Santa Cruz, 1987, ICMC Proceedings, pp. 174-181, (8 Page). |
| David Cope, “Techniques of the Contempory Composer”, Schirmer Thomson Learning, 1997, (123 Pages). |
| Donya Quick, “Kulitta: A Framework for Automated Music Composition”, Dec. 2014, Yale University, US, (229 Pages). |
| Donya Quick, Paul Hudak, “Grammar-Based Automated Music Composition in Haskell”, 2013, Department of Computer Science, Yale University, USA, (20 Pages). |
| Donya Quick, Paul Hudak, “Grammar-Based Automated Music Composition in Haskell”, 2013, Yale University, USA, (12 Pages). |
| Evening Standard, Samuel Fischwick, “Robot rock: how AI singstars use machine learning to write harmonies”, Mar. 2018, (pp. 1-3). |
| Flow Machines, “‘Happy’ With the Reflexive Looper”, Jun. 2016, (pp. 1). |
| Francois Pachet, Pierre Roy, Julian Moreira, Mark D'Inverno, “Reflexive Loopers for Solo Musical Improvisation”, Apr. 2013, (pp. 1-5). |
| G. Scott Vercoe, “Moodtrack: Practical Methods for Assembling Emotion-Driven Music”, 2006, Massachusetts Institute of Technology, Massachusetts, (86 Pages). |
| George Sioros, Carlos Guedes, “Automatic Rhythmic Performance in Max/MSP: the kin.rythmicator”, published in 2011 International Conference on New Interfaces for Musical Expression, Oslo, Norway, May 30-Jun. 1, 2011, (4 Pages). |
| Guilherme Ludwig, “Topics in Statistics: Extracting Patterns in Music for Composition via Markov Chains”, May 11, 2012, University of Wisconsin, US, (18 Pages). |
| Gustavo Diaz-Jerez, “Algorithmic Music: Using mathematical Models in Music Composition”, Aug. 2000, The Manhattan School of Music, New York, (284 Pages). |
| Hanna Jarvelainen, “Algorithmic Musical Composition”, Apr. 7, 2000, Helsinki University of Technology, Finland, (12 Pages). |
| Heinrich Konrad Taube, “Notes from the Metalevel: An Introduction to Computer Composition”, first published online by Swets Zeitlinger Publishing on Oct. 5, 2003 at http://www.moz.ac.at/sem/lehre/lib/bib/software/cm/ Notes from the Metalevel/intro.html, then later by Routledge, Taylor & Francis in 2005 (ISBN 10: 9026519575 ISBN 13: 9789026519574 Hardcover), (313 Pages). |
| Heinrich Taube, “An Introduction to Common Music”, Computer Music Journal, Spring 1997, vol. 21, MIT Press, USA, pp. 29-34. |
| Horacio Alberto Garcia Salaa, Alexander Gelbukh, Hiram Calvo, Fernando Gal in Do Soria, Automatic Music Compositon with Simple Probabilistic Generative Grammars, Polibits, 2011 ,vol. 44, pp. 57-63, Center for Technological Design and Development in Computer Science, Mexico City, Mexico. |
| Horacio Alberto Garcia Salas, Alexander Gelbukh, Musical Composer Based on Detection of Typical Patterns in a Human Composer's Style, 2006, Mexico, (6 Pages). |
| Iannis Xenakis, Formalized Music: Thought and Mathematics in Composition, Pendragon Press, 1992, (201 Scanned Pages). |
| IBM, “IBM Watson Beat”, Nov. 2011, (pp. 1-9). |
| IEEE Access, Luca Turchet, “Smart Musical Instruments: Vision, design Principles, and Future Directions”, Oct. 2018, (pp. 1-20). |
| Image Line Software, “FL Studio: Getting Started Manual”, Jan. 2017, (pp. 1-89). |
| International Search Report and Written Opinion of the International Searching Authority, dated Feb. 7, 2017 PCT/US2016/054066, (37 Pages). |
| Isabel Lacatus, “Composing Music to Picture”, Nov. 2017, (pp. 1-8). |
| Isabel Lacatus, “How to Compose Like Hans Zimmer”, Dec. 2017, (pp. 1-5). |
| Jacob M. Peck, Explorations in Algorithmic Composition: Systems of Composition and Examination of Several Original Works, Oct. 2011, (63 Pages). |
| James Harkins, A Practical Guide to Patterns, 2009, Supercollider, (72 Pages). |
| Joel Douek, “Music and Emotion—A Composer's Perspective”, vol. 7, Article 82, Frontiers in Systems Neuroscience, Nov. 2013, (4 Pages). |
| Joel L. Carbonera, Joao L. T. Silva, An Emergent Markovian Model to Stochastic Music Composition, 2008, University of Caxias do Sul, Brazil, (10 Pages). |
| John Brownlee, “Can Computers Write Music That Has a Soul?”, FastCompany, Aug. 2013, (11 Pages). |
| John J. Dubnowski, Ronald W. Schafer, Lawrence R. Rabiner, Real-Time Digital Hardware Pitch Detector, vol. 24, IEEE IEEE Transactions on Acoustics, Speech, and Signal Processing, Feb. 1976, (7 pages). |
| Jon Brantingham, “How to Spot a Film”, Aug. 2017, (pp. 1-12). |
| Jon Sneyers, Danny De Schreye, “Apopcaleaps: Automatic Music Generation with CHRiSM”, 2010, K.U. Leuven, Belgium, (8 Pages). |
| Kento Watanabe et al, “Modeling Structural Topic Transitions for Automatic Lyrics Generation”, PACLIC 28,2014, pp. 422-431, Graduate School of Information Sciences Tohoku University, Japan, (10 Pages). |
| Kris Goffin, “Music Feels Like Moods Feel”, vol. 5, Article 327, Frontiers in Psychology, Apr. 2014, (4 Pages). |
| Kristine Monteith, Tony Martinez and Dan Ventura, “Automatic Generation of Melodic Accompaniments for Lyrics”, 2012, Proceedings of the Third International Conference on Computational Creativity, pp. 87-94, 15 Pages. |
| Kristine Monteith, Virginia Francisco, Tony Martinez, Pablo Gervas and Dan Ventura, “Automatic Generation of Emotionally-Targeted Soundtracks”, 2011 Proceedings of the Second International Conference on Computational Creativity, pp. 60-62, 3 Pages. |
| Kristine Monteith, Virginia Francisco, Tony Martinez, Pablo Gervas Dan Ventura, “Automatic Generation of Music for Inducing Emotive Response”, Computer Science Department, Brigham Young University, Proceedings of the First International Conference on Computational Creativity, 2010, pp. 140-149, (10 Pages). |
| Kurt Kleiner,“Is that Mozart or a Machine? Software can Compose Music in Classical, Pop, or Jazz Styles”, Dec. 16, 2011, Phys.org, (1 page). |
| LBB Online, “Music Machines: Jukedeck is Using AI to Compose Music”, Sep. 2017, (pp. 1-5). |
| Leon Harkleroad, “The Math Behind Music”, Aug. 2006, Cambridge University Press, UK, (139 Pages). |
| Lorenzo J. Tardon, Carles Roig, Isabel Barbancho, Ana M Barbancho, Automatic Melody Composition Based on a Probabilistic Model of Music Style and Harmonic Rules, Aug. 2014, Knowledge Based Systems. |
| M D Plumbley, S A Abdallah, Automatic Music Transcription ans Audio Source Separation, 2001, Dept of Electronic Engineering, University of London, London, (20 pages). |
| Maia Hoeberechts, Ryan Demopoulos and Michael Katchabaw, “A Flexible Music Composition Engine”, Department of Computer Science, Middlesex College, The University of Western Ontario, London, Ontario, Canada, published in Audio Mostly 2007, 2nd Conference on Interaction with Sound, Conference Proceedings, Sep. 27-28, 2007, Rontgenbau, Ilmenau, Germany, Fraunhofer Institute for Digital Media Technology IDMT, (6 Pages). |
| Marco Marchini, Francois Pachet, Benoit Carre, “Reflexive Looper for Structured Pop Music”, May 2017, (pp. 1-6). |
| Marco Scirea, Mark J. Nelson, and Julian Togelius, “Moody Music Generator: Characterizing Control Parameters Using Crowsourcing”, published in 2015 Proceedings of the 4th Conference on Evolutionary and Biologically Inspired Music, Sound, Art and Design, and republished at http://julian.togelius.com/Scirea2015Moody.pdf , (12 Pages). |
| Michael C. Mozer, Todd Soukup, Connectionist Music Composition Based on Melodic and Stylistic Constraints, 1990, Department of Computer Science and Institute of Cognitive Science, University of Colorado, Boulder Colorado, (8 Pages). |
| Michael Chan, John Potter, Emery Shubert, Improving Algorithmic Music Composition with Machine Learning, 9th International Conference on Music Perception and Cognition, Aug. 2006, pp. 1848-1854, University of New South Wales, Sydney, Australia, (7 Pages). |
| Michael Kamp, Andrei Manea, Stones: Stochastic Technique for Generating Songs, Jan. 2013, Fraunhofer Institute for Intelligent Analysis Information Systems, Germany, (6 Pages). |
| Miguel Febrer et al, Aneto: A Tool for Prosody Analysis of Speech, 1998, Polytechnic University of Catalunya, Barcelona, Spain, (4 Pages). |
| Miguel Haruki Yaimaguchi, An Extensible Tool for Automated Music Generation, May 2011, Department of Computer Science, Lafayette College, Pennsylvania, (108 Pages). |
| Mixonline, Michael Cooper, “Sonicsmiths The Foundary: Virtual Instrument Takes Fresh Approach to Sound Design”, Apr. 2016, (pp. 1-3). |
| Motu, “Digital Performer 10 User Guide”, Jan. 2019, (pp. 1-1036). |
| Motu, “Digital Performer 8 Screenshots”, Sep. 2012, (pp. 1-6). |
| Musictech, Andy Jones, “The Essen tial Guide to DAWs”, Jun. 2017, (pp. 1-8). |
| Native Instruments, “Session Horns Pro Manual”, May 2014, (pp. 1-68). |
| Nonetwork LLC, Rob Hardy, “The Process of Scoring Your Own Films Just Became Insanely Simple”, Nov. 2014, (pp. 1-3). |
| Notice of Allowanace dated May 23, 2018 for U.S. Appl. No. 15/489,693 (pp. 1-8). |
| Notice of Allowance dated May 28, 2019 for U.S. Appl. No. 15/489,701 (pp. 1-8). |
| Notice of Allowance dated Aug. 7, 2018 for U.S. Appl. No. 15/489,707 (pp. 1-8). |
| Notice of Allowance dated Jan. 24, 2019 for U.S. Appl. No. 15/489,672 (pp. 1-7). |
| Notice of Allowance dated Mar. 27, 2019 for U.S. Appl. No. 15/489,709 (pp. 1-5). |
| Office Action dated Aug. 30, 2018 for U.S. Appl. No. 15/489,672 (pp. 1-6). |
| Office Action dated Dec. 3, 2018 for U.S. Appl. No. 15/489,709 (pp. 1-5). |
| Office Action dated Jan. 12, 2018 for U.S. Appl. No. 15/489,707; (pp. 1-6). |
| Office Action dated Nov. 30, 2018 for U.S. Appl. No. 15/489,672 (pp. 1-5). |
| Office Action dated Sep. 26, 2019 for U.S. Appl. No. 16/253,854 (pp. 1-9). |
| Owen Dafydd Jones, “Transition Probabilities for the Simple Random Walk on Seirpinski Graph, Stochastic Processes and Their Applications”, 1996, pp. 45-69, Elsevier, (25 Pages). |
| Patricio Da Silva, “David Cope and Experiments in Musical Intelligence”, 2003, Spectrum Press, 86 Pages, (93 Pages). |
| Patrik N. Juslin, Daniel Vastfjall, “Emotional Responses to Music: The Need to Consider Underlying Mechanisms, Behavioral and Brain Sciences”, 2008, pp. 559-621, vol. 31, Cambridge University Press, (63 Pages). |
| Paul Doornbusch, “Gerhard Nierhaus: Algorithmic Composition: Paradigms of Automated Music Generation (Review)”, CMJ Reviews, 2012, vol. 34 Issue 3 Reviews, Computer Music Journal, Melbourne, Australia, (5 Pages). |
| Paul Nelson, “Talking About Music—A Dictionary” (Version Sep. 1, 2005), published at http://www.composertools.com/ Dictionary/, (50 Pages). |
| Philippe Martin, “A Tool for Text to Speech Alignment and Prosodic Analysis”, 2004, Paris University, Paris, France, (4 Page). |
| “Pop Music Automation” published on Mar. 8, 2016, on Wikipedia, at https://en.wikipedia.org/wiki/Pop_music_automation Last modified on Dec. 27, 2015, at 14:34, (4 Pages). |
| Presonus, “Studio One 4 Reference Manual”, Jan. 2019, (pp. 1-336). |
| Propellerhead Software, “Reason Essentials Operation Manual”, Jan. 2011, (pp. 1-742). |
| Rebecca Dias, “A Mathematical Melody: An Introduction to Fractals and Music”, Dec. 10, 2012, Trinity University, (26 Pages). |
| Ricardo Miguel Moreira Da Cruz, “Emotion-Based Music Composition for Virtual Environments”, Apr. 2008, Technical University of Lisbon, Lisbon, Portugal, (121 Pages). |
| Ripple Training, “Music Scoring for Video in Logic Pro X”, Jan. 2016, (pp. 1-6). |
| Robert Cookson, “Jukedeck's computer composes music at the touch of a button”, published in the Financial Times Ltd, on Dec. 7, 2015, (3 Pages). |
| Robert Plutchik,“Plutchik Wheel of Emotions”, reprinted on http://www.6seconds.org by permission of American Scientist magazine of Sigma XI, The Scientific Research Society, Feb. 2020, (3 Pages). |
| Roger B. Danneberg, Course Outline for “Week 5—Music Generation and Algorithmic Composition”, Carnegie Mellon University (CMU), Spring 2014, (29 Pages). |
| Roger Dannenberg, Music Generation and Algorithmic Composition, Spring 2014, Carnegie Mellon University, Pennsylvania, (29 Pages). |
| Ruoha Zhou, Feature Extraction of Musical Content, for Automatic Music Transcription, Oct. 2006, Federal Institute of Technology, Lausanne, (169 Pages). |
| Ryan Demopoulos and Michael Katchabaw, “MUSIDO: A Framework for Musical Data Organization to Support Automatic Music Composition”, Department of Computer Science, The University of Western Ontario, London, Ontario Canada, published in Audio Mostly 2007, 2nd Conference on Interaction with Sound, Conference Proceedings, Sep. 27-28, 2007, Rontgenbau, Ilmenau, Germany, Fraunhofer Institute for Digital Media Technology IDMT, (6 Pages). |
| Satoru Fukayama et al, Automatic Song Composition from the Lyrics Exploiting Prosody of Japanese Language, 2010, The University of Tokyo, Nagoya Institute of Technology, Japan, (4 Pages). |
| Score Cast Online, “ESP and Music”, Jun. 2009, (pp. 1-6). |
| Score Cast Online, Deane Ogden, “‘Roadmapping’ a Score”, Jul. 2009, (pp. 1-9). |
| Score Cast Online, Deane Ogden, “Tools for Studio Organization”, Oct. 2010, (pp. 1-8). |
| Score Cast Online, James Olszewski, “Your First Spotting Experience”, Mar. 2010, (pp. 1-5). |
| Score Cast Online, Lee Sanders, “Spotting Content”, Mar. 2010, (pp. 1-6). |
| Score Cast Online, Lee Sanders, “Everything *BUT* Spotting”, Mar. 2010, (pp. 1-10). |
| Score Cast Online, Leon Willett, “Spotting for Video Games”, Mar. 2010, (pp. 1-7). |
| Score Cast Online, Nikola Jeremie, “Scoring With PreSonus Studio One—Setting Up”, Nov. 2011, (pp. 1-6). |
| Score Cast Online, Yaiza Varona, “Scoring to Picture in Logic 9 (Part 1)”, Jan. 2013, (pp. 1-8). |
| Score Cast Online, Yaiza Varona, “Scoring to Picture in Logic 9 (Part 2)”, Feb. 2013, (pp. 1-7). |
| Score Cast Online,, David E. Fluhr, “Spotting With the Composer and Sound Designer”, Apr. 2012, (pp. 1-11). |
| Simone Hill, “Markov Melody Generator”, Computer Science Department, University of Massachusetts Lowell, Published on Dec. 11, 2011, at http://www.cs.uml.edu/ecg/pub/uploads/Alfall11/SimoneHill.FinalPaper. MarkovMelodyGenerator.pdf, (4 Pages). |
| Simpsons Music 500, “Music Editing 101—Music Spotting Notes”, Aug. 2011, (pp. 1-6). |
| Siwei Qin et al, Lexical Tones Learning with Automatic Music Composition System Considering Prosody of Mandarin Chinese, 2010, Graduate School of Information Science and Technology, The University of Tokyo, Japan, (4 Page). |
| Sonicsmiths, “The Foundary”, Aug. 2015, (pp. 1). |
| Sound on Sound, “A Touch of Logic”, Jun. 2014, (pp. 1-4). |
| Sound on Sound, Jayne Drake, “What Does Artificial Intelligence Mean for Musicians and Producers?”, Sep. 2018, (pp. 1-13). |
| Steinberg Media Technologies, “Cubase Pro 10 Operation Manual”, Nov. 2018, (pp. 1-1156). |
| Steve Engels, Fabian Chan, and Tiffany Tong, Automatic Real-Time Music Generation for Games, 2015, Department of Computer Science, Department of Engineering Science, and Department of Mechanical and ndustrial Engineering, Toronto, Ontario, Canada, (3 Pages). |
| Steve Rubin, Maneesh Agrawala, Generating Emotionally Relevant Musical Scores for Audio Stories, UIST 2014, Oct. 2014, pp. 439-448, (10 Pages). |
| Supplemental Notice of Allowability dated May 2, 2017 for U.S. Appl. No. 14/869,911; (pp. 1-4). |
| Sweetwater, “Spotting Session”, Dec. 1999, (pp. 1-2). |
| Thomas M. Fiore, “Music and Mathematics”, University of Michigan, 2004, published on http://www-personal.umd. Umich.edu/˜tmfiore/1/musictotal.pdf, (36 Pages). |
| Virginia Francisco, Raquel Hervas, “EmoTag: Automated Mark Up of Affective Information in Texts”, Department of Software Engineering and Artificial intelligence, Complutense University, Madrid, Spain, published at http://nil.fdi.ucm.es/sites/default/files/FranciscoHervasDCEUROLAN2007.pdf, 2007, ( 8 Pages). |
| Website Pages from Audio Network Limited, covering the directory structure of its “Production Music Database Prganized by Musical Styles, Mood/Emotion, Instrumentation, Production Genre, Album Listing and Artists & Composers”, https://www.audionetwork.com, (7 Pages). |
| Xsample, “Xsample Acoustic Intruments Library”, Jan. 2015, (pp. 1-40). |
| Xsample, “Xsample AI Library: Notation Guide Part I”, Jan. 2015, (pp. 1-8). |
| Xsample, “Xsample AI Library: Notation Guide Part II”, Jan. 2015, (pp. 1-49). |
| Xsample, “Xsample Player Edition”, Jan. 2016, (pp. 1-16). |
| Yu-Hao Chin, Chang-Hong Lin, Ernestasia Siahaan, Jia-Ching Wang, “Music Emotion Detection Using Hierarchical Sparse Kernel Machines”, 2014, Hindawi Publishing Corporation, Taiwan, (8 Page). |
| Office Action dated Sep. 26, 2019 for U.S. Appl. No. 16/219,299 (pp. 1-11). |
| Communication Pursuant to Rules 70(2) ane 70a(2) EPC dated Jan. 10, 2019 issued in EP Application No. 16852438.7 (1 Page). |
| Extended European Search Report dated Dec. 9, 2019 issued in EP Application No. 16852438.7 (20 Pages). |
| Banshee in Avalon, “Xhail, Innovative Automatic Composing Solution: Score Music Interactive is a AE3 in Boston where they are introducing a new system for multimedia music composers,” published by AudioFanZine at https://en.audiofanzine.com/misc-music-software/score-music-interactive/xhail/medias/videosfitid:35534 on Sep. 24, 2014 (1 Page). |
| Captured Screenshots from the “Xhail Preview” by Score Music Interactive Ltd., published on AudioFanZine at https://en.audiofanzine.com/misc-music-software/score-music-interactive/xhail/medias/videos/#id:35534 on Sep. 24, 2014 (35 Pages). |
| Captured Screenshots from the “Xhail Preview” by Score Music Interactive Ltd., published on Vimeo.com on Sep. 24, 2014 (34 Pages). |
| Jacqui Cheng, “Virtual Composer Makes Beautiful Music—and Stirs Controversy: Can A Computer Program Really Generate Musical Compositions that Are Good . . . ”, published by Arstechnia at https://arstechnica.com/science/2009/09/virtual-composer-makes-beautiful-musicand-stirs-controversy/ on Sep. 29, 2009 (3 Pages). |
| Music Marcom, “Are You A Professional Muscian or Talented Composer? Help Xhail Find You” published by Prosound Network at https://www.prosoundnetwork.com/the-wire/are-you-a-professional-musician-or-talented-composer-help-xhail-find-you on May 19, 2015 (2 Pages). |
| Prosoundnework Editorial Staff, “Xhail Recruiting Music Talent” published by Prosound Network at https://www.prosoundnetwork.com/business/xhail-recruiting-music-talent on May 21, 2015 (1 Page). |
| Crunchbase Profile on Score Music Interactive Ltd., summarized as “Score Music Interactive: A Music Publishing Software Platform That Creates Original, Copyrighted Music from A Centralized Database of Tagged Musical Stems,” published by Crunchbase at https://www.crunchbase.com/organization/score-music-interactive on Dec. 2, 2019 (1 Page). |
| Linkedin Profile on Score Music Interactive Ltd, summarized as “Xhail is the most advanced music creation platform in the world. Unique one-ofa-kind tracks created instantly with incredible flexibility. Real performances by real musicians, combining for the very first time, creating the perfect music solution. Xhail's platform gives editors, music supervisors and other professionals extreme creative control in a most intuitive way without the requirement of music skill. Our patented technology creates desired music in a fraction of the time it would take to search for a suitable standard track from a traditional music library”, published at Linkedin.com on Dec. 2, 2019 (1 Page). |
| Screenshots taken from the Xhail WWW Site by Score Music Interactive Ltd., captioned “The Evolution of Music Creation & Licensing” and published at https://www.xhail.com/#whatis on Dec. 2, 2019 (10 Pages). |
| Roger B. Dannenberg, “An On-Line Algorithm for Real-Time Accompaniment,” in Proceedings of the 1984 International Computer Music Conference, Computer Music Association, Jun. 1985, 193-198, (6 Pages). |
| Johan Sundberg, et al, “Rules for Automated Performance of Ensemble Music”, Contemporary Music Review, 1989, vol. 3, pp. 89-109, Harwood Academic Publishers GmbH, (12 Pages). |
| Roberto Bresin, “Articulation Rules for Automatic Music Performance”, Department of Speech, Music and Hearing, Royal Institute of Technology, Stockholm, Jan. 2002, (4 Pages). |
| Mitsuyo Hashida, et al., Rencon: Performance Rendering Contest for Automated Music Systems, Proceedings of the 10th International Conference on Music Perception and Cognition (ICMPC 10), Sapporo, Japan, Aug. 25, 2008, (5 Pages). |
| One Page Love, “Jukedeck, Interactive Landing Page—Beta” built by Qip Creative, Reviewed by Rob Hope on Jan. 6, 2014, (4 Pages). |
| Press Release by Aiva Technologies, “Composing the music of the future”, Nov. 2016, (7 Pages). |
| Barry L. Vercoe, “New Dimensions in Computer Music,” Trends and Perspectives in Signal Processing II/2, Apr. 1982, pp. 15-23 (9 Pages). |
| Barry L. Vercoe , “Computer Systems and Languages for Audio Research,” The New World of Digital Audio (Audio Engineering Society Special Edition), 1983, pp. 245-250 (6 Pages). |
| Barry L. Vercoe, “Extended Csound,” in Proceedings, 1996, ICMC, Hong Kong, pp. 141-142, (2 Pages). |
| R. B. Dannenberg, “An On-Line Algorithm for Real-Time Accompaniment”, Proceedings of the 1984 International Computer Music Conference, 1985 International Computer Music Association, p. 193-198, http://www.cs.cmu.edu/˜rbd/bib-accomp.html#icmc84, (6 Pages). |
| Barry L. Vercoe, “The Synthetic Performer in the Context of Live Performance,” in Proceedings, International Computer Music Conference, 1984, Paris, pp. 199-200, (2 Pages). |
| Bloch and Dannenberg, “Real-Time Accompaniment of Polyphonic Keyboard Performance,” Proceedings of the 1985 International Computer Music Conference, Vancouver, BC Canada, Aug. 19-22, 1985, San Francisco: International Computer Music Association, 1985. pp. 279-290, (11 Pages). |
| Barry L. Vercoe, and Puckette, M.S. (1985) “Synthetic Rehearsal: Training the Synthetic Performer,” in Proceedings, ICMC, Burnaby, BC, Canada, 1985, pp. 275-278, (4 Pages). |
| Roger B. Dannenberg and Mukaino, “New Techniques for Enhanced Quality of Computer Accompaniment,” in Proceedings of the International Computer Music Conference, Computer Music Association, Sep. 1988, pp. 243-249, (7 Pages). |
| Barry L. Vercoe, “Hearing Polyphonic Music with the Connection Machine,” in Proceedings, First Workshop on Artificial Intelligence and Music, 1988, AAA-88, St. Paul, MN, pp. 183-194, (12 Pages). |
| Roger B. Dannenberg, “Real-Time Scheduling and Computer Accompaniment,” in Current Research in Computer Music, edited by Max Mathews and John Pierce, MIT Press, 1989, (37 Pages). |
| Barry L. Vercoe,“Synthetic Listeners and Synthetic Performers,” Proceedings, International Symposium on Multimedia Technology and Artificial Intelligence (Computerworld 90), Kobe Japan, Nov. 1990, pp. 136-141, (6 Pages). |
| Barry L. Vercoe, and D.P.W Ellis, “Real-time Csound: Software Synthesis with Sensing and Control,” in Proceedings, ICMC, 1990, Glasgow, pp. 209-211. (3 Pages). |
| Grubb and Dannenberg, “Automated Accompaniment of Musical Ensembles,” in Proceedings of the Twelfth National Conference on Artificial Intelligence, AAAI, 1994, pp. 94-99, (6 Pages). |
| Grubb and Dannenberg, “Automating Ensemble Performance,” in Proceedings of the 1994 International Computer Music Conference, Aarhus and Aalborg, Denmark, Sep. 1994. International Computer Music Association, 1994. pp. 63-69, (7 Pages). |
| Grubb and Dannenberg, “Computer Performance in an Ensemble,” in 3rd International Conference for Music Perception and Cognition Proceedings, Liege, Belgium. Jul. 23-27, 1994. Ed. Irene Deliege. Liege: European Society for the Cognitive Sciences of Music Centre de Recherche et de Formation Musicales de Wallonie, 1994. pp. 57-60, (2 Pages). |
| Barry L. Vercoe, ., “Computational Auditory Pathways to Music Understanding,” in Deliege I. and Sloboda J (Eds.), 1997, Perception and Cognition of Music , East Sussex, UK: Psychology Press, pp. 307-326, (20 Pages). |
| Grubb and Dannenberg, “Enhanced Vocal Performance Tracking Using Multiple Information Sources,” in Proceedings of the International Computer Music Conference, San Francisco: International Computer Music Association, 1998) pp. 37-44, (8 Sheets). |
| Tristan Jehan and Bernd Schoner, “An Audio-Driven, Spectral Analysis-Based, Perceptual Synthesis Engine”, Audio Engineering Society Convention Paper Presented at the 110th Convention, May 12-15, 2001 Amsterdam, The Netherlands, (10 Pages). |
| Brian Whitman, Gary Flake and Steve Lawrence, “Artist Detection in Music with Minnowmatch,” Computer Science NEC Research Institute, Princeton NJ, NNSP—Sep. 2001, (17 Pages). |
| Brian Whitman and Ryan Rifkin, “Musical Query-by-Description as a Multiclass Learning Problem”, Jan. 1, 2003, 2002 IEEE Workshop onMultimedia Signal Processing, (4 Pages). |
| Daniel P. W. Ellis, Brian Whitman, Adam Berenzweig, and Steve Lawrence, “The Quest for Ground Truth in Musical Artist Similarity”, ISMIR 2002, 3rd International Conference on Music Information Retrieval, Paris, France, Oct. 13-17, 2002, Proceedings, (8 Pages). |
| Roberto Bresin, “Articulation Rules For Automatic Music Performance”, Proceedings of the 2001 International Computer Music Conference : Sep. 17-22, 2001, Havana, Cuba, pp. 294-297, (4 Pages). |
| Brian Whitman and Paris Smaragdis, “Combining Musical and Cultural Features for Intelligent Style Detection”, ISMIR 2002, 3rd International Conference on Music Information Retrieval, Paris, France, Oct. 13-17, 2002, Proceedings, (6 Pages). |
| Brian Whitman and Steve Lawrence, “Inferring Descriptions and Similarity for Music from Community Metadata”, Proceedings of the 2002 International Computer Music Conference, Jan. 2002, (8 Pages). |
| Adam Berenzweig, Beth Logan, Daniel P. W. Ellis, and Brian Whitman, “A Large-Scale Evaluation of Acoustic and Subjective Music Similarity Measures”, Computer Music Journal, vol. 28(2), Nov. 2003, (7 Pages). |
| Roberto Bresin, “Artificial Neural Networks Based Models for Automatic Performance of Musical Scores,” Journal of New Music Research, 1998, vol. 27, No. 3, pp. 239-270, (32 Pages). |
| Brian Whitman, Deb Roy and Barry Vercoe, “Learning Word Meanings and Descriptive Parameter Spaces from Music”, Computer SciencePublished in HLT-NAACL 2003, (8 Pages). |
| Tristan Jehan, “Perceptual Segment Clustering for Music Description and Time-Axis Redundancy Cancellation”, ISMIR 2004, 5th International Conference on Music Information Retrieval, Barcelona, Spain, Oct. 10-14, 2004, Proceedings, (4 Pages). |
| Barry Vercoe, “Audio-Pro with Multiple DSPs and Dynamic Load Distribution,” BT Technology Journal, vol. 22, No. 4, Oct. 2004, (7 Pages). |
| Brian Whitman and Daniel P. W. Ellis, “Automatic Record Reviews,” In Proceedings of ISMIR 2004—5th International Conference on Music Information Retrieval. (8 Pages). |
| Brian A. Whitman, “Learning the Meaning of Music”, Jun. 2005, Phd., Doctoral dissertation, MIT, (104 Pages). |
| Brian A. Whitman, “Learning the Meaning of Music”, Apr. 14, 2005, MIT, (65 Pages). |
| Tristan Jehan, “Creating Music by Listening”, Sep. 2005, Phd. Doctoral dissertation, MIT (137 Pages). |
| Tristan Jehan, “Downbeat Prediction by Listening Tristan Jehan, Downbeat Prediction By Listening and Learning”, 2005 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, Oct. 16-19, 2005, New Paltz, NY, (4 Pages). |
| Roger B. Dannenberg, “New Interfaces for Popular Music Performance,” in Seventh International Conference on New Interfaces for Musical Expression: NIME 2007 New York, New York, NY: New York University, Jun. 2007, pp. 130-135. (6 Pages). |
| Youngmoo E. Kim et al, “Music Emotion Recognition: State of The Art Review”, 11th International Society for Music Information Retrieval Conference (ISMIR 2010), (12 Pages). |
| Nicholas E. Gold and Roger B. Dannenberg, “A Reference Architecture and Score Representation for Popular Music Human-Computer Music Performance System,” Proceedings of the International Conference on New Interfaces for Musical Expression, May 30-Jun. 1, 2011, Oslo, Norway, (4 Pages). |
| Roberto Bresin and Anders Friberg, “Emotion Rendering in Music: Range and Characteristics Values of Seven Musical Variables”, May 17, 2011, c Cortex vol. 4 7 ( 2 0 1 1 ), Pagerds 1 0 6 8 -1 0 8 1, (14 Pages). |
| Roger B. Dannenberg, “A Virtual Orchestra for Human-Computer Music Performance,” Proceedings of the International Computer Music Conference 2011, University of Huddersfield, UK, Jul. 31-Aug. 5, 2011, (4 Pages). |
| Marius Kaminskas and Francesco Ricci, “Contextual music information retrieval and recommendation: Stateof the Art and Challenges,” Computer Science Review, vol. 6, Issues 2-3, May 2012, pp. 89-119, (31 Pages). |
| Roger B. Dannenberg, Nicolas E. Gold, Dawen Liang and Guangyu Xia, “Active Scores: Representation and Synchronization in Human-Computer Performance of Popular Music,” Computer Music Journal, 38:2, pp. 51-62, Summer 2014,(12 Pages). |
| Roger B. Dannenberg and Andrew Russell, “Arrangements: Flexibly Adapting Music Data for Live Performance,” Proceedings of the International Conference on New Interfaces for Musical Expression, Baton Rouge, LA, USA, May 31-Jun. 3, 2015, (2 Pages). |
| Roger B. Dannenberg, “Time-Flow Concepts and Architectures For Music and Media Synchronization,” in Proceedings of the 43rd International Computer Music Conference, International Computer Music Association, 2017, pp. 104-109, (6 Pages). |
| Josh McDermott and March Hauser, “The Origins of Music: Innateness, Uniqueness, and Evolution”, published in Music Perception vol. 23, Issue 1, Mar. 2005, pp. 29-59, (32 Pages). |
| Josh McDermott, “The evolution of music”, published in Nature, vol. 453, No. 15, May 2008, pp. 287-288, (2 Pages). |
| Anne Trafton, “Why We Like The Music We Do”, MIT News Office, Jul. 13, 2016, (4 Pages). |
| Eric Nichols, Dan Morris, Sumit Basu and Christopher Raphael, “Relationsips Between Lyrics and Melody In Popular Music”, Proceedings of the 11th International Society for Music Information Retrieval Conference, Oct. 2009, (6 Pages). |
| Francois Panchet, “The Continuator: Musical Interaction With Style”, In Proceedings of International Computer Music Conference, Gotheborg (Sweden), ICMA, Sep. 2002, (10 Pages). |
| Francois Panchet, Pierre Roy and Gabriele Barbieri, “Finite-Length Markov Processes with Constraints”, Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, 2011, (8 Pages). |
| Jonathan Cabreira, “A Music Taste Analysis Using Spotify API and Python: Exploring Audio Features and building a Machine Learning Approach,” published on Toward Data Science at https://towardsdatascience.com/a-music-taste-analysis-using-spotify-api-and-python-e52d186db5fc , Aug. 17, 2019, (7 Pages). |
| Ben Popper, “Tastemaker: How Spotify's Discover Weekly cracked human curation at internet scale”, published in the Verge, at https://www.theverge.com/2015/9/30/9416579/spotify-discover-weekly-online-music-curation-interview , Sep. 30, 2015, (18 Pages). |
| Eric Drott, “Why the Next Song Matters: Streaming, Recommendation, Scarcity”, Twentieth-Century Music 15/3, 325-357, Cambridge University Press, 2018, (33 Pages). |
| Form F-1 Registration Statement Under the Securities Act of 1933, United States Securities and Exchange Commission, by Spotify Technology S.A, Feb. 28, 2018, (265 Pages). |
| Ipshita Sen, “How AI helps Spotify win in the music streaming world,” published in outsideinsight.com, https://outsideinsight.com/insights/how-ai-helps-spotify-win-in-the-music-streaming-world/ , May 22, 2018 (12 Pages). |
| Ramon Lopez de Mantaras and Josep Lluis Arcos, “AI and Music: From Composition to Expressive Performance”, American Association for Artificial Intelligence, Fall 2002, pp. 43-57 (16 Pages). |
| “User Manual for Synclavier V, Version 2.0”, Arturia SA, published Oct. 15, 2018, (133 Pages). |
| “User Manual for Omnisphere Power Synth Version 2.6”, Spectrasonics.net, Jan. 2020, (944 Pages). |
| “User Guide for Note Performer 3”, Wallander Instruments AB, Sep. 12, 2019, (64 Pages). |
| “WIVI Documentation”, Wallendar Instruments AB, Dec. 18, 2014, (85 Pages). |
| Masataka Goto and Roger B. Dannenberg, “Music Interfaces Based On Automatic Music Signal Analysis: New Ways to Create and Listen To Music”, IEEE Signal Processing Magazine, Jan. 2019, pp. 74-81, Date of Publication Dec. 24, 2018, (8 Pages). |
| Gus G. Xia and Roger B. Dannenberg, “Improvised Duet Interaction: Learning Improvisation Techniques for Automatic Accompaniment,” in Copenhagen, May 2017, pp. 110-114, (5 Pages). |
| Guangyu Xia, Mao Kawai, Kei Matsuki, Mutian Fu, Sarah Cosentino, Gabriele Trovato, Roger Dannenberg, Salvatore Sessa, Atsuo Takanishi, “Expressive Humanoid Robot for Automatic Accompaniment”, Carnegie Mellon Univserity, https://www.cs.cmu.edu/˜rbd/papers/robot-smc-2016.pdf , 2016, (6 Pages). |
| Guangyu Xia, Yun Wang, Roger Dannenberg, Geoffrey Gordon. “Spectral Learning for Expressive Interactive Ensemble Performance”, 16th International Society for Music Information Retrieval Conference, 2015, (7 Pages). |
| Mutian Fu, Guangyu Xia, Roger Dannenberg, Larry Wasserman, “A Statistical View on the Expressi 0.0Timing of Piano Rolled Chords”, 16th International Society for Music Information Retrieval Conference, 2015, (6 Pages). |
| Roger B. Danneberg and Andrew Russell, “Arrangements: Flexibly Adapting Music Data for Live Performance”, Proceedings of the International Conference on New Interfaces for Musical Expression, Baton Rouge, LA, USA, May 31-Jun. 3, 2015, (2 Pages). |
| Roger B. Dannenberg, Nicolas E. Gold, Dawen Liang, and Guangyu Xia, “Methods and Prospects for Human-Computer Performance of Popular Music,” Computer Music Journal, 38:2, pp. 36-50, Summer 2014, (15 Pages). |
| Tongbo Huang, Guangyu Xia, Yifei Ma, Roger Dannenberg, Christos Faloutsos, “MidiFind: Fast and Effective Similarity Searching in Large MIDI Databases”, Proc. of the 10th International Symposium on Computer Music Multidisciplinary Research, Marseille, France, Oct. 15-18, 2013, (16 Pages). |
| Roger B. Dannenberg, Zeyu Jin, Nicolas E. Gold, Octav-Emilian Sandu, Praneeth N. Palliyaguru, Andrew Robertson, Adam Stark, Rebecca Kleinberger, “Human-Computer Music Performance: From Synchronized Accompaniment to Musical Partner”, Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden, (7 Pages). |
| Roger B. Dannenberg, “A Vision of Creative Computation in Music Performance”, Proceedings of the Second International Conference on Computational Creativity, published at https://www.cs.cmu.edu/˜rbd/papers/dannenberg_1_iccc11.pdf , Jan. 2011, (6 Pages). |
| Roger Dannenberg, and Sukrit Mohan,“Characterizing Tempo Change In Musical Performances”, Proceedings of the International Computer Music Conference 2011, University of Huddersfield, UK, Jul. 31-Aug. 5, 2011, (7 Pages). |
| Özgür İzmirli and Roger B. Dannenberg, “Understanding Features and Distance Functions for Music Sequence Alignment”, 11th International Society for Music Information Retrieval Conference (ISMIR 2010), (6 Pages). |
| Roger B. Dannenberg, “Style in Music”, published in The Structure of Style: Algorithmic Approaches to Understanding Manner and Meaning, Shlomo Argamon, Kevin Burns, and Shlomo Dubnov (Eds.), Berlin, Springer-Verlag, 2010, pp. 45-58, (12 Pages). |
| Roger B. Dannenberg and Masataka Goto, “Music Structure Analysis from Acoustic Signals”, in Handbook of Signal Processing in Acoustics, pp. 305-331, Apr. 16, 2005, (19 Pages). |
| Byeong-jun Han, Seungmin Rho Roger B. Dannenberg Eenjun Hwang, “SMERS: Music Emotion Recognition Using Support Vector Regression”, 10th International Society for Music Information Retrieval Conference (ISMIR), 2009, (6 Pages). |
| Roger B. Dannenberg, “Computer Coordination With Popular Music: A New Research Agenda,” in Proceedings of the Eleventh Biennial Arts and Technology Symposium at Connecticut College, Mar. 2008, (6 Pages). |
| William D. Haines, Jesse R. Vernon, Roger B. Dannenberg, and Peter F. Driessen, “Placement of Sound Sources in the Stereo Field Using Measured Room Impulse Responses,” in Proceedings of the 2007 International Computer Music Conference, vol. I. San Francisco: The International Computer Music Association, Aug. 2007, pp. I-496-499, (5 Pages). |
| Roger B. Dannenberg. “An Intelligent Multi-Track Audio Editor.” In Proceedings of the 2007, International Computer Music Conference, vol. II. San Francisco: The International Computer Music Association, Aug. 2007, pp. II-89-94, (7 Pages). |
| Ning Hu and Roger B. Dannenberg, “Bootstrap learning for accurate onset detection”, Machine Learning, May 6, 2006, vol. 65, pp. 457-471 (15 Pages). |
| William Birmingham, Roger Dannenberg, and Bryan Pardo, “Query by Humming With the Vocalsearch System”, Communications of the ACM, Aug. 2006, vol. 49, No. 8, pp. 49-52, (4 Pages). |
| Ning Hu and Roger B. Dannenberg, “A Bootstrap Method for Training an Accurate Audio Segmenter”, in Proceedings of the Sixth International Conference on Music Information Retrieval, London UK, Sep. 2005, London, Queen Mary, University of London & Goldsmiths College, University of London, 2005, pp. 223-229 (7 Pages). |
| Roger B. Dannenberg, Ben Brown, Garth Zeglin, Ron Lupish, “McBlare: A Robotic Bagpipe Player,” in Proceedings of the International Conference on New Interfaces for Musical Expression, Vancouver: University of British Columbia, (2005), pp. 80-84. |
| Roger B. Dannenberg, “Toward Automated Holistic Beat Tracking, Music Analysis, and Understanding,” in ISMIR 2005 6th International Conference on Music Information Retrieval Proceedings, London: Queen Mary, University of London, 2005, pp. 366-373, (8 Pages). |
| Roger B. Dannenberg, William P. Birmingham, George Tzanetakis, Colin Meek, Ning Hu, and Bryan Pardo, The MUSART Testbed for Query-by-Humming Evaluation, Computer Music Journal, 28:2, pp. 34-48, Summer 2004, (15 Pages). |
| Ning Hu, Roger B. Dannenberg and George Tzanetakis, “Polyphonic Audio Matching and Alignment for Music Retrieval”, 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, Oct. 19-22, 2003, New Paltz, NY, (4 Pages). |
| Dannenberg, Roger B. and Ning Hu, “Polyphonic Audio Matching for Score Following and Intelligent Audio Editors.” Proceedings of the 2003 International Computer Music Conference, San Francisco: International Computer Music Association, pp. 27-33, (7 Pages). |
| Ning Hu, Roger B. Dannenberg, and Ann L. Lewis, “A Probabilistic Model of Melodic Similarity,” In Proceedings of the International Computer Music Conference. San Francisco, International Computer Music Association, 2002, (4 Pages). |
| Ning Hu and Roger B. Dannenberg, “A Comparison of Melodic Database Retrieval Techniques Using Sung Queries,” in Joint Conference on Digital Libraries, 2002, New York: ACM Press, pp. 301-307, (7 Pages). |
| Dannenberg and Hu. “Pattern Discovery Techniques for Music Audio” in ISMIR 2002 Conference Proceedings, Paris, France, IRCAM, 2002, pp. 63-70, appears in Journal of New Music Research, Jun. 2003, pp. 153-164, (14 Pages). |
| Roger B. Dannenberg and Ning Hu, “Pattern Discovery Techniques for Music Audio,” In ISMIR 2002 Conference Proceedings: Third International Conference on Music Information Retrieval, M. Fingerhut, ed., Paris, IRCAM, 2002, pp. 63-70, (8 Pages). |
| Roger B. Dannenberg, “Listening to ‘Naima’: An Automated Structural Analysis of Music from Recorded Audio,” In Proceedings of the International Computer Music Conference, 2002, San Francisco, International Computer Music Association, (7 Pages). |
| Roger B. Dannenberg and Ning Hu, “Discovering Musical Structure in Audio Recordings” in Anagnostopoulou, Ferrand, and Smaill, eds., Music and Artificial Intelligence: Second International Conference, ICMAI 2002, Edinburgh, Scotland, UK. Berlin: Springer, 2002. pp. 43-57, (11 Pages). |
| Mazzoni and Dannenberg, “Melody Matching Directly from Audio,” in ISMIR 2001 2nd Annual International Symposium on Music Information Retrieval, Bloomington: Indiana University, 2001, pp. 73-82, (2 Pages). |
| Masataka Goto, “An Audio-based Real-time Beat Tracking System for Music With or Without Drum-sounds”, Journal of New Music Research, 2001, vol. 30, No. 2, pp. 159-171,(14 Pages). |
| Roger B. Dannenberg, “Music Information Retrieval as Music Understanding,” in ISMIR 2001 2nd Annual International Symposium on Music Information Retrieval, Bloomington: Indiana University, 2001, pp. 139-142, (4 Pages). |
| Lorin Grubb and Roger B. Dannenberg, “Enhanced Vocal Performance Tracking Using Multiple Information Sources,” Proceedings of the 1998 International Computer Music Conference, San Francisco, International Computer Music Association, pp. 37-44, (8 Pages). |
| Grubb, L. and Dannenberg, R.B., “A Stochastic Method of Tracking a Vocal Performer”, in 1997 International Computer Music Conference, 1997, International Computer Music Association. http://www.cs.cmu.edu/˜rbd/bib-accomp.html# icmc97, (8 Pages). |
| Roger B. Dannenberg, Belinda Thom, and David Watson, “A Machine Learning Approach to Musical Style Recognition”, School of Computer Science, Carnegie Mellon University, 1997, (4 Pages). |
| Grubb and Dannenberg, “Computer Performance in an Ensemble,” in 3rd International Conference for Music Perception and Cognition Proceedings, Liege, Belgium. Jul. 23-27, 1994. Ed. Irene Deliege. Liege: European Society for the Cognitive Sciences of Music Centre de Recherche et de Formation Musicales de Wallonie, 1994. pp. 57-60, 1994, ( 2 Pages). |
| Lorin Grubb and Roger B. Dannenberg, “Automating Ensemble Performance”, Machine Recognition of Music, ICMC Proceedings 1994, pp. 63-69, (7 Pages). |
| Lorin Grubb and Roger B. Dannenberg, “Automated Accompaniment of Musical Ensembles”, AAAI-94 Proceedings, 1994, pp. 94-99, (6 Pages). |
| Allen and Dannenberg, “Tracking Musical Beats in Real Time,” in 1990 International Computer Music Conference, International Computer Music Association, Sep. 1990, pp. 140-143, (4 Pages). |
| Allen and Dannenberg, “Tracking Musical Beats in Real Time,” in Proceedings of the International Computer Music Conference, Glasgow, Scotland, Sep. 1990. International Computer Music Association, 1990. pp. 140-143, (12 Pages). |
| Dannenberg and Mukaino, “New Techniques for Enhanced Quality of Computer Accompaniment,” in Proceedings of the International Computer Music Conference, Computer Music Association, Sep. 1988, pp. 243-249, (7 Pages). |
| Roger B. Dannenberg and Bernard Mont-Reynaud, “Following an Improvisation in Real Time,” in 1987 ICMC Proceedings, International Computer Music Association, Aug. 1987, pp. 241-248, (8 Pages). |
| Roger B. Dannenberg, “An On-Line Algorithm for Real-Time Accompaniment”, In Proceedings of the 1984 International Computer Music Conference, 1985, International Computer Music Association, 193-198. http://www.cs.cmu.edu/˜rbd/bib-accomp.html#icmc84, (6 Pages). |
| Bloch, J. B. and Dannenberg, R.B., “Real-Time Computer Accompaniment of Keyboard Performances”, In Proceedings of the 1985 International Computer Music Conference, 1985, International Computer Music Association, 279-289. http://www.cs.cmu.edu/˜rbd/bib-accomp.html#icmc85, (11 Pages). |
| Ethan Hein, “Scales and Emotions” from the Ethan Hein Blog, Posted Mar. 2, 2010, (31 Pages). |
| Anastasia Voitinskaia, “Scales, Genres, Intervals, Melodies, Music Theory”, published on www.musical-u.com , at https://www.musical-u.com/learn/the-many-moods-of-musical-modes/ , on Feb. 6, 2020, (5 Pages). |
| Score Music Interactive, Sampled Workflow of 2018-Version of XHail Automatic Loop-Based Music Composing System, Dec. 2018, (25 Pages). |
| Supplementantary Partial European Search Report issued in EP Application No. EP 16852438.7 dated Dec. 9, 2019 (20 Pages). |
| Communication Pursuant to Rules 70(2) and 70a(2) EPC issued in EP Application No. EP 16852438.7 dated Jan. 10, 2019 (1 Page). |
| Written Opinion Issued in International Patent Application No. PCT/US2020/014639 dated Jul. 21, 2020, (21 Pages). |
| PCT International Serve Report issued in International Patent Application No. PCT/2020/014639 dated Jul. 21, 2020, (2 Pages). |
| The Reason Essentials Operation Manual, by Propellerhead Software AB, 2011, (742 Pages). |
| Bitwig Studio 2.0 User Guide, Fourth Edition 2017, written by Dave Linnenbank, Bitwig GmbH, Germany, (383 Pages). |
| Ableton Reference Manual Vdersion 10, Windows and Mac, written by Dennis SeSantis et al, Ableton AG, 2018, Berlin, Germany (759 Pages). |
| Cubase Pro 10 Cubase Artist 10—Operation Manual , by Steinberg Media Technologies GmbH, Nov. 14, 2018, (1156 Pages). |
| Protools® Reference Guide, Version 2018.12, by Avid Technology, Inc., 2018, (1489 Pages). |
| Sample Robot Pro—User Manual, Version 6.0, Sep. 2018, by Skyline, Halten & Zweiling Gbr, Glinde, Germany, (88 Pages). |
| “This is SampleRobot: Your Personal Sampling Assistant”, published at https://samplerobot.com/pages/samplerobot , by Skylife, Apr. 12, 2019, (6 Pages). |
| FL Studio: Getting Started Manual, by Scott Fisher and Frank Van Biesen of Image Line BVBA, Apr. 2019, (89 Pages). |
| “Machines Can Create Art, but Can They Jam?” by Ken Weiner, published at on the Scientific American Blog Network, https://blogs.scientificamerican.com/observations/machines-can-cr/ on Apr. 29, 2019, (13 Pages). |
| “Making a Custom Sampler Instrument” by Griffin Brown, IZotope Blog Contributor, https://www.izotope.com/en/blog/music-production/making-a-cus , Jan. 28, 2019, (10 Pages). |
| Reference Manual for PreSonus Studio One 4 , Version 4.1 , Presonus, Apr. 2019 (336 Pages). |
| AWS Deep Composer: Press Play on Machine Learning, published on AWS Amazon Site, https://aws.amazon.com/deepcomposer/ , Dec. 2019 (9 Pages). |
| Notice of Reasons for Refusal dated Oct. 6, 2020, issued in Japanese Patent Application No. 2018-536083 which is a National Stage of PCT Appliction No. PCT/US2016/054066 filed 28 Sep. 28, 2016 (9 Pages). |
| “Movie Pro” Software, by AHS Co. Ltd, Japan, published in Gigazine.net, 2010 (15 Pages). |
| Chordana Composer App for the Apple iPhone/ iPad, by Casio Computer Co. Ltd., published on Jan. 30, 2015, https://www.dtmstation.com/archives/51927504.html, (15 Pages). |
| Yamaha News Release on VOCALOID™ Virtual Singing Voice Synthesizer Software, by Yamaha Corporation, https://www.vocaloid.com/en/, Japan, Published Apr. 24, 2014, (4 Pages). |
| Kat Agres, Jamie Forth and Geraint A. Wiggins, “Evaluation of Musical Creativity and Musical Metacreation Systems,” Comput. Entertain. 14, 3, Article 3 , Dec. 2016, (33 Pages). |
| Response to Office Action dated Apr. 17, 2020 filed in European Patent Application No. 16852438.7 (6 Pages). |
| Communication Pursuant to Article 94(3) EPC issued in in European Patent Application No. 16852438.7 on Jun. 29, 2020 (1 Page). |
| Office Action dated Jul. 24, 2020 for U.S. Appl. No. 16/653,747 (pp. 1-6). |
| Notice of Allowance dated Jul. 29, 2020 for U.S. Appl. No. 16/653,759 (pp. 1-9). |
| Office Action dated Sep. 22, 2020 for U.S. Appl. No. 16/664,814 (pp. 1-7). |
| Office Action dated Jun. 1, 2020 for U.S. Appl. No. 16/664,816 (pp. 1-11). |
| Office Action dated Jun. 1, 2020 for U.S. Appl. No. 16/664,817 (;pp. 1-11). |
| Office Action dated Sep. 17, 2020 for U.S. Appl. No. 16/664,824 (pp. 1-15). |
| Officer Action dated Oct. 6, 2020 for U.S. Appl. No. 16/672,997 (pp. 1-13). |
| Office Action dated Oct. 6, 2020 for U.S. Appl. No. 16/673,024 (pp. 1-12). |
| Notice of Allowance dated Nov. 16, 2020 for U.S. Appl. No. 16/653,759 (pp. 1-5). |
| Examination Report dated Nov. 20, 2020 issued in corresponding Indian Patent Application No. 201837009930 (7 Pages). |
