Patent Grant
9076418
If a Digital Audio Workstation (DAW) user desires to create a musical pattern such as a complex drum pattern they can utilize MIDI programming and/or arranging audio loops. However, utilizing MIDI programming and/or arranging audio loops requires in-depth expertise in MIDI and audio editing, as well as some musical background including knowledge of complex rhythms to achieve musically pleasing results where all elements of a sequence fit together.
Therefore users, particularly novice users, can benefit from a method and system for presenting a graphical user interface that allow a user to quickly program a musical sequence without music sequence programming expertise where all elements of the sequence fit together musically. Furthermore, experienced sequence programmers can benefit from a quick and efficient workflow for creating music sequences.
Disclosed are systems, methods, and non-transitory computer-readable storage media for presenting a graphical user interface that allows a user to arrange icons within a grid to program musical sequences.
An example graphical programming interface system includes a processor. A grid matrix defined by a plurality of coordinate axes, having selectable matrix positions is displayed on a display device. Multiple movable object icons, each representing an object having a predefined output sound are also displayed on the display device. In one aspect, a single object data file is associated with each matrix position on the grid matrix. In this aspect, once a user places an object icon on a matrix position, the processor causes the predefined output sound associated with the object icon in accordance with the object data file associated with the matrix position at which the object icon is placed, and outputs the processed sound to an output device. This allows a user to program musical sequences by placing one or more object icons each on the selectable matrix positions.
In a further aspect, a plurality of object data files is associated with each matrix position. In this aspect, each of the plurality of object data files associated with each matrix position is associated with an icon. In this aspect, once a user places an object icon on a matrix position, the processor causes the predefined output sound associated with the object icon in accordance with the object data file associated with the matrix position and the object icon, and outputs the processed sound to an output device
The plurality of object data files are organized or classified, such that different positions along a coordinate axis correspond to different characteristics of the object data files. The object data files are stored in a memory device.
In one aspect, the object data files are MIDI patterns or audio loop patterns. In another aspect, each object icon represents a component of a drum kit. In another aspect, a characteristic of the object data file is a rhythmic pattern of the object data file. In another aspect, the rhythmic pattern increases in complexity along a direction of a first coordinate axis of the grid matrix. In another aspect, a characteristic of the object data file for each object is intensity of notes in the object data file. In another aspect, the intensity of the object data file increases along a direction of a second coordinate axis of the grid matrix. In another aspect, the plurality of object data files are soundwave files.
Many other aspects and examples will become apparent from the following disclosure.
In order to facilitate a fuller understanding of the exemplary embodiments, reference is now made to the appended drawings. These drawings should not be construed as limiting, but are intended to be exemplary only.
The method, system, and computer-readable medium for providing a graphical user interface for sequence programming can be implemented on a computer. The computer can be a data-processing system suitable for storing and/or executing program code. The computer can include at least one processor that is coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories that provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution. Input/output or 110 devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers. Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data-processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters. In one or more embodiments, the computer can be a desktop computer, laptop computer, or dedicated device.
Although the exemplary environment described herein employs the hard disk, it should be appreciated by those skilled in the art that other types of computer-readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, digital versatile disks, cartridges, random access memories (RAMs), read only memory (ROM), a cable or wireless signal containing a bit stream and the like, may also be used in the exemplary operating environment.
To enable user interaction with the computing device 100, an input device 190 represents any number of input mechanisms such as a touch-sensitive screen for gesture or graphical input, accelerometer, keyboard, mouse, motion input, speech and so forth. The device output 170 can also be one or more of a number of output mechanisms known to those of skill in the art, such as a display or speakers. In some instances, multimodal systems enable a user to provide multiple types of input to communicate with the computing device 100. The communications interface 180 generally governs and manages the user input and system output. There is no restriction on the disclosed technology operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.
For clarity of explanation, the illustrative system embodiment is presented as comprising individual functional blocks (including functional blocks labeled as a “processor”). The functions these blocks represent may be provided through the use of either shared or dedicated hardware, including but not limited to hardware capable of executing software. For example the functions of one or more processors shown in
The technology can take the form of an entirely hardware-based embodiment, an entirely software-based embodiment, or an embodiment containing both hardware and software elements. In one embodiment, the disclosed technology can be implemented in software, which includes but may not be limited to firmware, resident software, microcode, etc. Furthermore, the disclosed technology can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computerreadable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium (though propagation mediums in and of themselves as signal carriers may not be included in the definition of physical computer-readable medium). Examples of a physical computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk, and an optical disk. Current examples of optical disks include compact disk read only memory (CD-ROM), compact disk read/write (CD-R/w), and DVD. Both processors and program code for implementing each as aspects of the technology can be centralized and/or distributed as known to those skilled in the art.
MIDI (Musical Instrument Digital Interface) is an industry-standard protocol that enables electronic musical instruments, such as keyboard controllers, computers, and other electronic equipment, to communicate, control, and synchronize with each other. MIDI does not transmit an audio signal or media, but rather transmits “event messages” such as the pitch and intensity of musical notes to play, control signals for parameters such as volume, vibrato and panning, cues, and clock signals to set the tempo. As an electronic protocol, MIDI is notable for its widespread adoption throughout the industry.
In another embodiment, organization of MIDI patterns according to complexity could mean that number of notes will increase, plus the rhythmic relationship of the notes are more intricate and the pattern contains more ornaments like ghost notes.
In another embodiment, organization of MIDI patterns according to intensity could mean that MIDI notes in the MIDI patterns have a higher velocity. In addition, in this embodiment, drum kit pieces played by the MIDI notes may switch to more intense sounding kit pieces, like moving from a closed hi-hat to a half-open or even playing the hi-hat pattern on a different sound generator such as a ride or crash. Changing instruments, this example can make a dramatic impact on the perceived intensity of a pattern.
As another example, MIDI patterns of a complete rhythm section consisting of drums, bass, rhythm guitar, percussion and so on can be organized according to “density” and “tightness”. Under the hood, MIDI patterns organized according to a “tightness” axis may change the “feel” of the pattern in numerous aspects going from a “loose, jamming type of feel” to a something that is perceived as “tight, in-the-pocket” playing by all instruments. This “tightness” can correspond to how closely MIDI notes in a MIDI pattern are to a particular grid. MIDI patterns organized according to a “density” axis can control how many instruments of the rhythm section play together the same chords or how “busy” the notes are. Busy can be defined as more notes in a given time frame.
A group of icons 230 are displayed. The icons 230 represent each instrument available for placing on the matrix 202. In one aspect, the group of icons 230 represent each piece of a drum kit software instrument. Each icon has a pre-defined output sound. The icons include Instr 1, 204, Instr 2, 206, Instr 3, 208, Instr 4, 210, Instr 5, 212, Instr 6, 214, Instr 7, 216, Instr 8, 218, Instr 9, 220, and Instr 10, 222.
In an example for drum programming, Icon 204 represents a kick drum sound, Icon 206 represents a snare sound, Icon 208 represents a rim shot sound, Icon 210 represents a clap sound, Icon 212 represents a shaker sound, Icon 214 represents a hi-hat sound, Icon 216 represents a conga sound, Icon 218 represents a second kick drum sound, Icon 220 represents a cymbal sound, and Icon 222 represents a cow bell sound.
When the icons 230 are not on the matrix 202, the pre-defined sounds corresponding to each icon are muted. A user can drag an icon to a selection grid point on matrix 202. This un-mutes the pre-defined sounds for that icon and causes the predefined sounds, such as a hi-hat, to play the pattern located on that grid point. The pattern for the icon's sound will now loop. The characteristics of the pre-defined sounds playing the pattern will have a complexity and intensity corresponding with the selected grid point's location on matrix 202. The user can continue dragging icons on matrix 202. Each time an icon is moved onto the matrix 202, and the predefined sounds corresponding to the icon are un-muted, the rhythms of the patterns played by the predefined sounds are mixed with any already looping instruments thus building and layering sound outputs from more and more icons. Likewise removing an icon off the grid will mute its respective pre-defined output sounds again. In this example, each icon can only be placed once on the matrix 202. In this example, at any point in the process the user may drag an already placed icon to a different grid position. If a user drags an already placed icon to a different grid position the pre-defined output sound will stop playing the pattern located on the first grid point and start playing the pattern located on the second grid point.
In this example, as moving grid position means a change in rhythm complexity and/or intensity, one advantage of this programming facility is an intuitive and easy to visualize way for novices to program drum patterns. Additionally, users can program musical sequences other than drum patterns using the disclosed technology.
In the example shown in
The pre-defined second kick drum sound associated with icon 218 playing the pattern associated with position (2,7), the pre-defined snare sound associated with icon 206 playing the pattern associated with position (3,2), the pre-defined shaker sound associated with icon 212 playing the pattern associated with position (5,5), and the predefined hi-hat sound associated with icon 214 playing the pattern associated with position (6,4) combine to form a musical drum pattern with four elements. Thus, a user can create a drum pattern or other music sequence by dragging icons within matrix 202 where all elements of the sequence fit together musically.
The pre-defined second kick drum sound associated with icon 218 playing the pattern associated with position (2,7), the pre-defined snare sound associated with icon 206 playing the pattern associated with position (3,2), the pre-defined shaker sound associated with icon 212 playing the pattern associated with position (8,5), and the predefined hi-hat sound associated with icon 214 playing the pattern associated with position (6,4) combine to form a new modified musical drum pattern with the four elements. All elements of the new modified musical drum pattern also fit together musically.
The pre-defined second kick drum sound associated with icon 218 playing the pattern associated with position (2,7), the pre-defined snare sound associated with icon 206 playing the pattern associated with position (3,2), the pre-defined shaker sound associated with icon 212 playing the pattern associated with position (8,5), and the predefined hi-hat sound associated with icon 214 playing the pattern associated with position (7,7) combine to form a new modified musical drum pattern with the four elements. All elements of the new modified musical drum pattern also fit together musically.
Although in
In another embodiment, instead of associating MIDI patterns with grid points on a matrix, audio loops or wave files are associated with each grid point. In this embodiment each audio loop has a set length. Furthermore, in this embodiment the set length of each audio loop is set to an equal length. In this embodiment, the audio loops are chosen so that they fit together musically. In this embodiment, all audio loops can be time-stretched to fit a defined tempo. A user can place an object icon at any matrix position, thereby causing the processor to process the predefined output sound associated with the object in accordance with an audio loop pattern associated with the matrix position at which the object icon is placed, and to output the processed sound to an output device.
At block 304, the method includes providing multiple movable object icons, each representing a component of a drum kit having a pre-defined output sound.
At block 306, the method includes providing multiple MIDI patterns, each MIDI pattern associated with a particular matrix position on the grid matrix, such that different positions along a coordinate axis correspond to different characteristics of the MIDI patterns. The MIDI patterns are stored in a memory device.
At block 308, the method includes receiving a user input to place an object icon at any matrix position, thereby causing the processor to process the predefined output sound associated with the object in accordance with the MIDI pattern associated with the matrix position at which the object icon is placed, and to output the processed sound to an output device.
Another embodiment includes the method includes providing multiple audio loop patterns, each audio loop pattern associated with a particular matrix position on the grid matrix, such that different positions along a coordinate axis correspond to different characteristics of the audio loop patterns. The audio loop patterns are stored in a memory device. This embodiment includes receiving a user input to place an object icon at any matrix position, thereby causing the processor to process the predefined output sound associated with the object in accordance with the audio loop pattern associated with the matrix position at which the object icon is placed, and to output the processed sound to an output device.
In one example, a characteristic of the MIDI/audio loop patterns is a rhythmic pattern of the MIDI/audio loop patterns. In one embodiment, the rhythmic pattern increases in complexity along a direction of a first coordinate axis of the grid matrix.
In one example, a characteristic of the MIDI/audio loop patterns is intensity of the MIDI/audio loop patterns. In one embodiment, the intensity of the MIDI/audio loop patterns increases along a direction of a second coordinate axis of the grid matrix.
At block 402, the method includes providing a grid matrix defined by multiple coordinate axes, having selectable matrix positions identified by coordinate values along the axes. The grid matrix is displayed on a display device.
At block 404, the method includes providing multiple movable object icons, each representing a component of the drum kit having a predefined output sound.
At block 406, the method includes providing multiple MIDI patterns associated with each object icon, each MIDI pattern being also associated with a particular matrix position on the grid matrix, such that different positions along a coordinate axis correspond to different characteristics of the predefined sound for each object represented by the object icons. The MIDI patterns are stored in a memory device.
At block 408, the method includes receiving a user input to place an object icon at any matrix position, thereby causing the processor to process the predefined output sound associated with the object in accordance with the MIDI pattern associated with the matrix position at which the object icon is placed, and to output the processed sound to an output device.
Another embodiment includes providing multiple audio loop patterns associated with each object icon, each audio loop pattern being also associated with a particular matrix position on the grid matrix, such that different positions along a coordinate axis correspond to different characteristics of the predefined sound for each object represented by the object icons. The audio loop patterns are stored in a memory device. This embodiment includes receiving a user input to place an object icon at any matrix position, thereby causing the processor to process the predefined output sound associated with the object in accordance with the audio loop pattern associated with the matrix position at which the object icon is placed, and to output the processed sound to an output device.
One embodiment for a graphical programming interface system includes a processor. The system further includes a grid matrix defined by multiple coordinate axes, having selectable matrix positions identified by coordinate values along the axes. In this embodiment, the grid matrix is displayed on a display device.
The system embodiment further includes multiple movable object icons, each representing an object having a predefined output sound.
The system embodiment further includes multiple object data files associated with each object icon, each object data file being also associated with a particular matrix position on the grid matrix, such that different positions along a coordinate axis correspond to different characteristics of the predefined sound for each object represented by the object icons. The multiple object data files are stored in a memory device.
Further in this system embodiment, placement of an object icon at any matrix position causes the processor to process the predefined output sound associated with the object in accordance with the object data file associated with the matrix position at which the object icon is placed, and output the processed sound to an output device.
In a further example, the object data files are MIDI patterns. In another example, the object data files are audio loop patterns. In another example, each object icon represents a component of a drum kit. In another example, a characteristic of the predefined sound for each object data file is a rhythmic pattern of the drum kit component. In this example, the rhythmic pattern can increase in complexity along a direction of a first coordinate axis of the grid matrix. In another example, a characteristic of the predefined sound for each object data file is intensity or volume of the predefined sound of the drum kit component.
In another example, the intensity of the predefined sound increases along a direction of a second coordinate axis of the grid matrix. In another example, the plurality of object data files are soundwave files. In another example, the object icons are movable on the display device by a user using a device such as a keyboard, mouse, or touch-sensitive mechanism on the display device. In another example, the grid matrix is a two-dimensional matrix.
Embodiments within the scope of the present disclosure may also include tangible and/or non-transitory computer-readable storage media for carrying or having computer executable instructions or data structures stored thereon. Such non-transitory computer readable storage media can be any available media that can be accessed by a general purpose or special purpose computer, including the functional design of any special purpose processor as discussed above. By way of example, and not limitation, such nontransitory computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer executable instructions, data structures, or processor chip design. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or combination thereof) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of the computer-readable media.
Computer-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, components, data structures, objects, and the functions inherent in the design of special-purpose processors, etc. that perform particular tasks or implement particular abstract data types. Computer executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.
Those of skill in the art will appreciate that other embodiments of the disclosure may be practiced in network computing environments with many types of computer system configurations, including personal computers, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network pes, minicomputers, mainframe computers, and the like. Embodiments may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
The above disclosure provides examples within the scope of claims, appended hereto or later added in accordance with applicable law. However, these examples are not limiting as to how any disclosed embodiments may be implemented, as those of ordinary skill can apply these disclosures to particular situations in a variety of ways.
This is a continuation of International Application No. PCT/US2011/051177, filed Sep. 12, 2011, which claims priority to U.S. Ser. No. 12/881,061, filed Sep. 13, 2010, now U.S. Pat. No. 8,330,033, all hereby incorporated herein by reference in their entirety. The following relates to a graphical user interface for programming musical sequences.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5557057 | Starr | Sep 1996 | A |
| 5741990 | Davies | Apr 1998 | A |
| 5812675 | Taylor | Sep 1998 | A |
| 6740802 | Browne, Jr. | May 2004 | B1 |
| 7010760 | Arnstein et al. | Mar 2006 | B2 |
| 7536257 | Nishibori et al. | May 2009 | B2 |
| 7538267 | Puryear | May 2009 | B2 |
| 7663049 | Puryear | Feb 2010 | B2 |
| 8063296 | Copeland et al. | Nov 2011 | B2 |
| 8173884 | Gatzsche et al. | May 2012 | B2 |
| 8822803 | Terry | Sep 2014 | B2 |
| 20060016322 | Randle et al. | Jan 2006 | A1 |
| 20100064880 | Takehisa et al. | Mar 2010 | A1 |
| 20100064881 | Takehisa | Mar 2010 | A1 |
| 20100132536 | O'Dwyer | Jun 2010 | A1 |
| 20110063206 | Karaoguz et al. | Mar 2011 | A1 |
| 20110088535 | Zarimis | Apr 2011 | A1 |
| 20110100198 | Gatzsche et al. | May 2011 | A1 |
| 20110107192 | Ge et al. | May 2011 | A1 |
| 20120060668 | Lengeling et al. | Mar 2012 | A1 |
| 20130112067 | Lengeling et al. | May 2013 | A1 |
| 20140053710 | Serletic et al. | Feb 2014 | A1 |
| Entry |
|---|
| Apple iTunes App Store, “moxMatrix,” mode of expression, LLC, released May 13, 2010 (Available online at http:// itunes.apple.com/us/app/moxmatrixlid369087304 ?mt=8, downloaded Jul. 19, 2010). |
| Apple iTunes App Store, “Matrix Music Pad,” Yudo Inc., Japan, updated Jul. 2, 2010 (Available online at http://itunes. apple.com/us/app/matrix-music-pad/id333221 071 ?mt=8, downloaded Jul. 19, 2010). |
| Apple iTunes App Store, “O-GAWA—The Future Beat Machine,” Yudo Inc., Japan, updated Apr. 23, 2010 (Available online at http://itunes.apple.com/us/app/o-gawa-the-future-beat-machine/id363731655?mt=8, downloaded Jul. 19, 2010). |
| Apple iTunes App Store, “Sound Matrix II—ToneMatrix for iPhone,” John Holdsworth, updated Dec. 3, 2009 (Available online at http://itunes.apple.com/us/app/soundmatrix-ii-tonematrixlid313032258?mt=8, downloaded Jul. 19, 2010). |
| PCT/US2011/051177 International Search Report and Written Opinion of the International Searching Authority; mailed on Mar. 29, 2012 (12 pages). |
| Beepstreetapps: “i Sequence tutorial: selections in pattern.” May 14, 2010, XP55022374, Retrieved from the Internet :U RL:http://www.youtube.com/watch?v=HQqVFy8Td 1c&feature=related [retrieved on Mar. 20, 2012]. |
| Joeyparanoia: “Ableton Launchpad Roland Clone Step Sequencer”,Youtube.com, Jan. 5, 2010, XP55022269 ,Retrieved from the Internet:URL:http://www.youtube.com/watch?v=kH8QeZuXnZg&feature=related [retrieved on Mar. 19, 2012]. |
| Gaudina et al., “Distributed Multimodal Interaction Driven Framework: Conceptual Model and Game Example”, Haptic Audio-Visual Environments and Games (Have), 2010 Ieee International Symposium on, Ieee, Piscataway, NJ, USA, Oct. 16, 2010, XP031791845, ISBN: 978-1-4244-6507-1. |
| Cong Ning Et Al: “The Musicpatterns”, Educational and Information Technology (Lceit), 2010 International Conference on, Ieee, Piscataway, NJ, USA, Sep. 17, 2010, pp. V3-9, XP031779929, ISBN: 978-1-4244-8033-3. |
| 123synthland: “Boss Dr. Rhythm DR-1ID”, youtube.com, Oct. 26, 2006, XP55022320, Internet Retrieved from the Internet: URL:http://www.youtube.com/watch?v=J1—RWnG43q4 [retrieved on Mar. 20, 2012]. |
| International Preliminary Report on Patentability, International Application No. PCT/US2011/051177, 10 pages, Mar. 19, 2013. |
| Number | Date | Country | |
|---|---|---|---|
| 20130112067 A1 | May 2013 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/US2011/051177 | Sep 2011 | US |
| Child | 13722464 | US | |
| Parent | 12881061 | Sep 2010 | US |
| Child | PCT/US2011/051177 | US |
