The present invention relates generally to the creation of music, and more particularly to a system and method for producing a more harmonious musical accompaniment.
Music is a reputably well-known form of human self-expression. However, a person's firsthand appreciation for this artistic endeavor may be derived in different manners. Often, the person can more easily enjoy music by listening to the creations of others rather than generating it by himself or herself. For many people, the ability to hear and recognize an appealing musical composition is innate, while the ability to manually create a proper collection of notes remains out of reach. A person's ability to create new music may be inhibited by the time, money, and/or skill necessary to learn an instrument well-enough to accurately reproduce a tune at will. For most people, their own imaginations may be the source of new music, but their ability to hum or chant this same tune limits the extent to which their tunes can be formally retained and recreated for the enjoyment of others.
Recording a session musician's performance can also be a laborious process. Multiple takes of the same material are recorded and painstakingly scrutinized until a single take can be assembled with all of the imperfections ironed out. A good take often requires a talented artist under the direction of another to adjust his or her performance accordingly. In the case of an amateur recording, the best take is often the result of serendipity and consequently cannot be repeated. More often than not, amateur performers produce takes with both good and bad portions. The recording process would be much easier and more fun if a song could be constructed without having to meticulously analyze every portion of every take. Thus, it is with respect to these considerations and others that the present invention has been made.
Moreover, the music that a person desires to create may be complex. For example, an envisioned tune can have more than one instrument, which may be played concurrently with other instruments in a potential arrangement. This complexity further adds to the time, skill, and/or money required for a lone person to generate a desired combination of sounds. The physical configuration of most musical instruments also requires a person's full physical attention to manually generate notes, further requiring additional personnel to play the additional parts of a desired tune. Additionally, extra review and management may then be necessary to ensure proper interaction of the various involved instruments and elements of a desired tune.
Even for people who already enjoy creating their own music, those listeners may lack the type of expertise that enables proper composition and music creation. As a result, the music created may contain notes that are not within the same musical key or chord. In most musical styles, the presence of off-key or off-chord notes, often referred to as “inharmonious” notes, causes the music to be unpleasing and jarring. Accordingly, because of their lack of experience and training, music listeners often create music that sounds undesirable and unprofessional.
For some people, artistic inspiration is not bound by the same time and location limitations that are typically associated with the generation and recording of new music. For example, a person may not be in a production studio with a playable instrument at hand when an idea for a new tune materializes. After the moment of inspiration passes, the person may not be able to recall the complete extent of the original tune, resulting in a loss of artistic effort. Moreover, the person may become frustrated with the time and effort applied in recreating no more than an inferior and incomplete version of his or her initial musical revelation.
Professional music composing and editing software tools are currently generally available. However, these tools project an intimidating barrier to entry for a novice user. Such complex user interfaces can soon sap the enthusiasm of any beginner who dares venture their way on an artistic whim. Being tethered to a suite of pro-audio servers also cramps the style of the mobile creative, wanting to craft a tune on the move.
What is needed is a system and method of music creation that can easily interface with a user's most basic ability, yet enable the creation of music that is as complex as the user's imagination and expectations. There is also an associated need to facilitate the creation of music free from notes that are inharmonious. In addition, there is a need in the art for a music authoring system that can generate a musical compilation track by aggregating portions of multiple takes based on automated selection criteria. It is also desirable that such a system further be implemented in a manner that is not limited by the location of a user when inspiration occurs, thereby enabling capture of the first utterances of a new musical composition.
There is an associated need in the art for a system and method that can create a compilation track from multiple takes by automatically evaluating the quality of previously recorded tracks and selecting the best of the previously recorded tracks, recorded via an electronic authoring system.
It is also desirable to implement a system and method for music creation that is based in the cloud whereby processing-intensive functions are implemented by a server remote from a client device. However, because digital music creation relics on vast amounts of data, such configurations are generally limited by at several factors. For the provider, processing, storing and serving such large amounts of data may be overwhelming unless the central processor is extremely powerful and thus expensive from a cost and latency point-of view. Given current costs for storing and sending data, the transmission of data from a rendering server to a client can quickly become cost prohibitive and may also add undesirable latency. From the client perspective, bandwidth limitations may also lead to significant latency issues, which detract from the user experience. Thus, there is also a need in the art for a system that can address and overcome these drawbacks.
Non-limiting and non-exhaustive embodiments are described in reference to the following drawings. In the drawings, like reference numerals refer to like parts through all the various figures unless otherwise specified.
For a better understanding of the present disclosure, a reference will be made to the following detailed description, which is to be read in association with the accompanying drawings, wherein:
The present invention now will be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific exemplary embodiments by which the invention may be practiced. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Among other things, the present invention may be embodied as methods or devices. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. The following detailed description is, therefore, not to be taken in a limiting sense.
Throughout the specification and claims, the following terms take the meanings explicitly associated herein, unless the context clearly dictates otherwise. The phrase “in one embodiment” as used herein does not necessarily refer to the same embodiment, although it may. Furthermore, the phrase “in another embodiment” as used herein does not necessarily refer to a different embodiment, although it may. Thus, as described below, various embodiments of the invention may be readily combined, without departing from the scope or spirit of the invention.
In addition, as used herein, the term “or” is an inclusive “or” operator, and is equivalent to the term “and/or,” unless the context clearly dictates otherwise. The term “based on” is not exclusive and allows for being based on additional factors not described, unless the context clearly dictates otherwise. In addition, throughout the specification, the meaning of “a,” “an,” and “the” include plural references. The meaning of “in” includes “in” and includes plural references. The meaning of “in” includes “in” and “on.”
As used herein the term “musical input,” refers to any signal input that contains musical and/or control information transmitted over any of a variety of mediums, including, but not limited to air, microphones, line-in mechanisms, or the like. Musical inputs are not limited to signal input frequencies which might be heard by a human ear, and may include other frequencies outside of that which may be heard by the human ear, or in a form not readily heard by the human ear. Moreover, the use of the term “musical” is not intended to convey an inherent requirement for a beat, rhythm, or the like. Thus, for example, a musical input, may include various inputs such as a tapping, including a single tap, clicking, human inputs (such as voice (e.g do, re, mi), percussive inputs (e.g. ka, cha, da-da), or the like) as well as indirect inputs through an instrument or other amplitude and/or frequency generation mechanism via a transport including, but not limited to, a microphone input, a Line-In input, a MIDI input, a file having signal information useable to convey a musical input, or other inputs that enable a transported signal to be converted into music.
As used herein, the term “musical key” is a group of musical notes that are harmonious. Keys are usually major or minor. Musicians frequently speak of a musical composition as being “in the key of” C major, for instance, which implies a piece of music harmonically centered on the note C and making use of a major scale whose first note, or tonic, is C. A major scale is an eight note progression consisting of the perfect and major semitones (e.g. C D E F G A B or do re mi fa so la ti). With respect to a piano, for instance, middle-C (sometimes called “C4”) has a frequency of 261.626 Hz, while D4 is 293.665 Hz; E4 is 329.628 Hz; F4 is 349.228 Hz; G4 is 391.995 Hz; A4 is 440.000 Hz; and B4 is 493.883 Hz. While the same notes on other musical instruments will play at the same frequencies, it is also understood that some instruments naturally play in one key or another.
As used herein, the term “inharmonious note” is a note that is not in a correct musical key or chord, where the correct musical key and the correct chord are the musical key or chord currently being played by another musician or musical source.
As used herein, the term “blue note” is a note that is not in a correct musical key or chord, but which is allowed to be played without transformation.
As used herein, the term “note of accompaniment musical input” is a note played by an accompanying musician that is associated with a note played in a corresponding lead melody.
The following briefly describes various embodiments in order to provide a basic understanding of some aspects of the invention. This brief description is not intended as an extensive overview. It is not intended to identify key or critical elements, or to delineate or otherwise narrow the scope. Its purpose is merely to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
Briefly stated, various embodiments are directed toward generating a multi-track recording by looping through a set of previously recorded audio tracks and receiving a new audible input for each added audio track. In one embodiment, each of the audio tracks in the multi-track recording may be generated from an audible vocal input from an end-user. Each new audible input may be provided after the current recording is repeatedly played back, or looped, one or more times. This recording sequence, separated by loop periods during which no new track input is received may permit a user to listen to the current recording thoroughly, continuously, and without a time-related pressure of an immediately required additional input. The looped playback, independent from a loop in which an additional track is input, may also permit other actions to be performed, such as modifying a previous track or changing parameters of the recording system.
Further, at least one of the audio tracks in the multi-track recording may comprise one or more musical instrument sounds generated based on one or more different sounds provided in the audible input. Various forms of processing may be performed on the received audible input to create the audio track, including aligning and adjusting of the timing of the audible input, frequency recognition and adjustment, converting the audible input to a timbre associated with a musical instrument, adding known aural cues associated with the musical instrument, and the like. Further, each of these processes may be performed in real-time, permitting almost instantaneous playback of a generated audio track and enabling another audible input to be immediately and subsequently received for processing and overlay as an audio track onto one or more previously recorded tracks in a multi-track recording.
In one embodiment, the looped or repeated portion of the multi-track recording may comprise a single bar of music. The length of this bar may be determined by a tempo and a time signature associated with the track. In another embodiment, the number of bars, or looping point for playback of the multi-track recording, may be dynamic. That is, the repetition of a first audio track in the multi-track recording may occur at a different time than that of a second audio track in the multi-track recording. The adjustment of this dynamic looping point, for example, may be automatically determined based on the length of an audible input for subsequent tracks.
Various embodiments are also directed toward automatically producing a single, “best” take that is derived from a collection of takes. In one embodiment, multiple takes of a performance are recorded during one or more sessions onto a multi-track recorder. Each take is automatically partitioned into segments. The quality of each partition of each of the multiple takes is scored, based on selectable criteria, and a track is automatically constructed from the best quality segments of each take. In one embodiment, a best segment is defined by the segment having a highest score from within a plurality of segment scores.
Various embodiments are also further directed toward protecting a musician from playing an inharmonious note. In one embodiment, the notes of an accompaniment musical instrument are received as well as from a lead musical instrument. The notes from the accompaniment musical instrument are then modified based on the key, chord, and/or timing of the lead. In one embodiment, a virtual instrument, where the input keys of the instrument dynamically map onto safe notes, may be provided. Thus, if a player of the virtual instrument is accompanying a melody, the virtual instrument may identify safe notes that comprise notes that are either for the current chord of the melody being accompanied or in the musical key of the melody.
Device Architecture
As shown in
Continuing with
Audio interface 2912 (
Device 50 also comprises input/output interface 2920 for communicating with external devices, such as a headset, a speaker 51, or other input or output devices. Input/output interface 2920 may utilize one or more communication technologies, such as USB, infrared, Bluetooth™, or the like. The optional haptic interface 2922 is arranged to provide tactile feedback to a user of device 50. For example, in an embodiment, such as that shown in
Optional GPS transceiver 2924 may determine the physical coordinates of device 100 on the surface of the Earth, which typically outputs a location as latitude and longitude values. GPS transceiver 2924 can also employ other geo-positioning mechanisms, including, but not limited to, triangulation, assisted GPS (AGPS), E-OTD, CI, SAI, ETA, BSS or the like, to further determine the physical location of device 50 on the surface of the Earth. In one embodiment, however, mobile device may, through other components, provide other information that may be employed to determine a physical location of the device, including for example, a MAC address, IP address, or the like.
As shown in
One or more data storage modules 132 may be stored in memory 2904 of device 50. As would be understood by those of ordinary skill in the art having the present specification, drawings and claims before them, a portion of the information stored in data storage modules 132 may also be stored on a disk drive or other storage medium associated with device 50. These data storage modules 132 may store multiple track recordings, MIDI files, WAV files, samples of audio data, and a variety of other data and/or data formats or input melody data in any of the formats discussed above. Data storage modules 132 may also store information that describes various capabilities of system 100, which may be sent to other devices, for instance as part of a header during a communication, upon request or in response to certain events, or the like. Moreover, data storage modules 132 may also be employed to store social networking information including address books, buddy lists, aliases, user profile information, or the like.
Device 50 may store and selectively execute a number of different applications, including applications for use in accordance with system 100. For example, application for use in accordance with system 100 may include Audio Converter Module 140, Recording Session Live Looping (RSLL) Module 142, Multiple Take Auto-Compositor (MTAC) Module 144, Harmonizer Module 146, Track Sharer Module 148, Sound Searcher Module 150, Genre Matcher Module 152, and Chord Matcher Module 154. The functions of these applications are described in more detail below.
The applications on device 50 may also include a messenger 134 and browser 136. Messenger 132 may be configured to initiate and manage a messaging session using any of a variety of messaging communications including, but not limited to email, Short Message Service (SMS), Instant Message (IM), Multimedia Message Service (MMS), internet relay chat (IRC), mIRC, RSS feeds, and/or the like. For example, in one embodiment, messenger 243 may be configured as an IM messaging application, such as AOL Instant Messenger, Yahoo! Messenger, .NET Messenger Server, ICQ, or the like. In another embodiment, messenger 132 may be a client application that is configured to integrate and employ a variety of messaging protocols. In one embodiment, messenger 132 may interact with browser 134 for managing messages. Browser 134 may include virtually any application configured to receive and display graphics, text, multimedia, and the like, employing virtually any web based language. In one embodiment, the browser application is enabled to employ Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized Markup Language (SMGL), HyperText Markup Language (HTML), eXtensible Markup Language (XML), and the like, to display and send a message. However, any of a variety of other web-based languages, including Python, Java, and third party web plug-ins, may be employed.
Device 50 may also include other applications 138, such as computer executable instructions which, when executed by client device 100, transmit, receive, and/or otherwise process messages (e.g., SMS, MMS, IM, email, and/or other messages), audio, video, and enable telecommunication with another user of another client device. Other examples of application programs include calendars, search programs, email clients, IM applications, SMS applications, VoIP applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, and so forth. Each of the applications described above may be embedded or, alternately, downloaded and executed on device 50.
Of course, while the various applications discussed above are shown as being implemented on device 50, in alternate embodiments, one or more portions of each of these applications may be implemented on one or more remote devices or servers, wherein inputs and outputs of each portion are passed between device 50 and the one or more remote devices or servers over one or more networks. Alternately, one or more of the applications may be packaged for execution on, or downloaded from a peripheral device.
Audio Converter
Audio converter 140 is configured to receive audio data and convert it to a more meaningful form for use within system 100. One embodiment of audio converter 140 is illustrated in
Track recorder 202 enables a user to record at least one audio track from either vocal or a musical instrument. In one embodiment, the user may record the track without any accompaniment. However, track recorder 202 may also be configured to play audio, either automatically or upon a user's request, comprising a click track, a musical accompaniment, an initial tone against which a user can judge his/her pitch and timing, or even previously recorded audio. “Click track” refers to a periodic clicking noise (such as the periodic clicking noise made by a mechanical metronome) intended to assist the user to keep a consistent tempo. Track recorder 202 may also enable a user to set the length of time to record—as either a time limit (i.e. a number of minutes and seconds) or a number of musical bars. When used in conjunction with MTAC module 144, as discussed below, track recorder 202 may also be configured to graphically indicate a score associated with various portions of a recorded track, so as to indicate, for instance, when a user is off-key, or the like.
In general, a musical compilation is comprised of multiple lyrical sections. For example,
Track partitioner 204 divides a recorded audio track into separate partitions that may then be addressed and potentially stored as individually addressable separate sound clips or files. The partitions are preferably chosen so that segments spliced end-to-end result in few or no audio artifacts. For example, let us assume that an audible input comprises the phrase “pum pa pum”. In one embodiment, division of this audible input may identify and distinguish each syllable of this audible input into separate sounds, such as “pum,” “pa,” and “pum.” However, it should be understood that this phrase can be delineated in other ways, and a single partition may include more than one syllable or word. Four partitions (numbered “1,” “2,” “3,” and “4”) each including more than one syllable are illustrated on display 75 in
To perform the division of an audible track into separate partitions track partitioner 204 may utilize one or more processes running on the processor 2902. In one exemplary embodiment illustrated in
Stop detector 404, on the other hand, may be configured to use speech analysis, such as formant analysis, to identify vowels and consonants in the track. For example, consonants such as T, P, B, G, K, and nasals are delimited by stoppages of airflow in their vocalization. The location of certain vowels or consonants may then be used to detect and identify preferably partitioning points. Similar to silence detector 402, the types of vowels and consonants utilized by stop detector 404 for identifying partitioning points may be configurable. Manual partitioner 406 may also be provided to enable a user to manually delimit each partition. For example, a user may simply specify a time length for each partition causing the audio track to be divided into numerous partitions each of equal length. The user may also be permitted to identify a specific location in the audio track at which a partition is to be created. The identification may be performed graphically using a pointing device, such as a mouse or game controller, in conjunction with the type of graphical user interface illustrated in
Of course, although the functions of silence detector 402, stop detector 304, and manual partitioner 406 have been described individually, it is contemplated that track partitioner 204 may use any combination of the silence detector, stop detector, and/or manual partitioner to partition or divide an audio track into segments. It would also be understood by those of ordinary skill in the art having the present specification, drawings and claims before them that other techniques to partition or divide an audio track into segments may also be used.
Quantizer 206 is configured to quantize partitions of a received audio track, which may utilize one or more processes running on the processor 2902. The process of quantization, as the term is used herein, refers to the time shifting of each previously created partition (and consequently the notes contained within the partition), as may be necessary in order to align the sounds within the partitions with a certain beat. Preferably, quantizer 206 is configured to align the beginning of each partition chronologically with a previously determined beat. For example, a meter may be provided where each bar may comprise four beats and alignment of a separate sound may occur relative to quarter-beat increments of time, thus providing sixteen time points in each four-beat bar to which a partition may be aligned. Of course, any number of increments for each bar (such as three beats for a waltz or polka effect, two beats for a swing effect, etc.) and beat may be used and, at any time during process, may be adjusted either manually by a user or automatically based on certain criteria such as a user selection of a certain style or genre of music (e.g. blues, jazz, polka, pop, rock, swing, or waltz).
In one embodiment, each partition may be automatically aligned by quantizer 206 with an available time increment to which it was most closely received at the time of recording. That is, if a sound begins between two time increments in the beat, then the playback timing of the sound will be shifted chronologically forward or backward to either one of these increments to which its initial starting time is closer. Alternately, each sound may be automatically shifted in time to each time increment that immediately precedes the relative time in which the sound was initially recorded. In yet another embodiment, each sound may be automatically shifted in time to each time increment that immediately follows the relative time in which the sound was initially recorded. A time shift, if any, for each separate sound may also be alternately or additionally influenced based on a genre selected for the multi-track recording, as further discussed below with regards to genre matcher 152. In another embodiment, each sound may also be automatically time aligned with a previously recorded track in a multi-track recording, enabling a karaoke-type effect. Moreover, the length of a separate sound may be greater than one or more time increments and time shifting of quantizer 206 may be controlled to prevent separate sounds from being time shifted so that they overlap within the same audio track.
Frequency detector 208 is configured to detect and identify the pitches of the one or more separate sounds that may be contained within each partition, which may utilize one or more processes running on the processor 2902. In one embodiment, a pitch may be determined by converting each separate sound to a frequency spectrum. Preferably, this is accomplished by using a Fast Fourier transform (FFT) algorithm, such as the FFT implementation by iZotope. However, it should be understood that any FFT implementation may be used. It is also contemplated that a Discrete Fourier Transform (DFT) algorithm may also be used to obtain the frequency spectrum.
For illustration,
In some instances, the identification of a pitch may be complicated due to additional noise. For example, as shown in
In addition to the frequency domain approaches discussed above, it is contemplated that the pitch of one or more sounds in a partition could also be determined using one or more time-domain approaches. For example, in one embodiment, the pitch may be determined by measuring the distance between zero crossing points of the signal. Algorithms such as AMDF (average magnitude difference function), ASMDF (Average Squared Mean Difference Function), and other similar autocorrelation algorithms may also be used.
For judgments in pitch to be most effective, the pitched content may also be grouped into notes (of constant frequency) and glisses (of steadily increasing or decreasing frequency). However—unlike instruments with frets or keys that naturally produce steady, discrete pitches—the human voice tends to slide into notes and wavers in a continuous fashion, making conversion to discrete pitches difficult. Consequently, frequency detector 208 may also preferably utilize pitch impulse detection to identify shifts or changes in pitch between separate sounds within a partition.
Pitch impulse detection is one approach of delimiting pitch events that focuses on the ballistics of the control loop formed between the singer's voice and his perception of his voice. Generally, when a singer utters a sound, the singer hears that sound a moment later. If the singer hears that the pitch is incorrect, be immediately modifies his voice toward the intended pitch. This negative feedback loop may be modeled as damped harmonic motion driven by periodic impulses. Thus, a human voice may be considered as a single oscillator: the vocal cord. One example illustration of a pitch changing and settling for a singer's voice 602 can be seen in
After the pitch events within a partition of an audio track have been determined, they may be converted and/or stored into a morphology, which is a plot of pitch events over time. One example of a morphology (without partitioning) is depicted in
Frequency shifter 210 may be configured to shift the frequency of the audible input, which may utilize one or more processes running on the processor 2902. For example, the frequency of one or more sounds within a partition of an audible input may be automatically raised or lowered in order to align with fundamental frequency of audible inputs or separate sounds that have been previously recorded. In one embodiment, the determination of whether to raise or lower the frequency of the audible input depends upon the closest fundamental frequency. In other words, assuming the composition was in the key of C major, if the audible frequency captured by track recorder 202 is 270.000 Hz frequency shifter 210 would shift the note down to 261.626 Hz (middle-C), whereas if the audible frequency captured by the track recorder 202 is 280.000 Hz frequency shifter 210 would shift the note up 293.665 Hz (or the D above middle-C). Even when the frequency shifter 210 primarily adjusts audible input to the closest fundamental frequency, the shifter 210 may also be further programmed to make different decisions on close calls (i.e. where the audible frequency is approximately half-way between two notes) based on the musical key, genre and/or chord. In one embodiment, the frequency shifter 210 may adjust audible inputs to other fundamental frequencies that make more musical sense based on the musical key, genre and/or chord based on controls provided by the genre matcher 260 and/or chord matcher 270, as further discussed below. Alternately or additionally the frequency shifter 210—in response to input from the instrument converter 212—may also individually shift one or more portions of one or more partitions to correspond with a predetermined set of frequencies or semitones such as those typically associated with a selected musical instrument, such as a piano, guitar or other stringed instrument, woodwind, or brass.
Instrument converter 212 may be configured to perform conversion of one or more portions of the audible input into one or more sounds that have a timbre associated with a musical instrument. For example, one or more sounds in an audible input may be converted into one or more instrument sounds of one or more different types of percussion instruments, including a snare drum, cowbell, bass drum, triangle, and the like. In one embodiment, the conversion of an audible input into one or more corresponding percussion instrument sounds may comprise adapting the timing and amplitude of one or more sounds in the audible input into a corresponding track comprising one or more sounds of the percussion instrument, the percussion instrument sound comprising a same or similar timing and amplitude as the one or more audible input sounds. For other instruments enabled to play different notes, such as a trombone or other types of brass, string, woodwind instrument or the like, the instrument conversion may further correlate one or more frequencies of audible input sounds with one or more sounds with the same or similar frequencies played by the instrument. Further, each conversion may be derived and/or limited by the physical capabilities of actually playing the corresponding physical instrument. For example, the frequencies of instrument sounds generated for an alto saxophone track may be limited by the actual frequency range of a traditional alto saxophone. In one embodiment, the generated audio track may comprise a MIDI formatted representation of the converted audible input. The data for the various instruments used by instrument converter 212 would preferably be stored in memory 2904 and may be downloaded from optical or magnetic media, removable memory, or via the network.
Gain control 214 may be configured to automatically adjust the relative volume of the audible input based on the volume of other, previously recorded tracks and may utilize one or more processes running on the processor 2902. Harmonics generator 216, may be configured to incorporate harmonics into the audio track, which may utilize one or more processes running on the processor 2902. For example, different, additional frequencies of the audible input signal may be determined and added to the generated audio track. Determining the additional frequencies may be also based on a genre from genre matcher 260 or through the use of other, predetermined parameter settings input by a user. For instance, if the selected genre were a waltz the additional frequencies may be selected from major chords harmonious to the lead music in the octave immediately below the lead, in 3/4's time with an “oom-pa-pa” beat, as follows:
Special effects editor 218 may be configured to add various effects to the audio track, such as an echo, reverberation, and the like preferably utilizing one or more processes running on the processor 2902.
Audio Converter 140 may also include a manual adjustment control 220 to enable a user to manually alter any of the settings automatically configured by the modules discussed above. For instance, manual adjustment control 220 may enable a user to alter the frequency of an audio input, or portions thereof; enable a user to alter the onset and duration of each separate sound; increase or decrease the gain for an audio track; select a different instrument to be applied to instrument converter 212, among other options. As would be understood by those of ordinary skill in the art having the present specification, drawings and claims before them, this manual adjustment control 220 may be designed for use with one or more graphical user interfaces. One particular graphical user interface will be discussed below in association with
Recording Session Live Looping
Recording Session Live Looping (RSLL) Module 142 implements a digital audio workstation that, in conjunction with audio converter 140, enables recording of audible input, generation of separate audio tracks, and the creation of multi-track recordings. Thus, RSLL Module 142 may enable any recorded audio tracks, either spoken, chanted, or otherwise, to be combined with previously recorded tracks to create a multi-track recording. As further discussed below, RSLL Module 142 is also preferably configured to loop at least one bar of a previously recorded multi-track recording for repeated playback. This repeated playback may be executed while new audible inputs are being recorded or the RSLL Module 142 is otherwise receiving instructions for a recording session currently being conducted. As a result, RSLL module 142 allows a user to continue editing and composing musical tracks while playing and listening to previously recorded tracks. As will be understood from the discussion below, the continuous looping of previously recorded tracks also minimizes the user's perception of any latency that may result from the processes that are applied to an audio track that is being currently recorded by the user, as such processes are preferably completed.
In either arrangement, process 900 begins, after a start block, at decision block 910, where a user determines whether a currently recorded multi-track recording is to be played back. The process of playing back the current multi-track recording, while enabling other actions to be performed, is generally referred to herein as “live looping.” The content and duration of a portion of the multi-track recording currently being played back, without explicit repetition, is referred to as a “live loop.” During playback, the multi-track recording may be accompanied by a click track, which generally comprises a separate audio track, not stored with the multi-track recording, that provides a series of equally spaced reference sounds or clicks that audibly indicate a speed and measure for a track for which the system is currently configured to record.
In an initial execution of process 900, an audio track may not yet have been generated. In such a state, playback of the empty multi-track recording in block 910 may be simulated and the click track may provide the only sounds played back to a user. However, in one embodiment, a user may select to mute the click track, as further discussed below with respect to block 964. Visual cues may be provided to the user during recording in conjunction with audio playback. Even when an audio track has not been recorded, and the click track is muted, indication of simulated playback and current playback position may be limited solely to those visual cues, which may include, for example, a changing display of a progress bar, pointer, or some other graphical indication (see, e.g.,
The live looped multi-track recording played back in decision block 910 may comprise one or more audio tracks that have been previously recorded. The multi-track recording may include an overall length as well as a length which is played back as a live loop. The length of a live loop may be selected to be less than the overall length of a multi-track recording, permitting a user to separately layer different bars of the multi-track recording. The length of a live loop, relative to the overall length of a multi-track recording, may be manually selected by a user or, alternately, automatically determined based on received audible input. In at least one embodiment, the overall length of the multi-track recording and the live loop may be the same. For example, the length of the live loop and the multi-track recording may be a single bar of music.
When the multi-track recording is selected for playback at decision block 910, additional visual cues, such as a visual representation of the one or more tracks, may be provided in synch with the audio playback of a live loop comprising at least a portion of the multi-track recording played back for the user. While the multi-track recording is played, process 900 continues at decision block 920 where a determination is made by an end-user whether an audio track for the multi-track recording is to be generated. The recording may be initiated based on the receipt of an audible input, such as a vocal audible input generated by an end-user. In one embodiment, a detected amplitude of an audible input may trigger the sampling and storage of an audible input signal received in system 100. In an alternate embodiment, such a track generation may be initialized by a manual input received by system 100. Further, generating a new audio track may require both a detected audible input, such as from a microphone, and a manual indication. If a new audio track is to be generated, processing continues at block 922. If the generation of an audio track is not initiated, process 900 continues at decision block 940.
At block 922, an audible input is received by the track recorder 202 of audio converter 140 and the audible input is stored in memory 2904 in one or more data storage modules 132. As used herein, “audible” refers to a property of an input to device 50 wherein, as the input is being provided, it may concurrently, naturally, and directly be heard by at least one user without amplification or other electronic processing. In one embodiment, the length of the recorded audible input may be determined based on the remaining amount of time within a live loop when the audible input is first received. That is, the recording of an audible input may be ended after a length of time at the end of a live loop, regardless if a detectable amount of audible input is still being received. For example, if the length of the loop is one bar long at four beats per bar and receipt of the audible input is first detected or triggered at the beginning of the second beat, then three beats worth of audible input may be recorded, corresponding to the second, third, and fourth beats of the bar and, thus, those second, third, and fourth beats would be looped in the multi-track recording playback continually processed in block 910. In such an arrangement, any audible input received after the end of the single bar may be recorded and processed as a basis for another separate track for the multi-track recording. Such additional processing of the separate track may be represented as a separate iteration through at least blocks 910, 920, and 922.
In at least one alternate embodiment, the length of the looped playback may be dynamically adjusted based on the length of audible input received at block 922. That is, the audible input might automatically result in an extension of the length of the track of the multi-track recording that is currently being played in block 910. For example, if additional audible input is received after a length of a current live loop has been played back, then this longer audible input may be further recorded and maintained for derivation as the new audio track. In such an arrangement, previous tracks of the multi-track recording may be repeated within subsequent live loops in order to match the length of the received audible input. In one embodiment, the repetition of the shorter, previous multi-track recording may be performed an integral number of times. This integral number of repetitions retains the relationship, if any, between multiple bars of the shorter multi-track recording previously recorded. In such a way, the looping point of a multi-track recording and live loop may be dynamically altered.
Similarly, the length of the received track at block 922 may be shorter than the length of the currently playing live loop (i.e. receiving only one bar of audible input during the playback of a four bar long live loop). In such an arrangement, the end of the audible input may be detected when no additional audible input has been received after a predetermined time (e.g. a selected number of seconds) following the receipt and recording of an audible input of at least a threshold volume. In one embodiment, the detection of this silence may be based on the lack of input above the threshold volume of the current live loop. Alternately or additionally, the end of an audible input may be signaled by receipt of a manual signal. The associated length of this shorter audible input may be determined in terms of a number of bars with a same number of beats as the multi-track recording. In one embodiment, this number of bars is selected as a factor of the length of the current live loop. In each case, an audible input, once converted to a track at block 924, may be manually or automatically selected for repetition for a number of times sufficient to match a length of the multi-track recording currently being played back.
In block 924, the received audible input may be converted into an audio track by audio converter 140. As discussed above, the audio conversion process may include various operations including partitioning, quantization, frequency detection and shifting, instrument conversion, gain control, harmonics generation, adding special effects, and manual adjustment. The order of each of these audio conversion operations may be altered, and may, in at least one embodiment, be configured by an end-user. Also, each of these operations may be selectively applied, enabling the audible input to be converted to an audio track with as much or as minimal additional processing as required. For example, instrument conversion may not be selected, thus permitting one or more original sounds from an audible input to be substantially included in the generated audio track with its original timbre. In block 924, an echo cancellation process may be applied to filter out the audio of other tracks being played during live looping from the audio track being actively recorded. In one embodiment, this may be accomplished by identifying the audio signal being played during the live loop, determining any delay between the output audio signal and the input audio signal; filtering and delaying the output audio signal to resemble the input audio signal; and subtracting the output audio signal from the input audio signal. One preferred echo cancellation process that may be used is one implemented by iZotope, although other implementations may also be used. The processes of block 924 may be subsequently applied or removed as further discussed herein with respect to block 942. After converting the audible input into a generated audio track at block 924, process 900 continues at block 926.
At block 926, the generated audio track from block 924 may be added in real-time to a multi-track recording. This may be a multi-track already initiated or, alternately, a new multi-track with the audio track included as the first track thereof. After block 926, process 900 may begin again at decision block 910, wherein the multi-track may be played back with the most recently generated audio track included. While the operations of 922, 924, and 926 are shown as being performed in series in
At decision block 940, a determination is made whether one or more audio tracks in the multi-track recording are to be modified. For example, an input may be received that indicates an end-user desires to modify one or more of the previously recorded audio tracks. In one embodiment, the indication may be received through a manual input. As noted above, this modification may also be performed during playback of the currently recorded multi-track recording, permitting immediate appreciation of a current state of the multi-track recording for the end-user. In one embodiment, the indication may include one or more tracks of the multi-track recording to which an adjustment is desired to be applied. These tracks may also include one or more new tracks manually added to the multi-track recording. If indication of a track modification is received, process 900 continues at block 942; otherwise, process 900 continues at decision block 960.
At block 942, the parameters of the one or more previously converted tracks are received and adjusted parameters may be input by an end-user. Parameters for modification may include any adjustments that can be done using the processes of audio converter 140, which may include among other examples, muting or soloing a track, removal of an entire track, adjusting the strike velocity of an instrument in a track, adjusting the volume level of a track, adjusting a tempo of playback of all tracks in the live loop, adding or removing separate sounds from selected time increments of a track, adjusting the length of a live loop and/or overall length of the multi-track recording. Adjusting the length of the live loop may comprise altering start and end points of the loop with respect to the overall multi-track recording and/or may also comprise adding more bars to the tracks currently being repeated in a live loop, adding and/or appending previously recorded bars of the multi-track recording with at least a subset of the tracks previously associated with these bars, or deleting bars from the multi-track recording. The addition of a new track may require various aspects of this new 10, track to be manually input by an end-user. Also at block 942, a search may be conducted for an additional track through the use of sound searcher module 150 to facilitate an end-user's reuse of previously recorded audio tracks.
At block 944, the adjusted parameters are applied to one or more tracks indicated at decision block 940. The application may include converting the adjusted parameter into a format compatible with the adjusted one or more tracks. For example, one or more numerical parameters may be adjusted to correspond to one or more values applicable to the MIDI or other protocol format. After block 944, process 900 may begin again at decision block 910, wherein at least a portion of the multi-track recording that corresponds to the live loop may be played back with the one or more modified audio tracks included.
At decision block 960, a determination is made whether a recording setup is to be modified. For example, an input may be received that indicates if a user desires to modify one or more aspects of the recording setup. This indication may also be received through a manual input. The indication may further one or more parameter settings of a recording setup that are to be adjusted. If the end user desires to modify the recording setup process 900 continues at block 962; otherwise, process 900 continues at decision block 980.
At block 962, the recording system may be calibrated. Particularly, the recording circuit, comprising at least an audio input source, audio output source, and audio track processing components, may be calibrated to determine latency of system 100 in conjunction with device 50, preferably measured in thousandths of a second, between a playback of a sound through the audio output source and receipt of an audible input through the audio input source. For example, if a recording circuit comprises a headset and a microphone, the latency may be determined by the RSLL 142 to improve receipt and conversion of an audible input, particularly a determination of a relative timing between beats of a multi-track recording being played back and a received audible input. After calibration at block 962, if any, process 900 continues to block 964.
At block 964, other recording system parameters settings may be changed. For example, playback of the click track may be turned on or off. Also, default settings for new tracks or new multi-track recordings may be modified, such as a default tempo and a default set of conversions for an audible input for block 924 may be provided. The time signature of a current multi-track recording may also be changed at block 964. Other settings associated with a digital audio workstation may also be provided so they may be modified by an end-user as would be understood by those of ordinary skill in the art having the present specification, drawings and claims before them. After block 964, process 900 may return to decision block 910, wherein adjustments to the recording system may be applied to the subsequent recording and modification of audio tracks for a multi-track recording.
At block 980, a determination is made whether the recording session is to be ended. For example, an input indicating the end of the session may be received from a manual input. Alternately, device 50 may indicate the end of the session if, for example, data storage 132 is full. If an end-of-session indication is received, the multi-track recording may be stored and/or transmitted for additional operations. For example, a multi-track recording may be stored in data storage 132 for future retrieval, review, and modification in a new session or a continuation of the session in which the multi-track recording was initially created. The multi-track recording may also be transmitted from a device 50 to another device 50 over a network for storage in at least one remote data store associated with a user account. A transmitted multi-track recording may also be shared through a network server with an online music community or shared in a game hosted by a network server.
If the recording session is not ended, process 900 returns again to decision block 910. Such a sequence of events may represent periods in which a user is listening to a live loop while deciding which, if any, additional tracks are to be generated or other modifications, if any, are to be performed. It will be understood by those of ordinary skill in the art having the present specification, drawings and claims before them that each block of the flowchart illustration in
The operation of certain aspects of the invention will now be described with respect to various screen displays that may be associated with a user interface implementing audio converter 140 and RSSL module 142. The illustrated embodiments are non-limiting, non-exhaustive example user interfaces that may be employed in association with the operations of system 100. The various screen displays may include many more or less components than those shown. Furthermore, the arrangement of the components are not limited to that shown in these displays, and other arrangements are also envisioned, including the disposition of various components on different interfaces. However, the components shown are sufficient to disclose an illustrative embodiment for practicing the present invention.
Tempo control 1012 displays a tempo of the multi-track recording in beats per minute. Tempo control 1012 may be directly, manually modified by a user. Bar control 1014 displays a bar number for a multi-track recording. Bar control 1014 may be configured to display a current bar number during a live loop, an overall number of bars, or alternately be used to select a certain bar number of the multi-track recording for further display in the interface 1000.
Beat control 1016 displays a beat number for a multi-track recording. Beat control 1016 may be configured to display a total number of beats for each bar, or, alternately, a current beat number during playback of the multi-track recording. Time control 1018 displays a time for the multi-track recording. This time control 1018 may be configured to display an overall time for the multi-track recording, a length of time for a currently selected live loop, an absolute or relative time during a live loop, or be used to jump to a certain absolute time of a multi-track recording. Operations of controls of interface 1000, such as controls 1012, 1014, 1016, 1018, and 1021-1026, may be changed in block 964 of
Add track control 1021 enables a user to manually add a track to a multi-track recording. Upon selection of control 1021, a new track is added to the multi-track recording and the interface is updated to include additional controls 1040-1054 for the added track, the operations of which are discussed as follows. Render WAV control 1022 generates and stores a WAV file from at least a portion of a multi-track recording. The portions of the multi-track recording rendered in this WAV file, as well as other storage parameters, may be further entered by a user upon selection of Render WAV control 1022. Further, other audio file formats, in addition to WAV, may also be available through a control such as control 1022.
Click track control 1023 toggles playback of the click track. Armed control 1024 toggles on and off the recording component of RSLL 142 and a device's ability to record an audible input. Armed control 1024 enables an end user to speak with other users, practice a vocal input, and create other audible sounds during a recording session without having those sounds converted into an audible input that is further processed by RSLL 142.
Circuit parameters control 1025 enables a user calibrate recording circuit parameters as is further discussed with regards to
Grid 1050 on user interface 1000 represents the playback and timing of separate sounds within one or more tracks of a multi-track recording, wherein each row represents an individual track and each column represents a time increment. Each row may, for example, include a box for each time increment in a single bar. Alternately, each row may include enough boxes to represent time increments for an overall duration of a live loop. Boxes with a first shading or color in the grid 1050, such as box 1052, may represent a relative timing of where a sound is played back during a live loop, while other boxes, such as boxes 1054, each indicate a time increment within a track where a separate sound is not played back. A track added via manual control 1021 initially includes boxes such as box 1054. Selection of a box, such as a box 1052 or box 1054 may add or remove a sound from the track at the time increment associated with the selected box. Sounds added via manual input to a box in grid 1050 may comprise a default sound for an instrument selected for the track, or alternately, a copy of at least one sound quantized from an audible input for a track. This manual operation with grid 1050 enables an audible input to generate one or more sounds for a track, yet add copies of one or more of these sounds at manually chosen locations within the track.
A progress bar 1056 visually indicates a time increment of a current playback position of a multi-track recording. Each track in grid 1050 is associated with a set of track controls 1040, 1042, 1044, 1046, and 1048. Remove track control 1040 enables removal of a track from multi-track recording and may be configured to selectively remove a track from one or more bars of a multi-track recording.
Instrument selection control 1042 enables selection of an instrument to which sounds of an audible input are converted in the generated audio track. As illustrated in
Mute/Solo control 1044 mutes an associated track or mutes all other tracks except for the track associated with the control 1044. Velocity control 1046 enables adjustment of an initial attack or strike strength of instrument sounds generated for a converted audio track, which may influence the peak, duration, release, and overall amplitude shape of each instrument sound generated for the associated audio track. Such velocity may be manually entered or, alternately, extracted based on properties of the audible input sounds from which one or more instrument sounds are generated. Volume control 1048 enables individual control of the playback volume of each track in the multi-track recording.
Thus, as illustrated, interfaces 1000 and 1100 present users with a control space that is welcoming and non-threatening, powerful, and consistent, yet intuitive to learn, which is particularly important for a lay user who is not a professional musician or otherwise familiar with digital audio authoring tools.
In the embodiment illustrated in
As depicted by comparing
Each box in the grid represents the timing increments for sounds associated with the related percussion instrument, where an unshaded box indicates that no sound is to be played at that time increment, and a shaded box indicates that a sound (associated with the timbre of the related percussion instrument) is to be played at that time increment. Thus,
Multiple Take-Auto Composition Module
MTAC Module 144 (
Partition scorer 1702 may be configured to score partitions based on any one or more criteria, which may utilize one or more processes running on the processor 2902. For example, a partition may be scored based on the key of the partition relative to a key selected for the overall composition. Often, a performer may sing an off-key note without knowing it. Thus, notes within a partition may also be scored based on the difference between the key of the note and the proper key for the partition.
In many cases, however, a novice end-user may not know what music key be wants to sing in. Consequently, partition scorer 1702 may also be configured to automatically identify a key, which may be referred to as “Automatic Key Detection.” With “Automatic Key Detection,” partition scorer 1702 may determine the key closest to that of the end-user's recorded audio performance. The system 50 may highlight any notes that are off-key from the automatically detected key and may further automatically adjust those notes to fundamental frequencies that are in the automatically determined key signature.
One illustrative process for determining the musical key is depicted in
As shown in the third block of
In one embodiment, a note's pitch class in a key, represented by the value “index” in
It may be desirable to improve the accuracy of the musical key determination than that achieved with the foregoing methodologies. Where such improved accuracy is desired, the partition scorer 1702 (or alternatively the harmonizer 146 (discussed below)) may determine whether each of the top four most probable keys (determined by the initial key signature determination methodology (described above) has one or more major or minor modes. As would be understood by those of ordinary skill in the art having the present specification before them, it is possible to determine the major or minor modes of any plurality of probable keys to achieve improvement in key signature accuracy with the understanding that the greater the number of probable keys analyzed the greater the processing requirements.
The determination whether each of the probable keys has one or more major or minor modes may be done by performing interval profiling on the notes fed to the partition scorer 1702 (or to the harmonizer 146 by the lead music source 2404 in some embodiments). As shown in
That would result in the matrix values depicted in
Then, the minor and major key sums may be calculated, as follows:
After completing these product-sum calculations for every index in the matrix, the values of the major and minor key sums are compared to the scores assigned to the plurality of most probable keys determined the initial key signature determination and a determination is made as to which key/mode combination is the best. After completing these product-sum calculations for every index in the matrix, the values of the major and minor key sums are multiplied by their corresponding matrix index in each of the interval profiles. Subsequently, the sum of those products constitutes the final valuation of the likelihood that the given set of notes is in the mode. So, for the example set forth in
Then, to determine the value for whether this mode is minor, however, we need to shift the minor interval profile into the relative minor. The reason for this is that the interval profile is set up to consider the tonic of the mode (not the root of the key signature) to be our first column and first row. We can understand why this is true by looking at the underlying music. Any given key signature may be either major or minor. For example, the major mode that is compatible with the key signature of C Major is the C Major mode. The minor mode that is compatible with the key signature of C Major is the A (natural) minor mode. Since the top-left numerical value in our minor interval represents the transition from C to C when considering the C minor mode, all the indices of comparison would shift by 3 steps (or, more specifically, 3 columns to the right, and 3 rows down), since the tonic/root of a minor key signature relative to the tonic/root of the major key signature is 3 semitones down. Once shifted by 3 steps, the top left numerical value in our interval profile represents the transition from A to A in the A minor mode. To run the numbers using our example of
The same process described above would apply to any key signatures as long as the initial matrix of note transitions is relative to the key being considered. So using
In another embodiment where the end-user knows which musical key they wish to be in, the user may identify that key in which case, the process of
In another embodiment, a partition may also be judged against a chord constraint. A chord sequence is a musical constraint that can be employed when the user wishes to record an accompaniment. Accompaniments may be typically thought of as arpeggiations of the notes in the chord track and may also include the chords themselves. It is, of course, permissible to play notes that are out of the chord, but these must typically be judged on their musical merits.
One illustrative process for scoring the quality of a partition's harmony based on a chord sequence constraint is depicted in
In another embodiment, partition scorer 1702 may also judge a partition against the collection of certain allowed pitch values, such as semitones as are typical in Western music. However, quarter tones of other musical traditions (such as those of Middle Eastern Cultures) are similarly contemplated.
In another embodiment, a partition may also be scored based on the quality of transitions between the various pitches within the partition. For example, as discussed above, changes in pitch may be identified using pitch impulse detection. In one embodiment, the same pitch impulse detection may also be used to identify the quality of the pitch transitions in a partition. In one approach, system may utilize the generally understood concept that damped harmonic oscillators generally satisfy the following equation:
where ω0 is the undamped angular frequency of the oscillator and ζ is a system dependent constant called the damping ratio. (for a mass on a spring having a spring constant k and a damping coefficient c, ω0=√{square root over (k/m)} and ζ=c/2mω0.) It is understood that the value of the damping ratio ζ critically determines the behavior of the damped system (e.g. overdamped, critically damped (ζ=1), or underdamped). In a critically damped system, the system returns to equilibrium as quickly as possible without oscillating. A professional singer, in general, is able to change his/her pitch with a response that is critically damped. By using pitch impulse analysis, both the true start of the pitch change event and the quality of the pitch change may be determined. In particular, the pitch change event is the deduced step function, whereas the quality of the pitch change is determined by the ζ value. For instance,
Another exemplary method for scoring the quality of pitch transition is shown in
The scoring methods described above may be utilized to score a partition against either an explicit reference or an implicit reference. An explicit reference may be an existing or pre-recorded melody track, musical key, chord sequence, or note range. The explicit case is typically used when the performer is recording in unison with another track. The explicit case could analogized to judging Karaoke in that the music reference exists and the track is being analyzed using the previous known melody as the reference. An implicit reference, on the other hand, may be a “target” melody (i.e. the system's best guess at the notes that the performer is intending to produce) computed from multiple previously recorded takes that have been saved by track recorder 202 in data storage 132. The implicit case is typically used when the user is recording the lead melody of a song during which no reference is available, such as an original composition or a song for which partition scorer 1702 has no knowledge.
In the case where a reference is implicit, a reference may be computed from the takes. This is typically achieved by determining the centroid of the morphologies for each of the N partitions of each previously recorded track. In one embodiment, the centroid of a set of morphologies is simply a new morphology constructed by taking the mean average pitch and duration for each event in the morphology. This is repeated for n=1 to N. The resulting centroid would then be treated as the morphology of the implicit reference track. One illustration of a centroid determined in this manner for a single note is depicted in
As would be understood by those of ordinary skill in the art having the present specification, drawings and claims before them, any number of the foregoing independent methodologies for scoring partitions may be combined to provide an analysis of a wider set of the considerations. Each score may be given identical or different weight. If the scores are given different weights it may be based on the particular genre of the composition as determined by genre matcher module 152. For instance, in some musical genre a higher value may be placed on one aspect of a performance over another. The selection of which scoring methodologies are applied may also be determined automatically or manually selected by a user.
As illustrated in
In one embodiment, the end-user may select the reference and/or one or more methods against which the recorded take(s) should be scored, step 2110. For instance, the user's configuration may indicate that the partition should be scored against a key, a melody, the chords, a target morphology constructed from the centroid of one or more tracks, or any other method discussed above. The guide selection may be made manually by the user or set automatically by the system.
The partitions of a take are scored in step 2112, and, in step 2114, an indication of the scoring for each partition in a track may be indicated to the user. This may benefit the end user by providing them with an indication of where the end user's pitch or timing is off so that the end user can improve in future takes. One illustration of a graphical display for illustrating the score of a partition is illustrated in
In step 2116, the end user manually determines whether to record another take. If the user desires another take, the process returns to step 2104. Once the end user has recorded all of the multiple takes for a track, the process proceeds to step 2118.
In step 2118, the user may be provided a choice as to whether a “best” overall track is to be compiled from all of the takes manually or automatically. If the user selects to create a manual composition, the user may, in step 2120, simply audition the first partition of the first take, followed by the first partition of the second take, until each of the candidate first partitions have been auditioned. One interface that be used to facilitate the auditioning and selection between the various takes of the partitions is shown in
One example of a virtual “best” track that is spliced together from partitions of actual recorded tracks is also illustrated in
Harmonizer
Harmonizer module 146 implements a process for harmonizing notes from an accompanying source with a musical key and/or chord of a lead source, which may be a vocal input, a musical instrument (real or virtual), or a pre-recorded melody that may be selectable by a user. One exemplary embodiment of this harmonizing process an accompanying source is described in conjunction with
The transform note module 2402 may also receive a musical key and/or a selected chord from the chord/key selector 2408. The control signal from controller 2410 indicates to transform note module 2402 whether the music output should be based on the lead music input, accompaniment music input and/or musical key or chord from the chord/key selector 2408 and how the transformation should be handled. For instance, as described above, the musical key and chord may be either derived from the lead melody or the accompaniment source or even from the manually selected key or chord indicated by chord/key selector 2408.
Based on the control signal, the transform note module 2402 may alternatively transform the lead music input into a note consonant with the chord or the musical key, producing a harmonious output note. In one embodiment, input notes are mapped to harmonious notes using a pre-established consonance metric. In an embodiment discussed in more detail below, the control signal may also be configured to indicate whether one or more “blue-notes” may be allowed in the accompaniment music input without transformation by the transform note module 2402.
At process 2506, a chord of the lead melody may be determined from the notes found in lead music buffer 2516. The chord of the lead melody may be determined by analyzing notes in the same manner forth in association with
At process 2508 of
Process 2510 (
At process 2512, the transformed note may be rendered by modifying the note of accompaniment musical input and modifying the timing of the note of accompaniment musical input. In one embodiment, the rendered note is played audibly. Additionally or alternatively, the transformed note may also be rendered visually.
Control consonance 2514 represents a collection of decisions that the process makes based on one or more inputs from one or more sources that control the selection of notes made by transform note process 2510. Control consonance 2514 receives a number of input control signals from controller 2410 (see
In one approach, control consonance 2514 may further include logic to find “bad sounding” notes (based on the selected chord progression) and snap them to the nearest chord tone. A “bad sounding” note would still be in the correct key, but it will sound bad over the chord being played. The notes are categorized into 3 different sets, relating to the chord over which they are played. The sets are defined as: “chordTones”, “nonChordTones” and “badTones”. All notes would still be in the correct key, but they would have varying degrees of how “bad” they sound over the chord being played; chordTones sound the best, nonChordTones sound reasonably well, and badTones sound bad. In addition, a “strictness” variable may be defined where notes are categorized based on how strictly they should adhere to the chords. These “strictness” levels may include: StrictnessLow, StrictnessMedium, and StrictnessHigh. For each “strictness” level, the three sets of chordTones, nonChordTones, and badTones vary. Further, for every “strictness” level, the three sets always relate to each other in this way: chordTones are always the tones of which the chord consists, badTones are the tones that will sound “had” at this strictness level, and the nonChordTones are the diatonic tones left over that have not been accounted for in either set. Because the chords are variable, the badTones may be specifically categorized for each strictness level, while the other two sets may be categorized when given a specific chord. In one embodiment, the rules for identifying “bad sounding” notes is static, as follows:
StrictnessLow (badTones):
StrictnessMedium (badTones):
StrictnessHigh (badTones):
Being a “bad” note alone may not be the sole basis for correction, basic counterpoint logic based on classical melodic theory may be used to identify those notes that would sound bad in context. The rules for whether notes are snapped to a chordTone may also be defined dynamically in terms of the strictness levels described above. Each level may use the note set definitions described above at its corresponding strictness level, and may further be determined in terms of “stepTones.” A stepTone is defined as any note that falls directly before a chordTone in time, and is 2 or less semitones away from the chordTone; and any note that falls directly after a chordTone in time, and is also 2 or less semitones away from the chordTone. In addition, each level may apply the following specific rules:
Another input to control consonance 2514 is the consonance metric, which is essentially a feedback path from the transform note process 2510. First, “consonance” is generally defined as sounds that make for pleasant harmony with respect to some base sound. Consonance can also be thought of as the opposite of dissonance (which includes any sounds used freely even if they are inharmonious). So, if an end user has caused control signals to be led into control consonance 2514 via controller 2410 that constrained the output note from transform note process 2510 to the chord or key manually selected via chord/key selector 2408, then it is possible that one or more of the output notes were inharmonious to the lead music buffer 2516. An indication that the output note was inharmonious (i.e. the consonance metric) will be ultimately fed back to control consonance 2514. While, control consonance 2514 is designed to force the output note track generated by transform note 2510 back into consonance with the lead music due to inherent latencies in feedback and programming systems, a number of inharmonious notes are expected to be allowed through into the music output. In fact, allowing at least some inharmonious notes and even inharmonious rifts in the music produced by the system should facilitate the system 50 making a less mechanical sounding form of musical composition, something desired by the inventors.
In one embodiment, another control signal that may also be input into control consonance 2514 indicates whether one or more “blue-notes” may be allowed in the music output. As noted above, the term “blue note” for purposes of this specification is given a broader meaning than its ordinary use in blues music as a note that is not in a correct musical key or chord, but which is allowed to be played without transformation. In addition to the harnessing the latencies of the system to provide some minimal insertion of “blue notes,” one or more blues accumulators (preferably software coded rather than hard wired) may be used to provide some additional leeway for blue-notes. So, for example, one accumulator can be used to limit the number of blue-notes within a single partition, another accumulator can be used to limit the number of blue-notes in adjacent partitions, still another accumulator can be used to limit the number of blue-notes per some predetermined time interval or total number of notes. In other words, control consonance via the consonance metric may be counting any one or more of the following: elapsing time, the number of blue-notes in the music output, the number of total notes in the music output, the number of blue-notes per partition, etc. Pre-determined, automatically determined, and real-time determined/adjusted ceilings can be programmed in real-time or as preset/predetermined values. These values may also be affected by the genre of the current composition.
In one embodiment, the system 100 may also include a super keyboard for providing an accompaniment music source. The super keyboard may be a physical hardware device, or a graphical representation that is generated and displayed by a computing device. In either embodiment, super keyboard may be thought of as the manual input for chord/key selector 2408 of
One embodiment of a super keyboard in accordance with the present invention is illustrated in
In one embodiment, the chord wheel 2700 depicts seven chords related to the currently selected musical key—three major chords, three minor chords, and one diminished chord. In this embodiment, the diminished chord is located at the center of the chord wheel; the three minor chords surround the diminished chord; and the three major chords surround the three minor chords. In one embodiment, a player is enabled to select a musical key using the outermost concentric circle, wherein each of the seven chords depicted by the chord wheel are determined by the selected musical key.
Track Sharer Module
Returning to the diagram of system 100 in
Sound Searcher Module
Sound searcher module 150, also shown in
Genre Matcher Module
Genre Matcher Module 152, also shown in
Chord Matcher Module
Chord Matcher 154 provides pitch and chord related services. For example, Chord Matcher 154 may perform intelligent pitch correction of a monophonic track. Such a track may be derived from an audible input and pitch correction may include modifying a frequency of the input to align the pitch of the audible input with a particular, predetermined frequency. The Chord Matcher 154 may also build and refine an accompaniment to an existing melody included in a previously recorded multi-track recording.
In one embodiment, Chord Matcher 154 may also be configured to dynamically identify the probability of appropriate future chords for an audio track based on the previously played chords. In particular, Chord Matcher 142 may, in one embodiment, include a database of music. Using a Hidden Markov Model in conjunction with this database, the probabilities for a future progression of chords may then be determined based on the previous chords occurring in the audio track.
Network Environment
As discussed above, device 50 may be any device capable of performing the processes described above, and need not be networked to any other devices. Nevertheless,
As shown, system 2800 of
Client devices 2801-2805 may also include virtually any computing device capable of communicating over a network to send and receive information, including track information and social networking information, performing audibly generated track search queries, or the like. The set of such devices may include devices that typically connect using a wired or wireless communications medium such as personal computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, or the like. In one embodiment, at least some of client devices 2803-2805 may operate over wired and/or wireless network.
A web-enabled client device may also include a browser application that is configured to receive and to send web pages, web-based messages, and the like. The browser application may be configured to receive and display graphics, text, multimedia, and the like, employing virtually any web-based language, including a wireless application protocol messages (WAP), and the like. In one embodiment, the browser application is enabled to employ Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized 25 Markup Language (SMGL), HyperText Markup Language (HTML), eXtensible Markup Language (XML), and the like, to display and send various content. In one embodiment, a user of the client device may employ the browser application to interact with a messaging client, such as a text messaging client, an email client, or the like, to send and/or receive messages.
Client devices 2801-2805 also may include at least one other client application that is configured to receive content from another computing device. The client application may include a capability to provide and receive textual content, graphical content, audio content, and the like. The client application may further provide information that identifies itself, including a type, capability, name, and the like. In one embodiment, client devices 3001-3005 may uniquely identify themselves through any of a variety of mechanisms, including a phone number, Mobile Identification Number (MIN), an electronic serial number (ESN), or other mobile device identifier. The information may also indicate a content format that the mobile device is enabled to employ. Such information may be provided in a network packet, or the like, sent to MND 108, or other computing devices.
Client devices 2801-2805 may further be configured to include a client application that enables the end-user to log into a user account that may be managed by another computing device, such as MND 2808, or the like. Such a user account, for example, may be configured to enable the end-user to participate in one or more social networking activities, such as submit a track or a multi-track recording, search for tracks or recordings similar to an audible input, download a track or recording, and participate in an online music community, particularly one centered around the sharing, review, and discussion of produced tracks and multi-track recordings. However, participation in various networking activities may also be performed without logging into the user account.
In one embodiment, a musical input comprising the melody may be received by client devices 2801-2805 over network 2806 or 2810 from MND 3008, or from any other processor-based device capable of transmitting such a musical input. The musical input containing the melody may be pre-recorded or captured live by MND 2808 or other such processor-based device. Additionally or alternatively, the melody may be captured in real-time by client devices 2801-2805. For example, a melody generating device may generate a melody, and a microphone in communication with one of client devices 2801-2805 may capture the generated melody. If the music input is captured live, the system typically seeks at least one bar of music before the musical key and chords of the melody are calculated. This is analogous to musicians playing in a band, where an accompanying musician may typically listen to at least one bar of a melody to determine the musical key and chords being played before contributing any additional music.
In one embodiment, the musician may interact with client devices 2801-2805 in order to accompany a melody, treating a client device as a virtual instrument. Additionally or alternatively, the musician accompanying the melody may sing and/or play a musical instrument, such as user played instrument, to accompany a melody.
Wireless network 2810 is configured to couple client devices 2803-2805 and its components with network 2806. Wireless network 2810 may include any of a variety of wireless sub-networks that may further overlay stand-alone ad-hoc networks, and the like, to provide an infrastructure-oriented connection for client devices 2803-2805. Such sub-networks may include mesh networks, Wireless LAN (WLAN) networks, cellular networks, and the like. Wireless network 2810 may further include an autonomous system of terminals, gateways, routers, and the like connected by wireless radio links, and the like. These connectors may be configured to move freely and randomly and organize themselves arbitrarily, such that the topology of wireless network 2810 may change rapidly.
Wireless network 2810 may further employ a plurality of access technologies including 2nd (2G), 3rd (3G), 4th (4G) generation radio access for cellular systems, WLAN, Wireless Router (WR) mesh, and the like. Access technologies such as 2G, 3G, 4G and future access networks may enable wide area coverage for mobile devices, such as client devices 2803-2805 with various degrees of mobility. For example, wireless network 2810 may enable a radio connection through a radio network access such as Global System for Mobil communication (GSM), General Packet Radio Services (GPRS), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), and the like. In essence, wireless network 2810 may include virtually any wireless communication mechanism by which information may travel between client devices 2803-2805 and another computing device, network, and the like.
Network 2806 is configured to couple network devices with other computing devices, including, MND 2808, client devices 2801-2802, and through wireless network 2810 to client devices 2803-2805. Network 2806 is enabled to employ any form of computer readable media for communicating information from one electronic device to another. Also, network 106 can include the Internet in addition to local area networks (LANs), wide area networks (WANs), direct connections, such as through a universal serial bus (USB) port, other forms of computer-readable media, or any combination thereof. On an interconnected set of LANs, including those based on differing architectures and protocols, a router acts as a link between LANs, enabling messages to be sent from one to another. In addition, communication links within LANs typically include twisted wire pair or coaxial cable, while communication links between networks may utilize analog telephone lines, full or fractional dedicated digital lines including T1, T2, T3, and T4, Integrated Services Digital Networks (ISDNs), Digital Subscriber Lines (DSLs), wireless links including satellite links, or other communications links known to those skilled in the art. Furthermore, remote computers and other related electronic devices could be remotely connected to either LANs or WANs via a modem and temporary telephone link. In essence, network 2806 includes any communication Method by which information may travel between computing devices.
In one embodiment, client devices 2801-2805 may directly communicate, for example, using a peer to peer configuration.
Additionally, communication media typically embodies computer-readable instructions, data structures, program modules, or other transport mechanism and includes any information delivery media. By way of example, communication media includes wired media such as twisted pair, coaxial cable, fiber optics, wave guides, and other wired media and wireless media such as acoustic, RF, infrared, and other wireless media.
Various peripherals, including I/O devices 2811-2813 may be attached to client devices 2801-2805. Multi-touch, pressure pad 2813 may receive physical inputs from a user and be distributed as a USB peripheral, although not limited to USB, and other interface protocols may also be used, including but not limited to ZIGBEE, BLUETOOTH, or the like. Data transported over an external and the interface protocol of pressure pad 2813 may include, for example, MIDI formatted data, though data of other formats may be conveyed over this connection as well. A similar pressure pad 2809 may alternately be bodily integrated with a client device, such as mobile device 2805. A headset 2812 may be attached to an audio port or other wired or wireless I/O interface of a client device, providing an exemplary arrangement for a user to listen to looped playback of a recorded track, along with other audible outputs of the system. Microphone 2811 may be attached to a client device 2801-2805 via an audio input port or other connection as well. Alternately, or in addition to headset 2812 and microphone 2811, one or more other speakers and/or microphones may be integrated into one or more of the client devices 2801-2805 or other peripheral devices 2811-2813. Also, an external device may be connected to pressure pad 2813 and/or client devices 101-105 to provide an external source of sound samples, waveforms, signals, or other musical inputs that can be reproduced by external control. Such an external device may be a MIDI device to which a client device 2803 and/or pressure pad 2813 may route MIDI events or other data in order to trigger the playback of audio from external device 2814. However, formats other than MIDI may be employed by such an external device.
Devices that may operate as network device 3000 include various network devices, including, but not limited to personal computers, desktop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, servers, network appliances, and the like. As shown in
The mass memory as described above illustrates another type of computer-readable media, namely computer-readable storage media. Computer-readable storage media may include volatile, nonvolatile, removable, and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer readable storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computing device.
As shown, data stores 3052 may include a database, text, spreadsheet, folder, file, or the like, that may be configured to maintain and store user account identifiers, email addresses, IM addresses, and/or other network addresses; group identifier information; tracks or multi-track recordings associated with each user account; rules for sharing tracks and/or recordings; billing information; or the like. In one embodiment, at least some of data store 3052 might also be stored on another component of network device 3000, including, but not limited to cd-rom/dvd-rom 3026, hard disk drive 3028, or the like.
The mass memory also stores program code and data. One or more applications 3050 are loaded into mass memory and run on operating system 3020. Examples of application programs may include transcoders, schedulers, calendars, database programs, word processing programs, HTTP programs, customizable user interface programs, IPSec applications, encryption programs, security programs, SMS message servers, IM message servers, email servers, account managers, and so forth. Web server 3057 and Music Service 3056 may also be included as application programs within applications 3050.
Web server 3057 represent any of a variety of services that are configured to provide content, including messages, over a network to another computing device. Thus, web server 3057 includes for example, a web server, a File Transfer Protocol (FTP) server, a database server, a content server, or the like. Web server 3057 may provide the content including messages over the network using any of a variety of formats, including, but not limited to WAP, HDML, WML, SMGL, HTML, XML, cHTML, xHTML, or the like. In one embodiment, web server 3057 may be configured to enable a user to access and manage user accounts and shared tracks and multi-track recordings.
Music Service 3056 may provide various functions related to enabling an online music community and may further include a Music Matcher 3054 a Rights Manager 3058, and melody data. The Music Matcher 3054 may match similar tracks and multi-track recordings, including those stored in data stores 3052. In one embodiment, such matching may be requested by Sound Searcher or MTAC on a client device which may, for example, provide an audible input, track, or multi-track to be matched. Rights Manager 3058 enables a user associated with an account to upload tracks and multi-track recordings. Such tracks and multi-track recordings may be stored in one or more data stores 3052. The Rights Manager 3058 may further enable a user to provide controls for the distribution of provided tracks and multi-track recordings, such as restrictions based on a relationship or membership in the online music community, a payment, or an intended use of a track or multi-track recording. Using Rights Manager 3058, a user may also restrict all access rights to a stored track or multi-track recording, thereby enabling an unfinished recording or other work-in-progress to be stored without community review before the user believes it to be ready.
Music Service 3056 may also host or otherwise enable single or multiplayer games to be played by and among various members of the online music community. For example, a multi-user role playing game hosted by Music Service 3056 may be set in the music recording industry. Users may select a role for their character that is typical of the industry. A game user may then progress their character through the creation of music using their client device 50 and, for example, RSLL 142, and MTAC 144.
Messaging server 3056 may include virtually any computing component or components configured and arranged to forward messages from message user agents, and/or other message servers, or to deliver messages. Thus, messaging server 3056 may include a message transfer manager to communicate a message employing any of a variety of messaging protocols, including, but not limited, to SMS messages, IM, MMS, IRC, RSS feeds, mIRC, any of a variety of text messaging protocols, or any of a variety of other message types. In one embodiment, messaging server 3056 may enable users to initiate and/or otherwise conduct chat sessions, VoIP sessions, text messaging sessions, or the like.
It is noted that while network device 3000 is illustrated as a single network device, the invention is not so limited. For example, in another embodiment, a music service, or the like, of network device 3000 may reside in one network device, while an associated data store might reside in another network device. In still another embodiment, various music and/or message forwarding components might reside in one or more client devices, operate in a peer-to-peer configuration, or the like.
Gaming Environment
To further facilitate the creation and composition of music,
Turning first to
To begin creating music, the player may select one of these rooms. In one embodiment, the user may simply select the room directly using a mouse or other input device. Alternatively, one or more buttons may be provided that correspond to the various studio rooms. For example, in the embodiment illustrated in
Other selectable buttons may also be provided, as shown in
In the example illustrated in
As shown in
The lead vocal/instrument room interface 3102 may further include a track selector 3216. The track selector 3216 enables a user to record or create multiple lead takes and the select one or more of those takes to be included within the musical compilation. For example, in
A track selection window 3218 is provided to enable the player to select one or more of the tracks to be included in the musical compilation by, for example, selecting and dragging one or more of the three track windows to the selection window 3218. In one embodiment, the selection window 3218 may also be used to engage MTAC module 144 in order to generate a single best take from multiple takes “1” “2” and “3.”
The lead vocal/instrument room interface 3102 may also include a plurality of buttons to enable one or more functions associated with the creation of a lead vocal or instrument track. For instance, a minimize button 3222 may be provided to permit a user to minimize grid 3202; sound button 3224 may be provided to enable a user to mute or unmute the sound associated with one or more audio tracks, a solo button 3226 may be provided to mute any accompaniment audio that has been generated by the system 100 based on the audio waveform 3210 or its morphology so as to allow the player to concentrate on issues associated with the lead audio, a new track button 3228 may be provided to enable the user to begin recording a new lead track; morphology button 3230 activates the operations of frequency detector and shifter 208 and 210 on the audio waveform in the control space 3202. A set of buttons may also be provided to enable a user to set a reference tone to aid in providing a vocal track. Thus, toggling tone button 3232 may enable and disable a reference tone, tone up button 3234 may increase the frequency of the reference tone, and tone down button 3236 may decrease the pitch of the reference tone.
A percussion segment selector 3308 may also be provided in order to enable a player to create and select multiple percussion segments. In the example illustrated in
As further illustrated in
The percussion room interface 3104 may also include a plurality of buttons to enable one or more functions associated with the creation of one or more percussion tracks. For instance, minimize button 3312 may be provided to enable a user to minimize grid 3302, a sound button 3314 may be provided to enable a user to mute or unmute the sound associated with one or more audio tracks, solo button 3316 may be provided to enable a user to toggle between mute and unmute to stop playback of the other audio tracks so the player can focus on the percussion track without distraction, additional percussion instrument button 3318 adds an additional sub-track corresponding to a percussion instrument that may be selected by the player, and swing button 3320 permits a user to swing (i.e. syncopate) notes.
For illustration purposes, let us assume that the player has selected the bass class 3404 in
After selecting a musician avatar in
After the player has selected a musician avatar, and instrument, system 100 creates an appropriate accompaniment track by generating a set of accompaniment notes based on the one or more lead tracks currently being played in the lead vocal/instrument room 3102 (even if the other rooms are muted), converting those notes to the appropriate genre, timbre, and musical style for the selected musician and instrument utilizing genre matcher module 152 and harmonizer module 146 to harmonize the one or more lead tracks. Thus, an accompaniment track for a specific instrument may have different sound, timing, harmony, blue note content, and the like depending on instrument and musician avatar chosen by the player.
The accompaniment room interface 3106 is also configured to enable the player to individually audition each of the multiple musician avatars and/or multiple instruments to aid in the selection of a preferred accompaniment track. As such, once a musical instrument and avatar have been selected by the user and the corresponding accompaniment track has been created as described above, the accompaniment track is automatically played in conjunction with other previously created tracks (lead, percussion, or accompaniment) during a live loop playback so that the player can, in virtually real time, assess whether the new accompaniment track is a good fit. The player may then choose to keep the accompanying track, select a different musician avatar for the same instrument, select a different instrument for the same musician avatar, pick an entirely new avatar and instrument, or delete the accompaniment track altogether. The player may also create multiple accompaniment tracks by repeating the steps described above.
As shown in
Using the graphical tools and the game based dynamic illustrated in
Render Cache
In one embodiment, the present invention may be implemented in the cloud where, in which the systems and methods described above are utilized within a client-server paradigm. By offloading certain functions onto a server, the processing power required by the client device is decreased. This increases both the number and type of devices on which the present invention can be deployed, this allowing interaction with a mass audience. Of course, the extent to which functions are performed by the server as opposed to the client may vary. For instance, in one embodiment the server may be utilized to store and serve relevant audio samples, while processing is performed in the client device. In an alternate embodiment, the server may both store the relevant audio samples and perform certain processing before serving the audio to the client.
In one embodiment, client-side operations may also be performed via a stand-alone application that operates on the client device and is configured to communicate with the server. Alternatively, the user may be capable of accessing the system and initiating communications with the server via an http browser (such as Internet Explorer, Netscape, Chrome, Firefox, Safari, Opera, etc.). In some instances, this may require a browser plug-in to be installed.
In accordance with the present invention, certain aspects of the systems and methods may be performed and/or enhanced through the use of an audio render cache. More specifically, as will be described in more detail below, the render cache enables improved identification, processing, and retrieval of the audio segments associated with requested or identified notes. As will be understood from the description below, the audio render cache has particular utility when the systems and methods described above are utilized with a client-server paradigm as described above. In particular, in such a paradigm the audio render cache would preferably be stored on the client side to improve latency and reduce server costs, although as described below, the render cache may also be stored remotely.
Preferably, the render cache is organized as an n-dimensional array, where n represents a number of attributes that are associated with, and used to organize, the audio within the render cache. One exemplary embodiment of a render cache 3800 in accordance with the present invention is illustrated in
The instrument type may represent the corresponding MIDI channel, the pitch may represent an integer index of the respective semitone, the velocity may represent the strength at which the note is played, and the duration may represent the duration of the note in milliseconds. Entries 3802 in the render cache 3800 may be stored within the array structure based on these four attributes, and may each include a pointer to the allocated memory containing the cached rendered audio samples. Each cache entry may also include an indicator identifying a time associated with the entry, such as the time the entry was first written, the time it was last accessed, and/or a time that the entry expires. This permits entries that are not accessed following the passage of a certain period of time to be removed from the cache. The render cache is also preferably maintained to a finite duration resolution, for example a 16th note, and is fixed in size in order to permit rapid indexing.
Of course, other structures may also be used. For example, the render cache may be maintained at a different finite resolution, or may not be fixed in size if rapid indexing is not necessary. Audio may also be identified using more or less than four attributes, thus requiring an array having more or less axes. For instance, rather than a 4-dimensional array, the entries in
It should also be understood that while an array is described as being the preferred embodiment for the render cache, other memory conventions may also be used. For example, in one embodiment, each audio entry in the render cache may be expressed as a hash value that is generated based on the associated attribute values. One exemplary system that may be employed to facilitate a caching system using this approach is Memcached. By expressing the audio in this way, the number of associated attributes can be increased or decreased without requiring significant changes to the associated code for cache entry look-up and identification.
In one exemplary embodiment, the notes may be specified as a MIDI ‘note on’ with a given duration, while the audio is returned as a Pulse-code modulated (PCM) encoded audio sample. However, it should be understood that the notes may be expressed using any attribute or attributes, and in any notation, including MIDI, XML, or the like. The retrieved audio sample may also be compressed or uncompressed.
As shown in
In step 4002, a requested note is received from the client 3902. In step 4004, it is determined whether the render cache 3800 contains an entry corresponding to the specific requested note. This may be accomplished by identifying the instrument that the requested note is to be associated with (i.e. a guitar, piano, saxophone, violin, etc.), as well as the duration, pitch, and velocity of the note, and then determining whether there is a cached entry that precisely matches each of these parameters. If there is, that audio is retrieved from the cache in step 4006 and provided to the client. If there is no exact match, the process proceeds to step 4008.
In step 4008, it is determined whether there is sufficient time to render a new audio sample for the requested note. For example, in one embodiment, the client may be configured to identify a specific time by which the audio for a note is to be provided. The time by which the audio is to be provided may be a preset amount of time after a request was made. In an embodiment in which live looping is employed, as described above, the time by which the audio is to be provided may also be based on the time (or number of bars) until the end of the loop and/or until the note is to be played back during the following loop.
In order to assess whether audio can be provided within the time limit, an estimate of the amount of time to render and send the note is identified and compared to the specific time limit. This estimate may be based on numerous factors, including a predetermined estimate of the processing time required to generate the audio, the length of any backlogs or processing queues present at the time of the request, and/or the bandwidth connection speed between the client device and the device serving the audio. To effectuate this step, it may also be preferable that the system clocks of the client and the device on which the cache control 3904 is operating be synchronized. If it is determined that there is sufficient time to render the note then, in step 4016 the note is sent to Render Note process 3906, where audio for the requested note is rendered. Once rendered, the audio may also be stored in the cache 3800 in step 4018.
However, if it is determined that there is not sufficient time to render the note, then the process proceeds to step 4010. In step 4010, it is determined whether a “near hit” entry is available. For purposes of this description, a “near hit” is any note that is sufficiently similar to the requested note that it can be reconstructed, using one or more processing techniques, into an audio sample that is substantially similar to the audio sample that would be rendered for the requested note. A “near hit” may be determined by comparing the instrument type, pitch, velocity, and/or duration of the requested note to those of already cached notes. Since different instruments behave differently, it should be understood that the range of entries that may be considered a “near hit” will differ for each instrument.
In one preferred embodiment, a first search for a “near hit” entry may look for a near cache entry along the “duration” axis of the render cache, (i.e. an entry with the same instrument type, pitch, and velocity). Even more preferably, the search would be for an entry having a longer duration (within a range determined to be acceptable for the given instrument) than the requested note, as shortening a note often produces a better result than lengthening a note. Alternatively, or if there is no acceptable entry along the duration axis, a second search may look for a near cache entry along the “pitch axis”, i.e. an entry within a certain range of semitones.
In yet another alternative, or if there are no acceptable entries on either the duration of pitch axes, a third search may look for a near cache entry within a range along the velocity axis. The acceptable range in differing velocities may, in some cases, depend on the specific software and algorithms used to perform the audio reconstruction. Most audio samplers user several samples mapped to different velocity ranges for one note, since most real instruments have significant timbral differences in the produced sound depending on how strongly the note is place. Thus, preferably a “near hit” along the velocity axis would be an audio sample that differs from the requested note only in amplitude.
In yet another alternative, or if there are no acceptable entries on the duration, pitch, or velocity axes a fourth search may look for a near cache entry within a range along the instrument axis. Of course, it is understood that this strategy may be limited to only certain types of instruments that produce similar sounds to other instruments.
It should also be understood that while it is preferable to identify a “near hit” entry that differs in only a single attribute (in order to limit the amount of processing required to reconstruct the audio sample), a “near hit” entry may also be an entry that differs in two or more of the duration, pitch, velocity, and/or instrument attributes. Additionally, if multiple “near hit” entries are available, the audio sample to be used may be selected based on any one or more of a number of factors including for example, the distance from the desired note in the array (for instance, by determining the shortest Euclidian distance in the ‘n’ dimensional space), the closest attribute-based hash value, a weighting of the priority of each axis in the array (e.g., audio differing in audio is preferred over audio differing in velocity, which is preferred over audio differing in pitch, which is preferred over audio differing in instrument and/or the speed in processing the audio sample.
In another embodiment, near hits may be identified using a composite index approach. In this embodiment, each dimension in the cache is folded. In one approach, this may be accomplished by folding a certain number of bits of each dimension. For example, if the lowest two bits of the pitch dimension are folded, all of the pitches can be mapped to one of 32 values. Similarly, the bottom 3 bits of the duration dimension can be folded. As a result, all duration can be mapped onto one of 16 values. Other dimensions can be similarly processed. In another approach, a non-linear folding method can be utilized where the instrument dimension is assigned a similar sounding instrument with the same folded dimension value. The folded dimension values may then be concatenated into a composite index, and the cached entries may be stored in a table that is sorted by the composite index. When a note is requested, relevant cached entries may be identified through a look-up based on the composite index. In this case, all results matching the composite index may be identified as ‘near hit’ entries.
If, in step 4010, a “near hit” entry is determined to be available, the process proceeds to step 4012 where the “near hit” entry is reconstructed (by Reconstruct Note process 3908) to generate an audio sample that substantially corresponds to the requested note. As shown in
The reconstruction techniques described below may also be performed at any device in the system. For instance, in one embodiment, the reconstruction techniques may be applied at a cache server or by a remote device coupled to the cache server, where the reconstructed note is then provided to the client device. However, in another embodiment, the cached note itself may be transmitted to client device, and the reconstruction may then be performed at the client. In this case, information identifying the note and/or instructions for performing the reconstruction may also be transmitted to the client along with the cached note.
Turning to the first technique, let us assume for example, that the “near hit” entry differed only in duration with the requested note. If the audio sample for the “near hit” is longer than the one being requested, the audio sample may be reconstructed using a “re-enveloping” technique where a new, shorter, envelope is applied to the audio sample.
If the requested note is longer than the “near hit” entry, the sustain portion of the envelope may be stretched to acquire the desired duration. Since attack and decay are generally considered to be what gives an instrument its sonic character, manipulations of the sustain may stretch the duration without a significant impact on the “color” of the note. This is referred to as “envelope-stretching.” Alternatively, a “looping” technique may be applied. In this technique, instead of stretching the sustain portion of an audio sample, a section of the sustain section may be looped in order to lengthen the duration of the note. However, it should be noted that randomly selecting a portion of the sustain section to loop may result in clock and pops in the audio. In one embodiment, this may be overcome by cross fading from one loop-end to the next loop-start. In order to lessen any effects that may result from processing and the addition of various effects, it is also preferred that the cached entry be a raw sample, and that any additional digital signal processing be performed after reconstruction is completed, for instance on the client device.
If the requested note is of a different pitch than the “near hit” entry, the cached audio sample may be pitch shifted to acquire the appropriate pitch. In one embodiment, this may be performed in the frequency domain using FFT. In another embodiment, pitch shifting may be performed in the time domain using auto-correlation. In a scenario where the requested note is an octave higher or lower, the cached note may also be simply stretched or shortened to acquire the appropriate pitch. This concept is similar to playing a tape recorder faster or slower. That is, if the cached entry is shortened to play twice as fast, the pitch of the recorded material becomes twice as high, or an octave above. If the cached entry is stretched to play twice as slow, the pitch of the recorded material is halved, or an octave below. Preferably, this technique is applied to cached entries that are within approximately two semitones of the requested note, since stretching or shortening an audio sample greater than that amount may cause an audio sample to lose is sonic character.
If the requested note is of a different velocity than the “near hit” entry, the cached entry may be shifted in amplitude to match the new velocity. For example, if the requested note is of a greater velocity, the amplitude of the cached entry may be increased by the corresponding difference in velocity. If the requested note is of a lower velocity, the amplitude of the cached entry may be decreased by the corresponding difference in velocity.
The requested note may also be of a different, but similar instrument. For instance, the requested note may be for a specific note played on a heavy metal guitar, while the cache may only include a note for a raw metal guitar. In this case, one or more DSP effects could be applied to the cached note in order to approximate a note from the heavy metal guitar.
After the “near hit entry has been reconstructed using one or more of the techniques described above, it may be sent back to the client. An indication may also be provided to the user to inform the user that a reconstructed note has been provided. For instance, in an interface such as the one shown in
If, in step 4010, a “near hit” cache entry was not present, the closest available audio sample (as determined based on the instrument, pitch, duration, and velocity attributes) may be retrieved. In one embodiment, this audio sample may be retrieved from the cache 3800. Alternatively, the client device may also be configured to store in a local memory a series of general notes to be used in circumstances when neither a rendered note nor a reconstructed “near hit” note is available. Additional processing, such as that described above, may also be performed on this audio sample. A user interface on the client may also be configured to provide a visual indication to the user that an audio sample has been provided that is neither rendered audio, nor a reconstructed “near-hit.”
In step 4016, a request is made to render note process 3906 to render the audio for the requested note using sample library 3910. Once the note is rendered, the audio is returned to cache control 3904, which provides the rendered audio to the client 3902, and writes the rendered audio to the render cache 3800 in step 4018.
In the example embodiment shown in
After receiving the response from the server 4102, the client 4108 sends a request (illustrated as signal 4310) to the edge cache 4202 to identify the appropriate cache entry (whether for the specific associated audio, a “near hit,” etc.) based on the information in the manifest. Again, this request may be formatted using http, although other languages or formats may also be used. In one embodiment, the client 4108 performs the determination of the appropriate cache entry, although the determination may also be performed remotely at the edge cache 4202. Signal 4310 represents the response from the edge cache server to the client 4108 that includes the identified cache entry. However, if the request identified a cache entry that is beyond its TTL, or is otherwise unavailable, the response will include an indication that the request has failed. This may cause the client 4108 to retry its request to the server 4102. If the response 4310 did contain the requested audio entry, it may then be decrypted and/or decompressed, as needed, by the client 4108. If the cached entry was a “near hit,” it may also be reconstructed using the processes described above or their equivalents.
Turning first to
In block 4506, the audio request is transmitted from the client to the relevant server or cache. The server or cache receives the audio request in block 4508 and determines that the received audio request includes a failure request in block 4510 is received by the server or cache. In block 4512, the receiving server or cache determines whether the requested audio can be provided to the client by the deadline time. This is preferably determined based on projected or previously determined times for identifying and obtaining cached audio, rendering the note, and/or transmitting the note back to the client. The time required to transmit the note back to the client may also be based on a latency time identified between the transmission time of the audio request and the time it was received.
If it is determined that the audio can be provided before the deadline time, the audio is place in a queue in block 4514, and the method for identifying, locating and/or rendering the audio proceeds as described above. If it is determined that the audio cannot be provided before the deadline time, a message is sent back to the client in block 4516 notifying the client that the audio will not be available by the deadline time. In one embodiment, the notification may be transmitted as an http 412 error message, although any other format can also be used. The client can then take any necessary actions in block 4518 to obtain and provide substitute audio. This may be accomplished by the client identifying an audio that is similar to the one required for the requested note from a local cache, and/or applying processing to previously stored or cached audio to approximate the requested note.
In block 4520, the server/cache checks whether the failure request had identified whether to abort or continue in the event audio could not be provided by the deadline time. If the failure request was set to abort, the audio request is discarded in block 4522 and no further actions are taken. If the failure request was set to continue, the audio request is placed into the queue for processing in block 4514. In this case, the audio can then be provided to the client once completed and used to replace the substitute audio that had been obtained by the client.
Returning to
The audio request is received in block 4608 and a time to service is determined is block 4610. For example in one embodiment, if the audio request only includes information regarding the duration of the loop, the time to service may be “calculated” by merely dividing the loop duration in half. This provides a statistical approximation of the length of time that is likely required before playback of the live loop at the client will reach the location of the note for which the audio requested.
In another embodiment, if the note start time and play head time information is included in the audio request, the time to service can be calculated with more precision. For example, in this case, it may be first determined whether the note start time is greater than the play head time (i.e. the note was at a later position in the loop than the playback bar at the time the audio request was made). If the note start time is greater, the time to service may be calculated as follows: time_to_service=note_start_time−play_head_time. If the play head time is greater than the note start time (i.e. the note was at an earlier position in the loop than the playback bar at the time the audio request was made), the time to service may be calculated as follows: time_to_service=(loop_length−play_head_time)+note_start_time. In another embodiment, the calculation of the time to service may also include the addition of the projected latency time required for transmission of the audio data back to the client. The latency time may be determined by identifying the time stamp of when the audio request was sent, and calculating an elapsed time identified between the time stamp and the time the audio request was received by the server or cache.
After the time to service value is determined, the audio request is placed in a queue based on its time to service. As a result, audio requests with a shorter time to service are processed before those with a longer time to service, thus increasing the likelihood that audio requests will be processed before the next playback of the associated note in the live loop.
As shown in
The new audio request is then positioned into the respective one of the plurality of the track queues 4722. This may be accomplished in one of several ways. Preferably, if an previous audio request with the same note ID had been located and dropped, the new audio request may replace the dropped request in the track queue 4720. Alternatively, in another embodiment, the new audio request may be position in the track queue based on the start time of the audio request. More specifically, notes with an earlier start time are placed earlier in the queue than notes with a later start time.
As a result of the method described in
Effects Chain Processing
Turning first to
After the distortion effects and/or kit selection is applied, the track is preferably transmitted to equalizer module 5006, which applies a set of equalizer settings to the track. The track is then preferably transmitted to compression module 5008 where a set of compression effects are applied. The equalizer and compression settings to be applied are preferably preconfigured for each musician avatar, although they may also be set or adjusted manually. By applying the above effects, the music track can be processed in order to be representatives of the style, sound; and musical tendencies of the virtual musician and instrument selected by the user.
Once the musician role effects have been applied, a series of producer role effects are applied, as illustrated in
As shown in
As shown in
In one embodiment, each virtual musician and producer may also be assigned a “influence” value indicative of their ability to influence a musical composition. These values may then be used to determine the manner in which the above described effects are applied. For instance, the stronger a musician's or producer's “influence” value, the greater the impact their settings can have on the music. A similar scenario may then also be applied for the producer role effects.” For effects that are applied both in the musician and producer roles, such as equalizer and compression settings, the “influence” value may also be used to determine how to reconcile differences between the effects settings. For example, in one embodiment a weighted average of the effect settings may be applied based on differences in the “influence” values. As an example, let us assume the “influence” value can be a number from 1 to 10. If a selected musician has an “influence” value of 10 is working with a producer with an “influence” value of 1, all of the effects associated with that selected musician may be applied in their entirety. If the selected musician has an “influence” value of 5 and is working with a producer with an “influence value of 5, the effect of any applied musician settings may be combined with the settings of the producer in a manner that may be random, but would preferably be pre-determined. If the selected musician has a “influence” value of 1, only a very minimal effect may be applied. If the selected musician has a “influence” value of 1, only a very minimal effect may be applied. In another embodiment, the effect settings associated may be chosen based solely on which of the virtual musician and producer has a greater “influence” value.
The effects described in
The foregoing description and drawings merely explain and illustrate the invention and the invention is not limited thereto. While the specification is described in relation to certain implementation or embodiments, many details are set forth for the purpose of illustration. Thus, the foregoing merely illustrates the principles of the invention. For example, the invention may have other specific forms without departing from its spirit or essential characteristic. The described arrangements are illustrative and not restrictive. To those skilled in the art, the invention is susceptible to additional implementations or embodiments and certain of these details described in this application may be varied considerably without departing from the basic principles of the invention. It will thus be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and, thus, within its scope and spirit.
This application is a continuation-in-part of U.S. patent application Ser. No. 12/791,792, filed Jun. 1, 2010; a continuation-in-part of U.S. patent application Ser. No. 12/791,798, filed Jun. 1, 2010; a continuation-in-part of U.S. patent application Ser. No. 12/791,803, filed Jun. 1, 2010; and a continuation-in-part of U.S. patent application Ser. No. 12/791,807, filed Jun. 1, 2010. Each of U.S. patent application Ser. Nos. 12/791,792; 12/791,798; 12/791,803 and 12/791,807 claim priority to U.S. Provisional Patent Application No. 61/182,982, filed Jun. 1, 2009, U.S. Provisional Patent Application No. 61/248,238, filed Oct. 2, 2009; and U.S. Provisional Patent Application No. 61/266,472, filed Dec. 3, 2009.
Number | Name | Date | Kind |
---|---|---|---|
4508002 | Hall et al. | Apr 1985 | A |
5350880 | Sato | Sep 1994 | A |
5663517 | Oppenheim | Sep 1997 | A |
5955691 | Suzuki et al. | Sep 1999 | A |
6172679 | Lim | Jan 2001 | B1 |
6495747 | Shimaya et al. | Dec 2002 | B2 |
6504090 | Tsai et al. | Jan 2003 | B2 |
6972363 | Georges et al. | Dec 2005 | B2 |
7026539 | Pennock et al. | Apr 2006 | B2 |
7179981 | Gayama | Feb 2007 | B2 |
7259316 | Kusumoto | Aug 2007 | B2 |
RE40543 | Aoki et al. | Oct 2008 | E |
7516232 | Barrett et al. | Apr 2009 | B2 |
7667123 | Phillips et al. | Feb 2010 | B2 |
7667126 | Shi | Feb 2010 | B2 |
7705231 | Morris et al. | Apr 2010 | B2 |
7772480 | Brennan | Aug 2010 | B2 |
7812241 | Ellis | Oct 2010 | B2 |
7849420 | Amidon et al. | Dec 2010 | B1 |
7902446 | Humphrey | Mar 2011 | B2 |
7985917 | Morris et al. | Jul 2011 | B2 |
8026436 | Hufford | Sep 2011 | B2 |
8338686 | Mann et al. | Dec 2012 | B2 |
20030014135 | Moulios | Jan 2003 | A1 |
20030110923 | Miyamoto | Jun 2003 | A1 |
20030128825 | Loudermilk | Jul 2003 | A1 |
20050144284 | Ludwig et al. | Jun 2005 | A1 |
20050256595 | Aiso et al. | Nov 2005 | A1 |
20060074649 | Patchet et al. | Apr 2006 | A1 |
20060075884 | Streitenberger et al. | Apr 2006 | A1 |
20060084218 | Lee et al. | Apr 2006 | A1 |
20060085182 | Eves et al. | Apr 2006 | A1 |
20060230909 | Song et al. | Oct 2006 | A1 |
20060230910 | Song et al. | Oct 2006 | A1 |
20070061729 | Terada | Mar 2007 | A1 |
20070245090 | King et al. | Oct 2007 | A1 |
20070289432 | Basu et al. | Dec 2007 | A1 |
20080013757 | Carrier | Jan 2008 | A1 |
20080229917 | Devalapalli et al. | Sep 2008 | A1 |
20080250315 | Eronen et al. | Oct 2008 | A1 |
20080289478 | Vella | Nov 2008 | A1 |
20080314228 | Dreyfuss et al. | Dec 2008 | A1 |
20090038467 | Brennan | Feb 2009 | A1 |
20090038468 | Brennan | Feb 2009 | A1 |
20090052471 | Pasqualino et al. | Feb 2009 | A1 |
20090064851 | Morris et al. | Mar 2009 | A1 |
20090067641 | Lengeling et al. | Mar 2009 | A1 |
20090132075 | Barry | May 2009 | A1 |
20090141910 | Mercs et al. | Jun 2009 | A1 |
20090217805 | Lee et al. | Sep 2009 | A1 |
20090304207 | Cooper | Dec 2009 | A1 |
20100018382 | Feeney et al. | Jan 2010 | A1 |
20100095829 | Edwards | Apr 2010 | A1 |
20100192755 | Morris et al. | Aug 2010 | A1 |
20100198760 | Maddage et al. | Aug 2010 | A1 |
20100300268 | Applewhite et al. | Dec 2010 | A1 |
20100300269 | Applewhite | Dec 2010 | A1 |
20100307320 | Hoeberechts et al. | Dec 2010 | A1 |
20100309153 | Terada | Dec 2010 | A1 |
20100313736 | Lenz | Dec 2010 | A1 |
20110009989 | Terada et al. | Jan 2011 | A1 |
20110009990 | Terada et al. | Jan 2011 | A1 |
20110011245 | Adam et al. | Jan 2011 | A1 |
20110013786 | Odom et al. | Jan 2011 | A1 |
20110058691 | Okabayashi | Mar 2011 | A1 |
20110120288 | Bignell et al. | May 2011 | A1 |
20110175915 | Gehring | Jul 2011 | A1 |
20110179940 | Humphrey | Jul 2011 | A1 |
20110179941 | Humphrey | Jul 2011 | A1 |
20110179942 | Humphrey | Jul 2011 | A1 |
20110228956 | Aiso et al. | Sep 2011 | A1 |
20110232461 | Taub et al. | Sep 2011 | A1 |
20110251842 | Cook et al. | Oct 2011 | A1 |
20110252949 | Lengeling | Oct 2011 | A1 |
20120020498 | Fujita et al. | Jan 2012 | A1 |
20120023406 | Fujita et al. | Jan 2012 | A1 |
20120027230 | Okabayashi | Feb 2012 | A1 |
20120059491 | Carroll | Mar 2012 | A1 |
20120132057 | Kristensen | May 2012 | A1 |
20120180618 | Rutledge et al. | Jul 2012 | A1 |
20120237038 | Hagiwara et al. | Sep 2012 | A1 |
20130123960 | Green | May 2013 | A1 |
20130233157 | Lengeling | Sep 2013 | A1 |
Entry |
---|
Sibelius S Reference [online]. Sibelius Software 2007 [Retrieved online Jul. 15, 2010]. Retrieved from internet: <URL: http://hub.sibelius.com/download/documentation/pdfs/sibeliuss-reference-en.pdf>. |
International Search Report and Written Opinion from PCT/US2010/036951, filed Jun. 1, 2010. |
International Search Report and Written Opinion dated Dec. 6, 2012 for PCT/US2012/048880. |
lnternatienal Search Report and Mitten Opinion dated Dec. 6, 2012 for PCT/US2012/048883. |
Larry Polansky, “Morpholological Metrics”, Journal of New Music Research, vol. 25, pp. 289-368, 1996. |
Logic Pro 8 User Manual,2007 Viewed online at hppt://manuals.info.apple.com/en—US/Logic—Pro—8—User—Manual.pdf. |
Pro Tools 7.4: Reference Guide, 2007, Digidesign Pages (5-6, 26-29, 72, 89, 96-99, 112, 114, 117-118,189, 218, 228, 317, 321-324, 343-344, 350-351, 359, 368-369, 372, 377, 403, 405, 437-438, 468, 534, 536, 557-558, 576-577, 683) http://akmedia.digidesign.com/support/docs/Pro—Tools—74—Reference—Guide—41784.PDF. |
RC-50 Loop Station: Owner's Manual, 2005, BOSS Pages (42 58-59) http://www.bossus.com/backstage/product—manuals/?group=2. |
WIPO; International Preliminary Report on Patentability for corresponding PCT/US2012/048880 issued Feb. 4, 2014, 6 pages. |
WIPO; International Preliminary Report on Patentability for corresponding PCT/US2012/048883 issued Feb. 4, 2014, 11 pages. |
Number | Date | Country | |
---|---|---|---|
20130025437 A1 | Jan 2013 | US |
Number | Date | Country | |
---|---|---|---|
61182982 | Jun 2009 | US | |
61248238 | Oct 2009 | US | |
61266472 | Dec 2009 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 12791792 | Jun 2010 | US |
Child | 13194819 | US | |
Parent | 12791798 | Jun 2010 | US |
Child | 12791792 | US | |
Parent | 12791803 | Jun 2010 | US |
Child | 12791798 | US | |
Parent | 12791807 | Jun 2010 | US |
Child | 12791803 | US |