Patent Application
20160358594
Singers, and more generally musicians of all types, often wish to modify the natural sound of a voice and/or instrument, in order to create a different resulting sound. Many such musical modification effects are known, such as reverberation (“reverb”), delay, pitch correction, scale correction, voice doubling, tone shifting, and harmony generation, among others. Complex technology has been developed to process live accompaniment music to analyze and change musical parameters in order to accomplish effects such as pitch and scale correction, tone shifting and harmony generation in real time.
Harmony generation involves generating musically correct harmony notes to complement one or more notes produced by a singer and/or accompaniment instruments. Examples of harmony generation techniques are described, for example, in U.S. Pat. No. 7,667,126 to Shi and U.S. Pat. No. 8,168,877 to Rutledge et al., each of which are hereby incorporated by reference. The techniques disclosed in these references generally involve transmitting amplified musical signals, including both a melody signal and an accompaniment signal, to a signal processor through signal jacks, analyzing the signals immediately to determine musically correct harmony notes, and then producing the harmony notes and combining them with the original musical signals.
Preexisting live pitch and harmony generation techniques have accuracy limitations for at least two reasons. First, different types of musical input or accompaniment are processed using the same methodology and without distinction. More specifically, because these products and algorithms were primarily designed to be applied with a live music input created by a reasonably experienced musician, they have inherent limitations when applied to pre-recorded accompaniment music and/or when used by an inexperienced musician such as an amateur karaoke singer.
The main goal of known techniques is to achieve near zero latency of the musical accompaniment, pitch correction and harmony generation. This harmony generation and pitch correction controlled by live instrument playing can be musically unstructured, for example, during a practice or creative writing session. Accordingly, existing techniques receive the musical input (live guitar or a prerecorded song) and attempt to analyze the music spectrum of the live guitar for lead note, chord, scale and key data for applying proper vocal harmony and pitch correction notes in real time, then immediately outputting the music accompaniment input source so it can be heard by the performer. This rapid analysis and response is necessary when applying harmony generation to live music, because adding any significant audio latency or delay to a live guitar accompaniment would make playing that guitar and performing very difficult or impossible. In some live techniques, a past lead note or spectral history can be stored and used to attempt to provide more accurate harmony. In any case, the real time or near real time analysis of live accompaniment music can result in undesirable errors when applied to pre-recorded music.
In addition, preexisting vocal processing systems typically receive relatively sonically “clean” harmonic information from a single instrument source, such as a guitar input. Because of the live performance requirement and clean accompaniment signal these algorithms provide immediate and generally unfiltered response to the input. This includes generating harmonies for any multiple quick interval key changes played by the musician. During live performance, practicing, and playing this spectral input can be intentionally musically unusual or unstructured. These vocal processing system algorithms rely on the accurate harmonic information from the musician's guitar or instrument input and generally do not interpret the musical intent of input source accompaniment and performer (e.g., a guitarist strumming chords). Therefore, if a guitar player sequentially strums five different chords in five different keys while singing with harmony voices and pitch correction turned on, the system will respond to that music input because the algorithm was designed not to significantly interpret the intent of the live performer.
Conversely, switching between five different musical keys in a sequence is not typical in pre-recorded commercial songs and music. Unlike live performance and practicing with a guitar input, the majority of pre-recorded music is highly structured, predicable, usually contains a detectable start and end point of the song, and follows certain general song and musical theory, norms, and principles. Accordingly, rapid or sequential key changes in pre-recorded music are likely to be errors that should be ignored for the purpose of generating harmony voices.
Unlike a guitar or other live single instrument input, a pre-recorded accompaniment track is much more difficult to analyze accurately for a vocal processing algorithm compared to a live accompaniment instrument, because a pre-recorded track typically involves multiple instruments, overlapping melodies, noise from percussion (non-harmonic sounds), sound effects and/or various vocals, and in some cases may be provided from a relatively poor quality recording. Unlike live performance and practice based musical accompaniment, pre-recorded songs typically follow very predictable key and scale patterns. For example, only a small percentage of all recorded music changes from its original starting musical key. Therefore, one identified the pitch correction notes of the identified key and scale will likely remain the same during an entire song.
In one aspect of the invention, vocal processing accompaniment music sources which drive the harmony generation and pitch correction, like a prerecorded musical track (e.g., a karaoke song) do not require the standard method of real-time analysis of the accompaniment music. Pre-recorded accompaniment can be delayed and allow for longer spectral analysis and utilize more song based statistical interpretation of that input data.
Utilizing the fastest potential non-interpretive vocal processing algorithms results in a technical limitation whereby the harmony or pitch correction cannot be synchronized precisely with the changing input chords in live music source. Using the fastest total processing and output speed possible, harmony voices can still be approximately 200 ms out of sync with the most recent identified live track audio chord. Using previously known harmony generation techniques, this gives rise to short periods of time after each chord change during which musically incorrect harmony notes are produced.
Accordingly, there is a need to distinguish the vocal processing techniques of live accompaniment music from pre-recorded accompaniment music. By employing the novel act of delaying output of only pre-recorded accompaniment signals and extending the time to analyze the accompaniment on the device or application, several significant improvements in harmony generation and pitch correction algorithms and techniques are possible and realized. These improvements can be used to avoid the significant shortcomings of the previous requirement to produce harmony notes and pitch correction in real time. In addition, there is significant reduction in errors while processing complex pre-recorded song spectral content for the required vocal processing data to drive the vocal processing system.
To overcome the issues described above, among others, the present teachings disclose improvements to the existing methods and apparatus for vocal processing live harmony and pitch correction effects. Specifically, the present teachings disclose (1) a new method of pre-recorded accompaniment track analysis, (2) delaying the audible output of a pre-recorded track for at least the time required to accurately synchronize harmony and pitch corrected voices to a spectrally detected chord in an associated pre-recorded accompaniment track, (3) utilizing the sync time buffer or delay or longer to reduce or eliminate harmony generation and pitch correction responses to short detected harmonics that are inconsistent with the playing pre-recorded accompaniment track and recorded track structure, statistics and theories, (4) scanning libraries of songs on a device or service and store the scale and key information associated with each song, (5) using advanced data to further inform the user about the detected key and scale information, and (6) providing the user the detected key(s) and scale(s), confirmation and selection of preferences of the detected key and scale information settings detected by the advanced scanning.
I. Distinguishing Live Input vs. Pre-recorded Processing
According to one aspect of the present teachings, two distinct types of musical inputs are identified separately. Live and pre-recorded accompaniment may be processed in a different manner for purposes of generating more accurate harmony notes and pitch correction. Live performance input, such as a live guitar player's guitar input, will continue to require the current standard of low latency and generally non-interpreted spectral processing response for accompaniment data. That data is typically a single instrument musical input source, such as a guitarist playing a live guitar and singing with live harmony and pitch correction from the device.
According to one aspect of the present teachings, accompaniment music received at a signal processor may not be immediately amplified and played through a loudspeaker, but rather amplification may be delayed for at least the time it takes for the spectral content of the received signal to be analyzed and harmony notes and pitch correction to be generated. As a result, harmony notes may be produced which are essentially now fully synchronized with the amplified accompaniment and melody notes, or pitch corrected notes even after a chord change.
In the new approach, pre-recorded accompaniment music is distinguished from live accompaniment as a different species of musical accompaniment input driving the vocal processing algorithm. Pre-recorded song accompaniment can also be spectrally processed differently for lead notes, chords, keys, and the like by analyzing the music before it is played to the performer whereby any musically inconsistent spectral data based on commercial song structure and other factors can be filtered and potentially rejected producing highly accurate and musically correct pitch and harmony generation data before the audio is audibly played to the user. In other words, buffering or delaying the accompaniment audio (e.g., analyzing the future accompaniment signal and comparing it to the dominant spectral data) provides more accurate harmonization and pitch correction for pre-recorded songs than previous minimally interpretive live methods. In the live accompaniment analysis process, the accuracy detection and processing of the musical source key and scale information will be less accurate because the window of time to analyze and produce a result is very narrow to achieve as close to zero latency as possible for live performance.
In some cases, with a sonically complex multi-instrument recording accompaniment, a momentary incorrect lead note, scale, or chord change can occur as the result of the system incorrectly detecting a momentary sonic combination of instruments and track vocals, noise, fidelity and other variables. That could result in the system changing the entire key of pitch correction and harmony voices to an incorrect key. With the proposed advanced song accompaniment processing method, incorrect brief, repeated and/or sudden detection of lead note, scale or key changes which resolve quickly to the previous or dominate key, note and scale data can potentially be filtered and ignored, whereby the current dominant key, scale or lead note, remains uninterrupted, resulting in significantly fewer unwanted harmonically dissonant system generated tones and harmonies.
In a further extension of the present teachings, scanning up to an entire pre-recorded accompaniment track or library of accompaniment tracks on a device and deriving note, key and scale data may be implemented. The extent and duration of this pre-scanning can have any desired time scale to suit a particular application. For example, it can be short in duration, such as 100-200 milliseconds, or it can be one second, three seconds or much longer, including pre-scanning the entire track to produce a data result. Any amount of advanced track scanning or delay techniques provide the most accurate harmony, pitch correction and time synchronization processing relative to the music accompaniment. Pre-scanning, buffering or delaying a playing track a song track to the performer can allow a larger “future” data segment to determine the most accurate spectral information for pre-recorded song accompaniment, including the omission of frequent brief or lengthy harmonic anomalies found during spectral analyses which are statically inconsistent with standard multi-instrument and vocal songs statistics such as rapid key changes or musically dissonant chord data.
As mentioned above, determining the current chord or other spectral data in an accompaniment signal takes a signal processor and harmony generator a finite amount of time, typically around 200 milliseconds. In preexisting harmony generation systems used with live music sources, that processing time is a source of inherent lack of synchronization of the generated harmony notes with the original melody and the accompaniment track. While this problem will always be present with live instrument accompaniment such as a guitar input, the present teachings overcome this problem for pre-recorded accompaniment by playing the track and delaying that musical output.
More specifically, harmony voices create a chord with the original melody voice. When chords in the pre-recorded accompaniment music change, the chords created by the melody and harmony voices ideally should change at the same time, rather than at some later time. However, in current live harmony generation systems, the input accompaniment signal is typically amplified immediately, whereas the harmony notes are determined and amplified later and are asynchronous. Therefore, in existing systems, synthesized harmony notes are generally not always synchronized with the detected chords in the original musical accompaniment signal. This can result in a certain discordant sound in the combined amplified output for a finite time after a chord change in the accompaniment audio.
According to the present teachings, the amplified output accompaniment signal 18, including both the original accompaniment audio and any synthesized harmony notes, may be delayed relative to the input audio signal by a predetermined time, as depicted in
The block diagram of
The singer then sings in conjunction with the delayed loudspeaker output, so that the singer's melody signal 62 will be highly synchronized with the latest accompaniment chord that has already been analyzed. The singer's current melody note may be used in conjunction with the analyzed chord to generate harmony notes and/or pitch-corrected melody notes, collectively indicated at 64, with a digital signal processor 66 virtually immediately, resulting in essentially synchronized amplification of the singer's melody note or pitch corrected note, the accompaniment chord or notes, and processor generated harmony notes generated using the present melody and accompaniment data.
In other words, the presently described system provides a sufficient delay or buffer of the pre-recorded accompaniment song so that the singer's output and the accompaniment output is synchronized. The additional buffer window further provides the accompaniment spectral algorithm significantly more time to accurately interpret and process complex multi-instrument music. Although two separate digital signal processors 54 and 66 are shown in
Method 100 allows for a comparatively longer analysis of spectral (i.e., musical note) information, which can even include future accompaniment spectral data and lead notes. Controlling harmony generation and pitch correction with the standard live method using pre-recorded accompaniment of any playable multi-instrument commercial song produces serious inaccuracies because this music source type is the most spectrally complex to analyze accurately in real time. Brief and quickly alternating spectral and harmonic interpretation errors occur due to the complex harmonics of a given music track or for other reasons. These errors are amplified immediately causing incorrect pitch correction and harmony generation. Unlike live performance and live music structure, these events in a pre-recorded song are highly likely to be incorrect data or noise and need to be buffered and filtered for a period of time while the system, for example, maintains the previous and musically correct consistent data. Therefore, in conjunction with the novel delay feature for harmony synchronization, further new methods of controlling and potentially limiting harmony and pitch correction responsiveness are required to greatly improve accuracy. Live instrument methods are insufficient.
This new method combines commercial song structure statistical data such as the fact that commercial songs generally stay in one key from the detected song start point. When most commercial songs change key, the key is maintained for a significant period of time. Incorrect musical spectral interpretation occurs frequently with pre-recorded songs, when inadvertent notes or other types of “noise” are incorrectly interpreted as a key change. The harmony and pitch algorithm in the new method analyzes the future segment of the audible track to omit these errors, relying on the consistency of pre-recorded music structure. Since a novice user can select any possible pre-recorded song in existence to sing along and be the source to control the harmony and pitch correction, the new method directs the pitch correction and harmony notes response to buffer sudden inconsistent accompaniment data following known commercial music standards.
Furthermore, sonically complex prerecorded accompaniment songs can be spectrally analyzed in a manner whereby musically inconsistent sonic analyses data moments (errors) are expected by the control algorithm, and the pitch correction and or harmony generation can be controlled to ignore spectral inconsistencies, maintain the current and future (music scanned in advance) dominant musical features, and ignore these brief errors.
At step 102, an accompaniment track or library of accompaniment tracks is provided. At step 104, a desired accompaniment track or set of provided accompaniment tracks is scanned and analyzed by a signal processor to determine its spectral information. Because there is no urgency to accomplish this in order to synchronize with live playing of accompaniment instruments, time is provided to confirm accurate spectral information and filter potentially erroneous and musically incorrect spectral data. In the case of a detected and potentially erroneous harmonic data point, both pitch correction and harmony generation can be maintained to the previous data point, or only the pitch or scale correction can be maintained to the previous data point while the harmony generation is allowed to follow the potentially erroneous chord data point, balancing the risk that at least one of the two will be musically correct. Moreover, with the additional time that can be spent on spectral analysis, confirming a song key or chord change can be performed accurately and consistently.
At step 106, melody notes are received, typically produced by a karaoke singer's voice, and harmony notes and pitch corrected notes are generated based on the melody notes in conjunction with the recently analyzed accompaniment music. The system maintains output of current key/scale and chord during the buffer period. Also, if a singer is detected as holding a note for a duration of time determined to be a held or sustained note, the algorithm can maintain at least the initial pitch corrected note steady and in some cases the harmony notes can also be maintained, briefly ignoring other conflicting spectral information.
More specifically, according to the present teachings, the performer's held note data may be interpreted by the effects processing algorithm as strongly intending to hold that distinct note, and possibly also to hold the current harmony combination, temporarily overriding any conflict with the key and chord data. The algorithm can resume processing after the held note is released. Rapidly adjusting or pitch correcting a held or sustained note and potentially an associated harmony drastically to another note in the scale or a different key would confuse the performer who obviously intended to maintain those notes and harmonies. Also during this time, additional techniques may be applied to avoid unpleasant harmony or pitch generation, such as by maintaining the output of the current or dominant scale, key and chord data.
At step 108, an evaluation is performed to determine if the current key and scale of the melody notes should be maintained, or if they should be adjusted, and any adjustment is performed. For example, step 108 may include determining if a current melody note is musically complementary with the current accompaniment note, i.e., falls within the same key. In addition, step 108 may include determining if the key of the current accompaniment note is a reliable indication of the accompaniment key, or if it is an anomaly based on a mistake or inadvertent key change in the accompaniment music. This can be accomplished by evaluating the duration of the accompaniment key and ignoring key changes of sufficiently short duration. Because the accompaniment music may be analyzed in advance, evaluating the duration of the accompaniment key can also be done in advance. It need not be done at the instant a particular melody note is sung and detected.
For example, key changes or detected dissonant chord detection anomalies in the accompaniment music of fewer than three seconds, fewer than two seconds, or under any other desired time threshold may be ignored for purposes of performing corrections to the current melody note and or harmony notes. If however, an accompaniment key change is determined to be an actual, intentional key change in the music, then the melody note can be adjusted into the proper key if necessary. Furthermore, if it is determined that the melody note is already in the proper key but is off-pitch (i.e., sharp or flat), the melody note also may be shifted to correct its sound. Pitch shifting of melody notes may be accomplished, for example, using the well known technique of pitch synchronous overlap and add (PSOLA). A description of this technique is found, for instance, in U.S. Patent Application Publication No. 2008/0255830, which is hereby incorporated by reference for all purposes. Additional pitch shifting methods are disclosed, for example, in U.S. Pat. No. 5,973,252, which is also hereby incorporated by reference for all purposes.
At step 110, the generated harmony notes and the melody, including any pitch correction, is synchronized with the accompaniment track. Finally, at step 112, the accompaniment track, the vocal harmonies, and the originally sung melody notes with possible pitch correction and/or other chosen sound effects, all are output, for instance through an output jack or directly from a speaker integrated with a harmony generating karaoke device.
At step 216, the user selects a single accompaniment track for an immediate performance. At step 218, the track accompaniment begins to play but is not audible to the user. Instead, at step 220, a delay buffer stores the track in memory for at least the time required to synchronize the harmony and pitch correction output with the latest detected chord accompaniment, and perhaps longer. During this time, at step 222, the spectral analysis algorithm of the effects processor attempts to determine the current key, scale and chord in the accompaniment song. Special pre-recorded song based filters and algorithms are enabled for this purpose, which are different from live guitar input algorithms. At step 224, the accompaniment is broadcast audibly to the user, for example through a loudspeaker, and at step 226, the processor receives melody notes sung by the user.
At step 228, the processor detects a key, chord, or lead note change in the accompaniment audio and/or in the melody notes, and evaluates the change to determine whether to accept the change for purposes of harmony generation and/or pitch correction. If the duration of the change is less than a predetermined threshold duration, such as three seconds, two seconds, one second, or any other desired threshold, the algorithm ignores the change and maintains the current or dominant key, chord or lead note data. On the other hand, if a change is detected for a consistent duration past the threshold, the algorithm may accept the change for purposes of harmony generation and pitch correction.
At step 230, the processor generates harmony notes and makes any pitch correction deemed necessary. Since the buffered delay of the audible audio is at least the time to spectrally analyze the accompaniment track and generate the harmony notes and pitch corrected notes, the harmony notes and accompaniment chords are synchronized. When the track accompaniment ends, at step 232 a duration of silence can be detected by the spectral algorithm. At step 234, the processor then can potentially reset or remove any previous spectral history. Upon recognition of a starting track from a period of silence, a new spectral history for that song can begin to be stored, returning to step 210 of the method.
System 300 includes a chord detection circuit 302, which also may be referred to simply as a chord detector, a harmony processing circuit 304, which may be referred to more generally as a note generator, and a delay circuit 306, which also may be referred to as a delay unit. In some cases, chord detection circuit 302, harmony processing circuit 304 and delay circuit 306 all may be portions of a digital signal processor, as indicated at 308. Furthermore, digital signal processor 308 may be integrated into a karaoke machine 310, along with other components such as an amplifier 312, a loudspeaker 314 and/or a microphone 316.
Chord detection circuit 302 is configured to receive and analyze an accompaniment audio signal, and to determine chord information corresponding to a chord of the accompaniment audio signal. In other words, the chord detector is configured to receive an accompaniment audio signal, to analyze the accompaniment audio signal to determine chords contained within the accompaniment audio signal, and to produce chord information corresponding to the chords that have been determined. This process generally takes a particular duration of time, which is typically on the order of hundreds of milliseconds, such as 200 ms.
Harmony processor circuit or note generator 304 is configured to receive and analyze the chord information produced by the chord detector along with melody notes received from a singer, and to produce a synthesized harmony signal corresponding to each detected chord and melody note. The harmony signal will be harmonized to the chord of the accompaniment audio signal and the melody note, and the harmony processing circuit is typically configured to transmit the harmony signal to a loudspeaker to produce harmony audio.
Delay circuit or unit 306 is configured to receive the accompaniment audio signal, and to store the accompaniment audio signal in memory for a predetermined delay time until the chord detector produces the chord information. The delay circuit is further configured to stream the accompaniment audio signal to the loudspeaker after the predetermined delay time has lapsed to produce accompaniment audio. In some cases, the predetermined delay time approximates the duration of time required for the chord detector to extract chord information from the accompaniment audio signal. In other cases, the delay time may be longer, and may allow for additional analysis of the accompaniment audio.
When system 300 or portions thereof are integrated into a karaoke machine such as machine 310, the accompaniment audio signal will typically be pre-recorded, and the melody notes will be received in real time from a karaoke singer using microphone 316. In this case, system 300 will be configured to generate harmony notes as quickly as possible after receiving each melody note, i.e., the system may be configured to produce the harmony signal substantially in real time with receiving and amplifying the melody note. To accomplish this, the harmony processing circuit may be further configured to transmit the melody note to the loudspeaker, along with the harmony notes and the accompaniment signal. According, system 300 may be configured to broadcast the accompaniment audio signal, the melody audio signal and any generated harmony notes through the loudspeaker substantially simultaneously.
Digital signal processor 308 also may be configured to perform other functions. For example, the digital signal processor may be configured to determine a musical key of the accompaniment audio signal and to create a pitch-corrected melody note by shifting the melody note received from the singer into the musical key of the accompaniment audio signal, and to transmit the pitch-corrected melody note to the loudspeaker. In other words, the digital signal processor (or a portion thereof, such as the note generator) may be configured to determine a pitch of the melody note and to generate a pitch-corrected melody note if the pitch of the melody note is musically inconsistent with the chord information. When pitch-shifted melody notes are generated, they may be broadcast through the loudspeaker in place of the corresponding original melody notes, which have presumably been determined to contain a pitch error. In some cases, however, the system may be configured to amplify and audibly produce both the original melody notes and the pitch-shifted notes, for instance as a method of allowing a karaoke singer to hear the correction.
In some cases, the note generator may be configured to generate a pitch-corrected melody note only based on chord information representing chord changes lasting longer than a predetermined threshold duration. That is, the note generator may be configured to ignore short-term chord changes that have a high probability of misrepresenting the overall pattern or intent of the accompaniment music. Similarly, the harmony generator may be configured to ignore such short-term chord changes. Generally speaking, short-term chord changes may be ignored for purposes of generating harmony notes, generating pitch-shifted melody notes, or both.
In addition to possibly ignoring chord changes that occur for less than a predetermined duration, signal processor 308 may be configured to ignore other types of chord information, such as chord information that is determined to represent sounds produced by percussion instruments or by other sources that are unlikely to embody a musician's intent to change chords. As in the case of short-term chord changes, such source specific chord information can be ignored for purposes of generating harmony notes, generating pitch-shifted melody notes, or both.
This application is a continuation of U.S. patent application Ser. No. 14/815,707, filed Jul. 31, 2015, which is a continuation of U.S. patent application Ser. No. 14/467,560, filed Aug. 25, 2014, which is a continuation of U.S. patent application Ser. No. 14/059,355, filed Oct. 21, 2013, which claims priority to U.S. Provisional Patent Application Ser. No. 61/716,427, filed Oct. 19, 2012, all of which are incorporated herein by reference into the present disclosure.
| Number | Date | Country | |
|---|---|---|---|
| 61716427 | Oct 2012 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14815707 | Jul 2015 | US |
| Child | 15237224 | US | |
| Parent | 14467560 | Aug 2014 | US |
| Child | 14815707 | US | |
| Parent | 14059355 | Oct 2013 | US |
| Child | 14467560 | US |
