Patent Application
20210350778
An important part of modern music production is the use of the samples. Samples are usually short audio files that contain some musical information. There are single shot samples that contain a single sound or loops that contain a short musical phrase that is performed typically by a single instrument (drums, guitar, bass, etc.) or sometimes two or more instruments. Loops are also call stems. An audio stem represents one or more audio sources mixed together. In the context of this technology we refer to loops and stems interchangeably.
Musicians and producers make heavy use of loops mainly in electronic music production. Percussive loops or beats form the rhythmic foundation of their tracks while melodic loops (e.g. guitar or piano loops) are used to create musical phrases. The main problem with the use of loops is that they are static, in the sense that they are audio files played back by a computer that are always the same. In contrast, when a musician plays a musical phrase with her instrument, it is dynamic, in the sense that it's never exactly the same. Electronic music producers are aware of this and go to great lengths to manually change the loop over time using advanced features of their Digital Audio Workstation (DAW) applications like automation. This process is very time-consuming and inefficient. Hence, there is a need for methods that produce loop variations automatically without user intervention or with minimal user intervention where she will setup one or a few parameters.
Because of the importance of loops in the modern music production workflow, there are several commercial libraries available. Musicians and producers typically have access to thousands of loops that they can use. There is a need for automatic methods that will automatically produce new variations of the loops inside the user libraries and hence allow users to re-use them for a very long time without being the same.
Aspects of the technology relate to processing an audio stem, including dividing a stem into a plurality of stem slices, classifying each of the plurality of stem slices into at least a first group or a second group and applying a stem effect.
Aspects of the technology relate to applying a stem effect comprising replacing at least one stem slice with an all-zero stem slice.
Aspects of the technology relate to applying a stem effect comprising replacing at least one stem slice belonging to the first group with a stem slice belonging to the second group.
Aspects of the technology relate to applying a stem effect comprising replacing at least one stem slice belonging to the second group with a stem slice belonging to the first group.
Aspects of the technology relate to applying a stem effect comprising replacing at least one stem slice belonging to the first group with a different stem slice belonging to the first group.
Aspects of the technology relate to applying a stem effect comprising replacing at least one stem slice belonging to the second group with a different stem slice belonging to the second group.
Aspects of the technology relate to applying a stem effect comprising replacing at least one stem slice with a time-reversed version of the at least one stem slice.
Aspects of the technology relate to applying a stem effect comprising replacing at least one stem slice by a time-reversed version of a second stem slice, wherein the second stem slice precedes the at least one stem slice.
Aspects of the technology relate to processing an audio stem, including dividing a stem into a plurality of stem slices, classifying each of the plurality of stem slices into at least a first group or a second group and applying a stem effect, wherein the first group is associated with a low energy level and the second group is associated with a high energy level.
Aspects of the technology relate to processing an audio stem, including dividing a stem into a plurality of stem slices, classifying each of the plurality of stem slices into at least a first group or a second group and applying a stem effect, wherein a stem effect comprises replacing at least one stem slice with a time-reversed version of the at least one stem slice and wherein the time-reversed version of the at least one stem slice belongs to the high energy group.
Aspects of the technology relate to processing an audio stem, including dividing a stem into a plurality of stem slices, classifying each of the plurality of stem slices into at least a first group or a second group and applying a stem effect, wherein a stem effect comprises replacing at least one stem slice with a time-reversed version of the at least one stem slice and wherein the time-reversed version of the at least one stem slice belongs to the low energy group.
Aspects of the technology relate to processing an audio stem, including dividing a stem into a plurality of stem slices, classifying each of the plurality of stem slices into at least a first group or a second group and applying a stem effect, wherein the stem effect comprises replacing at least one stem slice belonging to the first group with a different stem slice belonging to the first group, wherein the time-reversed version of the second stem slice belongs to the high energy group.
Aspects of the technology relate to processing an audio stem, including dividing a stem into a plurality of stem slices, classifying each of the plurality of stem slices into at least a first group or a second group and applying a stem effect, wherein the stem effect comprises replacing at least one stem slice belonging to the first group with a different stem slice belonging to the first group, wherein the time-reversed version of the second stem slice belongs to the low energy group.
Aspects of the technology relate to using a different audio property than the energy level for classifying stem slices into a first and second group. For example, the audio property could include one or more of the energy level, frequency content, sharpness, crest factor, and/or skewness of the stem slices and/or psychoacoustic features such as pitch, timbre, and/or loudness of the stem slices.
Aspects of the technology relate to using a Euclidean algorithm to determine which stem slices to replace.
Aspect of the technology relate to calculating the energy level of each stem slice of the plurality of stems, sorting each stem slice in ascending order or descending order or alternating order based on the energy level of each stem slice to create a sorted stem slice sequence, replacing the first n stems in the sorted stem slice sequence with an all-zero stem slice, wherein n is an integer greater 0.
Aspects of the technology relate to processing an audio stem, including dividing a stem into a plurality of stem slices, classifying each of the plurality of stem slices into at least a first group or a second group and applying a stem effect, wherein the first group is associated with high frequencies and the second group is associated with low frequencies.
Aspects of the technology relate to processing an audio stem, including dividing a stem into a plurality of stem slices, classifying each of the plurality of stem slices into at least a first group or a second group and applying a stem effect, wherein the first group and second groups are based on two or more different audio properties, wherein the audio properties include energy level and/or frequency content and/or sharpness and/or crest factor and/or skewness of the stem slices.
For a more complete understanding of the technology, reference is made to the following description and accompanying drawings, in which:
Consider an audio signal x(k). For the purposes of the present disclosure we refer to this signal as an audio stem. We refer to x(k) as an audio signal or an audio stem interchangeably. It is understood that the present technology can be applied to any audio signal or audio stem(s) that contain any number of audio sources.
The present disclosure provides a method for processing audio stems to produce stem variations. The exemplary steps of the method are shown in
The first step of the exemplary method is to divide a stem into stem slices or equivalently perform “stem slicing”. A stem slice represents a part of the audio signal and is an audio signal itself. The length of a stem slice is Ni. Here a stem slice is represented as a Ni×1 vector xi. Each element of the vector corresponds to a sample of the audio signal x(k)
x
i=[x(NP+1),x(NP+2), . . . ,x(NP+Ni)]T (1)
where NP=Σk=0PNk. The index i of each stem slice indicates its order in the stem. This way writing
x=[x1T,x2T, . . . ,xMT]T (2)
we can represent the complete original audio signal. M is the number of slices and depends on the length of the stem and the method we choose to divide the stem into slices.
In one technique each stem slice corresponds to a musical note duration. This way we divide the stem in slices of equal musical length (e.g. a quarter note or a triplet sixteenth note). An example is shown in
In a third technique the start and end points of a slice and hence its length could be defined according to a detection function d(x). An example is shown in
After a stem is divided in stem slices, we group the stem slices together based on some measure of “similarity”. The goal here is that each stem slice group can be meaningfully interpreted in the context of music creation or synthesis to help design and implement useful stem effects. For each slice xi we extract a F×1 feature vector fi and create the F×M feature matrix S. The features we choose to extract define the concept of “similarity”.
In one embodiment of this disclosure, we want to group stem slices according to how important they are to the rhythmic structure of the stem. In one exemplary embodiment we use the stem slice energy as an indication of its importance. We separate stem slices in groups with the following steps which are also shown in
Alternatively, or in addition, stem slice groups can be based on two or more features including frequency content or other audio signal properties (such as sharpness, crest factor, skewness, etc.) or psychoacoustic features such as pitch, timbre, loudness.
One is not limited to create two stem slice groups. We can create a plurality of stem slice groups either directly or hierarchically. For example, after creating a high energy and a low energy group based on the energy level, the high energy group and/or the low energy group could each be further divided into a high frequency group and a low frequency group. This additional grouping is, of course, not limited to frequency and energy. Any two or more audio features/properties can be combined in this manner to create a plurality of stem slice groups. It is also understood that any clustering method can be used to produce the stem slice feature groups from the stem slice feature matrix S, including but not limited to k-means clustering, Gaussian Mixture Model (GMM) clustering, non-negative factorization (NMF) clustering, etc. Supervised classification methods can also be used to group stem slices according to the feature matrix S if sufficient training data are available, including but not limited to Support Vector Machines (SVM), artificial neural networks and deep neural networks (ANN, DNN), naïve Bayes classifiers, etc.
The final step as shown in
A stem effect is a process that generates a new pattern vector and/or applies some processing on one or more stem slices to produce a new stem variation. A stem effect can have one or more parameters that control the behavior of the effect. We will describe several stem effects in the following paragraphs.
A stem effect that is very useful in music creation or synthesis is the “arrange” effect. “Arrange” is a stem effect that can produce slight or drastic variations of a stem, similar to those of a human musician when performing a musical phrase. To apply this effect to a stem, we generate a new pattern vector {circumflex over (p)} using an M×M permutation matrix T
{circumflex over (p)}=Tp (3)
and then use this pattern vector to generate the stem variation as in
We describe here an exemplary step by step process to construct the permutation matrix T.
We can repeat steps 2-4 if needed. As an option, a parameter v can define how many times the process is repeated. For example, if v=1 we repeat the process one time, if v=2 we repeat the process twice and so on and so forth. This parameter may be set by a user or it can depend on other parameters of other stem effects. In this case the maximum value of this parameter is equal to the number of slices.
An example of the “arrange” stem effect is shown in
Instead of starting from an existing pattern vector p and generating new ones via permutation matrices, we can choose to directly generate a new pattern vector and use the stem slices to produce radical variations of the original stem. There are two main questions here: a) How to generate a completely new pattern vector p and b) how to choose the stem slices that will be used to produce the stem variation.
In one embodiment, we use the Euclidean algorithm to produce stem variations employing the following steps:
Using this method, we can product stem variations that have consistent sonic characteristics but radically different rhythmic structures. An example is shown in
Another stem effect that is very useful to produce stem variations that are musically meaningful is the “filter” effect. The “filter” effect defines a processing function ƒ(x) that will process one or more of the stem slices. It is understood that ƒ(x) can describe any type of processing including but not limited to filtering, time-reversal, amplitude modification, dynamic range compression, saturation, pitch shifting, etc. The type of processing can be user defined or chosen depending on the properties of a stem slice. Again, the main issue here is how many and which stem slices will be chosen to apply the processing. We use the stem slice groups to choose slices and apply processing that will result in musically meaningful stem variations.
In one embodiment of the generic type of “filter” stem effect we define the “reverse” effect where the processing function ƒ(x) applies a time-reversal on the stem slice data. We define the Ni×Ni exchange matrix E and the time-reversed slice is i=Exi. To apply this effect we perform the following steps:
If rhythmic structure of the original stem is important and should be kept intact, we can use step 3b. This way the i-th high energy slice remains in place, unprocessed and is followed by a time-reversed copy of itself.
An example of using step 3a is shown in the 3.
An example of using step 3b is shown in the 3.
Steps 1-3 can be repeated a number of times. As with the “arrange” effect, a parameter v can define how many times the process is repeated. For example if v=1 we repeat the process one time, if v=2 we repeat the process twice and so on and so forth. This parameter may be set by a user or it can depend on other parameters of other stem effects. In this case the maximum value of this parameter is equal to the number of slices in CH.
In another embodiment of the “filter” stem effect, we define the “silence” effect where the processing function ƒ(x) applies a zero gain value to the stem slice data. This effect will produce stem variations that are more sparse and leave space in order to use the stem with other stems in a music creation or synthesis scenario. As we discussed before, the choice of which slices to process is not trivial. To obtain a musically meaningful stem variation when applying the silence effect we perform the following steps:
Of course, we can use different sorting orders in step 2 including but not limited to descending, alternating, etc. An example is shown in
One exemplary goal behind the “arrange” and “filter” effects as detailed above is that they are “guided” by the properties of the stem slices as defined in the stem slice grouping. This allows us to define stem effects that achieve specific musical results depending on the features we use in the stem slice grouping and how we use the groups to constrain the construction of permutation matrices or choosing slices to apply processing. While we have described embodiments of the “arrange” and “filter” effects that use a stem slice grouping with two groups, it is understood that one can devise generalizations with three or more groups. It is also understood, that we can combine a number of processing functions ƒ(x) to define more complex effects. For example, we can define two different processing functions ƒ1(x), ƒ2(x) and apply each function only to stem slices from a specific group, for example use ƒ1(x) to process the stem slices from CL and ƒ2(x) to process the stem slices from CH.
We can combine the steps and principles defined for the “arrange” and “filter” effects and create any process that produces musically meaningful stem variations.
We are not limited to the number of stem effects that are applied on a stem. We can choose to apply two or more effects to the stem, in series or in parallel or in any combination of these. The order of the application of effects can be predefined, user defined or automatically determined based on some properties of the stem. One important issue if we will perform a pre-processing (i.e. stem slice grouping) step before each stem effect to choose different stem slicing methods or to update the stem slice groups for the new stem variant. However, at least one preprocessing step must be performed before applying the first stem effect.
Users have often access to multiple stems from the same song for example the vocal stem, the percussion stem, the bass stem and the guitar stem. Alternatively, we can use source separation methods to automatically extract multiple stems from an existing stem or song.
When multiple stems are present, we can choose to apply any number of stem effects in each of the stems. The number and type of the stem effects applied on each stem can be different or the same for some or all stems. In the case of multiple stems, one or more global parameters can be defined that control the value of individual stem effect parameters. The global parameters can control the same stem effect parameter for all stems or different stem effect parameters for each stem.
While the above-described embodiments have been discussed in relation to a particular sequence of events, it should be appreciated that changes to this sequence can occur without materially effecting the operation of the technology. Additionally, the exemplary techniques illustrated herein are not limited to the specifically illustrated embodiments but can also be utilized and combined with any one or more of the other exemplary embodiments and each described feature is individually and separately claimable.
An exemplary embodiment of a system for processing stems is shown in
Additionally, the systems, methods and protocols of this technology can be implemented on a special purpose computer, a programmed micro-processor or microcontroller and peripheral integrated circuit element(s), an ASIC or other integrated circuit, a digital signal processor, a hard-wired electronic or logic circuit such as discrete element circuit, a programmable logic device such as PLD, PLA, FPGA, PAL, any comparable means, or the like. In general, any device capable of implementing a state machine that is in turn capable of implementing the methodology illustrated herein can be used to implement the various methods, protocols and techniques according to this disclosure.
Examples of the processors as described herein may include, but are not limited to, at least one of Qualcomm® Snapdragon® 800 and 801, Qualcomm® Snapdragon® 610 and 615 with 4G LTE Integration and 64-bit computing, Apple® A7 processor with 64-bit architecture, Apple® M7 motion coprocessors, Samsung® Exynos® series, the Intel® Core™ family of processors, the Intel® Xeon® family of processors, the Intel® Atom™ family of processors, the Intel Itanium® family of processors, Intel® Core® i5-4670K and i7-4770K 22 nm Haswell, Intel® Core® i5-3570K 22 nm Ivy Bridge, the AMD® FX™ family of processors, AMD® FX-4300, FX-6300, and FX-835032 nm Vishera, AMD® Kaveri processors, Texas Instruments® Jacinto C6000™ automotive infotainment processors, Texas Instruments® OMAP™ automotive-grade mobile processors, ARM® Cortex™-M processors, ARM® Cortex-A and ARM926EJ-S™ processors, Broadcom® AirForce BCM4704/BCM4703 wireless networking processors, the AR7100 Wireless Network Processing Unit, other industry-equivalent processors, and may perform computational functions using any known or future-developed standard, instruction set, libraries, and/or architecture.
Furthermore, the disclosed methods may be readily implemented in software using object or object-oriented software development environments that provide portable source code that can be used on a variety of computer or workstation platforms. Alternatively, the disclosed methods may be readily implemented in software on an embedded processor, a micro-processor or a digital signal processor. The implementation may utilize either fixed-point or floating point operations or both. In the case of fixed point operations, approximations may be used for certain mathematical operations such as logarithms, exponentials, etc. Alternatively, the disclosed system may be implemented partially or fully in hardware using standard logic circuits or VLSI design. Whether software or hardware is used to implement the systems in accordance with this disclosure is dependent on the speed and/or efficiency requirements of the system, the particular function, and the particular software or hardware systems or microprocessor or microcomputer systems being utilized. The systems and methods illustrated herein can be readily implemented in hardware and/or software using any known or later developed systems or structures, devices and/or software by those of ordinary skill in the applicable art from the functional description provided herein and with a general basic knowledge of the audio processing arts.
Moreover, the disclosed methods may be readily implemented in software that can be stored on a storage medium, executed on programmed general-purpose computer with the cooperation of a controller and memory, a special purpose computer, a microprocessor, or the like. In these instances, the systems and methods of this disclosure can be implemented as program embedded on personal computer such as an applet, JAVA® or CGI script, as a resource residing on a server or computer workstation, as a routine embedded in a dedicated system or system component, or the like. The system can also be implemented by physically incorporating the system and/or method into a software and/or hardware system, such as the hardware and software systems of an electronic device.
Any non-transitory computer-readable information storage media, having stored thereon instructions, that can be executed by one or more processors and cause to be performed the methods described above.
Finally, the disclosed methods may be readily implemented as services or applications accessible from the user via a web browser. The software can reside in a local server or a remote server. The software may be written in JavaScript utilizing JavaScript Web APIs such as the Web Audio API or make use of Web Assembly.
It is therefore apparent that there has been provided, in accordance with the present disclosure, systems and methods of processing audio stems. While this technology has been described in conjunction with a number of embodiments, it is evident that many alternatives, modifications and variations would be or are apparent to those of ordinary skill in the applicable arts. Accordingly, it is intended to embrace all such alternatives, modifications, equivalents and variations that are within the spirit and scope of this disclosure.
The present application claims the benefit of and priority under 35 U.S.C. § 119(e), to U.S. Provisional Application No. 62/743,680, filed Oct. 10, 2018, entitled “METHOD FOR PROCESSING AUDIO STEMS” which is incorporated herein by reference, in its entirety, for all that they teach and for all purposes.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2019/055548 | 10/10/2019 | WO | 00 |
| Number | Date | Country | |
|---|---|---|---|
| 62743680 | Oct 2018 | US |
