Patent Application
20210125593
This application claims priority to European Patent Application No. 19205553.1, filed Oct. 28, 2019, which is hereby incorporated by reference in its entirety.
Orchestration in general is a task consisting in distributing various musical voices or parts to musical instruments. As such, orchestration is not very different from composition. In practice however, orchestration is a task performed usually by arrangers, i.e. musicians able to compose music material that somehow reveals a given music target such as a melody, a motive, or a theme.
There is no real scientific basis for orchestration and most treatises consist of informed descriptions and analysis of existing examples. As a consequence, orchestration cannot be based on a model built from existing academic knowledge, as opposed to more constrained forms of musical polyphony.
Like most musical composition tasks, the orchestration problem (including its projective variant i.e., orchestration built from existing melodies) in general is ill-defined, as virtually all musical effects and means can be employed by the arranger to create a satisfying musical work. Even within the boundaries of tonal music, almost any instrument can be used. For a given instrument any musical production, provided they conform to the intrinsic limitations of the instrument such as its tessitura or playability constraints, can be employed.
This disclosure provides a new MIDI file based on a source MIDI file and a target MIDI file. In some embodiments, the new MIDI file may be regarded as a re-orchestration of the target MIDI file based on the source MIDI file.
In one aspect, there is provided a method of automatically preparing an MIDI file based on a target MIDI file and a source MIDI file. The method comprises segmenting the source MIDI file into source segments, reordering at least some of the source segments, concatenating the reordered source segments to obtain a new MIDI file, preferably having the same length as the target MIDI file, and outputting the new MIDI file.
In another aspect, there is provided a non-transitory computer readable medium comprising computer-executable components for causing an electronic device to perform an embodiment of the method of the present disclosure when the computer-executable components are run on processing circuitry comprised in the electronic device.
In another aspect, there is provided an electronic device for automatically preparing an MIDI file. The electronic device comprises processing circuitry, and data storage storing instructions executable by said processing circuitry whereby said electronic device is operative to segment a source MIDI file into source segments, reorder at least some of the source segments, concatenate the reordered source segments to obtain a new MIDI file, preferably having the same length as a target MIDI file, and outputting the new MIDI file.
As a result of the embodiments described herein, a new MIDI file can be automatically prepared based on two existing MIDI files, herein called target and source MIDI files. By the source segments being reordered in relation to the source MIDI file, the new MIDI file differs from the source MIDI file. By the new MIDI file having the same length (in time, i.e. duration) as the target MIDI file, the new MIDI file may be outputted (e.g., played) together with the target MIDI file, which may be preferred in some embodiments.
Embodiments will be described, by way of example, with reference to the accompanying drawings, in which:
Embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments are shown. However, other embodiments in many different forms are possible within the scope of the present disclosure. Rather, the following embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Like numbers refer to like elements throughout the description.
The embodiments described herein reference Musical Instrument Digital Interface (MIDI) as an example technical standard that describes a communications protocol and file format for an electronic music score. Reference to MIDI is not meant to be limiting; the concepts described herein may be applied to any other type of electronic music-based communications protocol and/or file format.
In accordance with some embodiments, a new MIDI file is generated as what is herein called an orchestration of a target MIDI file in the style of a source MIDI file. The target MIDI file may have a melody, a chord sequence, or both, and may generally be any multitrack MIDI file. Similarly, the source MIDI file may also be any multitrack MIDI file, typically a capture of a musical performance. Herein, orchestration may be seen as a sequence generation problem in which a good trade-off is found between 1) harmonic conformance of the generated new MIDI file to the target MIDI file and 2) sequence continuity with regards to the source MIDI file.
The generated MIDI file may be intended to be played along with the target MIDI file, e.g., as a combined MIDI file. However, other use cases are also envisioned.
On one hand, the new MIDI file may be in the style of the source MIDI file, e.g., preserving as much as possible of expression, transitions, groove, and idiosyncrasies. On the other hand, the new MIDI file may be harmonically, and, to some extent, rhythmically compatible with the target MIDI file.
In accordance with
The target and source MIDI files T and S are segmented, preferably in equal-length segments, e.g., one-beat-long or one-measure-long segments, such that the target MIDI file T is segmented into N target segments t and the source MIDI file S is segmented into P source segments s. Optionally, in order to be tonality-invariant, the source segments s may be transposed, for example 12 times (e.g., from five semitones down to six semitones up, depending on the pitch range of the source MIDI file S).
When reordering the source segments s to form the sequence of segments for the new MIDI file O, one or some segments may be used several times. Thus, the new MIDI file may in some cases be formed from fewer source segments s than there are target segments t in the target MIDI file. Also, domain augmentation may be used to generate a plurality of segments for the new MIDI file sequence of segments from a single source segment. Thus, the source MIDI file S need not have at least the same length in time as the target MIDI file T to form the new MIDI file having the same length as the target MIDI file. It is noted that when it is herein referred to MIDI files, it is often the audio encoded by the MIDI file which is intended. The length of a MIDI file, or a segment thereof, may thus be regarded as, e.g., the number of bars or beats of the audio encoded thereby, or a time duration of the audio when played at a predetermined tempo.
The new MIDI file O is produced by reordering at least some of the (optionally transposed) source segments s and then concatenating the reordered source segments to create a new sequence of the same duration as the target MIDI file T. In the example of
In some embodiments, the first and second source segments si and sj may be chosen so that either or both of properties (i) and (ii), below, are satisfied:
(i) Each source segment s in the new MIDI file O is harmonically conformant to the corresponding target segments t to which the source segments s are aligned, for instance H(si, tk) and H(sj, tk+1) are relatively small, where H is a harmonic distance that indicates the harmonic similarity between the MIDI segments. The harmonic distance H may correspond to a harmonic probability for choosing a source segment s to be aligned with a target segment t. Thus, a smaller harmonic distance H, corresponding to a higher harmonic similarity, results in a higher harmonic probability that a source segment s from the source MIDI file S is chosen to be included in the new MIDI file O and aligned with its corresponding target segment t. This is in
(ii) The transition between each two consecutive source segments s in the new MIDI file O is musically similar to a transition between two consecutive source segments s in the source MIDI file S, other than the two consecutive source segments s in the new MIDI file O (since the source segments s in the new MIDI file O are reordered compared with the source MIDI file S). Looking again at the two consecutive source segments si and sj in the new MIDI file O, they are in
Property (i) aims at ensuring that the new MIDI file O is conformant to the target MIDI file. The harmonic distance H(s, t) is typically close to zero if segments s and t use the same notes (or same pitch-classes). Conversely, H(s, t) is typically much more than zero if segments s and t contain different pitch-classes.
Property (ii) states that two source segments s, here si and sj, can be concatenated in this order if there exists an index l<P such that G(sl, si) is close to zero and G(sl+1, sj) is close to zero.
It can be noted that the graphical distance G may be endogenous to the source MIDI file S, whereas the harmonic distance H is computed between source and target segments s and t and is thus agnostic in terms of composition and performance style of the audio represented by the MIDI files.
The distances H and G may, each or both together, be used to compute costs, such that a harmonic cost is computed using the harmonic distance H and/or a transition cost is computed using the graphical distances G. These costs may be interpreted as probabilities, harmonic probability and graphical probability, respectively, to be used by a sampling algorithm, e.g., using Belief Propagation as discussed further below.
The harmonic distance H between source and target segments s and t may be based on a comparison between the pitch profiles of the two segments s and t. In order to remain as independent as possible from the music style of the sources and target MIDI files S and T, a simple pitch profile distance may be used which is not tuned for Western tonal music (e.g., taking into account the salience of pitches in a given scale). In practice, the harmonic distance H may be computed between Boolean matrices that represent corresponding piano rolls of the segments s and t. More precisely, for each segments and t of length b beats, all the tracks of the respective MIDI files may be mixed together, and a Boolean matrix may be computed of size (128, 12b), such that a number 1 at position (i, j) in the matrix indicates that at least one note of pitch i is playing at time j. These matrices may be referred to as merged piano rolls. Each matrix may then be folded modulo 12 (octave folding) as we only care about harmony, not absolute pitches, resulting in a Boolean matrix of dimensions (12, 12b), in which a number 1 at position (I, j) indicates that at least one note with pitch p such that p mod 12=i is playing at temporal position j. These matrices may be referred to as modulo 12 piano rolls.
The harmonic distance H(t, s) between a target segment t and a source segment s may be computed by considering three quantities extracted from the modulo 12 piano roll ps and pt, for segments t and s respectively:
1. Quantity c is the number of common active bits in ps and pt.
2. Quantity m is the number of active bits in pt that are inactive in ps, which corresponds to active notes in the target segment t that are missing in the source segment s.
3. Quantity f is the number of active bits in ps that are inactive in pt, which corresponds to active notes in the source segment s that are missing in the target segment t, which may be called foreign notes.
Then, the harmonic distance H(s, t) may be defined as
where wm and wf represent weights of missing and foreign notes respectively. These weights may allow, e.g., a user to tailor the harmonic distance H for achieving specific musical effects.
Embodiments of the method of the present disclosure automatically prepare a new MIDI file O by recombining source segments s of the source MIDI file S, which results in new transitions between existing segments s. The quality of such a new transition may be measured in relation to the transitions between source segments s in the source MIDI file S. For example, if the source MIDI file S has unusual transitions that do not appear in other existing music, it may be desirable to reproduce such transitions in the new MIDI file O. In contrast, a general model may rank such transitions with a low score and will therefore not reproduce them.
The quality of a transition may not depend only on harmonic features, but also on rhythm and on absolute pitches, e.g., to prevent very large melodic intervals in transitions. Therefore, contrarily to the harmonic distance H, which may rely on modulo 12 piano rolls, the graphical distance G may rely on merged piano rolls, which retain information about absolute pitches. The graphical distance G between any source segments sx and sy (see also property (i) and
where PR(s) is a Boolean matrix representing the piano roll of MIDI segments, and b is the length, in beats, of the segment s.
Using the harmonic and graphical distances H and G, reordered sequences of source segments s for the new MIDI file O may be generated, e.g., using Belief Propagation. This algorithm may sample solutions according to probabilities for harmonic conformance (unary factors or local fields) and for transitions (binary factors). The Belief Propagation typically requires two probabilities, which may be obtained from the harmonic and graphical distances H and G, respectively, e.g., as follows:
In practice, the number of sj may be in O(l), where l is the size of the source MIDI file, why computing the two normalization factors ZH and ZG is typically fast.
Thus, a plurality of possible source segment sequences for the new MIDI file may be ranked by means of the harmonic and/or graphical probabilities based on the harmonic and/or graphical distances H and G. Typically, a highly ranked source segment sequence, i.e., with high probabilities (low distance(s)), e.g., the most highly ranked, may be chosen for the new MIDI file O which is then outputted, e.g., to a storage internal to the electronic device preparing the new MIDI file or to another electronic device such as a smartphone or smart speaker.
In some embodiments, the source segments s used for the new MIDI file O may be adjusted (augmented) to provide more creatively novel versions of the new MIDI file. In domain augmentation, as used herein, each source segment s can be transformed to create better fits to a target segment t with which it is aligned. Formally, this may comprise generating samples s′ of a source segment s, for a given pair of aligned source and target segments (s, t) so that:
G(s,s′)<ε, for an ε>0 (4)
and
H(t,s′)<H(t,s) (5)
A possible mechanism to achieve this is by means of machine learning model, e.g., using a Variational Autoencoder (VAE), e.g., in accordance with Roberts, A., Engel, J., Raffel, C., Hawthorne, C., and Eck, D. “A hierarchical latent vector model for learning long-term structure in music”, CoRR abs/1803.05428 (2018). By training a VAE on a large set of MIDI files, it may be possible to explore the intersection between an imagined sphere around a source segment s and another sphere around a target segment t in the corresponding latent space. Another approach to domain augmentation may comprise exploring small variations around each source segment s using ad hoc variation generators. This may allow control of the amount of creativity of the system preparing the new MIDI file O.
In some embodiments, any transformation of the source segment s may be used for domain augmentation. For example, the “reversed” source segment may be used (produced by reversing the order of each note in the segment), any diatonic transposition of the source segment may be added in any key, the result of the basic version (non-augmented) of the source segment may be added, or any other transform of the source segment may be added, to the segment sequence of the new MIDI file O. Thus, augmented versions of the source segments s which may be “closer” harmonically to the target segments t with which they are aligned may be selected for the new MIDI file.
Below, some more specific augmentation mechanisms are presented as examples.
Domain augmentation may be based on harmonic adaptation (augmentation). Harmonic augmentation may comprise exploring variations defined by imposing a small number (e.g., 0, 1 or 2) of pitch changes to the pitches of the source segments. Only small pitch changes (e.g., ±1 semitones) may be considered, so that the resulting augmented source segments s′ are close to the original source segment s, i.e., G(s, s′)≈0.
For example, consider a source MIDI file S with P source segments {si, . . . , sP}. For each si, we explore the neighbourhood {si1, si2, . . . } of si, containing all segments s obtained by imposing a small number of pitch changes to the pitches of si. Note that, we avoid creating segments with duplicate pitches. Eventually, we keep the new segments sik which are such H(t, sik)<minl≤j≤P H(t, sj) for at least one target segment t.
Another example of an augmentation mechanism is to allow more transitions between source segments s (including their augmented variants s′). This may be achieved in principle with Deep Hash Nets, e.g., in accordance with Joslyn, K., Zhuang, N., and Hua, K. A. “Deep segment hash learning for music generation”, arXiv preprint arXiv: 1805.12176 (2018). In practice, it may be possible to use property (ii) discussed above, that the transition between each two consecutive source segments s in the new MIDI file O is musically similar to a transition between two consecutive source segments s in the source MIDI file S, applied to the augmented variants s′ of the source segments s.
In some embodiments, the method may further comprise segmenting (204) the target MIDI file into target segments t. Preferably, most or all of the target segments have the same length(s), e.g., in respect of number of bars or beats, as the source segments s, allowing source and target segments of the same lengths to be aligned with each other. Then, after the source segments have been reordered (206), some or each of the target segments t of the target MIDI file T may be aligned (208) with a corresponding source segments of the reordered source segments s, before the outputting (212) of the new MIDI file. The target segments t, typically remaining in the same order as in the target MIDI file T, forming a sequence of target segments, may be aligned (208) with a sequence of reordered source segments which may form the new MIDI file. Thus, the sequence of target segments may be aligned to a sequence of reordered source segments (typically both sequences having the same length). By aligning the target and source segment sequences with each other, a combined MIDI file may be obtained. Thus, in some embodiments, the aligning (208) of each segment t of the target MIDI file T with a corresponding source segment s of the new MIDI file O results in a combined MIDI file C comprising the target MIDI file T aligned with the new MIDI file O. Then, the outputting of the new MIDI file may be done by outputting the combined MIDI file comprising the new MIDI file.
In some embodiments, each source segment s of the new MIDI file O is harmonically similar to its aligned target segment t. Harmonic similarity may be determined by the harmonic distance H, optionally using harmonic probability, as discussed herein. In some embodiments, each source segment s is harmonically similar to its aligned target segment t based on a harmonic distance H between a pitch profile of the source segment and a pitch profile of the target segment.
In some embodiments, a transition between two consecutive source segments, e.g., si and sj, in the new MIDI file O is musically similar to a transition between two consecutive other source segments, e.g., sl and sl+1, in the source MIDI file S. In some embodiments, the transitions are musically similar based on graphical distances G, as discussed herein, e.g., dependent on Hamming distance. In some embodiments, the graphical distances G are such that a graphical distance between a first source segment si of the two consecutive source segments si and sj in the new MIDI file O and a first segment sl of the two consecutive other source segments sl and sl+1 in the source MIDI file S is low and a graphical distance between a second source segment sj of the two consecutive source segments in the new MIDI file and a second segment sl+1 of the two consecutive other source segments in the source MIDI file is also low, e.g., as illustrated in
As mentioned above, the reordering (206) may be based on Belief Propagation. In some embodiments, the Belief Propagation is dependent on a harmonic probability corresponding to the harmonic distance H between a pitch profile of a source segment s of the reordered source segments and a pitch profile of a target segment t with which the source segment is aligned. The steps of reordering (206) and aligning (208) may e.g., be done iteratively until a reordered source segment s is aligned with a target segment to which there is a relatively small harmonic distance H, corresponding to a high harmonic probability. This may be done for each of the target segments, e.g., until the sequence of target segments is aligned with a sequence of source segments where the combined harmonic distances H between all pairs of target and source segments is relatively small.
In some embodiments, the Belief Propagation is additionally or alternatively dependent on a graphical probability corresponding to graphical distances G of two consecutive source segments si and sj of the reordered source segments and two consecutive other source segments sl and sl+1 in the source MIDI file S. Again, this may be done for each pair of consecutive source segments of the reordered source segments to obtain a combined or average graphical distance which is relatively small.
In some embodiments, as discussed above, at least one of the reordered source segments s is augmented to an augmented source segment s′ (still being regarded as a source segment) before the concatenating. Thus, source segments fitting even better with the target segments and/or with each other may be obtained. In some embodiments, the at least one source segment s is augmented by means of a machine learning model, e.g., using a Variational Autoencoder (VAE) and/or by harmonic augmentation comprising imposing a pitch change to a pitch of the source segment.
The electronic device 300 comprises processing circuitry 310, e.g., a central processing unit (CPU). The processing circuitry 310 may comprise one or a plurality of processing units in the form of microprocessor(s). However, other suitable devices with computing capabilities could be comprised in the processing circuitry 310, e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD). The processing circuitry 310 is configured to execute one or more instructions (referred to as computer program(s) or software (SW)) 330 stored in a storage 320 of one or several storage unit(s) e.g., a memory. The storage unit is regarded as a non-transitory computer-readable storage medium, forming a computer program product together with the SW 330 stored thereon as computer-executable components, as discussed herein and may, e.g., be in the form of a Random Access Memory (RAM), a Flash memory or other solid state memory, or a hard disk, or be a combination thereof. The processing circuitry 310 may also be configured to store data in the storage 320, as needed.
In the example depicted in
For each of a plurality of consecutive pairs of first and second target segments (e.g., for a first pair (t1, t2), a second pair (t2, t3), and so forth), the electronic device identifies corresponding source segments. For example, for a particular consecutive pair of first (t5) and second (t6) target segments, the electronic device identifies a first source segment (s3) corresponding to the first target segment (t5) of the consecutive pair, and identifies a second source segment (s14) corresponding to the second target segment (t6) of the consecutive pair. The identifying operations may correspond with operations 206 and 208 described above.
Specifically, the electronic device identifies the first source segment (s3) based in part on a determination that the first source segment (s3) is harmonically conformant to the corresponding first target segment (t5), and identifies the second source segment (s14) based in part on a determination that the second source segment (s14) is harmonically conformant to the corresponding second target segment (t6).
In some implementations, the electronic device determines harmonic conformance using any of the harmonic distance functions and/or operations described above. For example, the electronic device may determine that the first source segment (s3) is harmonically conformant to the corresponding first target segment (t5) based on a comparison of a pitch profile of the first source segment (s3) with a pitch profile of the corresponding first target segment (t5). For example, the first source segment (s3) has a pitch profile characterized by a C major chord, which is harmonically conformant to a melody (C-E-G-C) of the first target segment (t5). In some implementations, the electronic device compares the pitch profile of the first source segment (s3) with the pitch profile of the corresponding first target segment (t5) by comparing Boolean matrices representing piano rolls of the first source segment (s3) and the corresponding first target segment (t5).
To continue the example, the electronic device may determine that the second source segment (s14) is harmonically conformant to the corresponding second target segment (t6) based on a comparison of a pitch profile of the first source segment (s14) with a pitch profile of the corresponding first target segment (t6). For example, the second source segment (s14) has a pitch profile characterized by a G minor chord, which is harmonically conformant to a melody (G-Bb-D-D) of the second target segment (t6). In some implementations, the electronic device compares the pitch profile of the second source segment (s14) with the pitch profile of the corresponding second target segment (t6) by comparing Boolean matrices representing piano rolls of the second source segment (s14) and the corresponding second target segment (t6).
In addition to identifying the first and second source segments (s3) and (s14) based in part on their harmonic conformance with the first and second target segments (t5) and (t6), the electronic device may identify the first and second source segments (s3) and (s14) based in part on a determination that a transition between the first and second source segments (s3) and (s14) is graphically conformant to a transition between any of the consecutive pairs of source segments of the first MIDI file (e.g., a transition between segments of a first consecutive pair (s1) and (s2), a transition between segments of a second consecutive pair (s2) and (s3), and so forth).
In some implementations, the electronic device determines graphical conformance using any of the graphical distance functions and/or operations described above. For example, the electronic device may determine graphical conformance of transitions based on a comparison of (i) a rhythm and/or pitch transition between the first and second source segments (s3, s14) with (ii) a rhythm and/or pitch transition between each of a plurality of consecutive pairs of source segments (e.g., (si, s2), (s2, s3), and so forth). In the example depicted in
Since there may be a plurality of source segments with adequate harmonic conformance to various target segments, the graphical conformance determinations ensure that the source segments that are identified for correspondence to respective target segments include stylistic components (e.g., similarities in musical transitions) of the source file. As a result, the orchestrated version of the target file (the orchestration file) may be characterized by a particular style or genre of the source file.
Upon identifying source segments corresponding to each of the target segments, the electronic device generating the third MIDI file (the orchestration file) using the identified first and second source segments for each of the plurality of consecutive pairs of first and second target segments. In some implementations, the electronic device generates the third MIDI file by reordering at least some of the source segments based on their correspondence to respective target segments, and concatenating the reordered source segments. The generating options may correspond with operations 208, 210, and 210 described above. For example, the generating options may correspond with any of the sequence generation and/or domain augmentation functions and/or operations described above.
The foregoing description has been described with reference to specific implementations. However, the illustrative discussions above are not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many variations are possible in view of the above teachings. The implementations were chosen and described to best explain principles of operation and practical applications, to thereby enable others skilled in the art.
The various drawings illustrate a number of elements in a particular order. However, elements that are not order dependent may be reordered and other elements may be combined or separated. While some reordering or other groupings are specifically mentioned, others will be obvious to those of ordinary skill in the art, so the ordering and groupings presented herein are not an exhaustive list of alternatives.
As used herein: the singular forms “a”, “an,” and “the” include the plural forms as well, unless the context clearly indicates otherwise; the term “and/or” encompasses all possible combinations of one or more of the associated listed items; the terms “first,” “second,” etc. are only used to distinguish one element from another and do not limit the elements themselves; the term “if” may be construed to mean “when,” “upon,” “in response to,” or “in accordance with,” depending on the context; and the terms “include,” “including,” “comprise,” and “comprising” specify particular features or operations but do not preclude additional features or operations.
The present disclosure has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the present disclosure, as defined by the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 19205553.1 | Oct 2019 | EP | regional |
