Patent Application
20070289432
The specific features, aspects, and advantages of the present invention will become better understood with regard to the following description, appended claims, and accompanying drawings where:
In the following description of the preferred embodiments of the present invention, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.
1.0 Exemplary Operating Environments:
For example,
The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers, server computers, hand-held, laptop or mobile computer or communications devices such as cell phones and PDA's, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer in combination with various hardware modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices. With reference to
Components of computer 110 may include, but are not limited to, a processing unit 120, a system memory 130, and a system bus 121 that couples various system components including the system memory to the processing unit 120. The system bus 121 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.
Computer 110 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer 110 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data.
Computer storage media includes, but is not limited to, RAM, ROM, PROM, EPROM, EEPROM, flash memory, or other memory technology; CD-ROM, digital versatile disks (DVD), or other optical disk storage; magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices; or any other medium which can be used to store the desired information and which can be accessed by computer 110. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.
The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation,
The computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media discussed above and illustrated in
Other input devices (not shown) may include a joystick, game pad, satellite dish, scanner, radio receiver, a television or broadcast video receiver, a piano-type musical keyboard, etc. These and other input devices are often connected to the processing unit 120 through a wired or wireless user input interface 160 that is coupled to the system bus 121, but may be connected by other conventional interface and bus structures, such as, for example, a parallel port, a game port, a universal serial bus (USB), an IEEE 1394 interface, a Bluetooth™ wireless interface, an IEEE 802.11 wireless interface, etc. Further, the computer 110 may also include a speech or audio input device, such as a microphone or a microphone array 198, as well as a loudspeaker 197 or other sound output device connected via an audio interface 199, again including conventional wired or wireless interfaces, such as, for example, parallel, serial, USB, IEEE 1394, Bluetooth™, etc.
A monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190. In addition to the monitor, computers may also include other peripheral output devices such as a printer 196, which may be connected through an output peripheral interface 195.
The computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be a personal computer, a server, a router, a network PC, a peer device, or other common network node, and typically includes many or all of the elements described above relative to the computer 110, although only a memory storage device 181 has been illustrated in
When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user input interface 160, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
With respect to
At a minimum, to allow a device to implement the functionality of the Concatenative Synthesizer, the device must have some minimum computational capability, some storage capability, and a communications interface 230 for allowing data input/output. In particular, as illustrated by
In addition, the simplified computing device of
The exemplary operating environment having now been discussed, the remaining part of this description will be devoted to a discussion of the program modules and processes embodying a Concatenative Synthesizer that generates a musical output from a database of musical notes and an input musical score based on a process of concatenative synthesis.
2.0 Introduction:
A “Concatenative Synthesizer,” as described herein, provides a unique method for generating a musical output from a database of musical notes and an input musical score based on a process of concatenative synthesis. Note that the term “notes” as used herein is intended to refer to both individual notes and to chords or any other simultaneous combination of notes.
In various embodiments, the aforementioned database of musical notes is generated from any desired musical score, or from one or more musical scores in combination with corresponding audio recordings representing any desired musical genre, performer, performance, or instrument recording. Note that this database generally represents a particular music “feel” or “texture” that the user wants to achieve, and as such, it is generally referred to herein as the “music texture database.”
Further, since the music texture database is generated from any desired musical score and/or audio recording representing different musical genres, performers, performances, instrument recordings, etc., in one embodiment, separate user selectable music texture databases are presented to provide the user with a selection of “music textures” upon which to build the musical output from the input musical score.
It should also be noted that when a corresponding musical score is not available in combination with an audio recording that is evaluated to populate the music texture database, the corresponding music score is directly generated from that audio recording using conventional audio analysis techniques. Such score generation techniques are well known to those skilled in the art, and will not be described in detail herein.
The input musical score is provided in a computer readable format, such as a conventional MIDI score, or any other desired computer readable musical score format. Furthermore, the input musical score may also be automatically generated by using conventional audio processing techniques to evaluate an existing musical recording to automatically construct the corresponding input musical score. As noted above, such score generation techniques are well known to those skilled in the art, and will not be described in detail herein.
2.1 System Overview:
As noted above, the Concatenative Synthesizer described herein provides a unique method for generating a musical output from a database of musical notes and an input musical score based on a process of concatenative synthesis.
In general, the Concatenative Synthesizer begins operation by receiving an input musical score, either directly, or by processing an audio file to construct the score, and a database of musical notes (i.e., the music texture database). In various embodiments, the music texture database is either provided as a predefined “music texture,” or is automatically constructed from one or more user provided sound samples.
The Concatenative Synthesizer then evaluates the music texture database to identify a unique set of candidate musical notes for every note represented in the input musical score. Furthermore, notes in the music texture database may be modified (such as by changing the pitch, duration, etc.) to better fit particular notes of the input musical score. There are a number of well known conventional techniques for changing the pitch and/or duration of audio signals such as musical notes, and as such, these note modification techniques will note be described in detail herein. Simple examples of such techniques include the use of conventional SOLA (synchronized overlap and add) techniques to change note duration or the use of conventional resampling techniques to change a note pitch.
An “optimal path” through the candidate notes is then identified by minimizing an overall cost function for picking the best path through the candidate notes relative to the input musical score. In various embodiments, the cost of each possible path through the candidate notes is computed using various factors, including, for example, a “match cost” for directly matching one note to another (i.e., a closeness metric that considers factors such as pitch and/or duration) and a “transition cost” for placing a particular candidate directly after the preceding candidate in the musical output. In addition, it should also be noted that while the optimal path is generally described in terms of minimizing the path cost, this minimum, or lowest cost, path may also be expressed in terms of maximizing the path cost by simply inverting the cost values when evaluating the various paths. Further, this path cost can also be expressed probabilistically, such that the match cost probability would be it's “goodness” (negative cost) and the transition probability would be the “transition goodness.” In this case, the optimal path would be identified by maximizing the probability/goodness. In any case, each of these basic ideas are generally intended to be included in the overall concept of finding a best path through the candidates, as described herein.
Further, in a related embodiment, a user-adjustable scale factor provides an adjustable tradeoff between “accuracy” and “coherence,” such that the musical output is either a more accurate match to the input musical score, or is more coherent (in terms of unit ordering) with respect to the original sounds used to construct the music texture database. This tradeoff is accomplished by scaling the match and transition costs as a function of the user adjustable scale factor. Note that this embodiment is described in further detail in Section 3.5.
Once the optimal path has been identified, the musical output is then constructed by concatenating the selected candidate notes corresponding to the optimal path. In various embodiments, the musical output is a music score, an analog or digital audio file or music recording, or a music playback via conventional speakers or other output devices, as desired.
For example, assume that the Concatenative Synthesizer is provided with an example pair (A, A′) of data inputs, where A represents a MIDI score (or other score format), and A′ represents the corresponding waveform (or audio file). The user then provides the Concatenative Synthesizer with the input musical score (B) which will be used to produce the musical output B′, where B′ is a realization of MIDI score B using the “texture” of the input waveform A′. In other words, given musical scores A and B, and a sound clip A′ corresponding to A, the Concatenative Synthesizer will create a new sound clip B′ that is the realization of MIDI score B, where the relationship between B and B′ approximates the relationship between A and A′ as closely as possible. Note that “closeness” can have a continuum of senses, from perfectly reproducing the score of B using sounds from A′ to perfectly preserving coherence in the samples drawn from A′ at the expense of manipulating the score of B.
Alternately, in a related embodiment, instead of constructing a musical output relative to a particular instrument, the Concatenative Synthesizer constructs a modification of a musical score by replacing notes in B with notes or note sequences from A that reflect the phrasing of a certain musical style or performer to output a new score Bnew. These concepts will be discussed in further detail in the following sections.
2.2 System Architectural Overview:
The processes summarized above are illustrated by the general system diagram of
In general, as illustrated by
Next, an input musical score B 320 is provided or selected by the user via a musical score input module 325. A candidate selection module then evaluates entries in the selected music texture database 315 to identify a set of candidate notes for each note of the input musical score B 320. In general, each acceptable candidate represents a potential match to a particular note of the input musical score B 320. Assuming that the size of the selected music texture database 315 is not too large, every sample in the database is selected as a candidate for every note in the input musical score B 320.
However, given that the computational overhead of choosing an optimal path through the candidate notes will increase with the number of candidates for each note, in an alternate embodiment, a predefined maximum number (k) of most closely matching candidates are selected for each note in the input musical score B 320. In this case, a candidate cost evaluation module 340 first determines a match cost (cmatch) for directly matching one note to another based on the pitch and duration of each candidate relative to every note in the input musical score B 320. These match costs are then used to select the k best candidates for each note of the input musical score B 320.
In either case, the candidate cost evaluation module 340 then computes the match cost (cmatch) for each candidate (if not already computed) and a transition cost (ctransition) for placing a particular candidate directly after preceding candidate in the musical output.
Next, an optimal path selection module 345 evaluates the candidates in terms of their costs (cmatch and ctransition) to identify a best path through the candidates relative to the input musical score B 320. However, as noted above, in one embodiment, the user adjustable cost scaling factor (α) is input or adjusted via the user control module 335 for scaling the match and transition costs. This scaling of the match and transition costs (cmatch and ctransition) causes the best path through the candidates to vary from one extreme, wherein the resulting output music is the most accurate match to the input musical score B 320, to the other extreme, wherein the resulting output music is more coherent with respect to the original sounds used to construct the music texture database 315. See Section 3.5 for additional discussion regarding the use of the user adjustable α value.
Next, a candidate assembly module 350 uses concatenative synthesis to combine the sequence of notes from the music texture database 315 corresponding to the optimal path. Finally, the candidate assembly module 350 then outputs either an audio music output sound B′ 355, or a new music score Bnew 360, or both.
3.0 Operation Overview:
The above-described program modules are employed for implementing the Concatenative Synthesizer. As summarized above, the Concatenative Synthesizer generates a musical output from a database of musical notes and an input musical score based on a process of concatenative synthesis. In general, the Concatenative Synthesizer focuses on high quality music synthesis from a single example instrument. However, as noted above, this music synthesis may be based on example inputs from one or more particular performers, different genres, song collections, etc. In other words, the music synthesis is based on whatever musical input is used to construct the music texture database. However, the more focused the input to the music texture database, the more that the final music output will correspond to the particular performer, genre, instrument, etc., that is represented by the music texture database.
The following sections provide a detailed discussion of the operation of the Concatenative Synthesizer, and of exemplary methods for implementing the program modules described in Section 2 with respect to
3.1 Operational Details of the Concatenative Synthesizer:
The following paragraphs detail specific operational and alternate embodiments of the Concatenative Synthesizer described herein. In particular, the following paragraphs describe definitions of terms used to implement an operational embodiment the details of the Concatenative Synthesizer; data structures; and path construction for generation of musical outputs. Following the detailed description of the aforementioned features of the Concatenative Synthesizer, an operational flow diagram is described in Section 4, with respect to
3.2 Variable Definitions:
The terms defined below represent variables that are used for a description of various embodiments of the Concatenative Synthesizer. It should be appreciated that in view of the following discussion, not every described variable described below is required for operation of the Concatenative Synthesizer. Further, it should be clear that different variable definitions may be used without departing from the intended scope of the Concatenative Synthesizer.
3.3 Data Structures:
In addition to the three inputs described above (A, A′, and B), the Concatenative Synthesizer uses several intermediate data structures for generating the musical output B′. In particular, intermediate data structures employed by the Concatenative Synthesizer include:
3.4 Path Construction for Generation of Musical Outputs:
In view of the definitions of variables and data structures provided above, the following paragraphs detail specific operational and alternate embodiments of the Concatenative Synthesizer described herein. In particular, the following paragraphs describe steps for: construction of the music texture database and segmentation of the notes of the A, A′, and B into frames; choosing candidates for each frame of B; computing costs for each candidate; evaluating the cost and index matrices (Mcost and Mindex) to compute a globally optimal path through the candidates; and generating the musical output from notes corresponding to the optimal path.
3.4.1 Music Texture Database and Note Segmentation:
As noted above, the music texture database is generated from a musical audio sample A′ and a corresponding musical score A by segmenting those inputs into frames. Again, it should be noted that the corresponding musical score A can be automatically constructed from the musical audio sample A′ using conventional techniques.
In general, any piece of music played by a human musician will never be perfectly aligned with the original musical score that defines that piece of music. Consequently, given the musical audio sample A′ and the corresponding musical score A, improved segmentation results will be achieved by first aligning A and A′. In particular, a near-perfect alignment helps to minimize a problem wherein sound data from other notes in A′ manages to seep into the musical output, thereby causing audible “grace note” artifacts in the output waveform.
The process for aligning A and A′ uses conventional techniques, such as, for example, manual labeling, pitch tracking, or other automatic methods, for detecting note boundaries in A′, then modifying the duration and onset times for the notes of score A to accurately reflect the actual note boundaries. Then, since the musical score A is accurately aligned to the musical audio sample A′, segmentation of the inputs A and A′ into frames is straightforward.
In particular, to segment the inputs, the Concatenative Synthesizer breaks each audio and musical score input into discrete frames. As such, three types of frames are considered:
In general, a single frame (of any of the aforementioned types) corresponds to a single note (or rest, which can be treated the same as a note). However, it should be appreciated that sequences of notes can also be used in place of individual notes where sequences of notes in B may correspond to sequences of notes in A. In this case, the segmentation into frames may be performed an individual note basis and/or on a note sequence basis. Matching sequences may then be treated as individual notes for purposes of determining the optimal path through the candidate frames.
It should also be noted that segmentation of the audio input A′ can also be virtual rather than actual. In other words, rather than maintaining separate samples for every segmented frame, pointers to the frame positions within the original audio input A′ can be maintained in order to provide access to the individual frames, as needed.
In one embodiment, after the frame segmentation points have been determined, the input musical score B is modified to make matches with A more likely. In particular, the input musical score B is transposed so that it has maximal overlap with A in terms of pitch values. This is as simple as trying all possible transpositions of the notes of B, and keeping the one which has the most pitch overlaps with A. In addition, the tempo of B is uniformly modified so that the median note durations of B and A are the same. Other musical score tempo distance metrics may also be used, if desired, to provide the uniform tempo change.
3.4.2 Candidate Selection:
As noted above, once the input frames have been segmented, the next step is to choose the candidates zij for each target frame bi of the input musical score B. Assuming the musical texture database is small enough, or the computer processing time is not a primary concern, zij is constructed from note a′j for all j. In other words, k=|A| candidates are used to populate zij for each frame bi, and r(i,j)=j. In one embodiment, the pitch and/or duration of each candidate is also transformed to match the pitch and duration of bi.
In the case where the music texture database is very large, computation of an optimal path through the candidates in a reasonable amount of time requires a reasonable limitation on the number of candidates. Consequently, in one embodiment, a predefined or user adjustable value for k<|A| is used. In this case, the best k candidates for each frame bi are selected with respect to cmatch in order to populate zij for each frame bi.
3.4.3 Cost Computation:
Once the audio input A′ has been split into frames, and the candidates identified for each frame bi of B, the values of cmatch and ctransition are computed for every candidate for each frame. In order to compute these scores, it is necessary to consider the cost of transforming a frame (pitch and/or duration), i.e., cmatch, and and the cost of placing two candidate frames in succession, i.e., ctransition. There are many factors that can be considered in “scoring” these elements. Consequently, it should be understood that the Concatenative Synthesizer is not intended to be limited to computation of these costs in the manner described in this section, and that the costs described below are provided solely for purposes of example and explanation.
For example, in a tested embodiment, the Concatenative Synthesizer computed scores based on distance metrics, where the function dtransform(s1,s2) represents the cost of transforming from frame s1 to frame s2 (such as by using SOLA for pitch modification and resampling for duration modification), and the function dtransition(s1,s2) represents the cost of placing two frames (frame s1 and frame s2) in succession. Given these functions, cmatch and ctransition can be computed as follows:
c
match(i,j)=dtransform(ar(i,j),zij) Equation 1
c
transition(i,j,k)=dtransition(zij,zi+1k) Equation 2
In a tested embodiment, dtransform(s1,s2) was determined as a weighted function of the pitch and duration change. Note that any desired function of the pitch and/or duration can be used here. For example, in a tested embodiment, dtransform(s1,s2) was determined as follows:
The first term in the sum illustrated in Equation 3 is the cost of changing the duration of a note (i.e., using SOLA) and is proportional to the logarithm of the ratio of the durations. Note that pitch terms are also included, since the pitch is changed before applying SOLA. The second term illustrated in Equation 3 is the cost of changing the pitch of a note using resampling, and is proportional to the difference in pitch (or the logarithm of the ratio of the frequencies). Note that the β and γ terms illustrated in Equation 3 are optional variables that allow the user to place relative weights on the pitch modification and resampling terms, if desired.
Similarly, in a tested embodiment, dtransition(s1,s2) was determined as a weighted function of the pitch of the note candidates—note that the duration doesn't appear here because it is already covered in the match cost. Note that any desired function of the pitch can be used here. For example, in a tested embodiment, dtransition(s1,s2) was determined as follows:
The transition cost defined in Equation 5 is straightforward. In particular, if the two consecutive candidates do not come from two consecutive frames of A (i.e., r(i+1,k)≠r(i,j)), then a cost of λ+μ is incurred, where λ and μ are greater than 1. On the other hand, if the two candidates come from consecutive frames, but must be resampled at different rates to match the target pitch, a cost of λ is incurred. Finally, if the two candidates come from consecutive frames, and are transposed by the same interval, no cost is incurred. Note that this cost function for dtransition means that sequences that include more sets of consecutive frames from A have a lower cost than those that contain fewer such sets. This acts to improve the coherence of the resulting Bnew and or B′, since when adjacent frames in B′ come from adjacent frames in A′, the transition will sound more “natural” since in fact it is coming directly from the original.
In a related embodiment, the transition cost can also be lower when candidate notes have matching “note contexts,” as opposed to necessarily being adjacent in the original score. For instance, if the desired note transition is “C to G” and the first candidate is followed by a “G” and/or the second candidate is preceded by a “C”, even though they are not adjacent in the score, they still have the same note transition. More formally, if pitch(ar(i,j)+1)=pitch(ar(i+1,k)) and/or pitch(ar(i,j))=pitch(ar(i+1,k)−1), the cost could be between 1 and λ+μ.
3.4.4 Computing a Globally Optimal Path:
In general, once the costs have been computed for each candidate, the next step is to compute a globally optimal path through those candidates.
As noted above, the musical output B′ is constructed using a sequence of frames from A′. Each frame in the sequence should match the corresponding frame in B (i.e., minimize match cost), and the sequence should be coherent with respect to A′ (i.e., minimize transition cost). Given the above-describes cost functions for these two objectives, and the value α (i.e., the user adjustable scaling factor described in Section 3.5), the optimal sequence is well-defined, and can be computed with a dynamic programming algorithm.
For example, given cmatch, ctransition, and the value α, the Concatenative Synthesizer computes a globally optimal sequence S of frame indices from A′, where the optimal sequence minimizes the following quantity:
This type of minimization problem can be solved using conventional minimization techniques, such as, for example, a Viterbi algorithm. In this case, for each frame bi in B, the Concatenative Synthesizer first computes the cost of the set of candidates to match bi (zij). It then computes the transition cost dtransition between each candidate zij and zi+1k. Once the costs have all been determined, the algorithm goes from the first frame to the last, at each point computing for each candidate the minimum cumulative cost to get to that candidate from any candidate from the previous frame, as well as a “backpointer” to the candidate in the previous frame that resulted in this lowest cost. When this process reaches the final frame, the optimal sequence is decoded by taking the candidate in the final frame with the lowest cumulative cost, and then following the backpointers recursively back to the first frame. This is an application of the Viterbi algorithm, and is illustrated in
3.4.5 Construction of Musical Output:
As noted above, the musical output of the Concatenative Synthesizer is either a waveform (or other audio recording or file) or is a new musical score. In the case of a new musical score, the musical output score Bnew is simply the input musical score B transformed as described above during computation of the optimal path.
In the case of an audio output B′, it is necessary to construct a new waveform (or other audio recording or file) from the frames of the musical texture database that correspond to the optimal path described above. In particular, given selected candidate zij for frame bi, and the frame a′r(i,j) from which the sound data is to be taken, the Concatenative Synthesizer optionally transforms the sound data of the selected candidate to match the pitch and duration specified for frame bi. As noted above, pitch modification and duration modification is accomplished using conventional techniques such as the use of resampling for changing the pitch of the waveform and the use of SOLA to change the duration of the waveform representing the frame.
As is known to those skilled in the art, SOLA is a technique for changing the duration of a signal independent of the pitch. The signal is broken up into overlapping segments, which are then shifted relative to each other and added back together. The desired signal length determines the amount by which the segments are shifted. In addition, the segments should be shifted to align the signal optimally, which can be measured by cross-correlation.
In general, the use of conventional SOLA techniques yield good results as long as the ratio of original signal length to new signal length is not too large or small. Generally, ratios between 0.9 and 1.1 sound very good for all sounds, but any ratio between 0.5 and 2 sound reasonable with respect to periodic signals. Since the core part of most instrument sounds (excepting the initial “attack” and final “decay”) are approximately periodic, with a large enough (A′, A) pair, it should usually be possible to find a candidate whose original signal length is close enough to the target signal length. In addition, SOLA results can be improved by stretching some portions of a note while leaving others alone. For example, in one embodiment, the “attack” of a note is left alone during SOLA processing, as the attack portion of a note typically contains energy at too many frequencies for good signal alignments to be found after shifting.
Finally, once the selected candidates have been optionally transformed, the sequence of frames corresponding to the candidates along the optimal path are simply concatenated to construct the output waveform. Note that in one embodiment, conventional audio concatenation techniques are used to prevent audible discontinuities at the junction between frames. Such techniques include cross fading the frames, weighted or windowed blending, shifting the frames with respect to each other to maximize the cross-correlation, etc.
3.5 Musical Texture Adjustments:
As noted above, in one embodiment, a user adjustable α value is provided to allow the user to customize the sound of the musical output constructed by the Concatenative Synthesizer. In general, this α value allows the user to customize the “texture” of the musical output.
For example, in the domain of image processing, texture transfer generally refers to the problem of texturing a given image with a sample texture. For music, a natural analogue is to play one piece using the style and phrasing of another (i.e., the musical “texture” of a particular instrument, artist, genre, etc.). In one embodiment, the Concatenative Synthesizer allows the user to control the extent to which musical “texture” is transferred from a musical input to a musical output as a function of an input musical score. At one extreme, the musical score is interpreted rigidly, and its notes are played exactly, with the best matches to the musical score being selected from the music texture database. At the other extreme, the input musical score is given less weight when choosing matches from the music texture database.
In this approach, there is a fundamental tradeoff between accuracy and coherence. The more faithful B′ is to B, the less likely it is that B′ is coherent with respect to A′. Conversely, the more coherent B′ is with respect to A′, the less likely it is that B′ is an accurate transformation of B. In a tested embodiment, the Concatenative Synthesizer uses a value α, between 0 and 1, to express this tradeoff. Values closer to 1 mean that B′ should match B more closely, while values closer to 0 mean that B′ should incorporate more of the style of A′. So, at the most general level, the input to the Concatenative Synthesizer is an example pair (A, A′) representing the music texture database, a new score B provided by the user, and the parameter α, with the output of the Concatenative Synthesizer being a new waveform B′ (and/or a new musical score Bnew).
In one embodiment, this concept is implemented in an electronic piano keyboard or the like with an “auto-stylization” dial. As a performer plays a piece of music, he/she can adjust this dial to control the α value of the sound coming from the keyboard relative to a user selectable music texture database. In other words, this embodiment provides users with a variable control for “importing” musical styles from other performers, genres, instruments, etc., into a new piece of music.
For example, the Concatenative Synthesizer described herein, when applied to music score realization, presents a balance between playing “Paul Desmond's saxophone”, and “playing Paul Desmond's saxophone like Paul Desmond.” This balance can be thought of as controlling the amount of “texture transfer” that takes place when constructing the musical output.
4.0 Concatenative Synthesizer Operational Embodiments:
The processes described above with respect to
In general, as illustrated by
Next, an input musical score B 320 is also segmented 605 into frames. All possible candidate frames from the selected music texture database 315 are then identified 610 for each frame of the input musical score B 320. As discussed above, assuming that the size of the selected music texture database 315 is not too large, every sample in the database is selected as a candidate for every frame in the input musical score B 320. Alternately, the number of possible candidates is limited by a user adjustable or predefined maximum value (k).
Once the candidates have been identified 610, match and transition costs, cmatch, and ctransition, respectively, are computed for each candidate for each frame of the input musical score B 320.
Next, a globally optimal path is computed 620 through the candidate sets corresponding to each frame of the input musical score B 320. As noted above, in one embodiment, the user control module allows the user to weight the costs (cmatch, and ctransition) that are used in computing 620 the optimal path. This weighting is accomplished by varying the adjustable cost scaling factor (α) via the user control module 335.
Once the optimal path has been computed 620, the frames corresponding to that path are optionally transformed 625 to match the pitch and/or duration of the musical output frames.
Finally, in either case, whether transformed 625, or not, the frames corresponding to the optimal path are then concatenated to combine the sequence of notes from the music texture database 315 corresponding to the optimal path. The concatenated sequence of notes is then output either as an audio music output sound B′ 355, or a new music score Bnew 360, or both.
The foregoing description of the Concatenative Synthesizer has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. Further, it should be noted that any or all of the aforementioned alternate embodiments may be used in any combination desired to form additional hybrid embodiments of the Concatenative Synthesizer. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.
