Patent Grant
9471271
The present invention relates to systems and methods for audio synthesis in a portable device. Furthermore, the present invention relates to systems and methods for creating, modifying, interacting with and playing music, and more particularly to systems and methods employing a top-down and interactive auto-composition process, where the systems/methods provide the user with a musical composition that may be modified and interacted with and played and/or stored (for later play) in order to create music that is desired by the particular user. Furthermore, the present invention relates to systems and methods for auto-generated music, and more particularly to systems and methods for generating a vocal track as part of an algorithmically-generated musical composition. Furthermore, the present invention relates to a file format suitable for storing information in a manner which preferably provides forward and/or backwards compatibility. Furthermore, the present invention relates to systems and methods for algorithmic music generation, and more particularly to improved sample format and control functions, which preferably enable the general conformance of a sound sample to the current pitch and/or rhythmic characteristics of a musical composition. Furthermore, the present invention relates to systems and methods for broadcasting music, and more particularly to systems and methods employing a data-file-based distribution system, where at least portions of the music can be generated by a node/subscriber unit upon reception of a data file, which is processed using a music generation system that preferably composes music based on the data file. Additionally, the present invention relates to such systems and methods wherein music data files can be authored or modified by a node/subscriber unit and shared with others, preferably over a cellular or other wireless network.
A large number of distinct musical styles have emerged over the years, as have systems and technologies for creating, storing, and playing back music in accordance with such styles. Music creation, particularly of any quality, typically has been limited to persons who have musical training or who have expended the time and energy required to learn and play one or more instruments. Systems for creating and storing quality musical compositions have tended towards technologies that utilize significant computer processing and/or data storage. More recent examples of such technologies include compact disc (CD) audio players and players of compressed files (for instance as per the MPEG-level 3 standard), etc. Finally, there exist devices incorporating a tuner, which permit reception of radio broadcasts via electromagnetic waves, such as FM or AM radio receivers.
Electronics and computer-related technologies have been increasingly applied to musical instruments over the years. Musical synthesizers and other instruments of increasing complexity and musical sophistication and quality have been developed, a “language” for conversation between such instruments has been created, which is known as the MIDI (Musical Instrument Digital Interface) standard. While MIDI-compatible instruments and computer technologies have had a great impact on the ability to create and playback or store music, such systems still tend to require substantial musical training or experience, and tend to be complex and expensive.
A sound generator system can incorporate samples of existing sounds that are played in combination with interactively generated sounds. As an example, a portable music generation product can preferably be used to interactively generate music according to certain musical rules. It is preferable to also enable the use of pre-recorded sound samples to facilitate a more compelling musical experience for the user.
One problematic aspect of supporting the use of pre-recorded sound samples is that the playback of the sample during a section of music can sometimes sound out of sync with the music in terms of pitch or rhythm. This is a result of the lack of a default synchronization between the sample and the music at a particular point in time. One way around this is to use samples that do not have a clear pitch or melody, e.g., a talking voice, or a sound effect. However, as the use of melodic samples, especially at higher registers, is desirable in many styles of music, it is desirable in certain cases to have a means for associating pitch and/or periodicity information (embedded or otherwise) into a sample.
Broadcast music distribution historically has involved the real-time streaming of music over the airwaves using an FM or AM broadcasting channel. Similarly, the Internet has been used for audio streaming of music data in an approximately real time manner. Both of these examples involve steadily sending relatively large amounts of data, and consume relatively large amounts of the available bandwidth. The number of music styles and the amount of bandwidth required to make effective use of these systems have limited the usefulness of these approaches to a broad range of new products incorporating wireless computing resources (e.g., cellular telephones and/or personal data assistants (PDAs)). In addition, the limitations of these approaches to music distribution make it inordinately difficult to enable a node/subscriber unit to share music, either as part of the radio-type distribution of music, or with other node/subscriber units directly, and in particular music that has been authored or modified by a user of the node/subscriber unit.
Furthermore, it is often the case that a file format suitable for storing information associated with the presently discussed inventions does not provide an optimized set of features. As one example, forward and/or backward compatibility is often not achievable, resulting in a music file that cannot be effectively used by a system with a different version than the system that created it. File formats that do provide some level of forwards and/or backwards compatibility often incorporate overhead (e.g., multiple versions of ‘same’ data) that may be undesirable, e.g., in certain preferred embodiments that are portable and that therefore have relatively limited resources.
In the field of the present invention it is difficult to provide high quality audio synthesis in an environment with relatively limited processing resources. Typically high quality audio synthesis may involve a specialized DSP chip that consumes power, and adds significantly to the cost of the overall system. For example, in a cellular telephone that provides MIDI-based ringtones, typically a specialized MIDI DSP is incorporated that may add to the overall cost of development and materials of the system, as well as typically having an adverse impact on the battery life of the product. Furthermore, in many cases such a system may not provide high quality audio synthesis, notwithstanding the specialized DSP hardware.
In the field of the present invention it is difficult to provide a high quality MIDI sound bank in a reduced memory size associated with portable applications (e.g., cellular telephones, etc.). Typically, to get high quality sounds using a MIDI synthesis processor, a MIDI sound bank with a relatively large memory area is required. In certain portable applications, such a relatively large memory area is highly undesirable, as it sharply reduces the number of MIDI instruments available, and in certain cases, the quality of the MIDI sounds.
Accordingly, it is an object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music employing a top-down process, where the systems/methods provide the user with a musical composition that may be modified and interacted with and played and/or stored (for later play) in order to create music that is desired by the particular user.
It is another object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music that enables a user to quickly begin creating desirable music in accordance with one or a variety of musical styles, with the user modifying an auto-composed or previously createdmusical composition, either for a real time performance and/or for storing and subsequent playback.
It is another object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music in which a graphical interface is provided to facilitate use of the system and increase user enjoyment of the system by having graphic information presented in a manner that corresponds with the music being heard or aspects of the music that are being modified or the like; it also is an object of the present invention to make such graphic information customizable by a user.
It is another object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music in which a graphical interface is provided that presents a representation of a plurality of musical lanes, below each of which is represented a tunnel, in which a user may modify musical parameters, samples or other attributes of the musical composition, with such modifications preferably being accompanied by a change in a visual effect.
It is another object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music in which music may be represented in a form to be readily modified or used in an auto-composition algorithm or the like, and which presents reduced processing and/or storage requirements as compared to certain conventional audio storage techniques.
It is another object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music in which music may be automatically composed in a variety of distinct musical styles, where a user may interact with auto-composed music to create new music of the particular musical style, where the system controls which parameters may be modified by the user, and the range in which such parameters may be changed by the user, consistent with the particular musical style.
It is another object of the present invention to provide systems and methods for using pre-existing music as input(s) to an algorithm to derive music rules that may then be used as part of a music style in a subsequent auto-composition process.
It is another object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music based on efficient song structures and ways to represent songs, which may incorporate or utilize pseudo-random/random events in the creation of musical compositions based on such song structures and ways to represent songs.
It is another object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music in which songs may be efficiently created, stored and/processed; preferably songs are represented in a form such that a relatively small amount of data storage is required to store the song, and thus songs may be stored using relatively little data storage capacity or a large number of songs may be stored in a given data storage capacity, and songs may be transmitted such as via the Internet using relatively little data transmission bandwidth.
It is another object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music in which a modified MIDI representation of music is employed, preferably, for example, in which musical rule information is embedded in MIDI pitch data, musical rules are applied in a manner that utilize relative rhythmic density and relative mobility of note pitch, and in which sound samples may be synchronized with MIDI events in a desirable and more optimum manner.
It is another object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music in which a hardware/software system preferably includes a radio tuner so that output from the radio tuner may be mixed, for example, with auto-composed songs created by the system, which preferably includes a virtual radio mode of operation; it also is an object of the present invention to provide hardware that utilizes non-volatile storage media to store songs, song lists and configuration information, and hardware that facilitates the storing and sharing of songs and song lists and the updating of sound banks and the like that are used to create musical compositions.
It is another object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music that works in conjunction with a companion PC software program that enables users to utilize the resources of a companion PC and/or to easily update and/or share Play lists, components of songs, songs, samples, etc.
It is another object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music in which songs may be generated, exchanged and disseminated, preferably or potentially on a royalty free basis.
It is another object of the present invention to provide systems and methods for creating, modifying, interacting with and/or playing music that may be adapted to a variety of applications, systems and processes in which such music creation may be utilized.
It is another object of the present invention to provide systems and methods for automatically generating a human vocal track as part of a musical piece that is being algorithmically generated.
It is another object of the present invention to provide systems and methods for improved sample format and control functions, preferably to enable the general conformance of a sound sample to the current pitch and/or rhythmic characteristics of a musical piece.
It is an object of the present invention to provide systems and methods for distributing, broadcasting, and/or sharing music employing a node-based music generation process, where the systems/methods enable the user to receive (via the node/subscriber unit) and/or author or modify a data file from which the music may be composed.
It is an object of the present invention to enable music data to be broadcast or transmitted over a cellular or other wireless network.
It is an object of the present invention to provide an efficient backward and/or forward compatible file format. The advantages of such a file format may be of particular benefit when used in association with certain of the preferred embodiments disclosed herein.
It is an object of the present invention to provide high quality audio synthesis in a portable environment (e.g., such as a cellphone, personal digital assistant, handheld video game, etc.), where quality is desired, and processing resources are limited.
Finally, it is an object of the present invention to provide a high quality MIDI sound bank with a relatively low memory size area or footprint.
The present invention addresses such problems and limitations and provides systems and methods that may achieve such objects by providing hardware, software, musical composition algorithms and a user interface and the like (as hereinafter described in detail) in which users may readily create, modify, interact with and play music. In a preferred embodiment, the system is provided in a handheld form factor, much like a video or electronic game. A graphical display is provided to display status information, graphical representations of musical lanes or components, which preferably vary in shape, color or other visual attribute as musical parameters and the like are changed for particular instruments or musical components such as a microphone input, samples, etc. The system preferably operates in a variety of modes such that users may create, modify, interact with and play music of a desired style, including an electronic DJ (“e-DJ”) mode, a virtual radio mode, a song/song list playback mode, sample create/playback mode and a system mode, all of which will be described in greater detail hereinafter.
Preferred embodiments employ a top-down process, where the system provides the user with in effect a complete musical composition, basically a song, that may be modified and interacted with and played and/or stored (for later play) in order to create music that is desired by the particular user. Utilizing an auto-composition process employing musical rules and preferably a pseudo random number generator, which may also incorporate randomness introduced by timing of user input or the like, the user may then quickly begin creating desirable music in accordance with one or a variety of musical styles, with the user modifying the auto-composed (or previously created) musical composition, either for a real time performance and/or for storing and subsequent playback.
A graphical interface preferably is provided to facilitate use of the system and increase user enjoyment of the system by having graphic information presented in a manner that corresponds with the music being heard or aspects of the music that are being modified or the like. An LCD display preferably is used to provide the graphical user interface, although an external video monitor or other display may be used as an addition or an alternative. In preferred embodiments, such graphic information is customizable by a user, such as by way of a companion software program, which preferably runs on a PC and is coupled to the system via an interface such as a USB port. For example, the companion software program may provide templates or sample graphics that the user may select and/or modify to customize the graphics displayed on the display, which may be selected and/or modified to suit the particular user's preferences or may be selected to correspond in some manner to the style of music being played. In one embodiment, the companion software program provides one or more templates or sample graphics sets, wherein the particular template(s) or sample graphic set(s) correspond to a particular style of music. With such embodiments, the graphics may be customized to more closely correspond to the particular style of music being created or played and/or to the personal preferences of the user.
The graphical interface preferably presents, in at least one mode of operation, a visual representation of a plurality of musical lanes or paths corresponding to components (such as particular instruments, samples or microphone input, etc.). In addition to allowing the user to visualize the various components of the musical composition, through user input (such as through a joystick movement) the user may go into a particular lane, which preferably is represented visually by a representation of a tunnel. When inside of a particular tunnel, a user may modify musical parameters, samples or other attributes of the musical composition, with such modifications preferably being accompanied by a change in a visual effect that accompany the tunnel.
In accordance with preferred embodiments, music may be automatically composed in a variety of distinct musical styles. The user preferably is presented with a variety of pre-set musical styles, which the user may select. As a particular example, in e-DJ mode, the user may select a particular style from a collection of styles (as will be explained hereinafter, styles may be arranged as “style mixes” and within a particular style mix one or more particular styles, and optionally substyles or “microstyles”). After selection of a particular style or substyle, with a preferably single button push (e.g., play) the system begins automatically composing music in accordance with the particular selected style or substyle. Thereafter, the user may interact with the auto-composed music of the selected style/substyle to modify parameters of the particular music (such as via entering a tunnel for a particular component of the music), and via such modifications create new music of the particular musical style/substyle. In order to facilitate the creation of music of a desirable quality consistent with the selected style/substyle, the system preferably controls which parameters may be modified by the user, and the range over which such parameters may be changed by the user, consistent with the particular musical style/substyle. The system preferably accomplishes this via music that may be represented in a form to be readily modified or used in an auto-composition algorithm or the like. The musical data representation, and accompanying rules for processing the musical data, enable music to be auto-composed and interacted with in a manner that presents reduced processing and/or storage requirements as compared to certain conventional audio storage techniques (such as CD audio, MP3 files, WAV files, etc.).
In accordance with certain embodiments, pre-existing music may be used as input(s) to an algorithm to derive music rules that may then be used as part of a music style in a subsequent auto-composition process. In accordance with such embodiments, a style of music may be generated based on the work of an artist, genre, time period, music label, etc. Such a style may then be used as part of an auto-composition process to compose derivative music.
In accordance with certain embodiments, the system operates based on efficient song structures and ways to represent songs, which may incorporate or utilize pseudo-random/random events in the creation of musical compositions based on such song structures and ways to represent songs. Songs may be efficiently created, stored and/processed, and preferably songs are represented in a form such that a relatively small amount of data storage is required to store the song. Songs may be stored using relatively little data storage capacity or a large number of songs may be stored in a given data storage capacity, and songs may be transmitted such as via the Internet using relatively little data transmission bandwidth. In preferred embodiments, a modified MIDI representation of music is employed, preferably, for example, in which musical rule information is embedded in MIDI pitch data, and in which sound samples may be synchronized with MIDI events in a desirable and more optimum manner.
The system architecture of preferred embodiments includes a microprocessor or microcontroller for controlling the overall system operation. A synthesizer/DSP is provided in certain embodiments in order to generate audio streams (music and audio samples, etc.). Non-volatile memory preferably is provided for storing sound banks. Preferably removable non-volatile storage/memory preferably is provided to store configuration files, song lists and samples, and in certain embodiments sound bank optimization or sound bank data. A codec preferably is provided for receiving microphone input and for providing audio output. A radio tuner preferably is provided so that output from the radio tuner may be mixed, for example, with auto-composed songs created by the system, which preferably includes a virtual radio mode of operation. The system also preferably includes hardware and associated software that facilitates the storing and sharing of songs and song lists and the updating of sound banks and the like that are used to create musical compositions.
In alternative embodiments, the hardware, software, musical data structures and/or user interface attributes are adapted to, and employed in, a variety of applications, systems and processes in which such music creation may be utilized.
In certain embodiments, the present invention involves improved systems and methods for formatting and controlling the playback of pre-recorded samples during the algorithmic generation of music. At least certain of the benefits of the present invention preferably can be achieved through the use of pitch and/or rhythmic characteristic information associated with a given sample. Preferably, such information can optionally be used during the playback of a sample in music, as part of a process that preferably involves using DSP-functionality to alter the playback of the sample, preferably to enable the progression of rhythm and/or pitch of the sample to more desirably conform to the music.
In accordance with certain preferred embodiments of the present invention, the problem of pre-recorded sound samples sounding out of sync with the music in terms of pitch or rhythm can be substantially addressed, without limiting the samples to those that do not have a clear pitch or melody, e.g., a talking voice, or a sound effect. As the use of melodic samples, especially at higher registers, is desirable in many styles of music, even algorithmic or other autocomposed music, it is desirable to have the ability to associate pitch and/or periodicity information (embedded or otherwise) into a sample. Such information can then be interpreted by the musical rules and/or algorithm of the music device to enable a synchronization of the sample to the particular pitch, melody, and/or periodic characteristics of the musical piece.
In accordance with certain preferred embodiments of the present invention, problems associated with broadcast music are addressed by providing systems and methods for broadcasting music, and more particularly systems and methods employing data-file-based distribution, in which at least portions of the music can be generated by a node/subscriber unit upon reception of a data file, which is processed using a music generation system, which preferably composes music based on the data file. The present invention preferably makes use of node-based music generation. By incorporating the generation of the music into a node/subscriber unit, the bandwidth-intensive techniques of the prior art can be avoided. Consequently, the bandwidth also can be used for things such as node-to-node and node-to-base music data transmission features. For example, the node may create or modify a previously received or generated data file from which music may be generated, and the data file created or modified data file may be transmitted from the node to another node, or from the node to a base station, where it may be broadcast or transmitted to one or a plurality of nodes. The present invention is characterized by a relatively small data file transmission that contains various parameters sufficient to describe or define the music that subsequently will be generated. Such a file is then received and used by one or more node/subscriber units to render the music using various music generation and signal processing functions.
In accordance with presently preferred embodiments of the present invention, problems associated with audio synthesis in a portable environment are addressed by providing systems and methods for performing audio synthesis in a manner that preferably simplifies design requirements, minimizes cost, while preferably providing quality audio synthesis features targeted for a portable system (e.g., portable telephone, personal digital assistant, portable video game, etc.).
In accordance with presently preferred embodiments of the present invention, problems associated with the tradeoff of quality and memory size in a MIDI sound bank are addressed by providing systems and methods for providing a MIDI sound bank that is optimized for a relatively low memory size application (e.g., portable telephone, personal digital assistant, portable video game, etc.).
Such aspects of the present invention will be understood based on the detailed description to follow hereinafter.
The above objects and other advantages of the present invention will become more apparent by describing in detail the preferred embodiments of the present invention with reference to the attached drawings in which:
The present invention will be described in greater detail with reference to certain preferred and certain other embodiments, which may serve to further the understanding of preferred embodiments of the present invention. As described elsewhere herein, various refinements and substitutions of the various elements of the various embodiments are possible based on the principles and teachings herein.
In accordance with the present invention, music may be created (including by auto-composition), interacted with, played and implemented in a variety of novel ways as will be hereinafter described via numerous exemplary preferred and alternative embodiments. Included in such embodiments are what may be considered as top-down approaches to musical creation. Top-down as used herein generally means that a complete song structure for quality music is created for the end user as a starting point. This enables the user to immediately be in position to create quality music, with the user then having the ability to alter, and thereby create new music, based on the starting point provided by the system. Where a particular user takes the music creation process is up to them. More conventional musical creation processes involve a bottom-up approach, wherein the rudiments of each instrument and musical Style are learned, and then individual notes are put together, etc. This conventional approach generally has the side-effect of limiting the musical creation to a small group of trained people, and has, in effect, barred the wider population from experiencing the creative process with music.
A useful analogy for purposes of understanding embodiments of the present invention is that of building a house. In the conventional means of house-building, the user is given a bunch of bricks, nails, wood, and paint. If you want a house, you need to either learn all the intricacies of how to work with each of these materials, as well as electrical wiring, plumbing, engineering, etc., or you need to find people who are trained in these areas. Similarly, in musical creation, if you want a song (that is pleasing), you need to learn all about various types of musical instruments (and each of their unique specialties or constraints), as well as a decent amount of music theory, and acquire a familiarity with specific techniques and characteristics in a given Style of music (such as techno, jazz, hip-hop, etc.).
It would, of course, be far more convenient if, when someone wanted a house, they were given a complete house that they could then easily modify (with the press of a button). For example, they could walk into the kitchen and instantly change it to be larger, or a different color, or with additional windows. And they could walk into the bathroom and raise the ceiling, put in a hot tub, etc. They could walk into the living room and try different paint schemes, or different furniture Styles, etc. Similarly, in accordance with embodiments of the present invention, the user desirably is provided with a complete song to begin with, they can then easily modify, at various levels from general to specific, to create a song that is unique and in accordance with the user's desires, tastes and preferences.
In accordance with the present invention, the general population of people readily may be provided with an easy approach to musical creation. It allows them the immediate gratification of a complete song, while still allowing them to compose original music. This top down approach to musical creation opens the world of musical creativity to a larger group of people by reducing the barriers to creating pleasurable music.
In accordance with the present invention, various systems and methods are provided that enable users to create music. Such systems and methods desirably utilize intuitive and easy to learn and to use user interfaces that facilitate the creation of, and interaction with, music that is being created, or was created previously. Various aspects of one example of a preferred embodiment for a user interface in accordance with certain preferred embodiments of the present invention will now be described.
In accordance with such preferred embodiments of the present invention, user interface features are provided that desirably facilitate the interactive generation of music. The discussion of such preferred embodiments to be herein after provided are primarily focused on one example of a handheld, entry-level type of device, herein called ‘Player’. However, many of the novel and inventive features discussed in connection with such a Player relate to the visual enhancement of the control and architecture of the music generation process; accordingly they can apply to other types of devices, such as computing devices, web server/websites, kiosks, video, or other electronic games and other entertainment devices that allow music creation and interaction, and thus also may benefit from such aspects of the present invention. A discussion of certain of the other types of devices is provided hereinafter. As will be appreciated by one of ordinary skill in the art, various features of the user interface of the Player can be understood to apply to such a broader range of devices.
Generally, the goal of the user interface is to allow intuitive, simple operation of the system and interaction with various parameters with a minimum number of buttons, while at the same time preserving the power of the system.
In accordance with preferred embodiments, a Home mode is provided. Home mode is a default mode that can be automatically entered when Player 10 is turned on. As the example of
Play can be used when in Home mode to enter a particularly important visual interface mode referred to herein as the I-Way mode (discussed in greater detail below). As shown in the example of
An important feature of Home mode is the ability to configure Player 10 to start playing music quickly and easily. This is because, although Player 10 is configured to be interactive, and many professional-grade features are available to adjust various aspects of the Style and sound, it is desirable to have a quick and easy way for users to use the Player in a ‘press-it-and-forget-it’ mode. Thus, with only very few button pushes, a user with little or no musical experience, or little or no experience with Player 10, may easily begin composing original music with Player 10 of a desired Style or SubStyle. An additional preferred way to provide an auto-play type of capability is to use a removable storage memory medium (e.g., Smart Media Card) to store a Play list, such as a file containing a list of song data structures that are present on the removable memory. Following this example, when the user inserts the removable memory, or when the system is powered on with a removable memory already inserted, preferably the system will scan the removable memory to look for such a file containing a Play list and begin to play the song data structures that are listed in the system file. Preferably, this arrangement can be configured such that the Auto-Play mode is selectable (such as via a configuration setting in the system file), and that the system will wait a short duration before beginning Auto-Play, to allow the user an opportunity to enter a different mode on the system if so desired.
As illustrated in
In alternative embodiments, e.g., as shown in
While in I-Way mode, the screen preferably is animated with sound waves or pulses synchronized with music beats. In the example of
In an auto composition mode such as the I-Way mode it is Player 10 itself preferably that decides about a song progression in that it can automatically add/remove instruments, do music breaks, drums progressions, chord progressions, filtering, modulation, play samples in sync with the music, select samples to play based on rules, etc., to end up sounding like in a real song on a CD or from the radio. After a few minutes, if nothing is done by the user, Player 10 preferably is configured to end the song, preferably with an automatic fade out of volume, and automatically compose and play a new song in the same Style, or alternatively a different Style. It also should be understood that I-Way mode also is applicable in preferred embodiments for music that is not auto-composed, such as a song that the user created/modified using Player 10 (which may have been created in part using auto-composition) and stored in Player 10 for subsequent playback, etc.
In certain embodiments, newly composed patterns are numbered from 1 to n. This number can be displayed in the status line to help the user remember a music pattern he/she likes and come back to it after having tried a few other ones. In certain embodiments, this number might only be valid inside a given song and for the current interactive session. In other words, for example, the Riff pattern number 5 for the current song being composed would not sound like the Riff pattern number 5 composed in another song. However, if this song is saved as a user song, although the Riff music will be the same when replayed later, the number associated to it could be different.
In one exemplary embodiment, Player 10 “remembers” up to 16 patterns previously composed during the current interactive session. This means, for example, that if the current pattern number displayed is 25, the user can listen to patterns from number 10 to 25 by browsing forward through the previously composed patterns (patterns 1-9, in this embodiment, having been overwritten or otherwise discarded). If the User wants to skip a given composed pattern that is currently being played, he/she can, and the pattern number will not be incremented, meaning that currently played pattern will be lost. This feature can be used to store only specific patterns in the stack of previously played patterns, as desired by the user. What is important is that the user can create musical patterns, and selectively store (up to some predetermined number of musical patterns), with the stored patterns used to compose music that is determined by the user based on the user's particular tastes or desires, etc. The views presented by I-Way mode desirably facilitate this user creation and interaction with, and modification of, the music that is be created/played by Player 10.
In certain preferred embodiments, if desired by a user, additional music parameters of an instrument associated with a particular lane in the I-Way mode may be “viewed” and interacted with by the user. For example, if a Down is pressed (such as by way of joystick 15) while in I-Way mode, the center of view is taken “underground,” to the “inside” of a particular lane (e.g., see
In certain embodiments it is preferable to provide a force-feedback mechanism (as discussed below in connection with
The far end of the tunnel preferably is comprised of a shape (for example, a rectangle or other geometric) that can change in correlation to the value of one or more of the parameters affecting the sound of that particular lane. By way of example, in the case of drums, a filter parameter can be changed by depressing the function or Fx button (see, again
While in Underground mode, Player 10 preferably is configured to continue looping with the same musical sequence while the user is able to interact with and modify the specific element (e.g., the drums) using the joystick and other buttons of Player 10. Also, while down in a lane corresponding to a particular component, preferably the left and right buttons of the joystick can be used to move from one component parameter to another. Alternatively, side to side joystick movements, for example, may enable the user to step through a series of preset characteristics or parameters (i.e., with simple joystick type user input, the user may change various parameters of the particular component, hear the music effect(s) associated with such parameter changes, and determine desirable characteristics for the particular music desired by the user at the particular point in time, etc.). In yet another alternative, side to side joystick movements, for example, may cause the view to shift from one tunnel to an adjacent tunnel, etc. All such alternatives are within the scope of the present invention.
In addition to other similar variations, the user can mute a particular lane in the I-Way mode preferably by use of Stop key (shown in
An additional desirable variation of the user interface preferably involves animating a change to the visual appearance, corresponding to a new song part. For example, if in the Underground mode shown in
In certain alternative embodiments, it is preferable to provide multiple layers of tunneling, as shown in
Alternatives to the I-Way and Underground concepts can also be advantageously used with the present invention. For example, a user interface that visually depicts the instruments that are in the current song, and allows the user to select one to go into a tunnel or level where parameters of the particular instrument may be adjusted. In this example, while the music is playing, the user interface provides visual representations of the instruments in the current song, with the active instruments preferably emitting a visual pulse in time with the music.
As a particular example of such an embodiment,
In certain embodiments, both or multiple types of user interfaces are provided, and the user may select an I-Way type of user interface, such as shown in
Additionally, in certain preferred embodiments, the use of an external video display device (e.g., computer monitor, television, video projector, etc.) is used to display a more elaborate visual accompaniment to the music being played. In such cases the I-Way graphical display preferably is a more detailed rendition of the I-Way shown in
In certain preferred embodiments, pressing Play preferably causes the lane instrument to enter Forced mode. This can be implemented to force Player 10 to play this instrument pattern at all times until Forced mode is exited by pressing Play again when the lane of that instrument is active. In this case, if the instrument was not playing at the time Forced mode is selected, Player 10 can be configured to automatically compose the instrument pattern and play it immediately, or starting at the end of the current sequence (e.g., 2 bars). In addition, pressing Play for a relatively long period (e.g., a second or more) can pause the music, at which time a “paused” message can flash in the status line.
In other preferred embodiments, where such a Forced mode may not be desired (e.g., for simplicity, and/or because it may not be needed for a particular type of music), pressing Play briefly preferably causes a Pause to occur. Such a pause preferably would have a ‘Paused’ message appear on the Display 20, and preferably can be rhythmically quantized such that it begins and ends in musical time with the song (e.g., rhythmically rounded up or down to the nearest quarter note).
In Solo mode, all other instruments are muted (except for those that may already be in Solo mode) and only this instrument is playing. Solo mode preferably is enabled by entering a tunnel or other level for a particular instrument, and, if the instrument is already playing entering Solo mode upon pressing of Play (e.g., the instrument is in Forced play and subsequent pressing of Play in Underground mode initiates Solo mode for that instrument; the particular key entry into Solo mode being exemplary). An instrument preferably remains soloed when leaving the corresponding tunnel and going back to the music I-Way. The user also preferably must re-enter the corresponding tunnel to exit Solo mode. Also, in certain embodiments multiple levels of Solo mode are possible in that you can solo several tracks, one at a time or at the same time, by going into different tunnels and enabling Solo mode. In addition, in certain embodiments the user preferably can enable/disable Solo mode from the I-Way by, for example, pressing Play for a long time (e.g., 2 seconds) while in a lane. Following this example, upon disabling Solo mode, any lanes that had previously been manually muted (before Solo mode was invoked) preferably will remain muted.
Preferably, from a Sample menu different sample parameters can be edited. From the Samples menu, the user can record, play and change effects on voice, music or sound samples. This menu also preferably permits the creation and edition of sample lists. The LCD preferably displays “e.Samples” in the status line and a list of available samples or sample lists in the storage media (for example, the SmartMedia card, discussed in connection with
When playing back a sample, the LCD preferably displays the play sample screen. The name of the sample preferably scrolls in a banner in the center right part of the LCD while the audio output level is indicated by a sizable frame around the name. The status line preferably shows the current effect.
Sample sets or lists preferably are used by the e.DJ, for user songs, as well as MIDI files. In the case of MIDI files, preferably a companion PC software program (e.g., a standard MIDI editing software program such as Cakewalk) is used to enable the user to edit their own MDI files (if desired), and use MIDI non-registered parameter numbers (NRPNs are discussed below in more detail) to effectuate the playing of samples at a specific timing point. Following this example, the companion PC software program can be enabled to allow the user to insert samples into the MIDI data, using NRPNs. When a new e.DJ song is created, Player 10 preferably picks one of the existing sample lists (sample sets preferably being associated with the particular Style/SubStyle of music) and then plays samples in this list at appropriate times (determined by an algorithm, preferably based on pseudo random number generation, as hereinafter described) in the song. When creating or editing a user song, the user preferably can associate a sample list to this user song. Then; samples in this list will be inserted automatically in the song at appropriate times. Each sample list can be associated with an e.DJ music Style/SubStyle. For instance, a list associated with the Techno Style can only be used by a Techno user song or by the e.DJ when playing Techno Style. In additional variations, the user preferably can specify specific timing for when a particular sample is played in a song, by way of NRPNs discussed below. This specification of the timing of a particular sample preferably can be indicated by the user through the use of a companion PC software program (e.g., a standard MIDI editing software program such as Cakewalk), and/or through a text interface menu on the Player 10 itself.
New Sample lists preferably are created with a default name (e.g., SampleList001). The list preferably can be renamed in the System-files menu. When the selected item is a sample, the current effect preferably is displayed in the status line. When the selected item is a sample list, “List” preferably is displayed in the status line.
Playback of preferably compressed audio, MIDI, Karaoke, and User songs (e.g., e.DJ songs that have been saved) preferably is accessible via the “Songs” mode. Songs can be grouped in so-called Play lists to play programs (series) of songs in sequence. The LCD preferably will display “e.Songs” in the status line and a list of available songs or Play lists on the SmartMedia card to choose from.
Depending on the type of the song (for example, user song, MIDI or WMA), different parameters can be edited. The type of the current selection preferably is indicated in the status bar: e.g., WMA (for WMA compressed audio), MID (for MIDI songs), KAR (for MIDI karaoke songs), MAD x (for user songs (x=T for Team( )Style, x=H for Hip-Hop, x=K for Cool, etc.)), and List (for Play lists).
The name of the song preferably scrolls in a banner in the center right part of the LCD while the audio output level is indicated by a sizable frame around the name. If the song is a karaoke song, the lyrics preferably are displayed on two (or other number) lines at the bottom of the LCD. The animated frame preferably is not displayed. If the song is a user song (i.e., composed by the e.DJ and saved using the Save/Edit button), the music I-Way mode is entered instead of the play song mode.
The edit screen preferably is then displayed, showing two columns; the left column lists the editable parameters or objects in the item, the right column shows the current values of these parameters. For example, a Play list edit screen preferably will display slot numbers on the left side and song names on the right side. The current object preferably is highlighted in reverse video.
Play lists are used to create song programs. New Play lists are preferably created with a default name (e.g., PlayList001), and preferably can be renamed by the user. When a list is selected and played in the song select screen, the first song on the list will begin playing. At the end of the song, the next song preferably will start and so on until the end of the list is reached. Then, if the terminating instruction in the list is End List, the program preferably stops and Player 10 returns to the song select screen. If the terminating instruction is Loop List, the first song preferably will start again and the program will loop until the user interrupts the song playing, such as by pressing the stop button.
In one embodiment of the present invention, the features of a conventional radio are effectively integrated into the user interface of the present invention (see, e.g., the FM receiver 50 of
In preferred embodiments, radio-type functionality involves the use of the same type of Radio interface, with virtual stations of different Styles. Each virtual station preferably will generate continuous musical pieces of one or more of a particular Style or SubStyle. In this v.Radio mode, the user can “tune-in” to a station and hear continuous music, without the use of an actual radio. Such an arrangement can provide the experience of listening to a variety of music, without the burden of hearing advertising, etc., and allows the user to have more control over the Style of music that is played. In such embodiments, a user will enter v.Radio mode and be presented with a list of v.Radio stations, each preferably playing a particular Style or SubStyle of music. The user then preferably “tunes” to a v.Radio channel by selecting a channel and pressing play, for example (see, e.g.,
In accordance with certain embodiments, another variation of the Radio feature integrates some aspects of the v.Radio with other aspects of the Radio. As one example, a user could listen to a Radio station, and when a commercial break comes on, Player 10 switches to the v.Radio. Then, when the real music comes back on, the device can switch back to a Radio. Another integration is to have news information from the Radio come in between v.Radio music, according to selectable intervals. For example, most public radio stations in the USA have news, weather, and traffic information every ten minutes during mornings and afternoons. The v.Radio can be configured to operate as a virtual radio, and at the properly selected interval, switch to a public station to play the news. Then it can switch back to the v.Radio mode. These variations provide the capability for a new listening experience, in that the user can have more control over the radio, yet still be passively listening. It is considered that such an arrangement would have substantial use for commercial applications, as discussed elsewhere in this disclosure.
Special functions can preferably be accessed from the System menu. These functions preferably include: file management on the SmartMedia card (rename, delete, copy, list, change attributes) (the use of such SmartMedia or other Flash/memory/hard disk type of storage medium is discussed, for example, in connection with
In certain embodiments a User Configuration interface preferably enables the user to enter a name to be stored with the song data on the removable memory storage (e.g., SMC), and/or to enable the user to define custom equalization settings, and/or sound effects. As an example of EQ settings, it is preferable to enable the user to select from a group of factory preset equalizer settings, such as flat (e.g., no EQ effect), standard (e.g., slight boost of lower and higher frequencies), woof (e.g., bass frequency boost), and hitech (e.g., high frequency boost). In addition to such preset EQ settings, it is preferable to enable the user to define their own desired settings for the EQ (as an example, a 4 band EQ with the ability to adjust each of the 4 bands by way of the joystick). Additionally, in certain embodiments it is preferable to enable the user to similarly customize sound effects to be used for particular samples. Following this example, in addition to a set of standard factory preset sound effects such as Lowvoice (e.g., plays the song with a slower speed and lower pitch to enable the user to sing along with a lower voice), reverb, Highvoice (e.g., plays the song with a faster speed and higher pitch), Doppler (e.g., varying the sound from Highvoice to Lowvoice), and Wobbler (e.g., simulating several voices with effects), it is preferable to make a customized effect capability available to the user that can incorporate various combinations of standard effects, and in varying levels and/or with varying parameter values. Furthermore, in certain embodiments it is preferable to use an equalizer as an additional filter and/or effect, e.g., an ‘AM Radio’ effect that simulates the equalization of an AM Radio station.
When the user saves a song that is being played in e-DJ mode, the song is preferably created with a default name (e.g. TECHNO001). The song can preferably be renamed in the System-files menu. When entering the Files menu, files present on the SmartMedia card and the free memory size are preferably listed in an edit screen format. The status line preferably indicates the number of files and the amount of used memory. The file management menu preferably offers a choice of actions to perform on the selected file: delete, rename, copy, change attributes, etc. The name of the current file preferably is displayed in the status line. Additionally, in certain embodiments it is preferable to enable the use of System parameter files that contain, for example, settings for radio presets (e.g., radio station names and frequencies), settings for certain parameters (e.g., pitch, tempo, volume, reverb, etc.) associated with music files such as WAV, WMA, MP3, MIDI, Karaoke, etc. In these embodiments it is preferable for the parameter setting to apply to the entire file.
When entering the Configuration menu, an edit screen preferably is displayed showing various configurable parameters.
With regard to
When selecting copy, a screen proposing a name for the destination file in a large font preferably is displayed. This name preferably is proposed automatically based on the type of the source file. For instance if the source file is a Hiphop user song, the proposed name for the destination file could be HIPHO001 (alternatively, the user preferably can use the rename procedure described above to enter the name of the destination file).
The Firmware Upgrade menu preferably permits the upgrade of the Player firmware (embedded software) from a file stored on the SmartMedia card. Preferably, it is not possible to enter the Upgrade firmware menu if no firmware file is available on the SmartMedia card. In this case a warning message is displayed and the Player preferably returns to Systems menu. In additional embodiments, the use of a bootstrap program preferably is enabled to allow the firmware to be updated from a removable memory location (e.g., SMC). Such a bootstrap program preferably can alternatively be used for upgrading the DSP 42 soundbank located in Flash 49.
The Player function keys, identified in
In certain embodiments, one or more of the user interface controls are velocity-sensitive (e.g., capable of measuring some aspect of speed and/or force with which the user presses the control. As an example, one or more of the Player function keys can detect such velocity-type of information and incorporate it into the sounds. In these exemplary embodiments, if the Play button is being used to trigger a sample, it is preferable to incorporate the velocity-type information derived from the button press into the actual sound of the sample. In some embodiments, the result preferably is that the sample will sound louder when the button is pressed quickly and/or more forcefully. In other embodiments, this will preferably result in a changed sound effect, i.e., depending how hard and or forcefully the control is pressed, the resulting sound will be modulated, filtered, pitch-changed, etc., to a different degree. As will be discussed later in more detail, many of the music events involve a MIDI-type (or MIDI-similar) event descriptor, and accordingly, in certain embodiments it is preferable to use the velocity portion (and/or volume, aftertouch, etc.) of a MIDI-type descriptor to pass on the velocity and/or force characteristics of a button push to the DSP (e.g., via the generation algorithms, etc.). Clearly, separate descriptor events can alternatively be used, such as system exclusive messages or MIDI NRPN messages.
When a list is selected in the song select screen, pressing Play preferably will start playing the first song in the list. While the sample lane is selected, Play preferably can be configured to start playing the selected sample. While in an instrument lane, Play preferably can be configured to enter solo mode for the current instrument, or Forced mode.
To create a song/sample list, Forward preferably can be used while in the song or sample select screen.
To leave an edit screen, Stop preferably can be used to discard the edits and exit. For example, in the sample selection screen press Stop to go back to the Home screen. Additionally, for any given instrument during playback, Stop preferably can be used as a toggle to mute/unmute the instrument.
Record preferably can be pressed once to start recording a sample (recording samples preferably is possible in almost any operating mode of the Player). Record preferably can be pressed again to end the recording (recording preferably is stopped automatically if the size of the stored sample file exceeds a programmable size, such as 500 Kbytes). The record source preferably is chosen automatically depending on the operating mode. If no music is playing, the record source preferably is the active microphone (e.g., either local or integrated into an external docking station). If music is playing songs, e.DJ or radio, the record source preferably is a mix of the music and the microphone input if not muted. Further, it is possible to use different sample recording formats that together provide a range of size/performance options. For example, very high quality sample encoding format may take more space on the storage medium, while a relatively low quality encoding format may take less space. Also, different formats may be more suited for a particular musical Style, etc.
In certain embodiments it is preferable to support a sample edit mode. In these cases, sample edit mode is preferably used by a user to edit a selected sample. The selected sample is displayed (e.g., as a simplified waveform representation on LCD 20) and the user is able to select a start point and an end point of the sample. Preferably, such graphical clipping functions enable a user to easily crop a selected sample, e.g., so as to remove an undesired portion, etc. After clipping/cropping the sample, the user is presented with the option of saving the newly shortened sample file, e.g., with a new name. Clearly, other similar edit-type functions in addition to or besides such clipping can be supported, to make use of LCD 20 to provide a graphic version of the waveform of a selected sample, and to provide the end user with a simple way to carryout basic operations on the selected sample.
In v-Radio mode, to listen to the selected station, Play preferably can be used. Press Play to mute the radio. Press Stop to go back to station preset selection screen. To launch an automatic search of the next station up the band, press Forward until the search starts. To launch an automatic search of the next station down the band, press Backward until the search starts. Press Forward/Backward briefly to fine-tune up/down by 50 kHz steps.
In eDJ Mode, while in Sample lane, Play preferably can be pressed to play a selected sample. As mentioned previously, in certain embodiments it is preferable to detect the velocity or force of a particular button press, and to impart the detected information into the resulting sound (e.g., in the case of a sample play event, preferably adjusting the volume, aftertouch, pitch, effect, etc. of the sample sound). If sample playback had previously been disabled, the first press on Play preferably will re-enable it. Subsequent presses preferably will play the selected sample. If a sample is playing, Stop preferably will stop it. If no sample is playing, pressing Stop preferably will mute the samples (i.e. disable the automatic playback of samples by the e-DJ when returning to I-Way mode). When muted, “Muted” preferably is displayed in the status bar and the round speaker preferably disappears on the I-Way sample lane.
In Song mode, to start the playback of selected song or Play list, preferably press Play and the LCD will preferably display the play song screen. In Song mode, Stop preferably can be pressed to stop the music and preferably go back to song selection screen. Preferably press Forward briefly to go to next song (if playing a Play list, this preferably will go to the next song in the list; otherwise, this preferably will go to the next song on the SmartMedia). Preferably press Forward continuously to fast forward the song. Preferably press Backward briefly to go to the beginning of the song and a second press preferably takes you to the previous song (if playing a Play list, this preferably will go to the previous song in the list; otherwise, this preferably will go to the previous song on the SmartMedia). Preferably press Backward continuously to quickly go backward in the song.
Pressing Stop can be a way to toggle the muting of an instrument/lane. For example, when on a Drums lane, pressing Stop briefly preferably can mute the drums, and pressing it again briefly preferably can un-mute the drums. Additionally, pressing Stop for relatively long period (e.g., a second or so) preferably can be configured to stop the music and go back to Style selection screen.
Forward preferably can be configured to start a new song. Backward preferably can be used to restart the current song.
Forward or Backward preferably can be used to keep the same pattern but change the instrument playing (preferably only “compatible” instruments will be picked and played by the Player).
Preferably press Stop from within the MIC lane or tunnel to mute microphone. Preferably press Play to un-mute the microphone.
To start the playback of the selected sample, preferably press Play. Preferably press Stop to stop the sample and go back to sample selection screen.
In Song mode, preferably press Play to pause the music. Preferably press Play again to resume playback. Pressing Forward key in the song select screen preferably will create a new Play list. In the song selection screen, preferably press Stop to go back to the Home screen.
In the Style selection screen preferably press Stop to go back to the Home screen.
To enter the file management menu for the highlighted file, preferably press Play.
While browsing the file management list, preferably press Forward to scroll down to next page. Press Backward preferably to scroll up to previous page.
In the file management menu, to start a selected action, preferably press Play.
When selecting Delete, preferably a confirmation screen is displayed.
When selecting Rename, preferably a screen showing the name of the file in big font is displayed and the first character is preferably selected and blinking.
When copying a file, preferably press Play to validate the copy. If a file of the same type as the source file exists with the same name, preferably a confirmation screen asks if the file should be overwritten. Select YES or No and preferably press Play to validate. Press Stop to abort the copy and preferably return to file menu. It is a preferable feature of this embodiment to allow files to be copied from one removable memory storage location (e.g., SMC) to another by use of MP 36 RAM. In this example, it is a desirable to enable the copying of individual song or system files from one SMC to another without using a companion PC software program, however, in the case where an entire removable memory storage volume (e.g., all the contents of a particular SMC) is to be copied, it is desirable to use a companion PC software program to allow larger groups of data to be temporarily buffered (using the PC resources) by way of the USB connection to the PC. Such a feature may not be possible in certain embodiments without the PC system (e.g., using the MP 36 internal RAM) because it likely would involve the user repeatedly swapping the SMC target and source volumes.
The e-DJ, v-Radio, Songs, Samples and System direct access keys detailed in
The audio output control is identified in
The Up/Down/Left/Right keys preferably comprise a joystick that can be used for: menu navigation, song or music Style selection, and real time interaction with playing music. Additionally, Up/Down preferably can be used for moving between modes such as the Underground & I-Way modes in an intuitive manner.
When editing a list, objects preferably can be inserted or deleted by pressing Forward to insert an object after the highlighted one or pressing Backward to delete the highlighted object.
To browse the list or select parameters, preferably use Up/Down. To edit the highlighted object preferably press Right. Press Left preferably to go directly to first item in the list.
In instrument tunnels (i.e.; Drums, Bass, Riff and Lead), Right preferably can be configured to compose a new music pattern. Similarly, Left preferably can be used to return to previous patterns (see note below on music patterns). The new pattern preferably will be synchronized with the music and can start playing at the end of the current music sequence (e.g., 2 bars). In the mean time, preferably a “Composing . . . ” message can be configured to appear on the status line. Additionally, Down preferably can be used to compose a new music pattern without incrementing the pattern number. This preferably has the same effect as Right (compose and play another pattern), except that the pattern number preferably won't be incremented.
One benefit of these composition features is that they enable the user to change between patterns during a live performance. As can be appreciated, another reason for implementing this feature is that the user preferably can assemble a series of patterns that can be easily alternated. After pressing Right only to find that the newly composed pattern is not as desirable as the others, the user preferably can subsequently select Down to discard that pattern and compose another. Upon discovering a pattern that is desirable, the user preferably can thereafter use Right and Left to go back and forth between the desirable patterns. Additionally, this feature preferably allows the system to make optimum use of available memory for saving patterns. By allowing the user to discard patterns that are less desirable, the available resources preferably can be used to store more desirable patterns.
In the file management menu, to select a desired action, preferably use Up/Down. When renaming files, the user preferably can use Left/Right to select the character to be modified, and Up/Down to modify the selected character. Pressing Right when the last character is selected preferably will append a new character. The user preferably can also use the Forward/Backward player function keys at these times to insert/delete characters.
In the microphone tunnel, Left/Right preferably can be configured to change microphone input left/right balance. In the sample tunnel, Left/Right preferably can be used to select a sample. Pressing Forward in the sample select screen preferably will create a new sample list.
Down is an example of an intuitive way to enter the Underground mode for the current I-Way mode lane. In this mode, the user preferably can change the pattern played by the selected instrument (drums, bass, riff or lead) and preferably apply digital effects to it. Similarly, Up preferably can be configured to go back to music I-Way from the Underground mode.
In v-Radio mode, to select the desired station preset, preferably use Up/Down. Preferably use Up/Down to go to previous/next station in the preset list and preferably press Save/Edit while a station is playing to store it in the preset list.
The Save/Edit key preferably can be used to save the current song as a User song that can be played back later. Such a song preferably could be saved to a secondary memory location, such as the SmartMedia card. In the case of certain Player embodiments, this preferably can be done at any time while the e-DJ song is playing, as only the “seeds” that generated the song preferably are stored in order to be able to re-generate the same song when played back as a User song. In certain embodiments it is preferable to incorporate a save routine that automatically saves revised files as a new file (e.g., with the same name but a different suffix). Such a feature can be used to automatically keep earlier versions of a file.
While the use of seeds is discussed elsewhere in this disclosure, it may be helpful at this point to make an analogy on the use of the Save/Edit 17 key. This key is used to save the basic parameters of the song in a very compact manner, similar to the way a DNA sequence contains the parameters of a living organism. The seeds occupy very little space compared to the information in a completed song, but they are determinative of the final song. Given the same set of saved seeds, the Player algorithm of the present invention preferably can generate the exact same sequence of music. So, while the actual music preferably is not stored in this example (upon the use of the Save/Edit 17 key), the fundamental building blocks of the music is stored very efficiently. The desirability of such an approach can be appreciated in a system with relatively limited resources, such as a system with a relatively low-cost/low performance processor and limited memory. The desirability of such a repeatable, yet extremely compact method of storing music can also be contemplated in certain alternative embodiments, such as those involving the communication with other systems over a relatively narrow band transmission medium, such as a 56 kbps modem link to the internet, or an iRDA/bluetooth type of link to another device. Clearly this feature can be advantageously employed using other relatively low bandwidth connections between systems as well. Additionally, this feature allows the user to store many more data files (e.g., songs) in a given amount of storage, and among other advantages, this efficiency enhances other preferable features, such as the automatic saving of revised files as new files (as discussed above).
In certain embodiments, it is desirable to check the resources available to a removable memory interface (e.g., the SMC interface associated with SMC 40) to safeguard the user song in instances where a removable memory volume is not inserted, and/or there is not enough available storage on an inserted removable memory volume. In these cases, when the user saves a song (e.g., pushes the Save/Edit key 17 button) it is advantageous to prompt the user to insert an additional removable memory volume.
The name of the song preferably can be temporarily displayed in the status line, in order to be able to select this song (as a file) later on for playback. Of course the song file name preferably can be changed later on if the User wishes to do so. Once an item has been created, it preferably can be edited by selecting it in the song or sample selection screens and pressing Save/Edit. Pressing Save/Edit again will preferably save the edited item and exit. When the On/Off key is pressed for more than 2 seconds, the Player preferably can be configured to turn on or off, yet when this combination is pressed only briefly, the On/Off key can alternatively preferably be configured to turn the LCD backlight on or off.
When Pitch/Tempo is pressed simultaneously with Left or Right, it preferably can be used as a combination to control the tempo of the music. When Pitch/Tempo is pressed simultaneously with Up/Down, it preferably can control the pitch of the microphone input, the music, etc.
When Effects/Filters is pressed simultaneously with Left/Right or Up/Down, it preferably can control the effect (for example, cutoff frequency or resonance) and/or volume (perhaps including mute) applied on a given instrument, microphone input, or sample.
As will be appreciated by one of ordinary skill in the art, other related combinations can be employed along these lines to provide additional features without detracting from the usability of the device, and without departing from the spirit and scope of the present invention.
Various examples of preferred embodiments for the structuring of a song of the present invention will now be described. Preferably for a new song, the only user input needs to be an input Style. Preferably even this is not required when an auto-play feature is enabled that causes the Style itself to be pseudo-randomly selected. But assuming the user would like to select a particular Style, that is the only input preferably needed for the present embodiment to begin song generation.
Before moving into the actual generation process itself, it is important to note that preferably implicit in the user's Style selection can be a Style and a SubStyle. That is, in certain embodiments of the present invention, a Style is a category made up of similar SubStyles. In these cases, when the user selects a Style, the present embodiment will preferably pseudo-randomly select from an assortment of SubStyles. Additionally, it is preferably possible for the user to select the specific SubStyle instead, for greater control. In these particular embodiments, preferably whether the user selects a Style or a SubStyle, the result preferably is that both a Style and a SubStyle can be used as inputs to the song generation routines. When the user selects a SubStyle, the Style preferably is implicitly available. When the user selects a Style, the SubStyle preferably is pseudo-randomly selected. In these cases, both parameters are available to be used during the song generation process to allow additional variations in the final song.
As shown in
Also, in certain cases, the user experience preferably may benefit from having the display updated for a particular Part. For example, an indication of the current position within the overall length of the song may be helpful to a user. Another example is to alert the user during the ending part that the song is about to end. Such an alert preferably might involve flashing a message (i.e., ‘Ending’) on some part of the display, and preferably will remind the user that they need to save the song now if they want it saved.
Another optimization at this level is preferably to allow changes made by the user during the interactive generation of a song to be saved on a part-by-part basis. This would allow the user to make a change to an instrument type, effect, volume, or filter, etc., and have that revised characteristic preferably be used every time that part is used. As an example, this would mean that once a user made some change(s) to a chorus, every subsequent occurrence of the chorus would contain that modified characteristic. Following this particular example, the other parts of the song would contain a default characteristic. Alternatively, the characteristic modifications preferably could either be applied to multiple parts, or preferably be saved in real time throughout the length of the song, as discussed further below.
Each Part preferably can be a different length, and preferably can be comprised of a series of SubParts. One aspect of a preferred embodiment involves the SubPart level disclosed in
In certain embodiments, where a SubPart layer is implemented, each SubPart preferably can be of a different size. Such an approach can enhance the feel of the resulting musical composition, as it affords a degree of variety to the Parts.
Each SubPart preferably is comprised of a series of Sequences (SEQs). In keeping with the previous comment regarding the relationship between consistent sizing and flexibility of rule applications, each SEQ preferably can be the same length and time signature. In the example of
Following the example of
In this case, the multiple RPs preferably are merged together to comprise the SEQ. As will be recognized by those skilled in the art, this is analogous to the way a state-of-the-art MIDI sequencer merges multiple sets of MIDI Type 1 information into MIDI Type 0 file.
Further background detail on this can be found in the “General MIDI Level 2 Specification” (available from the MIDI Manufacturer's Association) which is hereby incorporated by reference.
One reason for allowing multiple RPs in parallel to define a SEQ, is that at certain times, certain lanes on the I-Way may benefit from the use of multiple RPs. This is because it may be desirable to vary the characteristics of a particular piece of the music at different times during a song. For example, the lead preferably may be different during the chorus and the solo. In this case it may be desirable to vary the instrument type, group, filtering, reverb, volume, etc., and such variations can be enacted through the use of multiple RPs. Additionally, this method can be used to add/remove instruments in the course of play. Of course, this is not the only way to implement such variations, and it is not the only use for multiple RPs.
Following the example of
Generally, it is cumbersome to allow notes to be held over multiple RPs. This is partly because of the characteristics of MIDI, in that to hold a note you need to mask out the Note Off command at the end of a pattern, and then mask out the Note On command at the beginning of the next pattern. Also, maintaining the same note across pattern boundaries is a concern when you switch chords, because the end of a pattern preferably is an opportunity to cycle through the chord progression, and you need to make sure that the old note being sustained is compatible with the new chord. The generation and merging of chord progression information preferably occurs in parallel with the activities of the present discussion, and shall be discussed below in more detail. While is considered undesirable to hold notes across patterns, there are exceptions.
One example of a potentially useful time to have open notes across multiple patterns is during Techno Styles when a long MIDI event is filtered over several patterns, herein called a ‘pad’. One way to handle this example, is to use a pad sequence indicator flag to check if the current SEQ is the beginning, in the middle, or the end of a pad. Then the MIDI events in the pad track can be modified accordingly so that there will be no MIDI Note Offs for a pad at the beginning, no MIDI Note Ons at the beginning of subsequent RPs, and the proper MIDI Note Offs at the end.
Continuing our discussion of
A VP preferably can be considered as a series of Blocks. In the example of
The Blockfx dimension described in
Assuming the example presented earlier wherein the time signature is 4/4 and the RP is two bars, all Blocks in a pattern preferably must add up to 8 quarter notes in duration. In this example, assuming n Blocks in a particular RP, the duration in quarter notes of each Block in the corresponding VP would be between 1 and (8-{n−1}). While this example describes 4/4 time with a quarter note being the basic unit of length for a Block, simple variations to this example preferably would include alternate time signatures, and alternate basic units for the Block (i.e., 13/16 time signature and 32nd note, respectively, etc.).
Getting at the bottom of
Additionally, in some applications of the present invention, it may be desirable to enable certain levels in
Various examples of preferred embodiments of the Music Rules used in the creation of a Song of the present invention will now be described.
In a presently preferred embodiment, the sub-block data is preferably created as needed during the algorithmic processing of music generation.
As illustrated in
As illustrated in
Referring back to
This process of applying various musical rules to generate a RP preferably can be a part of the overall song generation process mentioned above in connection with
Bearing in mind that the MIDI Specification offers a concise way to digitally represent music, and that one significant destination of the output data from the presently discussed musical rules is the MIDI digital signal processor, we have found it advantageous in certain embodiments to use a data format that has some similarities with the MIDI language. In the discussion that follows, we go through the steps of
In the present example it is considered advantageous to break down the rhythmic and musical information involved in the music into Virtual Notes and/or Controllers (VNC). In the example of
An important feature of this aspect of the present invention is that we have embedded control information for the music generation algorithm into the basic blocks of rhythmic data drawn upon by the algorithm. We have done this in a preferably very efficient manner that allows variety, upgradeability, and complexity in both the algorithm and the final musical output. A key aspect of this is that we preferably use a MIDI-type format to represent the basic blocks of rhythmic data, thus enabling duration, volume, timing, etc. Furthermore, we preferably can use the otherwise moot portions of the MIDI-type format of these basic blocks to embed the VNC data that informs the algorithm how to go about creating a part of the music. As an example, we preferably can use the pitch of each MIDI-type event in these basic sub-blocks of rhythmic data to indicate to the algorithm what VNC to invoke in association with that MIDI-type event. Thus, as this rhythmic data is accessed by the algorithm, the pitch-type data preferably is recognized as a particular VNC, and replaced by actual pitch information corresponding to the VNC function.
In the example of
In the discussion above, by ‘predictably-selected’ we refer to the process of pseudo-randomly selecting a result based on a seed value. If the seed value is the same, then the result preferably will be the same. This is one way (though not the only way) to enable reproducibility. Further discussion of these pseudo random and seed issues is provided elsewhere in the present specification.
Continuing with
Similar to the Magic Note, the Harmonic Note VNC preferably allows the algorithm to pseudo-randomly select a harmonic from a set of possible harmonics. This capability is useful when there are multiple notes sounding at the same time in a chord. When this VNC is used, it preferably can result in any of the relative harmonics described in
Last Note is a VNC that is very similar to the Base Note, except that it preferably only contains a subset of the possible values. This is because, as we understand musical phrasing for the types of music we address, the final note preferably is particularly important, and generally sounds best when it has a relative value of C or G (bearing in mind that in this example, all the notes preferably can subsequently be transposed up or down through additional steps). As with all the VNCs, the precise note that might be played for this value preferably depends on the Mode and Key applied subsequently, as well as general pitch shifting available to the user. However, in the music we address, we find this to be a useful way to add subtlety to the music, that provides a variety of possible outcomes.
One Before Last Note is a VNC that preferably immediately precedes the Last Note. Again, this is because we have found that the last two notes, and the harmonic interval between them, are important to the final effect of a piece, and accordingly, we find it advantageous with the Final Notes of C and G to use One Before Last Notes of E, G, or B. These values can be adapted for other Styles of music, and only represent an example of how the VNC structure can be effectively utilized.
The last example VNC in
In this manner, you can setup the resulting chord because the ALC value preferably will alert the software routine that is processing all of the VNCs to let it know that the following note is to be the basis of a chord, and that the next number of harmonic notes will be played at the same as the basis note, resulting in a chord being played at once. This example shows one way that this can be done effectively. Other values of VNC controllers preferably can be used to perform similar musical functions.
It is important to note that an additional variation can preferably be implemented that addresses the natural range, or Tessitura, of a particular instrument type. While the software algorithm preferably is taking the VNCs mentioned above and selecting real values, the real pitch value preferably can be compared to the real natural range of the instrument type, and the value of subsequent VNC outcomes preferably can be inverted accordingly. For example, if the Base Note of a given pattern is near the top of the range for a bass instrument Tessitura, any subsequent Magic Notes that end up returning a positive number can be inverted to shift the note to be below the preceding Base Note. This is a particular optimization that adds subtlety and depth to the outcome, as it preferably incorporates the natural range limitations of particular instrument types.
As a simplified example of Tessitura,
In certain alternative embodiments, it is preferable to perform the conversion between virtual note controllers (VNCs) and real notes by relying in part upon state machines, tables, and a “Magic Function” routine. In this manner, new musical styles preferably can subsequently be released without changing the magic function. As an example, new styles can be created that involve new and/or modified VNC types, by merely updating one or more tables and/or state machines. Such an approach is desirable in that it enables the magic function to be implemented in an optimized design (e.g., a DSP) without sacrificing forward compatibility. In certain embodiments, state machine data preferably may be stored as part of a song.
Starting at the top left of
In certain embodiments where a tessitura associated with a particular musical component (e.g., as described herein in connection with
In certain embodiments where a tessitura of allowable ranges IS desired, at the point of the magic function illustrated in
The use of halfsteps for encoding Keys is advantageous because, as mentioned previously, the MIDI language format uses whole numbers to delineate musical pitches, with each whole number value incrementally corresponding to a half step pitch value. Other means of providing an offset value to indicate Keys can be applied, but in our experience, the use of half steps is particularly useful in this implementation because we are preferably using a MIDI DSP, and so the output of the Musical Rules preferably will be at least partly MIDI based.
At this point we will go through a detailed example of the Musical Rule portion of the algorithm, using
Beginning at the top row, there is a collection of predefined VP Sub-Blocks that preferably can advantageously be indexed by music Style and/or length. These blocks preferably are of variable sizes and preferably are stored in a hexadecimal format corresponding to the notation of pitch (recognizing that in certain embodiments the pitch information of a VP does not represent actual pitch characteristics, but VNC data as discussed above), velocity, and duration of a MIDI file (the preferable collection of predefined VP-Sub-Blocks is discussed in more detail below with reference to
After step 2 of
The third row (NCP) depicts the same data after step 3 of
The fourth row in
Row 5 of
The final row of
There are instances where certain elements of the music preferably do not need the musical rules discussed above to be invoked. For example, drum tracks preferably do not typically relate to Mode or Key, and thus preferably do not need to be transposed. Additionally, many instrument types such as drums, and MIDI effects, preferably are not arranged in the MIDI sound bank in a series of pitches, but in a series of sounds that may or may not resemble each other: In the example of drums, the sound corresponding to C sharp may be a snare drum sound, and C may be a bass drum sound. This means that in certain cases, different levels of the process discussed above in reference to
The collection of sub-blocks discussed above, from which VPs preferably are constructed, can be better understood in light of
One way to create a set of rhythmic variations such as those in
The example of
Clearly, if any of the conditions described in
In addition to efficiency, such an approach to organizing the available rhythmic blocks preferably enables the use of rhythmic density as an input to a software (e.g., algorithmic function) or hardware (e.g., state table gate array) routine. Thus, one preferably can associate a relative rhythmic density with a particular instrument type and use that rhythmic density, possibly in the form of a desired block length, preferably to obtain a corresponding rhythmic block. This preferably can be repeated until a VP is complete (see
As will be apparent to one of ordinary skill in the art of MIDI, given the context of VP generation discussed herein, the rhythmic variations shown in
Similar to the concept of using relative rhythmic density as a deterministic characteristic in creating algorithmic music,
This concept preferably applies to most instrument types in a given musical Style as well, in that certain instruments have a higher relative mobility of note pitch than others. As an example, a bass guitar in a rock Style can be thought of as having a lower relative mobility of note pitch compared to a guitar in the same Style. The relationship between relative mobility of note pitch and relevant VNC type can be very helpful in creating the collection of predefined sub-blocks discussed above, in that it serves as a guide in the determination of actual VNC for each rhythmic pattern. When one wants to create a set of rhythmic building blocks for use in a particular musical Style and/or instrument type, it is advantageous to consider/determine the desired relative mobility of note pitch, and allocate VNC types accordingly.
As an additional variation, and in keeping with the discussion above regarding relative rhythmic density, an architecture that constructs a VP for a given instrument type and/or musical Style preferably can greatly benefit from a software (e.g., algorithmic function) or hardware (e.g., state table gate array) routine relating to relative mobility of note pitch. As an example, a particular music Style and/or instrument type can be assigned a relative rhythmic density value, and such a value can be used to influence the allocation or distribution of VNC types during the generation of a VP.
The use of relative rhythmic density and relative mobility of note pitch in the present context preferably provides a way to generate VPs that closely mimic the aesthetic subtleties of ‘real’ human-generated music. This is because it is a way of preferably quantifying certain aspects of the musical components of such ‘real’ music so that it preferably can be mimicked with a computer system, as disclosed herein. Another variation and benefit of such an approach is that these characteristics preferably are easily quantified as parameters that can be changeable by the user. Thus a given musical Style, and/or a given instrument type, preferably can have a relative mobility of note pitch parameter (and/or a relative rhythmic density parameter) as a changeable characteristic. Accordingly, the user preferably could adjust such a parameter during the song playback/generation and have another level of control over the musical outcome.
Various examples of preferred embodiments for the block creation aspects of the present invention will now be described. Again, as discussed above, in certain embodiments the VP sub-block data may be algorithmically generated based on parameter input data (e.g., as illustrated in
Continuing the example presented in
Certain presently preferred embodiments for the algorithmic generation of a style will now be discussed in connection with
As shown in
In certain embodiments, the input music data may be in other formats than MIDI. For example, certain approaches have previously been described for analyzing recorded music (e.g., music in a PCM, WAV, CD-Audio, etc. format). Such previously disclosed techniques may be incorporated into the context of the present invention, for example as part of the “analysis algorithm” of
As shown in
Patt_Info is a routine that preferably can be used to generate the pattern structure information as part of the creation of a particular VP from Blocks.
Shift is a multiplier that preferably can be used in a variety of ways to add variation to the composed VP; for example, it could be a binary state that allows different Block variations based on which of the 2 bars in the RP that a particular Block is in. Other uses of a Shift multiplier can easily be applied that would provide similar variety to the overall song structure.
Num_Types is the number of instruments, and Num_Sub_Drums is the number of individual drums that make up the drum instrument. This latter point is a preferable variation that allows an enhanced layer of instrument selection, and it can be applied to contexts other than the drum instrument. Conversely, this variation is not at all necessary to the present invention, or even the present embodiment.
Block_Ind is the Block index, FX No is for any effects number information. Combi_No is an index that preferably points to a location in a table called Comb_Index_List. This table preferably is the size of the number of Styles multiplied by the number of instrument types; each entry preferably contains: SubStyle_Mask to determine if the particular entry is suitable for the present SubStyle, Combi_Index to determine the Block length, and Group_Index to determine the group of individual MIDI patches (and related information) from which to determine the Block.
Combi_Index preferably points to a table called Style_Type_Combi that preferably contains multiple sets of Block sizes. Each Block_Size preferably is a set of Block sizes that add up to the length of the SEQ. An example SEQ length is 8 QN.
Group_Index preferably points to a table called Style_Group that preferably contains sets of MIDI-type information for each group of Styles, preferably organized by MIDI Bank. PC refers to Patch Change MIDI information, P refers to variably sized MIDI parameters for a given Patch, and GS stands for Group Size. GS for group 1 preferably would indicate how many instruments are defined for group 1.
One preferable optimization of the execution of this step is to incorporate a pseudo-random number generator (PRNG) that preferably will select a particular patch configuration from the group identified by GS. Then, as the user elects to change the instrument within a particular SubStyle, and within a particular lane, another set of patch information preferably is selected from the group identified by GS. This use of a PRNG preferably can also be incorporated in the auto-generation of a song, where, at different times, the instrument preferably can be changed to provide variation or other characteristics to a given song, Part, SubPart, SEQ, RP, VP, etc. There are other areas in this routine process that preferably could benefit from the use of a PRNG function, as will be obvious to one of ordinary skill in the art.
Once the Block duration and instrument patch information preferably are determined for a given VP, the virtual Block information preferably can be determined on a Block-by-Block basis, as shown in
Block_List preferably is a routine that can determine a virtual Block using the Block size, and the instrument type. As shown in
Again, as discussed above in connection with the pattern structure generation, the present steps of the overall process preferably can use an optional PRNG routine to provide additional variety to the Block. Another fairly straightforward extension of this example is to use ‘stuffing’ (i.e.; duplicate entries in a particular table) preferably to provide a simple means of weighting the result. By this we refer to the ability to influence the particular Block data that is selected from the Virtual_Block_Data table preferably by inserting various duplicate entries. This concept of stuffing can easily be applied to other tables discussed elsewhere in this specification, and other means of weighting the results for each table lookup that are commonly known in the art can be easily applied here without departing from the spirit and scope of the invention.
Additionally, as one of ordinary skill in the art will appreciate, though many of these examples of preferred embodiments involve substantial reliance on tables, it would be fairly easy to apply concepts of state machines, commonly known in the art, to these steps and optimize the table architecture into one that incorporates state machines. Such an optimization would not depart from the spirit and scope of the present invention. As an example, refer to the previous discussion regarding
Various examples of preferred embodiments for pseudo-random number generation aspects of the present invention will now be described.
Some of the embodiments discussed in the present disclosure preferably involve maximizing the limited resources of a small, portable architecture, preferably to obtain a complex music generation/interaction device: When possible, in such embodiments (and others), preferably it is desirable to minimize the number of separate PRNG routines. Although an application like music generation/interaction preferably relies heavily on PRNG techniques to obtain a sense of realism paralleling that of similarly Styled, human-composed music, it is tremendously desirable to minimize the code overhead in the end product so as to allow the technology preferably to be portable, and to minimize the costs associated with the design and manufacture. Consequently, we have competing goals of minimal PRNG code/routines, and maximal random influence on part generation.
In addition, another goal of the present technology is preferably to allow a user to save a song in an efficient way. Rather than storing a song as an audio stream (i.e.; MP3, WMA, WAV, etc.), it is highly desirable to save the configuration information that was used to generate the song, so that it preferably can be re-generated in a manner flawlessly consistent with the original. The desirability of this goal can easily be understood, as a 5 minute MP3 file is approximately 5 MB, and the corresponding file size for an identical song, preferably using the present architecture, is approximately 0.5 KB, thus preferably reduced by a factor of approximately 10,000. In certain preferred embodiments, the sound quality of a saved song is similar to a conventional compact disc (thereby demonstrably better than MP3). In this comparison, a 5 minute song stored on a compact disc might be approximately 50 MB; thus the file size of a song using the present invention is reduced from a compact disc file by a factor of approximately 100,000.
Saving the configuration information itself, rather than an audio stream, preferably allows the user to pick up where they left off, in that they can load a previously saved piece of music, and continue working with it. Such an advantage is not easily possible with a single, combined audio stream, and to divide the audio into multiple streams would exponentially increase the file size, and would not be realizable in the current architecture without significant trade-offs in portability and/or quality.
Additionally this aspect of the present invention preferably enables the user to save an entire song from any point in the song. The user preferably can decide to save the song at the end of the song, after experiencing and interacting with the music creation. Such a feature is clearly advantageous as it affords greater flexibility and simplicity to the user in the music creation process.
Turning now to
It is important to note that if the same seed input to the simple PRNG routine is used a plurality of times, the same list of values preferably will be output each time. This is because simple PRNG routines are not random at all, as they are a part of a computing system that is, by its very nature, extremely repeatable and predictable. Even if one adds some levels of complexity to a PRNG algorithm that take advantage of seemingly unrelated things like clocks, etc., the end user can discern some level of predictability to the operation of the music generation. As can be imagined, this is highly undesirable, as one of the main aspects of the device is to generate large quantities of good music.
One benefit of the preferably predictable nature of simple PRNGs is that, by saving the seed values, one preferably can generate identical results later using the same algorithm. Given the same algorithm (or a compatible one, preferably), the seeds preferably can be provided as inputs and preferably achieve the exact same results every time. Further discussion of the use of seeds in the music generation/interaction process is discussed elsewhere in this specification.
While it is a feature of the present invention to preferably incorporate PRNG that are repeatable, there are also aspects of the present invention that preferably benefit from a more ‘truly-random’ number generation algorithm. For purposes of clarity, we call this ‘complex PRNG’. Using the example of
One example of a complex PRNG that works within the cost/resource constraints we have set, is one preferably with an algorithm that incorporates the timing of an individual user's button-presses. For example, from time to time in the process of generating music and providing user interaction in that generative process, we preferably can initialize a simple timer, and wait for a user button press. Then the value of that timer preferably can be incorporated into the PRNG routine to add randomness. By way of example, one can see that, if the system is running at or around 33 MHz, the number of clocks between any given point and a user's button press is going to impart randomness to the PRNG. Another example is one preferably with an algorithm that keeps track of the elapsed time for the main software loop to complete; such a loop will take different amounts of time to complete virtually every time it completes one loop because it varies based on external events such as user button presses, music composition variations, each of which may call other routines and/or timing loops or the like for various events or actions, etc. While it preferably is not desirable to use such a complex PRNG in the generation of values from seeds, due to repeatability issues discussed above, it preferably can be desirable to use such a PRNG in the creation of seeds, etc., as discussed above. As an additional example, such a complex PRNG routine can be used to time interval, from the moment the unit is powered up, to the moment the ‘press-it-and-forget-it’ mode is invoked; providing a degree of randomness and variability to the selection of the first auto-play song in Home mode (discussed earlier in this disclosure). Of course, this type of complex PRNG preferably is a variation of the present invention, and is not required to practice the invention.
In certain embodiments, one desirable aspect of the present invention involves the limiting of choices to the end user. The various ways instruments can be played are limitless, and in the absence of a structure, many of the possible ways can be unpleasant to the ear. One feature of palatable music is that it conforms to some sort of structure. In fact, it can be argued that the definition of creativity is expression through structure. Different types of music and/or instruments can have differing structures, but the structure itself is vital to the appeal of the music, as it provides a framework for the listener to interpret the music. The present invention involves several preferable aspects of using seed values in the generation of a piece of music. One preferable way to incorporate seeds is to use two categories of seeds in a song: 1) seeds determining/effecting the higher-level song structure, and 2) seeds determining/effecting the particular instrument parts and characteristics. Preferably, the first category of seeds is not user-changeable, but is determined/effected by the Style/SubStyle and Instrument Type selections. Preferably, the second category of seeds is user-changeable, and relates to specific patterns, melodies, effects, etc. The point in this example is that, in certain embodiments, there are some aspects of the music generation that are preferably best kept away from the user. This variation allows the user to have direct access to a subset of the seeds that are used for the music generation, and can be thought to provide a structure for the user to express through. This preferable implementation of the present discussion of seeds enables a non-musically-trained end user to creatively make music that sounds pleasurable.
It is contemplated, however, that in certain cases it may be desirable to make some or all of the choices accessible to a user. As an example, while for a given style of music it may be desirable to limit some of the parameter values available to the generation algorithm to a particularly appropriate range, in certain cases it is desirable to allow a user to edit available range, and thereby have some influence on the style in question. As another example, the range of values associated with a particular instrument or other component may be associated with the typical role of that instrument/component in the song. Preferably, in certain embodiments, such ranges of acceptable parameters are editable by a user (e.g., via the companion PC program, etc.) to allow the user to alter the constraints of the style and/or instrument or component. As an example, while in certain styles there may be an instrument type “lead” that has a relatively high pitch mobility, it may be desirable to allow the user to lower the parameters associated with “lead” that may influence the pitch mobility. Other examples along the lines of the magic notes, etc., also can be used advantageously in certain of these embodiments.
Various examples of preferred embodiments for a simple data structure (SDS) to store a song of the present invention will now be described.
The use of PRNG seeds preferably enables a simple and extremely efficient way to store a song. In one embodiment of the present invention, the song preferably is stored using the original set of seeds along with a small set of parameters. The small set of parameters preferably is for storing real time events and extraneous information external to the musical rules algorithms discussed above. PRNG seed values preferably are used as initial inputs for the musical rules algorithms, preferably in a manner consistent with the PRNG discussion above.
‘Application Number’ is preferably used to store the firmware/application version used to generate the data structure. This is particularly helpful in cases where the firmware is upgradeable, and the SDS may be shared to multiple users. Keeping track of the version of software used to create the SDS is preferable when building in compatibility across multiple generation/variations of software/firmware.
‘Style/SubStyle’ preferably is used to indicate the SubStyle of music. This is helpful when initializing various variables and routines, to preferably alert the system that the rules associated with a particular SubStyle will govern the song generation process.
‘Sound Bank/Synth Type’ preferably indicates the particular sound(s) that will be used in the song. This preferably can be a way to preload the sound settings for the Midi DSP. Furthermore, in certain embodiments this preferably can be used to check for sonic compatibility.
‘Sample Frequency’ preferably is a setting that can be used to indicate how often samples will be played. Alternatively, this preferably can indicate the rate at which the sample is decoded; a technique useful for adjusting the frequency of sample playback.
‘Sample set’ preferably is for listing all the samples that are associated with the Style of music. Although these samples preferably may not all be used in the saved SDS version of the song, this list preferably allows a user to further select and play relevant samples during song playback.
‘Key’ preferably is used to indicate the first key used in the song. Preferably, one way to indicate this is with a pitch offset.
‘Tempo’ preferably is used to indicate the start tempo of the song. Preferably, one way to indicate this is with beats per minute (BPM) information.
‘Instrument’ preferably is data that identifies a particular instrument in a group of instruments. Such as an acoustic nylon string guitar among a group of all guitar sounds. This data is preferably indexed by instrument type.
‘State’ preferably is data that indicates the state of a particular instrument. Examples of states are: muted, un-muted, normal, Forced play, solo, etc.
‘Parameter’ preferably is data that indicates values for various instrument parameters, such as volume, pan, timbre, etc.
‘PRNG Seed Values’ preferably is a series of numerical values that are used to initialize the pseudo-random number generation (PRNG) routines. These values preferably represent a particularly efficient method for storing the song by taking advantage of the inherently predictable nature of PRNG to enable the recreation of the entire song. This aspect of the present invention is discussed in greater detail previously with respect to
Through the use of these example parameters in a SDS, a user song preferably can be efficiently stored and shared. Though the specific parameter types preferably can be varied, the use of such parameters, as well as the PRNG Seeds discussed elsewhere in this disclosure, preferably enables all the details necessary to accurately repeat a song from scratch. It is expected that the use of this type of arrangement will be advantageous in a variety of fields where music can be faithfully reproduced with a very efficient data structure.
At the start of
Various examples of preferred embodiments for a complex data structure to store a song of the present invention will now be described.
In another variation to the present invention, it is contemplated that, for purposes of saving and playing back songs, the reliance on seeds as inputs to the musical rule algorithms (see SDS discussion above) preferably may be exchanged for the use of Complex Data Structures (CDS). In part because of it's efficiency, the seed-based architecture discussed above is desirable when forward/backward compatibility is not an issue. However, it has some aspects that may not be desirable, if compatibility across platforms and/or firmware revisions is desired. In these cases, the use of an alternative embodiment may be desirable.
As described above, a seed preferably is input to a simple PRNG and a series of values preferably are generated that are used in the song creation algorithm. For purposes of song save and playback, the repeatability preferably is vital. However, if the algorithm is modified in a subsequent version of firmware, or if other algorithms would benefit from the use of the simple PRNG, while it is in the middle of computing a series (e.g.; DS0-DS3 in
In certain embodiments, a further distinction with a CDS is that it provides greater capabilities to save real time user data, such as muting, filtering, instrument changes, etc.
While both examples have their advantages, it may also be advantageous to combine aspects of each into a hybrid data structure (HDS). For example, the use of some seed values in the data structure, while also incorporating many of the more complex parameters for the CDS example, preferably can provide an appropriate balance between compatibility and efficiency. Depending on the application and context, the balance between these two goals preferably can be adjusted by using a hybrid data structure that is in between the SDS of
In the example of
However, in certain embodiments it is desirable to capture many real time events actuated upon by the user during a song. In these cases it is preferable to store the real time data as part of the CDS, i.e., via the ‘Parameter’ parameter. In this example, real time changes to particular instrument parameters preferably are stored in the ‘Parameter’ portion of the CDS, preferably with an associated time stamp. In such a manner, the user events that are stored can be performed in real time upon song playback. Such embodiments may involve a CDS with a significantly greater size, but provide a user with greater control over certain nuances of a musical composition, and are therefore desirable in certain embodiments.
‘Song Structure’ preferably is data that preferably lists the number of instrument types in the song, as well as the number and sequence of the parts in the song.
‘Structure’ preferably is data that is indexed by part that preferably can include the number and sequence of the sub-parts within that part.
‘Filtered Track’ preferably is a parameter that preferably can be used to hold data describing the characteristics of an effect. For example, it preferably can indicate a modulation type of effect with a square wave and a particular initial value. As the effect preferably is typically connected with a particular part, this parameter may preferably be indexed by part.
‘Progression’ preferably is characteristic information for each sub-part. This might include a time signature, number and sequence of SEQs, list of instrument types that may be masked, etc.
‘Chord’ preferably contains data corresponding to musical changes during a sub-part. Chord vector (e.g., +2, −1, etc.), key note (e.g., F), and progression mode (e.g., dorian ascending) data preferably are stored along with a time stamp.
‘Pattern’ and the sub-parameters ‘Combination’, ‘FX Pattern’, and ‘Blocks’, all preferably contain the actual block data and effects information for each of the instruments that are used in the song. This data is preferably indexed by the type of instrument.
‘Improv’ preferably is for specifying instruments or magic notes that will be played differently each time the song is played. This parameter preferably allows the creation of songs that have elements of improvisation in them.
Additional parameters can preferably be included, for example to enable soundbank data associated with a particular song to be embedded. Following this example, when such a CDS is accessed, the sound bank data preferably is loaded into non-volatile memory accessible to a DSP such that the sound bank data may be used during the generation of music output.
In certain preferred embodiments the Player 10 is accompanied by a companion PC software system designed to execute on a PC system and communicate with Player 10 via a data link (e.g., USB 54, Serial I/O 57, and/or a wireless link such as 802.11b, Bluetooth, IRDA, etc.). Such a PC software system preferably is configured to provide the user with a simple and effective way to copy files between the Player 10 and other locations (e.g., the PC hard drive, the Internet, other devices, etc.). For example, the companion PC software program preferably operates under the MS Windows family of Operating Systems and provides full access to the User for all Player 10 functions and Modes, as well as the local Player memory (e.g., SMC). Following this example, a user can connect to the Internet and upload or download music related files suitable to be used with the Player 10 (e.g., MIDI, -WMA, MP3, Karaoke, CDS, SDS, etc.) as well as user interface-related files such as customized user-selectable graphics preferably to be associated with music styles or songs on the Player 10. Such a companion PC program preferably is also used to enable hardware and/or software housekeeping features to be easily managed, such as firmware and sound bank updates. This companion PC software system preferably is used to provide the user with an easy way to share music components and/or complete songs with other users in the world (e.g., via FTP access, as attachments to email, via peer-to-peer networking software such as Napster, etc.). It is important to note the potentially royalty-free nature and extreme size efficiency of musical output from the Player 10 lends itself well to the Internet context of open source file sharing.
In addition to, or in combination with, the aforementioned embodiments involving a portable system linked to a PC system, certain additional features are advantageously employed in certain embodiments. For example, a companion PC software application provides a sample edit mode to depict a graphic associated with the waveform of the selected sample, e.g., via the PC display. In these embodiments, the user is provided with a simple way to easily select the start and end points of the selected sample, e.g., via a pointing device such as a mouse. Preferably, such graphical clipping functions enable a user to easily crop a selected sample, e.g., so as to remove an undesired portion, etc. After clipping/cropping the sample, the user is presented with the option of saving the newly shortened sample file, e.g., with a new name. Clearly, other similar functions in addition to or besides such clipping can be supported, to make use of the display and/or processing resources available on the PC system, to provide a graphic version of the waveform of a selected sample, and to provide the end user with a simple way to carryout basic operations on the selected sample. In these embodiments, the newly modified sample file is then transferable to the portable system, e.g., via connector 53 in
In certain embodiments where a link (e.g., wireless 802.11) is present between more than one device, the devices preferably can operate in a cooperative mode, i.e., wherein a first user and/or device is in control of parameters relating to at least one aspect of the music (e.g., one instrument or effect), and a second user/device is in control of parameters relating to a second aspect of the music (e.g., another instrument, or microphone, etc.). In certain of these embodiments, the plurality of devices preferably will exchange information relating to music style, as well as certain musical events relating to the music generation. In at least some embodiments it is preferably to commonly hear the resulting music, e.g., via a commonly accessible digital audio stream. Furthermore, as will be clear to one of ordinary skill in the art, the aforementioned cooperative mode can advantageously be utilized in embodiments where one or more of the devices are computers operating while connected to a network such as a LAN and/or the Internet.
Various examples of preferred embodiments for hardware implementation examples of the present invention will now be described.
The MP 36, DSP 42, FM receiver 50, and Microphone input 51 all preferably have some type of input to the hardware CODEC 52 associated with the DSP 42.
The connector 53 at the top left of
The MP 36 in this example is preferably the ARM AT91R40807, though any similar microprocessor could be utilized (such as versions that have on-board Flash, more RAM, faster clock, lower voltage/lower power consumption, etc.). This ARM core has 2 sets of instructions: 32 bit and 16 bit. Having multiple width instructions is desirable in the given type of application in that the 16 bit work well with embedded systems (Flash, USB, SMC, etc.), and 32 bit instructions work efficiently in situations where large streams of data are being passed around, etc. Other variations of instruction bit length could easily be applied under the present invention.
For 32 bit instructions, the system of the present invention preferably pre-loads certain instructions from the Flash memory 41 into the internal RAM of the MP 36. This is because the Flash interface is 16 bit, so to execute a 32 bit instruction takes at least 2 cycles. Also, the Flash memory 41 typically has a delay associated with read operations. In one example, the delay is approximately 90 ns. This delay translates into the requirement for a number of inserted wait states (e.g., 2) in a typical read operation. Conversely, the internal RAM of the MP 36 has much less delay associated with a read operation, and so there are less wait states (e.g., 0). Of course, the internal RAM in this case is 32 bits wide, and so the efficiencies of a 32 bit instruction can be realized.
As is shown above in the example regarding the wait states of Flash memory 41, there are many reasons why it is desirable to try to maximize the use of the internal MP RAM. As can be seen from
One example of a trade-off associated with complexity and portability is the use of a widely available WMA audio decoder algorithm from Microsoft. In this example, when operating the ARM MP of
In the example of
Another alternative embodiment can be an MP 36 with preferably more internal RAM (for example, 512 KB) which would preferably allow a reduction or elimination of the use of Flash memory 41. Such a system may add to the total cost, but would reduce the complexities associated with using Flash memory 41 discussed above.
Another variation is the example shown in
Continuing the discussion of the architecture shown in
In this example, when the Player is not operating in a WMA/MP3/etc. mode, the ‘multi-use’ mid section can preferably be used for at least three types of buffers. Block buffers are preferably used by the eDJ Block creation algorithms (e.g.,
SMC is a Flash memory technology that doesn't allow the modification of a single bit. To perform a write to the SMC, one must read the entire SMC Block, update the desired portion of the SMC Block, and then write the entire SMC Block back to the SMC. In the interests of efficiency, the currently used SMC Block is preferably maintained in the SMC buffers.
As one can appreciate, the system configuration described above cannot simultaneously playback large WMA/MP3 streams while also writing to the SMC. This is because the two functions, preferably alternatively use the same memory region. This is a creative use of limited resources, because it is preferably a relatively unusual condition to be reading WMA/MP3 while writing SMC at the same time. So the code is preferably arranged to swap in and out of the same location. Such an arrangement allows maximized use of the limited resources in a portable environment such as
However, in a more powerful environment (with additional resources, and/or faster clock speed), this ‘multi-use’ of a shared region of memory could preferably be eliminated, and simultaneous use of WMA/MP3 and the Record function could easily be implemented. Obviously, these additional enhancements for use in a portable environment do not limit the other aspects of the present invention.
The system discussed above is portable, but preferably has extremely high-quality sound. On a very basic level, this is partly due to the use of a sound chip that typically would be found in a high-end sound card in a PC system. The SAM9707 chip is preferable because of its excellent sound capabilities, but this has required it be adapted somewhat to work in the portable example discussed herein.
One characteristic of the SAM9707 is that it is typically configured to work with SDRAM in a sound card. This SDRAM would typically hold the MIDI sound banks during normal operation. Such sound banks are preferably a critical part of the final sound quality of music that is output from a DSP-enabled system. In fact, another reason why this particular chip is preferable is to allow custom sounds to preferably be designed.
In the example above of a portable system, SDRAM adds significantly to the power requirements, as well as the address logic. Accordingly, it is desirable to use a variation of the configuration, preferably using Flash as local DSP sound bank storage (see
The problem of reaching a proper balance between maintaining the low power/simple architecture on one hand, and providing high quality, upgradeable, music sound banks on the other hand, is preferably solved by adapting a mode of the DSP chip, and preferably customizing the address logic in such a way that the DSP can be “tricked” into providing the access from the MP side to the local DSP Flash memory.
So the first variation of the present invention, to the general use of the DSP chip, especially in its intended context of a sound card for a PC, is the address location of the RAMa. This region is selected to allow a very simple address decode logic arrangement (preferably external to the DSP) so that the assertion of A24 will preferably toggle the destination of RAMa addresses, between DSP-local RAM and DSP-local Flash memories. This variation preferably involves a firmware modification that will allow the specific location of RAMa to be configured properly preferably by default at startup time. There are other ways to modify this location after initialization, but they are more complicated, and therefore are not as desirable as the present method.
Another variation to the intended context of the DSP chip address map preferably involves a creative implementation of the DSPs BOOT mode to allow the sound banks to be upgraded, even though the sound banks are preferably located in the local Flash memory of the DSP chip; a location not typically accessible for sound bank upgrades.
In this example, the BOOT mode of the DSP causes an internal bootstrap program to execute from internal ROM. This bootstrap program might typically be used while upgrading the DSP firmware. As such, the internal bootstrap expects to receive 256 words from the 16 bit burst transfer port, which it expects to store at address range 0100H-01FFH in the local memory, after which the bootstrap program resumes control at address 0100H. This relatively small burst is fixed, and is not large enough to contain sound banks. Furthermore, it does not allow the complex Flash memory write activities, as discussed above in connection with the SMC. Since our design preferably uses Flash instead of SDRAM, we have found it highly desirable to use this bootstrap burst to load code that preferably ‘tricks’ the ROM bootstrap to effectuate the transfer of special code from the ARM MP bus to the RAM. This special code is then used to preferably effectuate the transfer of sound bank upgrade data from the ARM MP bus to the Flash memory.
In the present example, the A24 address line generated by the DSP is preferably altered by the BOOT signal controlled by the MP before being presented to the address decoding logic of the DSP local memory. This arrangement permits the MP to preferably invert the DSP's selection of RAM and Flash in BOOT mode, and thus allows the RAM to preferably be available at address 0x100 to receive the upgrade code.
Additional variations to the hardware arrangement discussed above can be considered. For example, if the power level is increased, and the MP performance increased, the DSP could be substituted with a software DSP. This may result in lower quality sounds, but it could have other benefits that outweigh that, such as lower cost, additional flexibility, etc. The DSP could similarly be replaced with a general-purpose hardware DSP, possibly with the result of lower quality sounds, possibly outweighed by the benefits of increased portability, etc. The MP could be replaced with one having a greater number of integrated interfaces (e.g., USB, SMC, LCD, etc.), and/or more RAM, faster clock speed, etc. With a few changes to some of the disclosed embodiments, one could practice the present invention with only a DSP (no separate MP), or a dual die DSP/MP, or with only an MP and software. Additionally, the SMC memory storage could be substituted with a Secure Digital (SD) memory card with embedded encryption, and/or a hard disk drive, compact flash, writeable CDROM, etc., to store sound output. Also, the LCD could be upgraded to a color, or multi-level gray LCD, and/or a touch-sensitive display that would preferably allow another level of user interface features.
Yet a further variation of the present discussion preferably can be the incorporation of a electromagnetic or capacitive touch pad pointing device, such as a TouchPad available from Synaptics, to provide additional desirable characteristics to the user interface. Both the touch pad and the touch sensitive display mentioned above can be used to provide the user with a way to tap in a rhythm, and/or strum a note/chord. Such a device preferably can be used to enable a closer approximation to the operation of a particular instrument group. For example, the touch pad can be used to detect the speed and rhythm of a user's desired guitar part from the way the user moves a finger or hand across the surface of the touch pad. Similarly, the movement of the users hand through the x and y coordinates of such a pointing device can be detected in connection with the pitch and/or frequency of an instrument, or the characteristics of an effect or sample. In another example, a touch pad pointing device can also be used to trigger and/or control turntable scratching sounds approximating the scratching sounds a conventional DJ can generate with a turntable.
As can be seen in
When incorporating the DSP into a generative/interactive music system, it is highly desirable to synchronize the MIDI and audio streams. A sample preferably has to play at exactly the right time, every time; when the audio stream components get even slightly out of sync with the MIDI events, the resulting musical output generally is unacceptable. This delicate nature of mixing audio streams and MIDI together in a generative/interactive context is worsened by the nature of the Flash read process, in that SMC technology is slow to respond, and requires complex read machinations. It is difficult to accurately sync MIDI events with playback of audio from a Flash memory location. Because of the delay in decoding and playing a sample (compared to a MIDI event), there is a tradeoff in either performing timing compensation, or preloading relatively large data chunks. Because of these issues, it is preferable to configure a new way to use MIDI and audio streams with the DSP chip. While this aspect of the present invention is discussed in terms of the DSP architecture, it will be obvious to one of ordinary skill in the art of MIDI/audio stream synchronization that the following examples apply to other similar architectures.
The two inputs to the Synth device preferably may actually share a multiplexed bus; but logically they can be considered as separately distinguishable inputs. In one example, the two inputs share a 16 bit wide bus. In this case, the MIDI input preferably may occupy 8 bits at one time, and the audio stream input preferably may occupy 16 bits at another time. Following this example, one stream preferably may pause while the other takes the bus. Such alternating use of the same bus can mean that relatively small pauses in each stream are constantly occurring. Such pauses are intended to be imperceptible, and so, for our purposes here, the two streams can be thought of as separate.
In this example, largely because of the constraints of the system architecture example discussed above, this is not a trivial thing to accomplish consistently and accurately using conventional techniques. Keeping in mind that the MIDI event is preferably generated almost instantly by the Synth chip, whereas the Audio Stream event could require one or more of the following assistance from the ARM MP: fetching a sound from SMC, decompressing (PCM, etc.), adding sound effects (reverb, filters, etc.).
In this example, it is highly desirable to create a special MIDI file preferably containing delta time information for each event, and specialized non-registered parameter numbers (NRPNs). This feature is especially advantageous when used with a Sample List (as mentioned above) because the name of a particular sample in a list is preferably implicit, and the NRPNs can preferably be used to trigger different samples in the particular sample list without explicitly calling for a particular sample name or type. This type of optimization reduces the burden of fetching a particular sample by name or type, and can preferably allow the samples used to be preloaded. In the following discussion, it should be understood that in certain embodiments, the use of MIDI System Exclusive messages (SYSEXs) may be used in place of (or perhaps in addition to) the NRPNs.
In
The top of the figure indicates that the first information in this file is a delta time of 250 ms. This corresponds to the 250 ms delay at the beginning of
In the previous example, the delta time preferably can be different (and often is) each time in the special MIDI type file. In our simplified example, and because we want to make the timing relationship with a quarter note, etc., more clear, we have used the same 250 ms value each time. Obviously, in a more complex file, the delta time will vary.
Additionally, as discussed earlier in connection with the example of the Player function keys in
As previously described, voice and other audio samples may be encoded, stored and processed for playback in accordance with the present invention. In certain preferred embodiments, voice samples are coded in a PCM format, and preferably in the form of an adaptive (predictive), differential PCM (ADPCM) format. While other PCM formats or other sample coding formats may be used in accordance with the present invention, and particular PCM coding formats (and ways of providing effects as will be hereinafter described) are not essential to practice various aspects of the present invention, a description of exemplary ADPCM as well as certain effects functions will be provided for a fuller understanding of certain preferred embodiments of the present invention. In accordance with such embodiments, a type of ADPCM may provide certain advantages in accordance with the present invention.
As will be appreciated by those of skill in the art based on the disclosure herein, the use of ADPCM can enable advantages such as reduced size of the data files to store samples, which are preferably stored in the non-volatile storage (e.g., SMC), thus enabling more samples, song lists and songs to be stored in a given amount of non-volatile storage. Preferably, the coding is done by a packet of the size of the ADPCM frame (e.g., 8 samples). For each packet, preferably a code provides the maximum value; the maximum difference between two samples is coded and integrated in the file. Each code (difference between samples (delta_max) and code of the packet (diff_max)) uses 4 bits. In accordance with this example, the data/sample is therefore (8*4+4)/8=4.5 bits/sample.
As will be appreciated, this type of coding attempts to code only what is really necessary. Over 8 samples, the maximum difference between two samples is in general much less than the possible dynamic range of the signal (+32767/−32768), and it is therefore possible to allow oneself to code only the difference between samples. Preferably, the ADPCM is chosen to be suitable for the voice that is relatively stationary. By predictive filtering, it is possible to reduce the difference between a new sample and its prediction. The better the prediction, the smaller the difference, and the smaller the coding (the quantization) that is chosen, taking into account the average differences encountered. While it will be appreciated that this approach requires additional computation ability for the prediction computation, it is believed that this approach provides significant advantages in reduced storage for samples with acceptable sample coding quality in accordance with the present invention. While more conventional or standardized ADPCM desires to offer a coding time without introducing delays, with the present invention it has been determined that such attributes are not essential.
A simple coding without prediction and taking into account only average values of differences encountered reacts very poorly to a non-stationary state (e.g., each beginning of a word or syllable). For each new word or syllable, a new difference much greater than the average differences previously encountered typically cannot be suitably coded. One therefore tends to hear an impulse noise depending on the level of the signal. Preferably, the solution is therefore to give the maximum value of the difference encountered (one therefore has a delay of 8 samples, a prediction is thus made for the quantizer only) for a fixed number of samples and to code the samples as a function of this maximum difference (in percentage). The coding tends to be more optimal at each instant, and reacts very well to a non-stationary state (each beginning of a word or syllable). Preferably, the coding is logarithmic (the ear is sensitive to the logarithm and not to the linear), and the Signal/Noise ratio is 24 db. In preferred embodiments, this function is put in internal RAM in order to be executed, for example, 3 times more rapidly (one clock cycle for each instruction instead of three in external flash memory).
Preferably certain effects may be included in the ADPCM coding used in certain embodiments of the present invention. For example, a doppler effect may be included in the ADPCM decoding since it requires a variable number of ADPCM samples for a final fixed number of 256 samples. As is known, such a doppler effect typically consists of playing the samples more or less rapidly, which corresponds to a variation of the pitch of the decoded voice accompanied by a variation of the speed together with the variation of pitch. In order to give a natural and linear variation, it is desirable to be able to interpolate new samples between two other samples. The linear interpolation method has been determined to have certain disadvantages in that it tends to add unpleasant high frequency harmonics to the ear.
The method traditionally used consists of over-sampling the signal (for example, in a ratio of 3 or 4) the signal and then filtering the aliasing frequencies. The filtered signal is then interpolated linearly. The disadvantage of this method is that it requires additional computational ability. Preferably, in accordance with certain embodiments, a technique is utilized that consists of interpolating the signal with the four adjacent samples. It preferably corresponds to a second order interpolation that allows a 4.5 dB gain for the harmonics created by a linear interpolation. While 4.5 db seems low, it is important to consider it in high frequencies where the voice signal is weak. The original high frequencies of the voice are masked by the upper harmonics of the low frequencies in the case of the linear method, and this effect disappears with second order interpolation. Moreover, it tends to be three times faster than the over-sampling method. Preferably, this function is put in internal RAM in order to be executed, for example, 3 times more rapidly (one clock cycle for each instruction instead of three in external flash memory).
Also in accordance with preferred embodiments, a frequency analysis function is included, which consists of counting the period number (the pitch) in an analysis window in order to deduce from this the fundamental frequency. Preferably, this function may be utilized to process samples in order to reveal the periods. In general, it is not feasible to count the peaks in the window because the signal tends to vary with time (for example, the beating of 1 to 3 piano strings that are not necessarily perfectly in tune); moreover, in the same period, there can be more than one peak. In accordance with such embodiments, the distance between a reference considered at the beginning of the analysis window and each of the panes shifted by one sample. For a window of 2*WINDOW_SIZE samples and a reference window of WINDOW_SIZE samples, one therefore may therefore carry out WINDOW_SIZE computations of distance on WINDOW_SIZE samples. Preferably, the computation of distance is done by a sum of the absolute value of the differences between reference samples and analysis samples. This function preferably is put in internal RAM in order to be executed, for example, 3 times more rapidly (one clock cycle for each instruction instead of three in external flash memory).
Also in accordance with such embodiments, special effects such as wobbler, flange, echo and reverb may be provided with the ADPCM encoding. Such special effects preferably are produced over 256 samples coming from the ADPCM decoder and from the doppler effect. Preferably, this function is put in internal RAM in order to be executed, for example, 3 times more rapidly (one clock cycle for each instruction instead of three in external flash memory). Preferably, the average value of the sample is computed, and it is subtracted from the sample (which can be present over the samples) in order to avoid executing the wobbler function on it, which would add the modulation frequency in the signal (and tend to produce an unpleasant hiss). Preferably, the method for the wobbler effect is a frequency modulation based on sample=sample multiplied by a sine function (based on suitable wobbler frequencies, as will be understood by those of skill in the art).
Also in accordance with the preferred embodiments, the purpose of the flange effect is to simulate the impression that more than one person is speaking or singing with a single source voice. In order to limit the computation power, two voices preferably are simulated. In order to provide this impression, preferably the pitch of the source voice is changed and added to the original source voice. The most accurate method would be to analyze the voice using a vocoder and then to change the pitch without changing the speed. In each case, one could have the impression that a man and a woman are singing together, although such a method typically would require DSP resources. A method that changes the pitch without changing the speed (important if one wants the voices to remain synchronous) consists of simulating the second voice by alternately accelerating and decelerating the samples. One then produces the doppler effect explained in the preceding, but with a doppler that varies alternately around zero in such a way as to have a slightly different pitch and the voices synchronous. With such embodiments, one may simulate, for example, a person placed on a circle approximately 4 meters in diameter regularly turning around its axis and placed beside another stationary person.
Also in accordance with such embodiments, the echo effect is the sum of a source sample and of a delayed sample, and the reverb effect is the sum of a source sample and a delayed sample affected by a gain factor. The delayed samples preferably may be put in a circular buffer and are those resulting from the sum. The formula of the reverb effect may therefore be:
Sample(0)=sample(0)+sample(−n)*gain+sample(−2*n)*gain^2+sample (−3*n)*gain^+ . . . +sample(−i*n)*gain^i. Preferably, the gain is chosen to be less than 1 in order to avoid a divergence. In accordance with preferred embodiments, for reasons of size of the buffer, which can be considerable, the echo effect preferably uses the same buffer as that of the reverb effect. In order to have a true echo, it is necessary to give reverb a gain effect that is zero or low. The two effects can function at the same time. The delay between a new sample and an old one is produced by reading the oldest sample put in the memory buffer. In order to avoid shifting the buffer for each new sample, the reading pointer of the buffer is incremented by limiting this pointer between the boundaries of the buffer. The size of the memory buffer therefore depends on the time between samples.
Also in accordance with such embodiments, an electronic tuner function may be provided, the aim of which is to find the fundamental of the sample signal coming from the microphone in order to give the note played by a musical instrument. Similar to what has been described previously, a preferred method will consist of computing the number of periods for a given time that is a multiple of the period in order to increase the accuracy of computation of the period. In effect, a single period will give little accuracy if the value of this period is poor because of the sampling. In order to detect the periods, preferably one uses a routine which computes the distance between a reference taken at the beginning of the signal and the signal. As will be understood, the period will be the position of the last period divided by the total number of periods between the first and the last period. The effective position of the last period is computed by an interpolation of the true maximum between two distance samples. The period thus computed will give by inversion (using a division of 64 bits/32 bits) the fundamental frequency with great precision (better than 1/4000 for a signal without noise, which is often the case).
Also in accordance with such embodiments, a low pass filter (or other filter) function may be provided as part of the effects provided with the ADPCM sample coding. Such a function may eliminate with a low-pass filter the high frequencies of the samples used for computation of the distance such for the routines previously described. These high frequencies tend to disturb the computations if they are too elevated. Filtering is done by looking for the highest value in order to normalize the buffer used for computation of the distance.
Also in accordance with the present invention, there are numerous additional implementations and variations that preferably can be used with many desirable aspects of the present invention. Exemplary ways to use the present invention to great effect include a software-based approach, as well as general integration with other products. Additionally, several valuable variations to the present invention can be used with great success, especially with regard to media content management, integration with video, and other miscellaneous variations.
Many aspects of the present invention can advantageously be employed in connection with a digital/video light show. As described in more detail in U.S. Pat. No. 4,241,295, entitled “Digital Lighting Control System,” and U.S. Pat. No. 4,797,795, entitled “Control System for Variable Parameter Lighting Fixtures,” hereby incorporated by reference in their entirety, the control of video and/or stage-type lighting systems can be performed using digital commands such as MIDI-type descriptor events. Accordingly, in certain of the presently described embodiments, it is preferable to integrate aspects of the music generation with such types of video/light control systems. As an example, the musical and visual events can be synchronized, temporally quantized, and/or otherwise coordinated using parameters such as the delta time, velocity, NRPN, system exclusive, etc. Such coordination may be advantageous, as significant changes in the music (e.g., going from Intro to Chorus) may be arranged to coincide with the timing and/or intensity, color, patterns, etc., of the visuals. In such embodiments, the experience for someone listening and viewing the audio/visual output will desirably be improved.
Furthermore, in certain embodiments the visual display preferably consist of or include animated characters (e.g., musicians) that are driven in their actions by the music (or in certain cases, a ringtone in a mobile phone, as described elsewhere in this specification) that preferably may be created by a user. Following this example, the visual display preferably may also include an animated background, such that the animated characters may be depicted as performing over the background animation.
In certain embodiments involving a telephone, a visual animation may launch upon the ringing of the phone call (and in certain cases, the phone call ring itself may use certain of the other embodiments described throughout this disclosure). Animated characters preferably can be pre-assigned to represent and identify a caller, e.g., based on the caller ID information transmitted. In this fashion, a user preferably has the capability to play a custom animation and/or music, that preferably is based on the identity of the caller.
Many aspects of the present invention can be incorporated with success into a software-based approach. For example, the hardware DSP of the above discussion can be substituted with a software synthesizer to perform signal processing functions (the use of a hardware-based synthesizer is not a requirement of the present invention). Such an approach preferably will take advantage of the excess processing power of, for example, a contemporary personal computer, and preferably will provide the quality of the music produced in a hardware-based device, while also providing greater compatibility across multiple platforms (e.g., it is easier to share a song that can be played on any PC). Configuring certain embodiments of the present invention into a software-based approach enables additional variations, such as a self-contained application geared toward a professional music creator, or alternatively geared towards an armchair music enthusiast. Additionally, it is preferable to configure a software-based embodiment of the present invention for use in a website (e.g., a java language applet), with user preferences and/or customizations to be stored in local files on the user's computer (e.g., cookies). Such an approach preferably enables a user to indicate a music accompaniment style preference that will ‘stick’ and remain on subsequent visits to the site. Variations of a software-based approach preferably involve a ‘software plug-in’ approach to an existing content generation software application (such as Macromedia Flash, Adobe Acrobat, Macromedia Authorware, Microsoft PowerPoint, and/or Adobe AfterEffects). It is useful to note that such a plug-in can benefit from the potentially royalty free music, and that in certain embodiments, it may be preferable to export an interactively generated musical piece into a streaming media format (e.g., ASF) for inclusion in a Flash presentation, a PDF file, an Authorware presentation, an AfterEffects movie, etc. Certain embodiments of the present invention can be involved in a Internet-based arrangement that enables a plurality of users to interactively generate music together in a cooperative sense, preferably in real time. Aspects of the present invention involving customized music can be incorporated as part of music games (and/or music learning aids), news sources (e.g., internet news sites), language games (and/or language learning aids), etc. Additionally, a software/hardware hybrid approach incorporating many features and benefits of the present invention can involve a hybrid “DSP” module that plugs into a high speed bus (e.g., IEEE 1394, or USB, etc.) of a personal computing system. In such an approach, the functionality of MP 36 can be performed by a personal computing system, while the functionality of DSP 42 can be performed by a DSP located on a hardware module attached to a peripheral bus such as USB. Following this example, a small USB module about the size of a automobile key can be plugged into the USB port of a PC system, and can be used to perform the hardware DSP functions associated with the interactive auto-generation of algorithmic music.
As will be appreciated, many advantageous aspects of the present invention can be realized in a portable communications device such as a cellular telephone, PDA, etc. As an example, in the case of a portable communications device incorporating a digital camera (e.g., similar in certain respects to the Nokia 3650 cellular telephone with a built-in image capture device, expected to be available from Nokia Group sometime in 2003), certain preferred embodiments involve the use of the algorithmic music generation/auto-composition functions in the portable communications device. Following this example, the use of a digital image capture device as part of such embodiments can allow a user to take one or more pictures (moving or still) and set them to music, preferably as a slideshow. Such augmented images can be exchanged between systems, as the data structure required to store music (e.g., SDS and CDS structures and features illustrated in
As will be appreciated, aspects of the present invention may be incorporated into a variety of systems and applications, an example of which may be a PBX or other telephone type system. An exemplary system is disclosed in, for example, U.S. Pat. No. 6,289,025 to Pang et al., which is hereby incorporated by reference (other exemplary systems include PBX systems from companies such as Alcatel, Ericsson, Nortel, Avaya and the like). As will be appreciated from such an exemplary system, a plurality of telephones and telephony interfaces may be provided with the system, and users at the facility in which the system is located, or users who access the system externally (such as via a POTS telephone line or other telephone line), may have calls that are received by the system. Such calls may be directed by the system to particular users, or alternatively the calls may be placed on hold (such aspects of such an exemplary system are conventional and will not be described in greater detail herein). Typically, on-hold music is provided to callers placed on hold, with the on-hold music consisting of a radio station or taped or other recorded music coupled through an audio input, typically processed with a coder and provided as an audio stream (such as PCM) and coupled to the telephone of the caller on hold.
In accordance with embodiments of the present invention, however, one or more modules are provided in the exemplary system to provide on-hold music to the caller on hold. Such a module, for example, could include the required constituent hardware/software components of a Player as described elsewhere herein (see, e.g.,
What is important is that, in accordance with such embodiments, one or more auto-composition engines are adapted for the exemplary system, with the command/control interface of the auto-composition engine being changes from buttons and the like to commands from the resources of the exemplary system (which are generated in response to calls being placed on hold, digit detection and the like). In accordance with variations of such embodiments, a plurality of auto-composition engines are provided, and the resources of the system selectively provide on-hold music to on hold callers of a style selected by the caller on hold (such as described above). In one variation, there may potentially be more callers on hold than there are auto-composition engines; in such embodiments, the callers on hold are selectively coupled to one of the output audio streams of the auto-composition engines provided that there is at least one auto-composition engine that is not being utilized. If a caller is placed on hold at a time when all of the auto-composition engines are being utilized, the caller placed on hold is either coupled to one of the audio streams being output by one of the auto-composition engines (without being given a choice), or alternatively is provided with an audio prompt informing the user of the styles of on-hold music that are currently being offered by the auto-composition engines (in response thereto, this caller on hold may select one of the styles being offered by depressed one or more digits on the telephone keypad and be coupled to an audio stream that is providing auto-composed music of the selected style).
Other variations of such embodiments include: (1) the resources of the exemplary system detect, such as via caller ID information or incoming trunk group of the incoming call, information regarding the calling party (such as geographic location), and thereafter directs that the on hold music for the particular on hold be a predetermined style corresponding to the caller ID information or trunk group information, etc.; (2) the resources of the exemplary system selectively determines the style of the on-hold music based on the identity of the called party (particular called parties may, for example, set a configuration parameter that directs that their on hold music be of a particular style); (3) the resources of the exemplary system may selectively determine the style of on-hold music by season of the year, time of day or week, etc.; (4) the exemplary system includes an auto-composition engine for each of the styles being offered, thereby ensuring that all callers on-hold can select one of the styles that are offered; (5) default or initial music styles (such as determined by the resources of the exemplary system or called party, etc., as described above) are followed by audio prompts that enable the caller on hold to change the music style; and (6) the resources of the exemplary system further provide audio prompts that enable a user to select particular music styles and also parameters that may be changed for the music being auto-composed in the particular music style (in essence, audio prompt generation and digit detection is provided by the resources of the exemplary system to enable the caller on hold to alter parameters of the music being auto-composed, such as described elsewhere herein.
Other examples of novel ways to generally integrate aspects of the present invention with other products include: video camera (e.g., preferably to enable a user to easily create home movies with a royalty free, configurable soundtrack), conventional stereo equipment, exercise equipment (speed/intensity/style programmable, preferably similar to workout-intensity-programmable capabilities of the workout device, such as a StairMaster series of hills), configurable audio accompaniment to a computer screensaver program, and configurable audio accompaniment to an information kiosk system.
Aspects of the present invention can advantageously be employed in combination with audio watermarking techniques that can embed (and/or detect) an audio ‘fingerprint’ on the musical output to facilitate media content rights management, etc. The preferable incorporation of audio watermarking techniques, such as those described by Verance or Digimarc (e.g., the audio watermarking concepts described by Digimarc in U.S. Pat. Nos. 6,289,108 and 6,122,392, incorporated herein by reference), can enable a user with the ability to monitor the subsequent usage of their generated music.
In another example, certain embodiments of the present invention can be incorporated as part of the software of video game (such as a PlayStation 2 video game) to provide music that preferably virtually never repeats, as well as different styles preferably selectable by the user and/or selectable by the video game software depending on action and/or plot development of the game itself.
Certain embodiments involve the use of the algorithmic music generation described herein to provide music to a video game player. In one embodiment, referring to selected portions of
In certain video game embodiments, the music being generated is updated in response to events in the game. For example, in the case of a game where a user encounters various characters during the game, one or more characters preferably have a style of music associated with them. In these embodiments, although the music may not repeat, the style of music preferably will indicate to the user that a particular character is in the vicinity.
In certain video game embodiments, a particular component of the music (e.g., the lead instrument, or the drums) is associated with a character, and as the particular character moves in closer proximity to the video game user, the corresponding component of the music preferably is adjusted accordingly. For example, as a video game user moves closer to a particular villain, the music component that corresponds to that villain (e.g., the lead instrument) is adjusted (e.g., raises in relative volume, and/or exhibits other changes such as increased relative mobility of note pitch as illustrated in
In certain video game embodiments, information relating to the musical piece (e.g., information such as the parameters illustrated in
Additionally, there are certain novel variations to the present invention that incorporate many advantages of the present invention to great effect. For example, in the portable hardware device 35 in
In certain embodiments it is preferable to enable a vocal chord mode that analyzes the vocal microphone input in real time and, as part of the music composition being generated, mimics the input vocal characteristics in a realtime manner. As one example, this feature can provide a vocal chord effect that combines the user's vocal input events with one or more artificially generated vocal events. In this fashion, preferably a user can sing a part and hear a chord based on their voice. This feature may advantageously be used with a pitch-correcting feature discussed elsewhere in the present disclosure.
In certain embodiments it is preferable to provide a reference note to the user in real time. Such a reference note preferably will provide a tonic context for the user, e.g., so that they are more likely to sing in tune. Further, in certain embodiments, such a reference note may serve as a reference melody, and provide the user with an accurate melody/rhythm line to sing along with. These features may be particularly advantageous in a karaoke context. In many of these embodiments, it is desirable to limit the reference note to the user's ear piece (e.g., allowing the user to bar the reference note information from any recording and/or performance audio output).
Certain embodiments directed to additional inventive concepts associated with the generation of a singing part will now be further described in greater detail. This discussion may also provide further context for certain vocal-related features discussed herein.
On one level, vocal communication can be considered to be various combinations of a limited number of individual sounds. For example, linguists have calculated a set of approximately 42 unique sounds that are used for all English language vocal communication. Known as ‘phonemes’, this set is of the smallest segments of sound that can be distinguished by their contrast within words. For example, a long ‘E’ sound is used in various spellings such as ‘me’, ‘feet’, ‘leap’, and ‘baby’. Also, a ‘sh’ sound is used in various spellings such as ‘ship’, ‘nation’, and ‘special’. As further examples, a list of English language phonemes is available from the Auburn University website at http://www.auburn.edu/˜murraba/spellings.html. Phonemes are typically associated with spoken language, and are but one way of categorizing and subdividing spoken sounds. Also, the discussion herein references only English language examples. Clearly, using the set of English phonemes as an example, one can assemble a set of sounds for any language by listening and preparing a phonetic transcription. Similarly, while the above discussion references only spoken examples, the same technique can be applied to sung examples to assemble a set of phonetic sounds that represent sung communication. While the list of assembled sound segments preferably will be longer than the set of English phonemes, this method can be used to identify a library of sung sound segments.
The Musical Instrument Digital Interface standard (MIDI) was developed to enable a communication protocol for digital-based musical products. It has since become the defacto standard for all digital-based music-related products that are designed to interact with each other. MIDI incorporates the capability to assign a bank of sounds to a particular ‘soundbank’ such that, for example, a set of grand piano sounds can be selected by sending a soundbank MIDI command identifying the sound bank. Thereafter, it is possible to sound individual notes of the soundbank using, for example, a ‘note-on’ command. Additionally, certain characteristics indicating how the individual note should sound are available (e.g., such as ‘velocity’ to indicate the speed at which the note is initiated). More information on MIDI can be received from the MIDI Manufacturer's Association in California.
In accordance with embodiments of the present invention, MIDI can be advantageously incorporated into a larger system for generating music including a singing part. Given the context of a music generation environment such as previously referenced, a vocal track can be added to the system, preferably with a library of sounds (e.g., phonemes), and preferably through the use of a control protocol (e.g., MIDI). Following this example, a library of phonemes can be associated with a MIDI soundbank (e.g., preferably via a particular channel that is set aside for voice). Preferably, individual phonemes in the library can be identified by the ‘program’ value for a given MIDI event. The amplitude of a given piece of sound (e.g., phoneme) preferably can be indicated by the velocity information associated with a given MIDI event. The pitch of each piece of sound preferably can similarly be identified using the ‘note’ value associated with a given MIDI event. In addition, the desired duration for each piece of sound can preferably be indicated in MIDI using the sequence of a note-on event associated with a certain time, followed by a note-off event associated with a delta time (e.g., a certain number of MIDI clocks after the note-on event was executed). Of course, the alternative usage of another note-on command with a velocity of zero can also be used to indicate the end of a note, as is well known in the MIDI field.
Previously disclosed embodiments for the autogeneration of music have been mentioned herein. The various features and implementations of these previous disclosures include the use of musical rules (e.g., computer algorithms) to generate a complete musical piece. These musical rules can include ways of automatically generating a series of musical events for multiple simultaneous parts (e.g., drums, bass, etc.) of a musical piece. Further, preferably a music rule that can effectively quantize the individual parts such that they are preferably harmony-corrected to be compatible with a musical mode (e.g., descending Lydian). In addition, these musical rules can preferably incorporate rhythmic components as well, to quantize generated events of each part such that they preferably are time-corrected in keeping with a particular groove, or style of music (e.g., swing, reggae, waltz, etc.). Finally, the previously referenced disclosures also provide certain embodiments for enabling user-interaction with an auto-generated musical composition.
Using the music autogeneration concepts previously discussed, vocal processing functions can similarly be used that, for example, generate a string of MIDI events incorporating auto-generated pitch, amplitude, and/or duration characteristics, to sound a series of vocal sounds from a library of fundamental vocal sounds. As an example, the series of pitches generated preferably can be harmony-corrected according to a particular musical mode, and the rhythmic duration and/or relative loudness preferably can be rhythm-corrected according to a particular desired groove, time-signature, style, etc. Furthermore, in certain embodiments it is preferable to impose additional rules relating more specifically to text. As an example, human voice communication can be observed to only change pitch during certain phonemes (e.g., vowel sounds). Accordingly, in certain embodiments it is desirable to check the phoneme type during the autogeneration process before allowing a pitch value to be changed. In another example, in certain embodiments it is preferable to only allow pitch change to occur to an accented syllable, or a ‘long’ syllable (e.g., the “o” sound as opposed to the “range” sound in “orange”). In these embodiments it is desirable to confirm that a particular sound is compatible with a pitch change during the autogeneration process. Such screening activity can be performed in a variety of ways; as an example, the vocal library can be arranged as a sound bank, with the ‘pitch-change-compatible’ sounds arranged to be together in one region of the addressable range associated with the soundbank. Following this example, when the music rules are being performed, and a sounding vocal note is being given a note change event, it is preferable to check the MIDI address of the current sound to determine if it is in the ‘acceptable’ range associated with a pitch change event. As will be clear to one of ordinary skill in the art, this is but one implementation example.
The discussion thus far has addressed how to generate and play a series of nonsensical sounds in a musical generation system, preferably in order to approximate the sound of a vocal track in a musical composition. The next section adds to the mixture the ideas of text-to-speech algorithms, in order to enable a sensical string of vocal sounds.
Text information preferably can readily be analyzed to identify individual phonetic components. For example, language analysis routines can be employed to identify context for each word, correct misspelling for commonly misspelled words, and identify commonly used abbreviations. Many text-to-phoneme schemes exist in the prior art; one example is the 1976 article authored by Honey S. Elovitz, Rodney Johnson, Astrid McHugh, and John E. Shore entitled “Letter-to-sound rules for automatic translation of English text to phonetics” (IEEE Transactions on Acoustics, Speech and Signal Processing). As illustrated in
While the text can be user-definable/loadable, it also can be nonsensical. Certain examples in the music world incorporate nonsense singing syllables, such as ‘scat’ in jazz music, and ambient music from ‘The Cocteau Twins’, such as the “Heaven or Las Vegas” album. In certain embodiments where a more simple effect or architecture is desirable, it is preferable to have a reduced number of available phonemes in a sound library.
One variation of preferred embodiments of the present invention can be practiced by enabling an end user to enter or load text, for example, via a user input interface such as a keyboard (or, in certain embodiments, via voice input), via a removable memory location, and/or via an interface to a personal computer (i.e., such as disclosed in the referenced and incorporated patent applications), and enable the inputted text to be used as a basis for an auto-generated vocal part, musically compatible with a musical composition. As an example in the context of the previously referenced and incorporated patent applications,
In other variations, in addition to, or in lieu of, the use of a text input interface, it is desirable to enable the use of prefabricated text, e.g., through an interface to another computer/the Internet (e.g., Data I/O interface 38 in
In certain embodiments it is preferable to support the use of MIDI Lyric commands in carrying out the text-to-vocal processing. As an example, the text information can be provided in the form of a series of one or more MIDI Lyric commands (e.g., one command for one or more text syllables). Each MIDI Lyric command in this series is then preferably analyzed by the algorithm to create/derive one or more MIDI Program Change messages associated with the syllable. Following this example, in certain embodiments it is preferable to support a MIDI Karaoke file incorporating a series of MIDI Lyric commands associated with a song. Additionally, in certain cases it is preferable to parse the MIDI Lyric commands in a particular Karaoke file to subdivide the descriptors further, affording greater rhythmic control, as each Lyric command includes one delta time parameter associated with it. Accordingly, greater rhythmic control is provided as a finer degree of delta time control is possible.
In yet another embodiment, visual output interface 32 has a touchscreen overlay, and a keyboard may be simulated on visual output interface 32, with the keys of the simulated keyboard activated by touching of the touchscreen at locations corresponding to images of the keys displayed on visual output interface 32. With such embodiments, the use of the touchscreen may also obviate the need for most, if not all, of the physical buttons that are described in connection with the embodiments described in the referenced and incorporated patent applications.
In yet another alternative embodiment, the use of such key entry enables the user to input a name (e.g., his/her name or that of a loved one, or some other word) into the automatic music generation system. In an exemplary alternative embodiment, the typed name is used to initial the autocomposition process in a deterministic manner, such that a unique song determined by the key entry, is automatically composed based on the key entry of the name. In accordance with certain disclosed embodiments in the referenced and incorporated patent applications, for example, the characters of the name are used in an algorithm to produce initial seeds, musical data or entry into a pseudo random number generation process (PRNG) or the like, etc., whereby initial data to initiate the autocomposition process are determined based on the entry of the name. As one example, add the ASCII representation of each entered character, perhaps apply some math to the number, and use the resulting number as an entry into a PRNG process, etc. As another example, each letter could have a numeric value as used on a typical numeric keypad (e.g., the letters ‘abc’ corresponds to the number ‘2’, ‘def’ to 3, etc.,) and the numbers could be processed mathematically to result in an appropriate entry to a PRNG process. This latter example may be particularly advantageous in situations where certain of the presently disclosed embodiments are incorporated into a portable telephone, or similar portable product (such as a personal digital assistant or a pager) where a keypad interface is supported.
As the process preferably is deterministic, every entry of the name would produce the same unique or “signature” song for the particular person, at least for the same release or version of the music generation system. While the autocomposition process in alternative embodiments could be based in part on the time or timing of entry of the letters of the name, and thus injecting user time-randomness into the name entry process (such human interaction randomness also is discussed in the referenced and incorporated patent documents) and in essence a unique song generation for each name entry, in preferred alternate embodiments the deterministic, non-random method is used, as it is believed that a substantial number of users prefer having a specific song as “their song” based on their name or some other word that has significance to them (a user may enter his/her name/word in a different form, such as backwards, no capital letters, use nick names, etc. to provide a plurality of songs that may be associated with that user's name in some form, or use the numbers corresponding to a series of letters as discussed herein in connection with a numeric keypad interface). As will be appreciated by those of skill in the art, this concept also is applicable to style selection of music to be autocomposed (as described in the referenced and incorporated patent documents; the style could be part of the random selection process based on the user entry, or the style could be selected, etc.). For example, for each style or substyle of music supported by the particular music generation system, a unique song for each style or substyle could be created based on entry of the user's name (or other word), either deterministically or based, for example, on timing or other randomness of user entry of the characters or the like, with the user selecting the style, etc.
As will be appreciated, the concept of name entry to initiate the autocomposition process is not limited to names, could be extended to other alphanumeric, graphic or other data input (a birthdate, words, random typed characters, etc.). With respect to embodiments using a touchscreen, for example, other input, such as drawn lines, figures, random lines, graphic, dots, etc., could be used to initiate the autocomposition process, either deterministically or based on timing of user entry or the like. What is important is that user entry such as keyboard entry of alphanumeric characteristics or other data entry such as drawings lines via the touchscreen (i.e., e.g., data entry that is generally not musical in nature), can be used to initiate the composition of music uniquely associated with the data entry events. Thus, unique music compositions may be created based on non-musical data entry, enabling a non-musically inclined person to create unique music based on non-musical data entry. Based on such non-musical data input, the music generation process picks seeds or other music generation initiation data and begins the autocomposition process. As will be appreciated, particularly with respect to entered alphanumeric data entry, such characters also could be stored (either alone or with music generation initiation data associated with the data entry), could be transmitted to another music generation system, whereby the transmission of the non-musical data is used to, in effect, transmit a unique song to another user/system, with the transmission constituting only a small number of bytes of data to transmit information determining the song to be created by the music generation system.
In yet other embodiments, the music generation system assists in the creation of lyrics. In one exemplary embodiment, the user selects a style or substyle of music, and preferably selects a category of lyric such as ballad/story, rap, gangsta rap, rhymes, emotional, etc. Based on the style/substyle of music and the lyric category, in response to entry of a word or phrase, the system attempts to create lyrics consistent with the user selections, such as via the use of cognates, rhymes, synonyms, etc. Words or phrases preferred are characterized by the lyric category (e.g., story, emotion, nonsensical, etc.), which enables words or phrases to be selected in a manner more consistent with the user selections. In accordance with such embodiments, lyrics and phrases could be generated by the music generation system, preferably in time with the music being generated, and the user could selectively accept (e.g., store) or reject (e.g., not-stored) the system generated lyrics.
In addition to the use of pre-fabricated text, and user input text as discussed above, in certain embodiments it is preferable to generate the text itself. As an example, through the use of a commonly available algorithmic approach to generating strings of sensical text such as the A.L.I.C.E AI Foundation's “Artificial Intelligence Markup Language (AIML) version 1.0.1 working draft dated Oct. 25, 2001 (available to the public on the internet at: http://alice.sunlitsurf.com/), hereby incorporated by reference in its entirety. AIML is one example of a draft standard for creating a simple automatic text creation algorithm. It is useful in the present invention as a means to algorithmically support the auto-generation of text, and consequently, the auto-generation of phoneme streams as a vocal track. In the context of
In combination with many of the various embodiments and features discussed herein, in certain embodiments it is preferable to create a vocal library (e.g., vocal library A in
Certain additional embodiments associated with the routing of audio information will now be discussed.
The present example is an audio device that can preferably generate audio information (e.g., music) using DSP 42. In certain embodiments the audio information may be encoded in a proprietary format (i.e., an efficient format that leverages the hardware details of the portable music device). In certain other embodiments (for example, when the audio device is capable of storing data and/or connecting to other systems), it may be desirable to generate a non-proprietary format of the audio information (for example, in addition to a proprietary format) such that the audio information can be more easily shared (e.g., with other different systems) and/or converted to a variety of other formats (e.g., CDROM, etc.). As an example, one preferable format for the audio information is the MP3 (MPEG Audio Layer 3). As will be obvious, other formats such as WAV, ASF, MPEG, etc., can also be used.
Continuing the discussion above in connection with
In certain embodiments, digital signal 62 output from DSP 42 during an audio generation operation of player 10 can preferably be routed via additional signal line(s) to MP address bus 37 and MP data bus 38. In this example USB I/F 39 (alternatively Serial I/O 57) can advantageously be a slave USB device such as the SL811S USB Dual Speed Slave Controller also available from ScanLogic Corporation. In this example, while certain cost savings can be realized with the use of a simpler USB I/F 39 (as opposed to a master USB device such as the SL811R mentioned above), a trade-off is that MP 36 will need to be capable of controlling the flow of digital signal 62. This is primarily because in this example MP 36 is the master of MP address bus 37 and MP data bus 38, and will need to perform the transfer operations involving this bus. In certain cases where MP 36 already has sufficient capabilities to perform these added functions, this embodiment may be preferable. As mentioned above in the previous embodiment, in other cases where price/performance is at a premium the use of a more capable USB I/F 39 (alternatively Serial I/O 57) part can be used with little detrimental effect to the available resources on MP address bus 37 and MP data bus 38.
In the examples described above, the audio information output from DSP 42, in the form of digital data, is sent over the connector 53 for reception by system 460. System 460 must be capable of receiving such digital data via a corresponding bus port (e.g., a USB port, or alternatively, another port such as, for example, a port compatible with at least one of the following standards: PCMCIA, cardbus, serial, parallel, IrDA, wireless LAN (e.g., 802.11), etc.
Such an arrangement will preferably involve the use of a control mechanism (e.g., synchronization between the audio playing and capturing) to allow a more user-friendly experience for the user, while the user is viewing/participating in operations such as generation/composition of music on player 10, routing of digital audio information from digital signal 62 to connector 53, receiving and processing of audio information on system 460, and recording the audio information on system 460. One example of such a control mechanism is a software/firmware application running on system 460 that responds to user input and initiates the process with player 10 via connector 53 using control signals that direct MP 36 to begin the audio generation process. Alternatively, the user input that initiates the procedure can be first received on player 10 as long as the control mechanism and/or system 460 are in a prepared state to participate in the procedure and receive the digital audio information.
In the foregoing discussion, control information preferably flows between player 10 and system 460 over connector 53 (e.g., in addition to digital audio information). Such control information may not be necessary in order to practice certain aspects of the present invention, but if used, will provide the end user with a more intuitive experience. For example, in certain embodiments such an arrangement which incorporates a controllable data link preferably will not require a connection on analog audio I/O 66 (e.g., an analog audio link using, for example, an eighth inch stereo phono plug), as digital audio data can be controllably directed over connector 53 (e.g., in lieu of analog audio information passing over analog audio I/O 66).
In certain alternative embodiments, e.g., with more processing resources, digital signal 62 output from DSP 42 during an audio generation operation of player 10 can preferably be routed to a local memory location on the player 10 (e.g., a removable memory such as via SMC 40, or a microdrive, RAM, other Flash, etc.). In this fashion, a digital audio stream can be saved without the use of an external system such as system 460. Possible digital formats that can be used in such an operation preferably include MP3, WAV, and/or CD audio.
In other embodiments, routing audio information to system 460 (e.g., to enable sharing, etc.) can be achieved by routing analog signal 64 through analog audio I/O 66 to a corresponding analog audio input (e.g., eighth inch stereo phono input plug) on system 460 (e.g., by use of an existing sound card, etc.). In this case, the alternative signaling embodiments discussed herein preferably may not required, in that the digital audio information output from DSP 42 does not need to be routed to connector 53. Such an embodiment may be advantageous in certain applications, as it may not require either a more capable MP 36, or a mastering type of USB I/F 39, and accordingly may provide a more cost-effective solution. Consequently, the present embodiment can easily and efficiently be incorporated into player 10. In spite of such ease and efficiency, the present approach may be less desirable in certain respects than the previous embodiments, as the format of the audio information being passed to system 460 is analog, and thus more susceptible to signal loss and/or signal interference (e.g., electromagnetic). In any event, this arrangement can additionally preferably involve control information passing between system 460 and player 10 via connector 53. Such an arrangement can provide the end user with a more intuitive experience (in the various ways referred to herein) in that the user can initiate the process, and the synchronization of the process can be achieved transparently to the user via control information passing between the devices through connector 53.
At this time, we address certain novel embodiments of a file format that is particularly useful to use in the present embodiment. However, it should be understood by one of ordinary skill in the field of file formats that the portions of the present disclosure concerning file formats can be easily utilized in a variety of other contexts than a portable music device. Accordingly, while at times examples are referenced that may be associated with a music device, it should be clear that other very similar examples can be readily envisioned that involve other contexts, such as files used in general computing systems, music files such as compact disks, files used in other types of portable devices, etc.
The presently described “Slotted” file format involves a collection of independent units called “slots”. This typically provides some flexibility in the organization of data within a file, because slots preferably can be added or removed without affecting the other slots, slots of a given type preferably can be enlarged or shrunk without affecting the other slots, and the slots that have an unknown type within a given firmware/operating system (OS) release, or within the current context, preferably are ignored, which typically helps solve backward compatibility issues.
Having the same generic file format for all proprietary files typically permits the use of the same code for verifying the checksum and the consistency of the file data, as well as the first level parsing (e.g., to access each individual slot).
Referring now to
The “Data Length” field preferably contains the number of bytes that the Data Length field of each slot contains. In certain embodiments, this field is used to optimize the size of the Slotted Files, because certain types of Slotted Files may use a very small Data field or no Data field at all (e.g., Lists), whereas other types of Slotted Files use a very big Data field (e.g., Samples).
The “Num Slots” field preferably holds the number of Slots (N) in the Slotted Structure. Decoding the file is easier if the number of slots is readily available. In certain embodiments, this redundant information is used to verify the consistency of the Slotted Structure.
The purpose of the checksum is to protect the Slotted Files against read/write errors. In many embodiments, it is expected that the best protection would be given by a CRC calculation, but this typically would be slow and complex, and likely is not really required anyway, because, since read/write errors are very rare, we do not need a protection against multiple errors. Accordingly, it is expected that a simple checksum calculation is sufficient for this purpose.
In certain embodiments, e.g., involving a 32-bit processor, the fastest and most efficient checksum computation typically is to add all the data of a file, dword by dword. Unfortunately dword by dword computation, as well as word by word computation, can create alignment problems. In such embodiments, the case where the number of bytes in a file may not be a multiple of dwords or words preferably can be fixed by adding null padding bytes at the end of the file for the checksum computation. However, in such embodiments, a more complex checksum situation is when the file is written in several chunks, unless the size of each chunk is typically constrained to be a multiple of dwords or words.
An alternative embodiment involves a compromise solution to this issue by forcing all the fields of Slotted Files to be word aligned. In these alternative embodiments, all the Slots preferably have an even size, e.g., an integral number of words. In this manner it becomes relatively easy to compute the checksum word by word. This may not be not as fast as dword by dword computation, but it is nevertheless typically faster than byte by byte computation. In certain embodiments where the file is located on a relatively slow medium, such as a flash memory card, the impact of this issue is not enormous, because by far the biggest contribution to the checksum computation delay may be the time it takes to read the data.
We refer now to the exemplary Slot Format embodiment illustrated in
In certain embodiments, a special type of slot may hold a reference to a file. In such a “File” slot, the Slot Type preferably is associated with the File Type of the file referenced by the slot, and the “Name” field preferably contains the name of the file. Thus a File slot may reference a file independently of the type given by the SLS Type.
Certain types of Slotted Files may contain slots that reference a file in a specific way. In this case, the Slot Type preferably has a fixed value, which is valid for this type of Slotted File only. The “Name” field of such a slot preferably contains the name of the file, and the File Type preferably is given by the context. Continuing this example, the “Data” field of File slots may advantageously be used to store the File Settings in the File Settings File.
In the present discussion, “Alien slots” (in a given type of Slotted File) are slots whose type is not recognized by the current firmware and/or operating system release. As an example, Alien slots may exist if we read a file on Release A that was originally written in Release B (where Release B is typically more recent than Release A). In certain cases, a Slotted File created or modified on a computer may add its own types of slots, which the portable system might regard as alien slots. Typically, it is advantageous for all the Slotted Files to be able to accept alien slots, no matter where they are placed in the Slotted Structure, without creating an incompatibility. This arrangement incurs two constraints in the Slotted File management: alien slots preferably are ignored (e.g., they must have no effect in the current firmware release), and alien slots preferably are preserved (e.g., not deleted) when the Slotted File is modified. This arrangement preferably permits complementary information to be added to any Slotted File in future firmware/OS releases without creating an incompatibility. Preferably, older releases will be able to read the Slotted Files created in the new release, and if an older release modifies such a Slotted File, the information relevant to the new release preferably will be preserved.
In certain Slotted Files, it is desirable for all the slots to have distinct Slot Type values. In these cases, the order in which the slots are placed in the Slotted Structure preferably should not matter because the firmware will scan/load the entire file to find the slots that it is looking for. In some other types of Slotted Files, the order in which the slots are placed should matter, as the order preferably can be used to determine how the file operates. As examples, such an arrangement is desirable in the case with lists, since the items (e.g., Songs or Samples) can be played/executed/accessed/etc., in the order in which they are located in the Slotted Structure. Accordingly, in a given implementation, the slot ordering may matter for a class of slots (for instance the File slots), and may not matter for other classes of slots.
In certain embodiments, for a Slotted Structure of a given type (e.g., as defined by the SLS Type), certain slots (e.g., as defined by a Slot Type value or a range of Slot Type values) preferably contain a complete Slotted Structure (e.g., placed in the “Data” field of the slot). This arrangement is advantageous because it permits nested slot structures.
When reading any Slotted File, it may be advantageous to perform verification at one or more of the following three points: the type of Slotted Structure must be a known type in the current Release, the SLS Version must be supported in the current Release, and/or the Slotted File data must be consistent. The data consistency check preferably consists in verifying that the size of the file matches the sum of the sizes of the N slots, where N is the number of slots read in the SLS Header. So as an example, the data consistency check will detect an error in the number of slots specified in the SLS. In certain embodiments, these verifications are performed after the file checksum. This verification preferably is not redundant with file checksum verification, because the latter typically will detect errors in writing or reading the file, whereas the verification of Slotted Structure consistency typically will detect things like a change in the format of the Slotted Structure (e.g., an older format which is not supported in the current firmware/OS release). In certain cases, checksum verification preferably may be skipped, e.g., if it takes too long for real-time operation. This might be desirable in the case of relatively large files such as samples, which can be accessed for real-time playback.
In certain embodiments a Song file preferably may have one or more of the following four slots: a Song Data Slot, an Instrument Slot (e.g., holding the instrument indexes that may be stored separately), a Sample Set Slot (e.g., holding any associated Sample set file), and a Settings Slot (e.g., holding the Song Settings (Volume, Speed and Pitch). In the case where Samples are stored in a Slotted File, any applicable Sample Settings (e.g., Sample Descriptor, Normalization factor, Effect index and Effect type) preferably can be stored in the same file as the Sample data. This feature typically affords great flexibility for future evolutions. In certain embodiments it may be desirable to store samples as slots in a file.
The “Sample data” preferably designates the data (e.g., PCM data) involved in playing the Sample. In certain embodiments, all of the Sample data may be stored in the “Data” field of one slot. In other implementations, Sample files preferably involve additional complementary information such as, for example, the definition of a custom effect that may not fit in the Sample Settings slot. This complementary information preferably is stored in a different slot, e.g., with an associated Slot Type. Another variation (not necessarily mutually exclusive with the previous one) involves splitting the Sample into smaller chunks that preferably can be played individually. As one example, such an implementation permits playing the Sample with a time interval between the chunks (e.g., rather than continuously).
In certain embodiments, Sample files preferably have two slots: a Sample Data slot, which preferably may hold the Sample data (e.g., PCM data), and a Sample Settings slot, which preferably holds the Sample Settings (e.g., Sample Descriptor, Normalization factor, Effect, etc.). It is further desirable to allow alien slots to be accepted in a sample file (e.g., ignored if not recognized by the firmware/OS). As an example, this could be used to contain comments or other data associated with the sample file data.
Ideally, the order in which the slots are placed in the Sample file should not matter. However, Sample files have special constraints because they are big, as explained in “Sample File evolution” above. If we place the Sample Settings slot after the Sample Data slot, then in certain situations (such as, for example, a heavily fragmented cluster allocation) the time to load the sample settings information may be undesirably long. For example, in the case of fragmentation, typically it is not easy to calculate the address of the end of the sample file where the settings may be stored (i.e. the “Data” field of the Sample Settings slot). To address this issue, in certain embodiments the area to be modified is always located in the primary VSB (Variable Size Block in dynamic memory allocation) of the Sample file. This may be achieved by making the Sample Settings slot the first slot in the Sample file, based on the fact that the length of this slot is small enough in comparison with the size of a cluster.
In certain embodiments that involve a List File, (e.g., in cases where each slot may identify an item or action, and the order of the slots affects the order of play and/or execution), it is advantageous for the last slot to be a “Terminating Instruction slot”, which tells what happens after executing the last item. As examples, such a slot might indicate “stop playing” or “loop to beginning” to keep on playing until there is a manual intervention. In certain of these embodiments, the order in which the File slots are placed in the List File preferably determines the order in which items of the List (e.g. Samples or Songs) are executed and/or played (e.g., even if a terminating instruction slot is not incorporated). On the other hand, if used, the Terminating Instruction slot typically can be placed anywhere in the S List File.
Following the above examples, a List File may hold references to one or more files that are no longer available. Each reference (e.g., held by a File slot) to a File that does not longer exist can be considered a “lost item”. When a List File is modified, lost item slots, unlike alien slots, preferably are deleted.
In certain embodiments that involve a list of items such as radio station presets, the presets of the radio stations preferably are stored in a dedicated List File. As an example, each slot of such a Radio List File preferably holds the name and frequency of a preset. In this example it preferably is possible to add more fields, e.g., within the “Data” field, without creating an incompatibility.
In certain embodiments that involve graphic files, it is desirable for graphics to be stored in a Graphics file. The Slotted File format presently discussed preferably permits several images/graphics to be stored, e.g., each with different attributes. This arrangement permits various still images (e.g., for multiple GUI display screens on a portable device) to be located in one file. In certain cases this arrangement can also be used to store animations, e.g., data associated with successive frames of the animation preferably can be located in individual slots, and the ordering of the slots preferably can be used to determine the ordering of successive frames of the animation.
Musical generation systems that generate music according to musical rules are typically fully aware of a specific musical mode, as well as a set time signature. The musical rules/algorithms that generate audio information typically are organized to operate around particular musical modes and time signatures so that the resulting output is musically compelling. As such, at any point in time during the generation or playback of a generatively created musical piece, there are defined variables that track the mode (e.g., a set of pitches that are compatible at any point in time) as well as the time signature (e.g., the beat and/or groove of the music, including the appropriate locations for accents, drum fills, etc.).
In addition, a final musical output is often favorably augmented with the use of selected samples that are played back in response to user-input or as part of the musical rules. As an example, a short sample of a female voice singing a simple melody may be favorably added to a particular hip-hop style song at certain key moments. Such an example illustrates how pre-recorded sounds can be utilized by a musical generator/playback system or algorithm to enhance the musical output.
The use of a sample format that is non-proprietary is also very desirable, as it enables an end user to easily share and/or generate samples of other sounds, possibly using other means, and include them in a generatively created musical piece. Such an arrangement allows a high degree of creativity and musical involvement on the part of the end user; even in cases where the end user has no musical background or experience. In fact, it is one of the aims of the present disclosure to effectively provide such a novice end user with a compelling experience in musical creativity.
Additionally, certain preferable embodiments of a musical system enable the use of signal processing on the samples used. This approach allows, for example, the user to easily adjust the pitch or speed of a sample with the use of certain DSP functionality in a preferred type of system. As one example, the Dream DSP chip available from Atmel Corporation, data sheets and application/user manuals for which are hereby incorporated by reference, allows samples to be adjusted along various lines, including pitch, speed, etc., as well as the application of various preferred sound effects, such as doppler, warbler, echo, etc. These aspects and features are described in greater detail herein.
One problematic aspect in the generative creation of audio content is that the playback of the sample during a section of music can sometimes sound out of sync with the music in terms of pitch or rhythm. This generally is a result of the lack of a default synchronization between the sample and the music at a particular point in time. One way around this is to use samples that do not have a clear pitch or melody, e.g., a talking voice, or a sound effect. However, as the use of melodic samples, especially at higher registers, is desirable in many styles of music, it is desirable in certain cases to have the capability for associating pitch and/or periodicity information (embedded or otherwise) into a sample. Such information can then be interpreted by the musical rules and/or algorithm of the music device to enable a synchronization of the sample to the particular pitch, melody, and/or periodic characteristics of the musical piece. A particular example of such an arrangement in accordance with certain preferred embodiments will now be described.
Preferably, tag ID 525 is used to identify optional header 530 to a compatible system, and may be used to provide backwards compatibility in a non-compatible system. The backward compatibility is achieved, for example, by providing a pointer to the start of sound sample data 510, in such a way that a legacy system reading the sample will read sound sample data 510 and disregard period info 520 or pitch info 515 (as examples). In certain embodiments, this preferably may be achieved via the slotted file format described herein. In certain additional embodiments, this preferably is achieved in a manner similar to the way in which mixed mode CDROMs are encoded to work on both native-Apple and native-IBM-compatible personal computer CDROM drives, in effect by providing a pointer at the beginning of the volume that transparently causes each system to skip to the correct portion of the volume. Such a technique fools the legacy system into believing that the volume (or file) is actually a legacy-formatted volume. So, while benefits of the present invention may be utilized when the sample data file is provided with period info 520 and/or pitch info 515 in header field 530 in a system that can benefit from the additional data in header field 530, the same sample data file can also be used to provide sample data in systems that cannot benefit from the additional data in the header field. Preferably, in the latter case the file will appear to the system as providing only sound sample data 510.
Pitch information 515 preferably contains parameter data indicating a pitch associated with the sample, preferably given a native periodic rate. This parameter data could indicate, for example, that if the sample is played at a normal, unadjusted rate, a pitch value is associated with it corresponding to C#. This preferably would indicate to the music rules and/or algorithms involved in the music generation process that the associated sound sample data should be treated as a C# note. In other words, if a C# note is deemed compatible, the sound sample preferably may be played without any pitch adjustment. One benefit to this arrangement is that in the event that C# is not deemed compatible, the sound sample data preferably can be pitch-transposed up or down to an appropriate pitch value. Since the algorithm/music rules will know the pitch info for a native playback speed, they can calculate an adjustment to pitch, and preferably use DSP resources and functionality (such as discussed herein with respect to the Dream chip) to adjust the perceived pitch of the sample during playback. Thus, sample pitch preferably will generally conform to an appropriate pitch, given the current mode (as one example).
Period information 520 preferably contains period-related parameter data. This data preferably identifies a timing characteristic of associated sound sample data. For example, if the sound sample has a rhythmic emphasis point 75 milliseconds from the start, this parameter preferably might indicate this. In this example, the period portion of the header will inform the music rules/algorithm generating a musical piece that this particular sound sample has a rhythmic emphasis point 75 milliseconds into it, when played at native speed. Accordingly, the music rule/algorithm preferably will know the rhythmic emphasis point of the sample when it is loaded into memory, and when the end user plays the sample during a song, the music rules/algorithm preferably can time-adjust the sample such that the rhythmic emphasis point occurs at a suitable point in the music (e.g., a downbeat). This time-adjustment preferably can be made using, as an example, DSP resources and functionality as discussed herein. The Period information 520 preferably can be conveyed in the form of beats per minute (BPM) information, e.g., in cases where the sample has an identifiable tempo.
As can be appreciated, such embodiments of the present invention can provide the music playing system with a way to improve the musical experience of the end user. Such an arrangement can preferably be employed in a hardware system, e.g., a handheld portable device, or a software system, e.g., a software musical system running on a personal computer. Samples preferably can thus be played during a song by an end user, with relatively little ill effects due to pitch or time incompatibilities.
As illustrated in
In this example, we assume for the sake of clarity that the musical mode is Lydian Descending at the point in a song wherein the sample is played.
Accordingly, the first rhythmic subpart identified as T0:P0 has a pitch value that is allowable in the Lydian Descending Mode. Therefore, preferably this section of the sample is not pitch-shifted. Similarly, as the rhythmic event of the start of the sample is preferably initiated by the end user, the time T0 is not adjusted.
Continuing this example, the second subpart identified as T1:P1 has a pitch value of F#, which is allowable in the Lydian Descending mode, and is therefore preferably not pitch-shifted. The rhythmic event T1 is associated with 50 ms, and accordingly this can be time-shifted to more closely match the tempo of the music (e.g., it can be slightly slowed down or speeded up so as to coincide with the rhythmic grid/resolution of the music being created). If the section of the song has a tempo wherein, as an example, an eighth note has a duration of 60 ms, then the start of T1:P1 preferably could be adjusted to occur in time with the beginning of an eighth note, and/or the duration of T1:P1 preferably could be slightly lengthened so as to occupy 60 ms, in keeping with the time signature of the music. Of course, these examples constitute various options that may not necessarily be used in order to practice the present invention.
The third subpart T2:P2 would begin at 110 ms, if the preceding subparts had been played at normal speed. As our example has the previous subpart being lengthened by 10 ms, preferably to be in synch with our tempo, the T2:P2 subpart begins at 120 ms. Accordingly, in our example of a 60 ms eighth note, the beginning of subpart T2:P2 preferably will occur in synch with the tempo, and as it has a duration of 90 ms, it preferably could be time-stretched 50% for a duration of 120 ms (in keeping with our 69 ms eighth note), be time-reduced to 60 ms, or remain at 90 ms (e.g., depending on the particular magic rules/algorithms in place for the style of music being played). As this subpart does not have an associated pitch value, it preferably is not pitch-adjusted.
The last subpart T3:P3 in
The information identifying the Pitch and Periodicity associated with a sample can be contiguously arranged, as illustrated in
In certain preferred embodiments of the present invention, note that the Pitch and Periodicity info does not have to be in a header field portion of a sample file, but alternatively could be in separate descriptor file (e.g., a general sample list file for multiple samples or a single second file for each sample with the same name, different suffix, etc.). One example of this is shown in
Upon accessing and/or loading the sample file (step 660),
Accordingly, as discussed in detail herein, in the case of a sample for which a series of pitches and rhythmic events have been detected, such events can be time-adjusted and/or pitch-adjusted so as to more closely match the music being generated. This is done using the digital signal processing resources or functions of the system, e.g., through the execution of DSP instructions to carry out DSP-type operations. As will be appreciated, the present invention may be applied in a simpler way, such as by assigning only a single pitch value and/or rhythmic event to a sample, as opposed to a series.
Previously herein, e.g., particularly in connection with
Preferred embodiments of the present invention provide a portable automatic music generation device, which may be implemented such as disclosed herein. In accordance with the present invention, however, it should be noted that a wireless capability, such as that used in a cell phone, a personal data assistant, etc., may easily be incorporated into the example architectures of the previously described portable automatic music generation device. As one example, a USB communication interface of the previous disclosure preferably could be substituted with a communication interface connecting to the data reception and/or broadcast circuitry of a preferably RF-amplifier-based cellular communication module. Other data interface arrangements can be effectuated while still achieving the benefits of the present invention. Similarly, the portable device may be part of an automobile-based system (e.g., radio or communications system), or even part of a home-based music system (e.g., for purposes of compatibility with bandwidth-poor portable technology). All such implementation examples are within the scope of the present invention.
As discussed elsewhere in this specification, the use of certain features of the present invention is particularly advantageous in a telephone device such as a portable telephone, where a ringtone may be algorithmically created in a manner consistent with many aspects of the present invention. In certain examples where it may be preferable to instill a degree of variety in a ringtone (e.g., so that it is noticeably different each time the telephone rings), it may also be preferable to retain a degree of recognizability in the theme of the ringtone. This feature may be particularly advantageous in connection with an artist-specific ringtone or the like, so that a song is recognizable each time the phone rings, but it is also a bit different each time. As an example, a technique may be used where a subset of the music lanes (tracks, instruments, etc.) may be inaccessible (e.g., to the user, the algorithm, etc.). In this example, this subset of lanes may contain the main theme of a song (or style, etc.). In this fashion, the other lanes may vary each time the ringtone is played, yet the theme will remain recognizable. Many of these features have applicability to other, non-ringtone, implementations as well, such as a portable music player, video game device, website plug-in, etc.
An additional feature that is preferable in certain portable environments such as telephone ringtone composition environments is to perform the algorithmic composition at the conclusion of a phone call. In this example, as discussed in more detail above, a ringtone may vary to some degree each time it is invoked (e.g., via an incoming phone call). In certain embodiments, the ringtone may vary to a limited extent, in a manner that allows a recognizable theme to be constant. In various of these examples, it is necessary to generate the ringtone using some autocomposition algorithms, at some point before the ringtone is invoked the next time. In certain cases where there are sufficient processing resources, it may be preferable to perform this autocomposition in real time as the incoming phone call triggers the ringtone process. However, it is expected that in certain situations where processing resources may be more limited, e.g., such as in certain portable environments that minimize processing resources to maximize battery life, etc., it is preferable to initiate the ringtone autocomposition process at a more ideal time, such as when the portable device is not in a particularly busy state of operation (e.g., participating in a phone call, or some other mode that occupies substantial processing resources). In one example, it is considered advantageous to trigger the ringtone autocomposition process at the time that a phone call is terminated (or shortly thereafter). A ringtone preferably is autocomposed upon termination of a phone call, resulting in a ringtone that will be played at the next ringtone event. In this fashion, the variable ringtone feature is provided in a manner that minimizes the required amount of processing resources.
Novel aspects of embodiments of the present invention include the usage of a particularly efficient music distribution and generation system. Unlike various FM Radio broadcast systems, or Internet streaming systems in conventional approaches, the present invention utilizes music that preferably can be generated by the Node, and preferably not by the Transmitter. The Node receives a data file that contains, in essence, data or instructions that define a song to be generated, and may be used by the hardware/software of the Node in order to generate a song (the data or instructions, however, may include some sub-components of the song that are akin to prior art broadcasting/streaming systems, such as samples). Examples of such information, data or instructions are discussed below with reference to
‘Application Revision’ is preferably used to store the firmware/application version used to generate the data structure. This is particularly helpful in cases where the firmware is upgradeable.
‘Style/SubStyle’ preferably is used to indicate the Style and/or SubStyle of music. This is helpful when initializing various variables and routines, to preferably alert the system that the rules associated with a particular Style and/or SubStyle will govern the song generation process. In certain preferred embodiments, Style and/or SubStyle can refer to a radio station style of music, such as ‘Hard Rock’, ‘Ambient’, ‘Easy Listening’, etc. In certain cases, for example as discussed below, the radio station style may be user-selectable prior to the reception of the music data file.
‘Sound Bank/Synth Type’ preferably indicates the particular sound(s) that will be used in the generation of the song. As an example, this can be a way to preload the sound settings for a MIDI DSP resource.
‘Sample Frequency’ preferably is a setting that can be used to indicate how often samples will be played, if samples are incorporated into the song. Alternatively, this preferably can indicate the rate at which the sample is decoded, which provides a technique useful for adjusting the frequency of sample playback.
‘Sample List’ preferably lists all of the samples that are associated with the data structure. This list preferably allows a user to further select and play relevant samples during song playback.
‘Key’ preferably is used to indicate the first key used in the song. Preferably, one way to indicate this is with a pitch offset.
‘Tempo’ preferably is used to indicate the start tempo of the song. Preferably, one way to indicate this is with beats per minute (BPM) information.
‘Instrument’ preferably is data that identifies a particular instrument in a group of instruments. For example, this could reference an acoustic nylon string guitar among a group of all guitar sounds. This data is preferably indexed by instrument type.
‘State’ preferably is data that indicates the state of a particular instrument. Examples of states are: muted, un-muted, normal, Forced play, solo, etc.
‘Parameter’ preferably is data that indicates values for various instrument parameters, such as volume, pan, timbre, etc.
‘PRNG Seed Values’ preferably is a series of numerical values that are used to initialize the pseudo-random number generation (PRNG) routines (such PRNG Seed Values are used in certain embodiments, but not in other embodiments; the present invention is not limited to the use of such PRNG Seed Values). These values preferably represent a particularly efficient method for storing the song by taking advantage of the inherently predictable nature of PRNG to enable the recreation of the entire song.
‘Song Structure’ preferably is data that preferably lists the number of instrument types in the song, as well as the number and sequence of the parts in the song.
‘Structure’ preferably is data that is indexed by part that preferably can include the number and sequence of the sub-parts within that part.
‘Filtered Track’ preferably is a parameter that preferably can be used to hold data describing the characteristics of an effect. For example, it preferably can indicate a modulation type of effect with a square wave and a particular initial value. As the effect preferably is typically connected with a particular part, this parameter may preferably be indexed by part.
‘Progression’ preferably is characteristic information for each sub-part. This might include a time signature, number and sequence of SEQs, list of instrument types that may be masked, etc.
‘Chord’ preferably contains data corresponding to musical changes during a sub-part. Chord vector (e.g., +2, −1, etc.), key note (e.g., F), and progression mode (e.g., dorian ascending) data preferably are stored along with a time stamp.
‘Pattern’ and the sub-parameters ‘Combination’, ‘FX Pattern’, and ‘Blocks’, all preferably contain the actual block data and effects information for each of the instruments that are used in the song. This data is preferably indexed by the type of instrument.
Additional parameters can preferably be included, for example to enable at least some of the soundbank data associated with a particular song to be embedded. Following this example, when such a broadcast music data file is accessed, at least some of the sound bank data preferably is loaded into non-volatile memory such that the sound bank data may be used during the generation of music output.
Additionally, many of these parameters preferably can incorporate data with associated timestamps. This optional feature can preferably be used to indicate the timing of each event, etc.
Through the use of such exemplary parameters in a broadcast song data structure, data from which a song can be generated preferably can be efficiently broadcast to a number of node music generator devices. Though the specific parameter types preferably can be varied, the use of such parameters preferably enables all the details necessary to accurately and faithfully regenerate a song from scratch at a node.
At the start of
In yet another alternative embodiment, referring back to
In certain embodiments, characteristics of the music output can be adjusted during the alarm operation. As an example, the style of music may progressively change from a quiet, relaxing ambiance to a more energetic and loud style of music. Preferably this progression occurs each time the user presses a snooze button (e.g., via user input interface 780) or at predetermined intervals of time (which may be without further user action). In this manner the alarm clock can first wake the user with a relaxing quiet and/or simple piece, and progressively become more lively the longer the user chooses to remain in bed (e.g., by continuing to press the snooze button, or alternatively, by simply remaining in bed without turning off the alarm).
In certain alarm clock embodiments it is preferable to similarly progressively adjust the music from a soothing chord progression to a more dissonant one. As an example, referring to
In certain alarm clock embodiments it is preferable to start with music with a relative lower mobility of note pitch (e.g., see
In yet another alternative embodiment, referring back to
As the process preferably is deterministic, every entry of the name would produce the same unique or “signature” song for the particular person, at least for the same release or version of the music generation system. While the autocomposition process in alternative embodiments could be based in part on the time or timing of entry of the letters of the name, and thus injecting user time-randomness into the name entry process (such human interaction randomness also is discussed in the referenced and incorporated patent documents) and in essence a unique song generation for each name entry, in preferred alternate embodiments the deterministic, non-random method is used, as it is believed that a substantial number of users prefer having a specific song as “their song” based on their name or some other word that has significance to them (a user may enter his/her name/word in a different form, such as backwards, upside down using numbers, no capital letters, use nick names, etc. to provide a plurality of songs that may be associated with that user's name in some form, or use the numbers corresponding to a series of letters as discussed herein in connection with a numeric keypad interface). As will be appreciated by those of skill in the art, this concept also is applicable to style selection of music to be autocomposed (as described in the referenced and incorporated patent documents; the style could be part of the random selection process based on the user entry, or the style could be selected, etc.). For example, for each style or substyle of music supported by the particular music generation system, a unique song for each style or substyle could be created based on entry of the user's name (or other word), either deterministically or based, for example, on timing or other randomness of user entry of the characters or the like, with the user selecting the style, etc.
As will be appreciated, the concept of name entry to initiate the autocomposition process in Node/Subscriber Unit Music Generator Device 720 is not limited to names, could be extended to other alphanumeric, graphic or other data input (a birthdate, words, random typed characters, etc.). With respect to embodiments using a touchscreen, for example, other input, such as drawn lines, figures, random lines, graphic, dots, etc., could be used to initiate the autocomposition process, either deterministically or based on timing of user entry or the like. What is important is that user entry such as keyboard entry of alphanumeric characteristics or other data entry such as drawing lines via the touchscreen (i.e., e.g., data entry that is generally not musical in nature), can be used to initiate the composition of music uniquely associated with the data entry events. Thus, unique music compositions may be created based on non-musical data entry, enabling a non-musically inclined person to create unique music based on non-musical data entry. Based on such non-musical data input, the music generation process picks seeds or other music generation initiation data and begins the autocomposition process. As will be appreciated, particularly with respect to entered alphanumeric data entry, such characters also could be stored (either alone or with music generation initiation data associated with the data entry), could be transmitted to another music generation system (e.g., via Transmitter 710), whereby the transmission of the non-musical data is used to, in effect, transmit a unique song to another user/system, with the transmission constituting only a small number of bytes of data to transmit information determining the song to be created by the music generation system.
Additionally, many aspects of the present invention are useful to enable a new concept in Firmware upgrades. Using aspects of the present invention, firmware updates can be made available to users, complete with embedded advertising, which provides the Firmware manufactures/distributors with a revenue source other than the user. This concept preferably involves the distribution of firmware (or other software-based programs such as sound bank data) upgrades that contain embedded advertising images (and/or sounds). Such images/sounds preferably can temporarily appear during the operation of the music product, and can fund the development of customized firmware for users to preferably freely download.
Presently preferred embodiments associated with interpolation aggregation during audio synthesis will now be described in connection with
An exemplary synthesis structure for performing audio synthesis is illustrated in
As the small processing loop that is performed repeatedly by the structure depicted in
As an example for purposes of comparison, for a given channel, the prior art approach depicted in
An additional benefit to the approach depicted in
In certain embodiments, a further optimization in*operation of (III) Gain depicted in
input signal*gain=gained signal (1)
gained signal*left modifier=left channel output (2)
gained signal*right modifier=right channel output (3)
As can be imagined, this processing can occupy significant resources, especially when multiple simultaneous audio events 901 are being processed (see
left modifier*gain=gained left channel modifier (4)
right modifier*gain=gained right channel modifier (5)
Equations (4) and (5) preferably are performed relatively seldomly, such as once every 5 milliseconds (e.g., in the case of 44.1 kHz). Then, the following second set of two equations preferably is performed more frequently, such as 44 times per millisecond:
input signal*gained left channel modifier=left channel output (6)
input signal*gained right channel modifier=right channel output (7)
In comparing the typical implementation (as discussed above in connection with equations (1), (2), and (3)) with the preferred embodiments (as discussed above in connection with equations (4), (5), (6), and (7)), it is evident that a substantial reduction in required MIPS is achieved (e.g., on the order of approximately 6%, in certain cases). This reduction is primarily because at most of the cycles (e.g., in the examples above, approximately 219 of every 220 cycles in a 5 millisecond period), only two equations are performed instead of three.
Accordingly, each of the two preferred embodiments associated with
Presently preferred embodiments associated with a MIDI sound bank with a reduced memory area or footprint will now be described in connection with
In certain situations, such as a portable environment, it is desirable for MIDI sound banks to be characterized with a relatively small memory footprint size. In the following discussion, a prior art sound bank called “GM” originally developed by Roland Corp., and now included as a standard part of Microsoft Windows, has a memory footprint of approximately 3,361 KB. Such a large size may work well in certain situations, but certainly it is undesirably large for a portable MIDI implementation such as a cellular telephone, handheld video game, portable keyboard, personal digital assistant, etc., where typically reduced resources (e.g., processing, battery life, memory, etc.) are available as compared to a desktop personal computer. Accordingly, set forth below are several techniques that together or separately preferably provide a high quality MIDI sound bank solution with a significantly reduced level of required resources. Of course, in certain situations it may be preferable to use one or more of these techniques in a non-portable environment as well, for a variety of reasons that may be discussed in more detail below.
Additionally, the discussion below references certain examples of waveforms that may comprise part of a MIDI-compatible sound bank such as a (DLS) compatible sound bank, as referenced above. As described in the DLS specification materials available from the MMA, DLS-compatible sound banks typically include waveform data and associated parameter data. Typically, the waveform data may have a “loop period” identified that sets forth the portion of the waveform that will continuously loop as needed during playback to support the duration of the MIDI note. Typically, prior art sound banks include waveforms for each instrument, and such waveforms typically have a section towards the end that is identified as a loop period.
In addition to the foregoing teachings, in certain preferred embodiments it may be preferable to use a relatively large sound bank, yet only select the needed sounds for a given use, such as on a song-by-song basis. In this fashion, only the sounds needed for a particular use (e.g., such as a particular song being played) need to be loaded in RAM. One advantage of this approach is that the greater sound quality preferably afforded by a larger sound bank may be used, while not occupying too much RAM space, which in certain situations may be the most critical part of memory (e.g., such as may be the case with mobile phones). Taking an example where 128 KB of ROM or Flash memory may be available for storing a sound bank, and wherein 32 KB of RAM may be available for storing the sound bank when the synthesizer is running, the advantage of the present technique is that a sound bank may preferably be sized at up to 128 KB, provided that one single MIDI song does not use more than 32 KB of data (e.g., source samples and parameters). A potential disadvantage of this technique may be that it may be problematic to guarantee that any one MIDI song will not use more than 32 KB of sound bank data. An additional technique that preferably addresses this potential problem is discussed below. The use of a sub-portion of an available sound bank on a song-by song basis is not limited to the exemplary sizes discussed herein, but rather may preferably be used in other situations where more or less resources are available for sound bank data. This technique provides a high quality sound bank, while reducing the impact on memory size footprint for a given application (e.g., song being played).
In addition to the foregoing teachings, an additional technique may be used to guarantee that all MIDI songs will not use more than a predetermined subset (e.g., 32 KB) of the available sound bank (e.g., 128 KB). This technique preferably involves associating a plurality of sounds in the available sound bank with a given instrument, wherein the sounds may preferably be of different quality and/or sizes, and wherein a lesser quality and/or smaller sized sound preferably may selectively be substituted for a higher quality and/or larger sized counterpart sound, in situations where the predetermined subset size is in danger of being exceeded. In this fashion, in the case where a given song can use the high quality and/or larger sized sounds for all the instruments used in the song, while still remaining within the predetermined memory size (e.g., 32 KB of RAM), then the song will be played using the high quality and/or larger sized sounds. However, in certain situations wherein a given MIDI song calls for instrument sounds that may collectively total more than the available size (e.g., 32 KB RAM) in their highest quality and/or largest sized versions, the present invention calls for an algorithmic determination to be made to determine which instruments preferably will be sounded with their lower quality, and/or lower sized sounds, preferably to stay within the predetermined memory size (e.g., 32 KB RAM). The algorithmic determination preferably can be based on randomness, importance of the instrument, tables, etc., and in certain situation preferably may be based on the Scalable Polyphony MIDI Specification (SP-MIDI) published by the MMA, and incorporated herein by reference.
In certain embodiments it may be preferable to use a redirection means (e.g., such as a table or state machine) to allow the redirection of individual instrument definitions, such that particular instruments may be redirected to other instrument definitions. As an example, if an instrument for General MIDI (GM) instrument #N is not included in a sound bank (e.g., to reduce the size of the sound bank), this redirection means preferably will indicate that instrument #N has been remapped to the instrument definition associated with instrument #P. In this example, both instruments #N and #P preferably will share the same sound (instrument) definition.
As will be understood by a person of ordinary skill in the art of portable electronic music design, the examples discussed here are representative of the full spirit and scope of the present invention. Additional variations, some of which are described here, incorporate many aspects of the present invention.
Although the invention has been described in conjunction with specific preferred and other embodiments, it is evident that many substitutions, alternatives and variations will be apparent to those skilled in the art in light of the foregoing description. Accordingly, the invention is intended to embrace all of the alternatives and variations that fall within the spirit and scope of the appended claims. For example, it should be understood that, in accordance with the various alternative embodiments described herein, various systems, and uses and methods based on such systems, may be obtained. The various refinements and alternative and additional features also described may be combined to provide additional advantageous combinations and the like in accordance with the present invention. Also as will be understood by those skilled in the art based on the foregoing description, various aspects of the preferred embodiments may be used in various subcombinations to achieve at least certain of the benefits and attributes described herein, and such subcombinations also are within the scope of the present invention. All such refinements, enhancements and further uses of the present invention are within the scope of the present invention.
This application is a continuation-in-part of U.S. application Ser. No. 337,753 filed on Jan. 7, 2003 and International Application No. PCT/US03/25813 filed on Aug. 8, 2003.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5450597 | Klappert | Sep 1995 | A |
| 20040044487 | Jung | Mar 2004 | A1 |
| 20140093105 | Okano | Apr 2014 | A1 |
| 20150025664 | Cory | Jan 2015 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20150148927 A1 | May 2015 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 13066538 | Apr 2011 | US |
| Child | 14469490 | US | |
| Parent | 10541640 | US | |
| Child | 13066538 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10337753 | Jan 2003 | US |
| Child | 10541640 | US |
