The present invention will now be described in detail below with reference to the drawings showing preferred embodiments thereof.
A main controller or comparison unit 13 inputs data indicating a reproduction timing of the musical characteristic from the characteristic extraction unit 12, and inputs data indicating a user input timing from a first operation unit 17, which can be a sensor, via an interface 18. Based on these data, the main controller 13 determines the suitability (accuracy) of timing of the user based operation, and outputs data representing a result of the determination to a reproduction controller 14. Furthermore, the main controller 13 can control the reproduction controller 14 based on a user instruction input from a second operation unit 16.
The reproduction controller 14 reads out music data stored in the memory 11, controls a reproduction condition based on the result of the determination input from the main controller 13, and causes a sound system or reproduction unit 15 to implement audio data reproduction in the reproduction condition that is made different depending on whether or not the user input timing is suitable. The reproduction controller 14 includes a control unit 141 for outputting an instruction on the reproduction condition to various parts of the reproduction controller 14 based on the determination result input from the main controller 13.
A readout control unit 142 reads out audio data stored in the memory 11 in accordance with the instruction from the control unit 141, and outputs the audio data to a filter unit 143. If the audio data has been compressed, a data extension process is implemented by the readout control unit 142.
In accordance with an instruction input from the control unit 141, the filter unit or erasing unit 143 filters the audio data, such as for cutting a predetermined frequency component of the audio data, and outputs the filtered audio data to an effector unit 144. In accordance with an instruction input from the control unit 141, the effector unit 144 adds an effect to the audio data. The effect can include volume control, low-pass/high-pass filtering, distortion, chorus, reverb, echo, etc.
The sound system 15 can reproduce the audio data output from the effector unit 144. The second operation unit 16 can be provided with a plurality of push buttons or actuators through which the user can input a user instruction. The second operation unit 16 can detect depression of any of the push buttons, and output a result of the detection (the user instruction) to the main controller 13.
The first operation unit 17 can be an acceleration sensor for detecting a motion of the audio reproduction apparatus 1 (for instance, at least a vertical motion input by the user) and for outputting a detection output. The interface 18 inputs the detection output of the sensor 17, and outputs the same to the main controller 13.
Next, with reference to
An envelope curve generator unit 122 detects crests and troughs of a waveform obtained from the audio data having been subjected to the filtering processing in the filter unit 121, and generates an envelope curve by connecting the waveform crests together and the waveform troughs together. It should be noted that it is not necessary to generate the envelope curve. However, as a result of the envelope curve generation, the waveform can be simplified, resulting in simplified subsequent processing.
The comparator unit 123 inputs waveform data corresponding to the envelope curve from the envelope curve generation unit 122, compares the level of the input waveform data with a predetermined threshold value, and determines a time period along reproduction time axis during which the threshold value is exceeded by the waveform data level. The threshold value is set so that it exceeds at the instance when a bass guitar or a bass drum is played for instance.
The threshold comparator unit 123 outputs a time information string consisting of pieces of time information each of which is at a low level during when the waveform data representing the envelope curve is at a level lower than the threshold value and at a high level during when the waveform data is at a level higher than the threshold value. In this embodiment, the time information string indicating whether or not the predetermined threshold value is exceeded is generated from the waveform data extracted in the filter unit 121 as described above, whereby the rhythm of audio data can be detected.
By broadening the width of the time information along the reproduction time axis, it is possible for the main controller 13 to determine that a user operation or input is implemented in proper timing, even if the user operation or input timing is somewhat deviated ahead of or behind the proper timing.
It should be noted that when the audio data is comprised of melody of single tones or notes, the melody can be detected by simply comparing the level of a waveform of audio data with a predetermined threshold value to create the time information string indicative of sound reproduction timings, without the need of implementing filtering processing or envelope curve generation processing.
Next, with reference to a flowchart in
When it is determined at the step S101 that the designation of audio data to be reproduced is input from the second operation unit 16, the main controller 13 instructs the characteristic extraction unit 12 to generate a time information string for the audio data. In response to this instruction, the characteristic extraction unit 12 reads out the audio data specified by the main controller 13 from the memory 11, and generates a time information string for the audio data (step S102).
It should be noted that if the time information string has once been generated, the generated time information string can be stored in a memory of the audio reproduction apparatus 1, such as the memory 11 or another memory (not shown). In the next reproduction of audio data, the procedure in steps S101 to S103 is omitted, and the already-generated time information string is used in the processing in step S104 and subsequent steps. The generation of the time information string in the step S102 can be performed in real time, while the audio data is being reproduced.
When it is determined that the generation of the time information string by the characteristic extraction unit 12 is completed (Yes to the step S103), the main controller 13 temporarily stores the time information string and gives the reproduction controller 14 an instruction to start audio data reproduction (step S104). When instructed from the main controller 13 to start the audio data reproduction, the control unit 141 of the reproduction controller 14 instructs the readout control unit 142 to read out from the memory 11 the audio data to be reproduced. At the start of the audio data reproduction, the audio data is reproduced by the sound system 15, without being subjected to the processing in the filter unit 143 and the effector unit 144. When and after the audio data reproduction is started, the main controller 13 inputs an output of the sensor 17 via the interface 18. If the user shakes the audio reproduction apparatus in a predetermined manner (for example, if the user vertically shakes the apparatus), the sensor 17 detects an acceleration in a predetermined axis direction (for example, an acceleration in a vertical direction), and outputs acceleration data (sensor output) to the main controller 13.
In this embodiment, the acceleration sensor is employed for detecting user operation/input. Alternatively, a magnetic sensor can be used to detect a change in earth magnetism in a predetermined axis direction when the user shakes the audio reproduction apparatus in a predetermined manner.
Based on the output of the sensor 17, the main controller 13 can generate detection information indicating a time when the detected acceleration exceeds a threshold value (i.e., user operation/input timing). This detection information can be saved in the memory 11 or another memory, such as an external memory device supplied by the user. The detection information can indicate the user operation/input timing. Next, the main controller 13 compares the detection information with time information indicating a reproduction timing (corresponding to a time period elapsed from the start of audio data reproduction) of the musical characteristic of the audio data and contained in the time information string generated from the audio data (step S105), thereby determining whether or not the detection information coincides with the time information (step S106). The main controller 13 outputs a result of this determination to the control unit 141 of the reproduction controller 14.
If it is determined at the step S106 that the detection information (user operation/input timing) coincides with the time information (reproduction timing) (No in the step S106), the control unit 141 causes the sound system 15 to reproduce the audio data, without performing predetermined processing on the audio data. On the other hand, if it is determined at the step S106 that the detection information does not coincide with the time information (Yes in the step S106), the control unit 141 instructs the filter unit 143 and the effector unit 144 to implement the predetermined processing on the audio data (step S107).
In the processing at the step S107, the filter unit 143 can cut a predetermined frequency region of the audio data, and the effector unit 144 can add a predetermined effect to the audio data or reduces the reproduction volume, for example. To implement such processing, it is possible to determine the desired type and intensity of processing and the desired number of processing to which the audio data is simultaneously subjected on the basis of a time difference between the user operation/input timing and the reproduction timing, the number of times of occurrences of non-coincidence between these timings, or the like. Subsequently, the steps S105 to S108 can be repeatedly executed until the audio data reproduction is completed.
It should be noted that in addition to applying effects to the audio data, the step S107 can temporarily terminate the audio data reproduction. The result of the comparison in the step S105 simply can be displayed on the display unit 19 or simply be stored in a memory device in the audio reproduction apparatus 1, with the reproduced sound not altered. In the time information string in
The audio reproduction apparatus according to this embodiment is configured such that the characteristic extraction unit 12 can generate, from the audio data, the time information string that includes pieces of time information each indicating a reproduction timing of the musical characteristic of the audio data, which eliminates the necessity of performing authoring process on the audio data in advance to prepare event data string synchronous with the audio data.
In the present embodiment, if the reproduction timing of the musical characteristic of the audio data indicated by each piece of time information in the time information string generated from audio data does not coincide with the timing of a corresponding user operation/input, the manner of audio data reproduction can be altered by temporarily terminating the audio data reproduction, by applying effects to the audio data, or the like. For this reason, to obtain the desired audio reproduction, the user is required to manipulate (for example, vertically shake) the audio reproduction apparatus 1 in proper timing with the audio data reproduction. As a result, the user can have an actual feeling of participating in music reproduction through the medium of operating the audio reproduction apparatus of this embodiment, rather than simply listening to the reproduced music.
Next, with reference to
The reproduction controller 14′ of the audio reproduction apparatus 2 includes a musical tone generator unit 145 that includes stored pieces of tone color data representative of tone colors of various instruments. If a detection signal generated when one of a plurality of push buttons or actuators of the operation unit 16 is depressed by the user is input from the unit 16, the tone generator 145 can generate musical tone data corresponding to the depressed push button. A mixer unit 146 mixes audio data output from the effector unit 144 with musical tone data output from the tone generator 145, and outputs the mixed data to the sound system 15.
In the second embodiment, the characteristic extraction unit 12 detects from the audio data the rhythm of predetermined musical instrument sound, and the tone generator 145 outputs musical tone data for the predetermined musical instrument sound in time with the depression of push buttons of the operation unit 16 (user operation/input timing) to the sound system 15, which reproduces corresponding musical tones. In the reproduction controller 14′, the readout control unit 142 reads out the audio data and outputs the same to the filter unit 143 under the control of the control unit 141, and the filter unit 143 cancels or deletes particular parts of the audio data corresponding to the predetermined instrument sound from the audio data, so that the audio data, from which the audio data parts corresponding to the predetermined instrument sound have been canceled, can be reproduced by the sound system 15.
It should be noted that the filter unit 143 can be designed, not only to cancel or remove audio data part corresponding to a particular musical instrument sound, but to cancel or remove audio data part corresponding to a particular frequency band component. For example, if an LPF (low pass filter) with cutoff frequency Hc is employed in the filter unit 121 of the characteristic extraction unit 12, then an HPF (high pass filter) with cutoff frequency Hc is employed in the filter unit 143 of the reproduction controller 14′.
Next, with reference to a flowchart shown in
When it is determined at the step S201 that the designation of audio data to be reproduced is input from the operation unit 16, the main controller 13 instructs the characteristic extraction unit 12 to generate a time information string for the audio data. The characteristic extraction unit 12 reads out the audio data specified by the main controller 13 from the memory 11, and generates a time information string for the audio data (step S202). In the second embodiment, the time information string indicates timings at which musical tone data generated in the tone generator 145 are to be sounded by the sound system 15.
It should be noted that if the time information string has once been generated, the generated time information string can be stored in a memory in the audio reproduction apparatus 2. At the next reproduction of audio data, the procedure in steps S201 to S203 is omitted, and the already-generated time information string is used in the processing in step S204 and subsequent steps.
When it is determined that the generation of the time information string by the characteristic extraction unit 12 is completed (Yes to step S203), the main controller 13 temporarily stores the time information string and gives the reproduction controller 14′ an instruction to cancel a predetermined musical instrument sound and then start the reproduction of the audio data (step S204). The predetermined musical instrument sound is a musical instrument sound with which the user performs push-button-based musical performance and which is reproduced in a step S206 described later.
When inputting the instruction to start the audio data reproduction from the main controller 13, the control unit 141 in the reproduction controller 14′ instructs the readout control unit 142 to read out from the memory 11 the audio data to be reproduced. The filter unit 143 performs filtering processing on the audio data read out by the readout control unit 142 so as to cancel audio data part corresponding to the predetermined musical instrument sound, and then outputs the audio data with the predetermined instrument sound part having been canceled to the effector unit 144.
When the audio data reproduction is started, the main controller 13 can input data indicating a user activating the operation unit 16 (step S205). If the user has depressed one of the push buttons of the operation unit 16 (Yes to step 205), the operation unit 16 detects the push button being depressed and outputs data indicating the push button depression to the main controller 13 and the tone generator 145.
In accordance with the data output from the operation unit 16, the tone generator 145 generates musical tone data for the predetermined musical instrument sound and outputs the same to the mixer unit 146. The mixer unit 146 mixes the audio data input from the effector unit 144 and the musical tone data input from the tone generator 145, and causes the sound system 15 to perform audio reproduction based on the mixed data (step S206).
Based on a result of the detection by the operation unit 16, the main controller 13 generates detection information indicating a time point at which the push button of the operation unit 16 has been depressed (i.e., a user operation/input timing). Next, the main controller 13 compares the detection information generated based on the result of determination by the operation unit 16 with time information indicating a sounding timing (corresponding to a time period elapsed from the start of audio reproduction) of the predetermined instrument sound and contained in the time information string generated from the audio data (step S207), thereby determining whether or not the detection information coincides with the time information (step S208). The main controller 13 outputs a result of this determination to the control unit 141 of the reproduction controller 14′.
If it is determined at the step S208 the detection information (user operation/input timing) coincides with the time information (sounding timing) (No in the step S208), the control unit 141 causes the sound system 15 to reproduce the audio data, without the audio data being subjected to processing other than the musical instrument sound cancellation. On the other hand, if it is determined at the step S208 that the detection information does not coincide with the time information (Yes in the step S208), the control unit 141 instructs the filter unit 143 and the effector unit 144 to implement predetermined processing on the audio data (step S209).
In the processing in the step S209, the filter unit 143 cuts a predetermined frequency region of the audio data, or the effector unit 144 adds predetermined effect or reduces the reproduction volume, for example. In a step S210, whether or not the audio data has reached its end is determined. Subsequently, the steps S205 to S210 are repeatedly carried out until completion of the audio data reproduction.
It should be noted that the audio reproduction can be temporarily stopped in the step S209 in addition to adding effects to the audio data. Furthermore, the reproduced sound need not be altered. In that case, the degree of coincidence in the comparison between user operation/input timing and sounding timing at the step S207 simply can be displayed in a display unit (not shown) or simply can be stored in a memory of the audio reproduction apparatus 2. A mixture of the original audio data and musical tone data generated by the tone generator 145 can be reproduced, without the musical instrument sound being canceled in the step S204. It is not necessary to use push buttons for user operation. Alternatively, a MIDI instrument, which is connected to the audio reproduction apparatus 2, can be used so that detection information generated based on performance (user operation) of the MIDI instrument can be sent to the main controller 13.
As described above, with the audio reproduction apparatus of this embodiment, the user is enabled to perform performance of a particular musical instrument by depressing one or more push buttons of the operation unit 16. Therefore, the user can participate in the audio data reproduction by depressing the push button in time with the rhythm of the musical instrument.
It is to be understood that the present invention can also be accomplished by supplying a system or an apparatus with a storage medium in which a program code of software, which realizes the functions of the above described embodiments is stored, and causing a computer (or CPU or MPU) of the system or apparatus to read out and execute the program code stored in the storage medium. In this case, the program code itself read from the storage medium can realize the functions of the above described embodiments, and therefore the program code/the storage medium in which the program code is stored constitute other aspects of the present invention. Examples of the storage medium for supplying the program code include a floppy (registered trademark) disk, a hard disk, a magnetic-optical disk, an optical disk such as a CD-ROM, a CD-R, a CD-RW, a DVD-ROM, a DVD-RAM, a DVD-RW, a DVD+RW, a magnetic tape, a nonvolatile memory card, and a ROM.
Further, it is to be understood that the functions of the above described embodiments can be accomplished not only by executing the program code read out by a computer, but also by causing an OS (operating system) or the like that operates on the computer to perform a part or all of the actual operations based on instructions of the program code.
Further, it is to be understood that the functions of the above described embodiments can be accomplished by writing a program code read out from the storage medium into a memory provided on an expansion board inserted into a computer or a memory provided in an expansion unit connected to the computer and then causing a CPU or the like provided in the expansion board or the expansion unit to perform a part or all of the actual operations based on instructions of the program code.
The present embodiments thus can be suitable for use in an audio reproduction apparatus for detecting whether a user operation is performed in time with the reproduction of audio data. The control unit can supply the reproduction unit with the audio data from which the particular audio data part has been erased and mixed with the musical tone data generated by the musical tone generator unit. The control unit can perform the predetermined control on the audio data from which the particular audio data part has been erased based on the result of comparison by the comparison unit. The characteristic extraction unit can increase a time width of each of pieces of time information forming the time information string by a predetermined value. The control unit can change a manner of the reproduction of the audio data based on the result of the comparison by the comparison unit during the reproduction of the audio data. The control unit can temporarily stop the reproduction of the audio data or can control a sound volume or can add an effect to the audio data based on the result of the comparison by the comparison unit during the reproduction of the audio data. The audio reproduction apparatus can further include a display unit, and the control unit can change a display on the display unit based on the result of the comparison by the comparison unit during the reproduction of the audio data.
According to the present embodiments, a musical characteristic of audio data can be extracted by the characteristic extraction unit or in the characteristic extraction step, and based thereon, can generate a time information string indicating reproduction timings of the musical characteristic, making it possible to control audio data reproduction in accordance with a user operation, without the need of creating in advance event data string synchronous with the audio data.
Furthermore, it is possible to perform control during the audio reproduction, such as changing the manner of audio data reproduction or changing a displayed content on a display unit, based on detection information indicating the manner of a user operation and generated by the detection unit or in the detection step. As a result, it is possible for the user to have an actual feeling of participating in the audio reproduction through the medium of performing the user operation, rather than merely listening to the audio reproduction.
While the present invention has been particularly shown and described with reference to preferred embodiment thereof, it will be understood by those skilled in the art that the foregoing and other changes in form and details can be made therein without departing from the spirit and scope of the present invention. All modifications and equivalents attainable by one versed in the art from the present disclosure within the scope and spirit of the present invention are to be included as further embodiments of the present invention. The scope of the present invention accordingly is to be defined as set forth in the appended claims.
This application is based on, and claims priority to, JP PA 2006-242674, filed on 07 Sep. 2006. The disclosure of the priority application, in its entirety, including the drawings, claims, and the specification thereof, is incorporated herein by reference.
Number | Date | Country | Kind |
---|---|---|---|
2006-242674 | Sep 2006 | JP | national |