Patent Application
20110174137
The present invention relates to apparatus and methods for reproducing a tone signal based on performance data, such as MID data, in synchronism with an audio signal.
Among examples of the conventionally-known techniques for reproducing, in synchronism with audio data, other music piece performance data is one disclosed in U.S. Pat. No. 7,026,537 corresponding to Japanese Patent Application Laid-open Publication No. 2003-316356 (hereinafter referred to as “the patent literature”). The technique disclosed in the patent literature is designed to store MIDI (Musical Instrument Digital Interface (registered trademark)) data in the LSBs (Least Significant Bits) of digital audio data and reproduce the individual data on the basis of association between these data.
In cases where music piece performance data and audio data are delivered via a communication network, such as the Internet, these data are sometimes compressed. However, with the technique disclosed in the patent literature, if audio data are compressed or converted into a tone signal of analog form, synchronized reproduction of the audio data may be undesirably prevented due to loss of necessary information for the synchronous reproduction.
In view of the foregoing, it is an object of the present invention to provide an improved technique for reproducing music piece performance data in synchronism with a tone signal on the basis of information obtained from the tone signal.
In order to accomplish the above-mentioned object, the present invention provides an improved tone reproduction apparatus for reproducing a first audio signal and a tone signal, based on music piece performance data, in synchronism with each other, which comprises: an acquisition section which acquires a second audio signal, the second audio signal being an audio signal obtained by embedding, into the first audio signal, watermark information indicative of content of the music piece performance data to be reproduced in synchronism with the first audio signal, the watermark information being embedded in the first audio signal in such a manner as to be positioned in a frequency band higher than frequency components of the first audio signal; an audio reproduction section which reproduces the first audio signal on the basis of the second audio signal; a decoding section which decodes the watermark information from the second audio signal acquired by the acquisition section; and a tone reproduction section which identifies, on the basis of the watermark information decoded by the decoding section, content of the music piece performance data to be reproduced and reproduces the tone signal based on the music piece performance data in accordance with the identified content.
According to the present invention, watermark information indicative of content of music piece performance data to be reproduced in synchronism with the first audio signal is embedded into the first audio signal, and the resultant signal is acquired as the second audio signal, and the first audio signal is reproduced on the basis of the second audio signal. In parallel to the above processing, the watermark information is decoded from the second audio signal, the content of music piece performance data to be reproduced is identified on the basis of the decoded watermark information, and then a tone signal based on the music piece performance data is reproduced in accordance with the identified content. In this way, the present invention can reproduce a tone signal based on the music piece performance data in synchronism with the first audio signal.
In an embodiment, a plurality of the watermark information is embedded in correspondence with a plurality of reproduction time positions along the time axis of the first audio signal, and each of the watermark information identifies at least one tone.
In an embodiment, the content of the music piece performance data indicated by the watermark information includes identification information identifying a music piece, and position information indicative of reproduction time positions of the music piece identified by the identification information. The tone reproduction section includes: a storage section which stores therein the music piece performance data; and a readout and reproduction section which reads out, from the storage section, the music piece performance data of the music piece identified by the identification information included in the decoded watermark information and then reproduces a tones signal, corresponding to any one of the reproduction time positions indicated by the position information included in the watermark information, on the basis of the read-out music piece performance data.
The present invention may be constructed and implemented not only as the apparatus invention as discussed above but also as a method invention. Also, the present invention may be arranged and implemented as a software program for execution by a processor such as a computer or DSP, as well as a storage medium storing such a software program.
The following will describe embodiments of the present invention, but it should be appreciated that the present invention is not limited to the described embodiments and various modifications of the invention are possible without departing from the basic principles. The scope of the present invention is therefore to be determined solely by the appended claims.
For better understanding of the object and other features of the present invention, its preferred embodiments will be described hereinbelow in greater detail with reference to the accompanying drawings, in which:
The control section 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory) and a RAM (Random Access Memory). Using the RAM as a working area, the CPU executes a program, stored in the ROM, to control various components of the tone signal supply apparatus 10. The storage section 12, which is a storage section including, for example, a hard disk device, stores therein programs for the control section 11 to perform various control operations and has an audio file storage area 121 having audio files stored therein. The operation section 13 includes, for example, a keyboard, mouse, etc., and once operated by a user or human operator or player, it outputs, to the control signal 11, an operation signal corresponding to the user's operation. The display section 14, which includes, for example, a liquid crystal display, display various information. The output interface 15, which is an example of an output section and to which is connected the audio cable 100, outputs a tone signal, supplied from the control section 11, to the automatic performance apparatus 20 via the audio cable 100.
The control section 21 includes components for realizing or implementing a function as a sequencer (MIDI sequencer in this case) as well as components similar to those of the control section 11 of the tone signal supply apparatus 10. A CPU of the control section 21 executes a program, stored in the ROM or storage section 23, to control various components of the automatic performance apparatus 20. The input interface 22 is an input terminal into which is connected the audio cable 100. A tone signal supplied from the tone signal supply apparatus 10 is input to the input interface 22. The storage section 23, which is a storage section including, for example, a hard disk device, stores therein programs for the control section 21 to perform various control operations and has a MIDI file storage area 231 having MIDI files stored therein. The speaker 24 is a sounding device for audibly generating a tone in accordance with a reproduced tone signal by the control section 21 and MIDI data that are an example of music piece performance data. The performance operation section 25, via which the human player executes performance operation or which accepts operation by the human player, includes a performance controller, such as a piano keyboard having a plurality of keys, sensors for detecting performance operation, a drive circuit for depressing any one of the keys even without human player's performance operation, etc. Once performance operation is executed by the human player, the performance operation section 25 detects the human player's performance operation via the sensor and outputs, to the control section 21, performance information indicative of the detected performance operation. The tone signal generation section 26, which includes a tone generator circuit and a DSP (Digital Signal Processor), generates a tone signal corresponding to the performance information supplied from the control section 21 by performing sound processing in accordance with the supplied performance information and then outputs the generated tone signal to the control section 21. The control section 21 outputs the tone signal to the speaker 24 so that the speaker 24 audibly generates a tone corresponding to the tone signal.
The MIDI file storage area 231 constructed in the aforementioned manner is an example of a music piece performance data storage section of the present invention.
The following describe functions implemented by the tone signal supply apparatus 10 and automatic performance apparatus 20.
[Tone Signal Supply Apparatus 10]
First, the functions implemented by the tone signal supply apparatus 10 will be described.
The reproduction time identification section 111 identifies a reproduction time (i.e., reproduction time position) indicative of a reproduction time position of the first tone signal. Let it be assumed that the reproduction time is indicative of a time position based on a start position (“0:00” in this case) of the audio data. The reproduction time identification section 111 can identify each reproduction time with reference to the audio file Aud.
The watermark information generation section 112 generates watermark information to be convoluted or embedded into the first tone signal. In this embodiment, the watermark information is a bit sequence including a song ID and position information indicative of reproduction times (i.e., reproduction time positions); such position information will hereinafter be referred to as “time codes”. The watermark information generation section 112 can identify the song ID from the audio file Aud.
The pre-processing section 113 includes, for example, an LPF (Low Pass Filter) and performs pre-processing on the first tone signal. More specifically, the pre-processing section 113 performs, as the pre-processing, a filtering process on the first tone signal using the LPF to thereby cut off, from among frequency components of the first tone signal, frequency components included in a frequency band into which the watermark information is to be embedded. Namely, the LPF has a cutoff frequency Fo.
The embedding section 114 convolutes or embeds, into the first tone signal having been subjected to the aforementioned pre-processing, watermark information indicative of content of music piece performance data that are to be reproduced in synchronism with the first tone signal. In the instant embodiment, the embedding section 114 is constructed to embed the watermark information into a frequency band higher than the frequency components of the first tone signal, by performing phase modulation of a carrier wave signal, indicative of a carrier wave for carrying the first tone signal, on the basis of the watermark information. The first tone signal having the watermark information embedded therein in the aforementioned manner will hereinafter referred to as “second tone signal” (second audio signal). In other words, the second tone signal (second audio signal) is a tone signal obtained by synthesizing together the first tone signal and a waveform signal indicative of a wave of the watermark information.
Note that, in the case where the audio data comprise a plurality of channels, the embedding section 114 only has to generate a second tone signal by embedding the watermark information into the first tone signal of at least one of the channels.
The output interface 15 of the tone signal supply apparatus 10 outputs the second tone signal, which is a tone signal embedded with the watermark information by the embedding section 114, in such a manner that the control section 21 of the automatic performance apparatus 20 can acquire the output second tone signal. The control section 11 of the tone signal supply apparatus 10 performs the processes, implemented by the aforementioned functions, on the first tone signal supplied per given processing unit, to thereby generate the second tone signal to be output to the automatic performance apparatus 20.
As shown in
As a modified example of the tone signal supply apparatus 10, a second tone signal, obtained by embedding watermark information in the first tone signal, may be prestored in a storage section so that the second tone signal are read out from the storage section. Alternatively, such a second tone signal may be acquired from outside via a network or the like.
[Automatic Performance Apparatus 20]
Next, the functions implemented by the automatic performance apparatus 20 will be described.
The tone signal acquisition section 211 acquires the second tone signal, output from the tone signal supply apparatus 10, via the input interface 22. Namely, the tone signal acquisition section 211 is an example of an acquisition section of the present invention.
The first reproduction section 212 includes an LPF having, for example, a cutoff frequency fo and reproduces the second tone signal, acquired by the tone signal acquisition section 211, after performing a filtering process on the acquired second tone signal using the LPF. In this way, the first reproduction section 212 reproduces the second tone signal with the frequency components of the watermark information removed therefrom, and thus, ideally, the tone signal reproduced by the first reproduction section 212 is similar in content to the first tone signal. Namely, the first reproduction section 212 reproduces the second tone signal in such a manner that a tone corresponding to the first tone signal can be audibly generated via the speaker 24. Namely, the first reproduction section 212 is an example of an audio reproduction section of the present invention.
Note that, in the case where the frequency components of the watermark information are included in a frequency band outside the audible range or where the frequency components of the watermark information are of a low sound pressure level, there would be little auditory influence on a sound corresponding to the watermark information, and thus, the above-mentioned filtering process may be dispensed with. Namely, the first reproduction section 212 may reproduce the second tone signal (acquired by the tone signal acquisition section 211) directly or as-is.
The decoding section 213 includes an HPF (High Pass Filter) having, for example, a cutoff frequency fo and decodes the watermark information from the second tone signal. As noted above, the frequency components of the watermark information are included in a frequency band higher than the first tone signal. Thus, the decoding section 213 can take out the frequency components of the watermark information from the second tone signal by performing a filtering process on the second tone signal using the HPF. Then, the decoding section 213 decodes the signal taken out through the filtering process, to thereby decode the watermark information.
The readout section 214 identifies the song ID on the basis of the watermark information decoded by the decoding section 213 and reads out, from the MIDI file storage area 231, MIDI data associated with the identified song ID.
The second reproduction section 215 identifies, on the basis of the watermark information, content of music piece performance data to be reproduced in synchronism with the tone signal reproduced by the first reproduction section 212 and reproduces the identified content of music piece performance data. Of the MIDI data read out by the readout section 214, the second reproduction section 215 reproduces MIDI data of the reproduction times indicated by the time codes decoded from the watermark information decoded by the decoding section 213. In this way, the second reproduction section 215 can reproduce the content of the MIDI data corresponding to the reproduction times of the second tone signal in synchronism with the tone signal reproduced by the first reproduction section 212. For example, a comparison is made between the reproduction times indicated by the time codes and reproduction times of tones to be reproduced on the basis of the MIDI data, so that each tone (tone based on the MIDI data) corresponding to any one of the reproduction times indicated by the time codes is reproduced. The reproduction times (absolute times) of the MIDI data can be identified through accumulation of delta times. The storage section 23, readout section 214 and second reproduction section 215 constitute a tone reproduction section of the present invention. Further, the readout section 214 and second reproduction section 215 constitute a readout and reproduction section of the present invention.
The control section 11 not only executes the synchronized reproduction in the aforementioned manner, but also synthesizes a tone signal generated in response to human player's performance operation performed via the performance operation section 25 and then causes the speaker 24 to audibly generate or sound a tone corresponding to the synthesized tone signal. In addition, the control section 11 causes each corresponding key even to be depressed, even without human player's operation, by driving the drive circuit of the performance operation section 25 in accordance with reproduced MIDI date.
According to the above-described embodiment of the invention, the tone signal supply apparatus 10 embeds, into the first tone signal, watermark information to be used for synchronized reproduction of the first tone signal and MIDI data, and outputs, as the second tone signal, the signal embedded with the watermark information. The automatic performed apparatus 20 identifies, on the basis of the watermark information, content of MIDI data to be reproduced in synchronism with the second tone signal and reproduces the identified content of MIDI data and the second tone signal. Further, in a case where the second tone signal has been compressed by the tone signal supply apparatus 10 and automatic performed apparatus 20, and if part of the song ID and/or time codes has been lost, the instant embodiment permits continued reproduction of the MIDI data although it cannot identify content of the MIDI data only in a time period for which the part of the song ID and/or time codes has been lost; thus, the loss of the song ID and/or time codes does not so influence the synchronized reproduction.
In the above-described manner, the reproduction system 1 of the present invention can achieve synchronized reproduction of a tone signal and music piece performance data on the basis of information obtained from the tone signal.
[Modification]
The present invention may be modified variously as exemplified below without being limited to the above-described embodiment, and the following modifications may be combined as desired.
(Modification 1)
In the above-described embodiment, the tone signal supply apparatus 10 is constructed to embed, into the first tone signal, watermark information including a song ID and time codes. As a modification, the tone signal supply apparatus 10 may embed, into the first tone signal. MIDI data themselves as watermark information. In this case too, the watermark information is information indicative of content of MIDI data themselves that are to be reproduced in synchronism with the first tone signal. In this modification, MIDI files stored by the automatic performance apparatus 20 in the above-described embodiment may be stored by the tone signal supply apparatus 10; that is, the automatic performance apparatus 20 need not store MIDI files in the modification.
In this modification, the watermark information generation section 112 identifies content of MIDI data to be reproduced in synchronism with a first tone signal and generates, as watermark information, a bit sequence indicative of the identified content of MIDI data. The embedding section 114 embeds the thus-generated watermark information into the first tone signal in the same manner as described above in relation to the embodiment. The other processes performed by the tone signal supply apparatus 10 in the modification are the same as those described above in relation to the embodiment. Further, in this modification, the second reproduction section 215 of the automatic performance apparatus 20 identifies content of MIDI data on the basis of the watermark information decoded from the second tone signal and then reproduces the identified content of MIDI data.
With such arrangements. MIDI data to be reproduced in synchronism with the first tone signal can be designated in the tone signal supply apparatus 10, and thus, a new file need not be stored in the automatic performance apparatus 20. In this case, a data quantity of the watermark information may become greater than that in the above-described embodiment, and thus, a processing time required for embedding and decoding the watermark information may become longer than that in the above-described embodiment. However, in this case too, synchronized reproduction of the tone signal and MIDI data can be realized like in the above-described embodiment, by the second reproduction section 215 appropriately controlling reproduction times in consideration of the processing time required for embedding and decoding watermark information.
(Modification 2)
Whereas the tone signal supply apparatus 10 and the automatic performance apparatus 20 are interconnected via the audio cable 100 in the above-described embodiment, the reproduction system 10 may be modified as follows.
The tone signal supply apparatus 10a includes, in place of the output interface 15 employed in the above-described embodiment, a speaker 16 detachably attachable, for example, to the apparatus 10a. The speaker 16 is a sounding device that audibly generates a tone in response to a tone signal supplied from the control section 11. The automatic performance apparatus 20a includes, in place of the input interface 22 employed in the above-described embodiment, a microphone 27 detachably attachable, for example, to the apparatus 20a. The microphone 27 is a sound pickup device that picks up a sound and outputs a sound signal indicative of the picked-up sound. The tone signal supply apparatus 10a and the automatic performance apparatus 20a are disposed somewhat close to each other in such a manner that a sound audibly generated via the speaker 16 can be picked up by the microphone 27.
In this modification, the control section 11 of the tone signal supply apparatus 10a outputs the generated second tone signal to the speaker 16 to cause the speaker 16 to audibly generate a tone corresponding to the second tone signal. The control section 21 of the automatic performance apparatus 20a causes the microphone 27 to pick up the tone audibly generated by the speaker 16 and acquires, as the second tone signal, an audio signal indicative of the picked-up tone. The control section 21 realizes synchronized reproduction as in the above-described embodiment by decoding the watermark information from the second tone signal in the same manner as in the above-described embodiment. The other processes are the same as those explained in relation to the above-described embodiment. Thus, the instant modification achieves the same advantageous benefits as the above-described embodiment.
Further, the reproduction system 1 of the present invention may be modified as follows.
The tone signal supply apparatus 10b is constructed as a transmitter having a function for transmitting the second tone signal, which is, for example a transmitter of a broadcasting station. The automatic performance apparatus 20b is a receiver having a function for receiving the second tone signal transmitted by the tone signal supply apparatus 10b and is possessed by a user who listens to sounds broadcast by a broadcasting apparatus. The tone signal supply apparatus 10b includes a transmission section 17 in place of the output interface 15 employed in the above-described embodiment. The transmission section 17 modulates a carrier wave signal, indicative of a carrier wave, in accordance with the second tone signal supplied from the control section 11 to thereby generate a transmission signal and then transmits the transmission signal. The automatic performance apparatus 20b includes a reception section 28 in place of the input interface 22 employed in the above-described embodiment. Once the reception section 28 receives the transmission signal transmitted by the tone signal supply apparatus 10b, it demodulates the transmission signal to take out the second tone signal and outputs the second tone signal to the control section 21. The control section 21 decodes, from the second tone signal, the watermark information in the same manner as in the above-described embodiment, to thereby realize the synchronized reproduction. The other processes are the same as those explained in relation to the above-described embodiment. Thus, the instant modification achieves the same advantageous benefits as the above-described embodiment.
With the aforementioned modification, a user who possesses the automatic performance apparatus 20b can use the apparatus 20b to enjoy synchronized performance (synchronized reproduction) with a tone indicated by a tone signal output from the tone signal supply apparatus 10b located remotely from the automatic performance apparatus 20b. On the other hand, even a user who does not possess the automatic performance apparatus 20b will not substantially suffer from adverse influence in listening to the tone.
Further, the basic principles of the present invention are applicable to services in which the tone signal supply apparatus 10b and the automatic performance apparatus 20b are interconnected via a communication network, such as the Internet, and which compress audio data etc. to deliver sounds and animations. In this case, the tone signal supply apparatus 10b is a server apparatus, such as a content provider. The automatic performance apparatus 20b is possessed by a client that receives content from a content provider. In this case, the tone signal supply apparatus 10b embeds watermark information into the second tone signal included in content to be delivered, and it transmits the second tone signal having the watermark information embedded therein. Then, the automatic performance apparatus 20b decodes the watermark information from the second tone signal received from the tone signal supply apparatus 10b, to thereby realize the synchronized reproduction.
(Modification 3)
The construction for the tone signal supply apparatus 10 to embed watermark information in the above-described embodiment may be modified as follows. For example, there may be employed a method that employs pseudo noise signals using any of the M and Gold sequences. In this case, the embedding section 114 generates a pseudo noise signal and phase-modulates the thus-generated pseudo noise signal on the basis of watermark information. The decoding section 213 generates the same pseudo noise signal as the embedding section 114 and determines a correlation between the pseudo noise signal and the second tone signal. In a case where a signal having a high autocorrelation is used as the pseudo noise signal to be embedded into the second tone signal, and if a correlation between the second tone signal and the pseudo noise signal is determined, a waveform where a steep peak appears is extracted. This waveform indicates watermark information, and the decoding section 213 can decode the watermark information by extracting the waveform. Further, spectrum spreading may be performed using the pseudo noise signal, or the watermark information may be embedded using the OFDM (Orthogonal Frequency-Division Multiplexing) modulation. In the latter case, the control section 11 removes part of a high-frequency band of frequency components of the first tone signal and then embeds the OFDM-modulated watermark information into the frequency band in accordance with a frequency distribution of the original tone signal. Even with such an embedding scheme, the instant modification can achieve the same advantageous benefits as the above-described embodiment because the tone signal supply apparatus 10 embeds the watermark information into a high-frequency band higher than the frequency components of the first tone signal.
In the above-described embodiment, the format of music piece performance data performance data to be reproduced by the automatic performance apparatus 20 may be other than the MIDI data format, such as the MP-3 format. Whereas the automatic performance apparatus 20 has been described above as being an electronic piano having an automatic performance function, it may be any other type of electronic musical instrument, such as an electric-acoustic guitar or electone (registered trademark). Further, the automatic performance apparatus 20 is not limited to an apparatus functioning as an electronic musical instrument, and the basic principles of the present invention are also applicable to a reproduction apparatus having at least a function for reproducing music piece performance data while reproducing a tone signal.
Furthermore, in the above-described embodiment, the function corresponding to the pre-processing section 113 may be dispensed with. Because, in a case where frequency components of a tone signal are not contained in a frequency band higher the frequency fo or where frequency components of a tone signal are considerably smaller in sound pressure level than frequency components of watermark information, for example, it is considered that there will be almost no auditory influence and almost no auditory influence on decoding of the watermark information even if the pre-processing is dispensed with.
The programs for execution by the control section 11 of the tone signal supply apparatus 10 and by the control section 21 of the automatic performance apparatus 20 may be provided stored in a computer-readable recording or storage medium, such as a magnetic recording medium (e.g., magnetic tape, or magnetic disk like an HDD or FD), optical recording medium (e.g., optical disk like a CD or DVD), optical-magnetic recording medium or semiconductor memory. Further, such programs may be downloaded via a network, such as the Internet. Furthermore, the functions to be performed by the control sections 11 and 21 may be implemented by one or a plurality of software, or one or a plurality of hardware.
This application is based on, and claims priority to, JP PA 2010-007150 filed on 15 Jan. 2010. The disclosure of the priority application, in its entirety, including the drawings, claims, and the specification thereof, are incorporated herein by reference.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2010-007150 | Jan 2010 | JP | national |
