Patent Grant
11854521
The present disclosure relates to electronic musical instruments, methods and storage media therefor.
In recent years, the usage scene of synthetic voice has been expanding. Under such circumstances, it is preferable to have an electronic musical instrument that can not only produce automatic performance but also advance the lyrics according to the key press of the user (performer) and output the synthetic voice corresponding to the lyrics, thereby providing more flexible synthetic voice expression.
For example, Patent Document 1 discloses a technique for advancing lyrics in synchronization with a performance based on a user operation using a keyboard or the like.
However, when a plurality of sounds can be simultaneously produced by a keyboard or the like, for example, if the lyrics are advanced each time a key is pressed, the lyrics will advance too much when a plurality of keys are pressed at the same time.
Therefore, the present disclosure aims at providing an electronic musical instrument, a method, and a storage medium capable of appropriately controlling the progress of lyrics during the performance.
Additional or separate features and advantages of the invention will be set forth in the descriptions that follow and in part will be apparent from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
To achieve these and other advantages and in accordance with the purpose of the present invention, as embodied and broadly described, in one aspect, the present disclosure provides an electronic musical instrument that can output stored lyrics of a song in accordance with operations by a user, comprising: a plurality of first operating elements that receive operations by the user, the plurality of first operating elements respectively specifying different pitches; a second operating element that can take one of the following two possible positions: a first position in which the lyrics will be advanced in accordance with the user's operation on the plurality of first operating elements and a second position in which the lyrics will not be advanced even if the user operates on the plurality of first operating elements; and one or more processors electrically connected to the plurality of first operating elements and the second operating element, the one or more processors performing the following: determining whether the second operating element is in the first position or in the second position when the user operates on the plurality of first operating elements; while the second operating element is in the first position, if a first operation by the user on the plurality of first operating elements is detected and thereafter a second operation by the user on the plurality of first operating elements is detected, causing a digitally synthesized voice with a first lyric to be produced in response to the first user operation and causing a digitally synthesized voice with a second lyric that is next to the first lyric to be produced in response to the second user operation; and while the second operating element is in the second position, if the first operation by the user on the plurality of first operating elements is detected and thereafter the second operation by the user on the plurality of first operating elements is detected, causing the digitally synthesized voice with the first lyric to be produced in response to the first user operation and causing the digitally synthesized voice with the second lyric that is next to the first lyric not to be produced in response to the second user operation.
According to this aspect of the present disclosure, the lyric progression can be appropriately controlled during the user performance.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory, and are intended to provide further explanation of the invention as claimed.
Singing with two or more notes in a part originally composed of one syllable to one note (syllable style) is called melisma singing. Melisma singing may also be referred to as fake, kobushi, etc.
The present inventors have focused on a feature of melisma that an immediately preceding vowel is maintained and while the pitch thereof is freely changed and have developed a lyrics progress control method applicable to an electronic musical instrument equipped with a singing voice synthesis sound source of the present disclosure.
According to one aspect of the present disclosure, it is possible to control the lyrics not to progress during melisma. Further, even when a plurality of keys are pressed at the same time, it is possible to appropriately control whether or not the lyrics progress.
Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same parts are designated by the same reference numerals. Since the same part has the same name and function, detailed explanation will not be repeated.
In this disclosure, “progress of lyrics”, “progress of position of lyrics”, “progress of singing position” and like expressions may be interchangeably used to express the same meaning. Further, in the present disclosure, “do not advance the lyrics”, “do not control the progress of the lyrics”, “hold the lyrics”, “suspend the lyrics” and like expressions may be interchangeably used to express the same meaning.
(Electronic Musical Instrument)
The electronic musical instrument 10 is a device that receives input from a user via playing elements such as a keyboard or switches, and that controls music performance, lyrics progression, and the like. The electronic musical instrument 10 may have a function of generating a sound according to performance information such as MIDI (Musical Instrument Digital Interface) data. The device 10 may be an electronic musical instrument (electronic piano, synthesizer, etc.), or may be an analog musical instrument equipped with a sensor or the like so as to process user performance electronically.
The switch panel 140b may include switches for operating a volume specification, a sound source, a tone color setting, a song (accompaniment) song selection (accompaniment), a song playback start/stop, a song playback setting (tempo, etc.), etc.
The keyboard 140k may have a plurality of keys as performance elements (operating elements). The pedal 140p may be a sustain pedal having a function of extending the sound of the pressed key while the pedal is being depressed, or may be a pedal for operating an effector that processes a tone, volume, or the like.
In the present disclosure, the sustain pedal, pedal, foot switch, controller (operator), switch, button, touch panel, etc. may be interchangeably used to mean the same functional element. Depressing the pedal in the present disclosure may be understood to mean operating the controller.
A key in a keyboard or the like may be referred to as a performance/playing/operating manipulator or element, a pitch manipulator or element, a tone manipulator or element, a direct manipulator or element, a first manipulator or element, or the like. A pedal or the like may be referred to as a non-playing element, a non-pitched element, a non-tone element, an indirect manipulator or element, a second operating manipulator or element, or the like.
The display 150d may display lyrics, musical scores, various setting information, and the like. The speakers 150s may be used to emit the sound generated by the performance.
The electronic musical instrument 10 may be configured to generate or convert at least one of a MIDI message (event) and an Open Sound Control (OSC) message.
The electronic musical instrument 10 may also be called a control device 10, a lyrics progression control device 10, and the like.
The electronic musical instrument 10 may be connected to a network (Internet, etc.) via at least one of wired and wireless (for example, Long Term Evolution (LTE), 5th generation mobile communication system New Radio (5G NR), Wi-Fi (registered trademark).
The electronic musical instrument 10 may hold singing voice data (may be called lyrics text data, lyrics information, etc.) related to lyrics whose progress is controlled in advance, or may transmit and/or receive such singing voice data via a network. The singing voice data may be text described by a musical score description language (for example, MusicXML), or may be a MIDI data storage format (for example, MusicXML). It may be written in Standard MIDI File (SMF) format), or it may be text given in a normal text file.
The electronic musical instrument 10 may also acquire the content of the user singing in real time through a microphone or the like provided in the electronic musical instrument 10, and may acquire the text data obtained by applying the voice recognition process to the electronic musical instrument 10 as singing voice data.
Central processing unit (CPU) 201, ROM (read-only memory) 202, RAM (random access memory) 203, waveform data output unit 211, key scanner 206 to which switch (button) panel 140b, keyboard 140k, and pedal 140p in
A timer 210 for controlling the sequence of automatic performance may be connected to the CPU 201. The CPU 201 may be referred to as a processor, and may include an interface with peripheral circuits, a control circuit, an arithmetic circuit, a register, and the like.
The CPU 201 performs various functions by loading predetermined software (program) from a storage device, such as ROM 202 or hard drive.
The CPU 201 executes control operation of the electronic musical instrument 10 of
The timer 210 used in the present embodiment is included in the CPU 201, and counts the progress of the automatic performance of the electronic musical instrument 10, for example.
The waveform data output unit 211 may include a sound source LSI (large-scale integrated circuit), a voice synthesis LSI, and the like. The sound source LSI and the voice synthesis LSI may be integrated into one LSI.
The singing voice waveform data 217 and the song waveform data 218 output from the waveform data output unit 211 are converted into an analog singing voice output signal and an analog music sound output signal by the D/A converters 212 and 213, respectively. The analog music sound output signal and the analog singing voice output signal are mixed by the mixer 214, and after the mixed signal is amplified by the amplifier 216, the mixed signal is emitted from the speaker 150s or outputted from an output terminal.
The key scanner (scanner) 206 constantly scans the key pressing/releasing state of the keyboard 140k in
The LCD controller 208 is an IC (integrated circuit) that controls the display state of the LCD, which is an example of the display 150d.
The system configuration explained above is an example and is not limited to this. For example, the number of each circuit included is not limited to this. The electronic musical instrument 10 may have a configuration that does not include a part of circuits (mechanisms), or may have a configuration in which the function of one circuit is realized by a plurality of circuits. It may also have a configuration in which the functions of a plurality of circuits are realized by one circuit.
In addition, the electronic instrument 10 may be constructed by various hardware, such as a microprocessor, a digital signal processor (DSP: Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field Programmable Gate Array), and the like. Such hardware may realize a part or all of each functional blocks. For example, the CPU 201 may be implemented on at least one of these types of hardware.
<Generation of Acoustic Model>
The voice learning unit 301 that realizes voice synthesis in the present disclosure and a waveform data output unit 211 described later may be implemented based on, for example, a statistical voice synthesis technique based on deep learning.
The voice learning unit 301 may include a training text analysis unit 303, a training acoustic feature extraction unit 304, and a model learning unit 305.
In the voice learning unit 301, as the training singing voice data 312, for example, a voice recording of a plurality of singing songs of an appropriate genre sung by a certain singer is used. Further, as the training singing data 311, the lyrics text of each song is prepared.
The training text analysis unit 303 receives the training singing data 311 that includes the lyrics text and analyzes the data. As a result, the training text analysis unit 303 estimates and outputs the training language feature sequence 313, which is a discrete numerical sequence expressing phonemes, pitches, etc., corresponding to the training singing data 311.
The training acoustic feature extraction unit 304 receives and analyzes the training singing voice data 312, which is acquired through a microphone or the like by a singer singing a lyrics text corresponding to the training singing data 311 in accordance with the input of the training singing data 311. As a result, the training acoustic feature extraction unit 304 extracts and outputs the learning acoustic feature sequence 314 representing the voice features corresponding to the training singing voice data 312.
In the present disclosure, the training acoustic feature sequence 314 and an acoustic feature sequence corresponding to an acoustic feature sequence described later include acoustic feature data (formant information, spectrum information, etc.) modeling the human vocal tract) and vocal cord sound source data (which may be called sound source information) that models a human vocal cord. As the spectrum information, for example, mel cepstral, line spectrum pairs (LSP) and the like may be used. As the sound source information, a fundamental frequency (F0) indicating the pitch frequency of human voice and power values can be used.
The model learning unit 305 estimates by machine learning an acoustic model that maximizes the probability that the training acoustic feature sequence 314 is generated from the training language feature sequence 313. That is, the relationship between the language feature sequence that is text and the acoustic feature sequence that is voice is expressed by a statistical model, which is an acoustic model. The model learning unit 305 outputs model parameters representing the acoustic model calculated as a result of the machine learning as a learning result 315. Therefore, the trained model constitutes the acoustic model.
HMM (Hidden Markov Model) may be used as the acoustic model expressed by the learning result 315 (model parameters).
An HMM acoustic model may learn how the characteristic parameters of the vocal cord vibration and vocal tract characteristics change over time when a singer utters lyrics along a certain melody. More specifically, the HMM acoustic model may be a phoneme-based model of the spectrum, fundamental frequency, and their time structure obtained from the training singing voice data.
First, the processing of the voice learning unit 301 of
The spectral parameters of the singing voice can be modeled by a continuous HMM. On the other hand, since the log fundamental frequency (F0) is a variable-dimensional time series signal that takes a continuous value in the voiced section and has no value in the unvoiced section, it cannot be directly modeled by a normal continuous HMM or a discrete HMM. Therefore, using a MSD-HMM (Multi-Space probability Distribution HMM), the spectral parameters of the singing voice are modeled by regarding mel cepstrum as a multidimensional Gaussian distribution, and the log fundamental frequency (F0) is modeled by regarding the logarithmic fundamental frequency (F0) in the voiced section as a one-dimensional Gaussian distribution and F0 in the unvoiced section as a zero-dimensional Gaussian distribution, at the same time.
Further, it is known that the characteristics of phonemes constituting a singing voice fluctuate under the influence of various factors even if the phonemes have the same acoustic characteristics. For example, the spectrum and the logarithmic fundamental frequency (F0) of a phoneme, which is a basic unit of vocal sounds, differ depending on the singing style and tempo, the lyrics before and after, the pitch, and the like. These factors that affect such acoustic features are called contexts.
In the statistical voice synthesis processing according to an embodiment of the present invention, an HMM acoustic model (context-dependent model) in consideration of context may be adopted in order to accurately model the acoustic features of voice sound. Specifically, the training text analysis unit 303 considers not only the phonemes and pitches for each frame, but also the phonemes immediately before and after, the current position, the vibrato immediately before and after, the accent, and the like when outputting the training language feature sequence 313. In addition, decision tree-based context clustering may be used to improve the efficiency of context combinations.
For example, the model learning unit 305 may output a state continuation length decision tree as the learning result 315 based on the training language feature sequence 313 that corresponds to the contexts of a large number of phonemes concerning the state continuation length that is extracted by the training text analysis unit 303 from the training singing data 311.
Further, the model learning unit 305 may output, for example, a mel cepstrum parameter decision tree for determining mel cepstrum parameters as the learning result 315, based on the training acoustic feature sequence 314, which corresponds to a large number of phonemes relating to the mel cepstrum parameters that is extracted by the training acoustic feature extraction unit 304 from the training singing voice data 312.
Further, the model learning unit 305 may output, for example, the log fundamental frequency decision tree for determining the log fundamental frequency (F0) as the learning result 315, based on the training acoustic feature sequence 314, which corresponds to a large number of phonemes relating to the log fundamental frequency (F0) that is extracted by the training acoustic feature extraction unit 304 from the training singing voice data 312. Here, the log fundamental frequency (F0) in the voiced section and that in the unvoiced section may be modelled by MSD-HMM that can handle variable dimensions as a one-dimensional Gaussian distribution and as a zero-dimensional Gaussian distribution, respectively, in generating the log fundamental frequency decision tree.
In addition, instead of or in addition to the acoustic model based on HMM, an acoustic model based on Deep Neural Network (DNN) may be adopted. In this case, the model learning unit 305 may generate model parameters representing the nonlinear conversion function of each neuron in the DNN from the language features to the acoustic features as the learning result 315. According to DNN, it is possible to express the relationship between the language feature sequence and the acoustic feature sequence by using a complicated nonlinear transformation function that is difficult to express with a decision tree.
Further, the acoustic model of the present disclosure is not limited to these, and any voice synthesis method may be adopted as long as it is a technique using statistical voice synthesis processing such as an acoustic model combining HMM and DNN.
As shown in
Alternatively, as shown in
<Voice Synthesis Based on Acoustic Model>
The waveform data output unit 211 includes a processing unit (may be called a text processing unit, a preprocessing unit, etc.) 306, a singing voice control unit (may be called an acoustic model unit) 307, a sound source 308, and a singing voice synthesis unit (may be called a vocal model unit) 309 and the like.
The waveform data output unit 211 receives singing data 215 including lyrics and pitch information, which is instructed by the CPU 201 via the key scanner 206 of
Further, when the song data is reproduced, the waveform data output unit 211 outputs the song waveform data 218 corresponding to the corresponding singing position.
The processing unit 306 receives the singing data 215 including information on the phonemes, pitches, etc., of the lyrics designated by the CPU 201 of
For example, the processing unit 306 determines whether the lyrics should progress based on a lyrics progress control method described later based on the note on/off data, pedal on/off data, etc., which are obtained from the operation of the keyboard 140k and the pedal 140p, and acquires singing data 215 corresponding to the lyrics to be output. Then, the processing unit 306 analyzes the language feature sequence expressing the phonemes, part of speech, words, etc., corresponding to the pitch data specified by the key press and the acquired singing data 215, and outputs the language feature sequence to the singing voice control unit 307.
The singing data may include at least one of lyrics (characters), syllable type (start syllable, middle syllable, end syllable, etc.), lyrics index, corresponding voice pitch (correct voice pitch), and corresponding uttering period (for example, utterance start timing, utterance end timing, utterance duration: correct uttering period).
For example, in the example of
The singing data 215 may include information (data in a specific audio file format, MIDI data, etc.) for playing the accompaniment (song data) corresponding to the lyrics. When the singing data is presented in the SMF format, the singing data 215 may have a track chunk in which data related to singing voice is stored and a track chunk in which data related to accompaniment is stored. The singing data 215 may be read from the ROM 202 into the RAM 203. The singing data 215 is stored in a memory (for example, ROM 202, RAM 203) before the performance.
The electronic musical instrument 10 may control the progress of automatic accompaniment based on an event indicated by the singing data 215 (for example, a meta event (timing information) that indicates the utterance timing and pitch of the lyrics, a MIDI event that instructs note-on or note-off, or a meta event that indicates a time signature, etc.).
Based on the language feature sequence input from the processing unit 306 and the acoustic model set as the learning result 315, the singing voice control unit 307 estimates the corresponding acoustic feature sequence. The formant information 318 corresponding to the acoustic feature sequence is then output to the singing voice synthesis unit 309.
For example, when the HMM acoustic model is adopted, the singing voice control unit 307 connects the HMMs with reference to the decision tree for each context obtained by the language feature sequence, and estimates the acoustic feature sequence (formant information 318 and the vocal cord sound source data 319) that makes the output probability from each connected HMM maximum.
When the DNN acoustic model is adopted, the singing voice control unit 307 may output the acoustic feature sequence for each frame with respect to the phoneme sequence of the language feature sequence that is inputted for each frame.
In
The sound source 308 generates a sound source signal (may be called instrumental sound waveform data) of musical instrument sound data (pitch information) corresponding to the sound to be produced (note-on) based on the note-on/off data inputted from the processing unit 306, and outputs it to the singing voice synthesis unit 309. The sound source 308 may execute control processing such as envelope control of the sound to be produced.
The singing voice synthesis unit 309 forms a digital filter that models the vocal tract based on the sequence of the formant information 318 sequentially inputted from the singing voice control unit 307. Further, the singing voice synthesis unit 309 uses the sound source signal input from the sound source 308 as an excitation source signal, applies the digital filter, and generates and outputs the singing voice waveform data 217, which is a digital signal. In this case, the singing voice synthesis unit 309 may be called a synthesis filter unit.
In addition, various voice synthesis methods, such as a cepstrum voice synthesis method and an LSP voice synthesis method, may be adopted for the singing voice synthesis unit 309.
In the example of
The sound source 308 may output the output of another channel as the song waveform data 218 together with the processing of the musical instrument sound wave data. As a result, the accompaniment sound can be produced with a regular musical instrument sound, or the musical instrument sound of the melody line and the singing voice of the melody can be produced at the same time.
As described above, the singing voice control unit 307 of
The singing voice synthesis unit 309 generates data (for example, the singing voice waveform data of the n-th lyric corresponding to the n-th note) that is for generating a signal obtained by applying a digital filter, which models the vocal cord based on the sequence of the formant information 318, to a pulse train that is periodically repeated with the fundamental frequency (F0) contained in the vocal cord sound source data 319 and its power values (in the case of voiced sound elements), white noise (in the case of unvoiced phonetic elements) having a power value contained in the vocal cord sound source data 319, or a signal of a mixture thereof, and outputs the generated data to the sound source 308.
The sound source 308 generates and outputs singing voice waveform data 217, which is a digital signal, from the singing voice waveform data of the n-th lyrics corresponding to the sound to be produced (note-on) based on the note-on/off data input from the processing unit 306.
In the example of
In this way, the voice synthesis of the present disclosure differs from the existing vocoder (a method of inputting words spoken by a human with a microphone and replacing them with musical instrument sounds) in that even if the user (performer) does not sing (in other words, the user does not sing and input a voice signal in real time to the electronic musical instrument 10), a synthesized voice can be output by operating the keyboard.
As described above, by adopting the technique of statistical voice synthesis processing as the voice synthesis method, it is possible to realize a much smaller memory capacity as compared with the conventional element piece synthesis method. For example, an electronic musical instrument of the elemental composition method requires a memory having a storage capacity of several hundred megabytes for voice elemental data, but in the present embodiment, in order to store the model parameters of the learning result 315, a memory with a storage capacity of only a few megabytes is required. Therefore, it is possible to realize a lower-priced electronic musical instrument, which makes it possible for a wider group of users group to use a high-quality singing voice performance system.
Further, in the conventional element data method, since the element data needs to be manually adjusted, it takes a huge amount of time (years or so) and labor to create the data for singing voice performance. However, in this embodiment, creating the model parameters of the training result 315 for the HMM acoustic model or the DNN acoustic model requires only a fraction of the creation time and effort because there is little data adjustment required. This also makes it possible to realize a lower-priced electronic musical instrument.
In addition, a general user can make the acoustic model learn his/her own voice, family's voice, celebrity's voice, etc., by using the learning function built in the server computer 300 that can be used as a cloud service, or in the voice synthesis LSI (in the waveform data output unit 211, for example), etc., and have the electronic musical instrument perform voice singing using the learned voice as the model voice. In this case as well, it is possible to realize a singing voice performance that is much more natural and has a higher sound quality than the conventional art as a lower-priced electronic musical instrument.
(Lyrics Progress Control Method)
A lyrics progression control method according to an embodiment of the present disclosure will be described below. The lyrics progress control method may be used by the processing unit 306 of the electronic musical instrument 10 described above.
Each of the following flowcharts may be performed by any one of the CPU 201, the waveform data output unit 211 (or the sound source LSI and/or voice synthesis LSI in the waveform data output unit 211), and any combinations thereof. For example, the CPU 201 may execute a control processing program loaded from the ROM 202 into the RAM 203 so as to execute each operation.
In addition, an initialization process may be performed at the start of the flow shown below. The initialization process includes interrupt processing, lyrics progression, derivation of TickTime, which is the reference time for automatic accompaniment, tempo setting, song selection, song reading, instrument sound selection, and other processing related to buttons, etc.
The CPU 201 can detect operations of the switch panel 140b, the keyboard 140k, the pedal 140p, and the like based on interrupts from the key scanner 206 at an appropriate timing, and can perform the corresponding processing.
In the following, an example of controlling the progress of lyrics is shown, but the target of the progress control is not limited to this. Based on this disclosure, for example, instead of lyrics, the progress of arbitrary character strings, sentences (for example, news scripts) and the like may be controlled. That is, the lyrics of the present disclosure may be replaced with characters, character strings, and the like.
First, the electronic musical instrument 10 substitutes 0 for the lyrics index (also expressed as “n”) indicating the current position of the lyrics (step S101). When the lyrics are started from the middle (for example, starting from the previous stored position), a value other than 0 may be assigned to n.
The lyrics index is a variable indicating at what position a given syllable (or character) is located as counted from the beginning when the entire lyrics are regarded as a character string. For example, the lyrics index n may indicate the singing voice data at the n-th playback position of the singing data 215 shown in
Step S101 may be triggered by the start of performance (for example, the start of playback of song data), the reading of the singing data, and the like.
In this embodiment, the electronic musical instrument 10 plays back song data (accompaniment) corresponding to the lyrics according to, for example, a user operation (step S102). The user can perform a key press operation in synchronization with the accompaniment so as to advance the lyrics.
The electronic musical instrument 10 determines whether or not the playback of the song data started in step S102 has been completed (step S103). When it is completed (step S103—Yes), the electronic musical instrument 10 may finish the process of the flowchart and return to the standby state.
Here, there may be no accompaniment. In this case, in step S102, the electronic musical instrument 10 may read the singing data that is designated based on the user's operation as the progress control target, and may determine whether or not all the singing data has been progressed in step S103.
When the reproduction of the song data is not completed (step S103—No), the electronic musical instrument 10 determines whether or not the pedal is on (the pedal is pressed or not) (step S111). If the pedal is on (step S111—Yes), the electronic musical instrument 10 determine whether a new key press occurred or not (note on event or not) (step S112). When the new key press occurred (step S112—Yes), the electronic musical instrument 10 increments the lyrics index n (S112). This increment is basically 1 increment (i.e., n+1 is input to n), but an integer greater than 1 may be used.
When the lyrics index is incremented, the electronic musical instrument 10 executes a sound production process of the n-th singing voice data (step S114). This process will be described in detail later. Then, the electronic musical instrument 10 decrements the lyrics index by the amount incremented in step S113 (step S115). That is, when the pedal is on, the value of n is maintained before and after the key press, and therefore, the lyrics is not advanced.
Next, the electronic musical instrument 10 determines whether or not the key is newly released (a note-off event has occurred) (step S116). When there is a new key release (step S116—Yes), the electronic musical instrument 10 performs a mute process of the corresponding singing voice data (step S117).
Next, the electronic musical instrument 10 determines whether or not the pedal is off and all the keys are off (step S118). When the pedal is off and all the keys are off (step S118—Yes), the electronic musical instrument 10 synchronizes the lyrics and the song (accompaniment) (step S119). The synchronization process will be described later.
On the other hand, when the pedal is off (step S111—No), the electronic musical instrument 10 determines whether or not there is a new key press (a note-on event has occurred) (step S122). When there is a new key press (step S122—Yes), the electronic musical instrument 10 increments the lyrics index n (step S123). This increment is basically 1 increment (n+1 is substituted for n), but a value larger than 1 may be added.
After incrementing the lyrics index, the electronic musical instrument 10 performs a sound production process for the n-th singing voice data (step S124). This process may be the same as the process of step S114.
That is, when the pedal is off, n is increased between before and after the key is pressed, so that the lyrics is advanced.
Next, the electronic musical instrument 10 determines whether or not the key is newly released (a note-off event has occurred) (step S126). When there is a new key release (step S126—Yes), the electronic musical instrument 10 performs a mute process of the corresponding singing voice data (step S127).
After steps S119, after S126—No and after S127, respectively, the process returns to step S103.
Note that S113 and S115 may be omitted. As a result, sound production process may be performed without advancing the lyrics. When there are S113 and S115, the singing voice data produced by S114 becomes the n+1st data, but when there are no S113 or S115, the singing voice data produced by S114 becomes the nth data.
The determination of S111 may be reversed, that is, whether or not the pedal is off (Yes if the pedal is off) may be determined instead.
The electronic musical instrument 10 may continuously output the same sound (or a vowel of the same sound) without advancing the lyrics for the sound already being produced, or may output a sound based on the advanced lyrics. When the electronic musical instrument 10 produces a sound corresponding to the same lyrics index as the sound already being produced, the electronic musical instrument 10 may output the vowel of the lyrics. For example, when the lyric “S1e” is already being uttered and the same lyric is to be newly uttered, the electronic musical instrument 10 may newly produce the sound “e”.
In the electronic musical instrument 10 of the present disclosure, when a plurality of sounds are simultaneously produced, each sound may be produced using a synthetic voice having a different voice color. For example, when the user presses four keys to produce four sounds, the electronic musical instrument 10 may perform voice synthesis and to produce the voices of soprano, alto, tenor, and bass in order from the highest sound.
<Sound Production Processing of n-th Singing Voice Data>
The sound production processing of the n-th singing voice data in step S114 will be described in detail below.
The processing unit 306 of the electronic musical instrument 10 inputs the pitch data designated by pressing the key and the n-th singing voice data to the singing voice control unit 307 (step S114-1).
Then, the singing voice control unit 307 of the electronic musical instrument 10 estimates the acoustic feature quantity sequence based on the input, and supplies the corresponding formant information 318 and the vocal cord sound source data (pitch information) 319 to the singing voice synthesis unit 309. Further, the singing voice synthesis unit 309 generates the n-th singing voice waveform data (which may be called the singing voice waveform data of the n-th lyrics corresponding to the n-th note) based on the inputted formant information 318 and the vocal cord sound source data (pitch information) 319, and outputs it to the sound source 308. This way, the sound source 308 acquires the n-th singing voice waveform data from the singing voice synthesis unit 309 (step S114-2).
The electronic musical instrument 10 performs a sound production process by the sound source 308 on the obtained n-th singing voice waveform data (step S114-3).
Further, it is assumed that the user turns on the pedal at the time t1 and turns off the pedal at t2. Similarly, it is assumed that the user turns on the pedal at the time as t3 and turns off the pedal before t5. Similarly, it is assumed that the user turns on the pedal at the time as t5 and turns off the pedal before the timing when the next bar is scheduled to start.
First, at timing t1, four keys were pressed. The electronic musical instrument 10 performs the determination process of
Next, at the timing t2, the user moves the left hand to the “Do # (C #)” key while continuously pressing the right hand key. The electronic musical instrument 10 performs the determination process of
Similarly, in t3, the electronic musical instrument 10 outputs the lyric “e” with the sound corresponding to the four keys, and at t4, updates only the sound newly pressed by the lyric “e”. Further, the electronic musical instrument 10 outputs the lyric “ping” with the sound corresponding to the four keys at t5, and updates only the sound newly pressed with the lyric “ping” at t6.
In the section t1-t6 of the example of
<Synchronous Processing>
The synchronization process is a process of matching the position of the lyrics with the playback position of the current song data (accompaniment). According to this process, the position of the lyrics can be appropriately moved when the position of the lyrics is exceeded due to excessive key pressing, or when the position of the lyrics does not advance as expected due to insufficient key pressing.
The electronic musical instrument 10 acquires the playback position of the song data (step S119-1). Then, the electronic musical instrument 10 determines whether or not the acquired playback position and the (n+1)th singing playback position coincide with each other (step S119-2).
The (n+1)th singing playback position may indicate a desirable timing for producing the (n+1)th note, which is derived in consideration of the total note length of the singing voice data up to the n-th singing voice.
When the playback position of the song data and the (n+1)th singing voice playback position match (step S119-2—Yes), the synchronization process is terminated. If not (step S119-2—No), the electronic musical instrument 10 acquires the X-th singing voice playback position that is closest to the playback position of the song data (step S119-3), and assign X−1 to n (step S119-4). Then the synchronization process may be completed.
If the accompaniment is not being played back, the synchronization process may be omitted. Alternatively, when the appropriate production timing of the lyrics can be derived based on the singing data, the electronic musical instrument 10 may adjust the position of the lyrics to be matched with the correct position based on the elapsed time from the start of the performance to the present, and the number of key pressing actions, even if the accompaniment is not played back.
According to the above-described embodiments, the lyrics can be appropriately advanced even when a plurality of keys are pressed at the same time.
(Modification Examples)
The voice synthesis processing shown in
In the flowchart of
The electronic musical instrument 10 only needs to be able to control at least the position of the lyrics, and does not necessarily have to generate or output the sound corresponding to the lyrics. For example, the electronic musical instrument 10 may transmit sound wave data generated based on a key press to an external device (such as a server computer 300), and the external device generates/outputs synthetic voice based on the sound wave data.
The electronic musical instrument 10 may control the display 150d to display lyrics. For example, the lyrics near the current lyrics position (lyric index) may be displayed, and the lyrics corresponding to the sound being pronounced, the lyrics corresponding to the pronounced sound, and the like may be displayed by coloring them so as to show the current lyrics position.
The electronic musical instrument 10 may transmit at least one of singing voice data, information on the current position of lyrics, and the like to an external device. The external device may perform control to display the lyrics on its own display based on the received singing voice data, information on the current position of the lyrics, and the like.
In the above example, the electronic musical instrument 10 is a keyboard instrument such as a keyboard, but the present invention is not limited to this. The electronic musical instrument 10 may be an electric violin, an electric guitar, a drum, a trumpet, or the like, as long as it is a device having a configuration in which the timing of sound generation can be specified by a user's operation.
Therefore, the “key” of the present disclosure may be a string, a valve, another performance operating element for specifying a pitch, any other adequately provided performance operating element, or the like. The “key press” of the present disclosure may be a keystroke, picking, playing, operation of an operator, or the like. The “key release” in the present disclosure may be a string stop, a performance stop, an operator stop (non-operation), or the like.
The block diagram used in the description of the above embodiments shows blocks of functional units. These functional blocks (components) are realized by adequate combination of hardware and/or software. Further, a specific manner that realizes each functional block is not particularly limited; each functional block or any combinations of functional blocks may be realized by one or more processors, such as one physically connected device, or two or more physically separated devices connected by wire or wirelessly and these plurality of devices.
The terms described in the present disclosure and/or the terms necessary for understanding the present disclosure may be replaced with terms having the same or similar meanings.
The information, parameters, etc., described in the present disclosure may be represented using absolute values, relative values from a predetermined value, or other corresponding information. Moreover, the names used for parameters and the like in the present disclosure are not limited in any respect.
The information, signals, etc., described in the present disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc., that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any combinations of them.
Information, signals, etc., may be input/output via a plurality of network nodes. The input/output information, signals, and the like may be stored in a specific location (for example, a memory), or may be managed using a table. Input/output information, signals, etc., can be overwritten, updated, or added. The output information, signals, etc., may be deleted. The input information, signals, etc., may be transmitted to other devices.
Regardless of whether called software, firmware, middleware, microcode, hardware description language, or another name, the term “software” used herein should broadly be interpreted to mean an instruction, instruction set, code, code segment, program code, program, subprogram, software module, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, or the like.
Further, software, instructions, information, and the like may be transmitted and received via a transmission medium. For example, when software is transmitted from a website, a server, or other remote source through wired technology (coaxial cable, fiber optic cable, twist pair, digital subscriber line (DSL: Digital Subscriber Line), etc.) and/or wireless technology (infrared, microwave, etc.), these wired and wireless technologies are included within the definition of the “transmission medium.”
The respective aspects/embodiments described in the present disclosure may be used alone, in combination, or switched in accordance with manners of execution. In addition, the order of the processing procedures, sequences, flowcharts, etc., of each aspect/embodiment described in the present disclosure may be changed as long as there is no contradiction. For example, the methods described in the present disclosure present elements of various steps using an exemplary order, and are not limited to the particular order presented.
The phrase “based on” as used in this disclosure does not mean “based only on” unless otherwise stated. In other words, the phrase “based on” means both “based only on” and “based at least on”.
Any reference to elements using designations such as “first”, “second” as used in this disclosure does not generally limit the quantity or order of those elements. These designations can be used in the present disclosure as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not mean that only two elements can be adopted or that the first element must somehow precede the second element.
When “include”, “including” and variations thereof are used in the present disclosure, these terms are as comprehensive as the term “comprising”. Furthermore, the term “or” used in the present disclosure is intended not to be an exclusive OR.
In the present disclosure, even if an article, for example “a,” “an,” of “the” in English, is added to a singular noun by translation, a case of a plural nouns may be included within the meaning of that expression.
Although the invention according to the present disclosure has been described in detail above, it is apparent to those skilled in the art that the invention according to the present disclosure is not limited to the embodiments described in the present disclosure. The invention according to the present disclosure can be implemented as a modified or modified mode without departing from the spirit and scope of the invention determined based on the description of the claims. Therefore, the description of the present disclosure is for purposes of illustration and does not bring any limiting meaning to the invention according to the present disclosure.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2019-231928 | Dec 2019 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5777251 | Hotta | Jul 1998 | A |
| 6245983 | Ishiguro | Jun 2001 | B1 |
| 10825433 | Danjyo | Nov 2020 | B2 |
| 11468870 | Danjyo | Oct 2022 | B2 |
| 20040040434 | Kondo | Mar 2004 | A1 |
| 20170025115 | Tachibana | Jan 2017 | A1 |
| 20180277075 | Nakamura | Sep 2018 | A1 |
| 20190318712 | Nakamura | Oct 2019 | A1 |
| 20190318715 | Danjyo | Oct 2019 | A1 |
| 20190392798 | Danjyo | Dec 2019 | A1 |
| 20190392799 | Danjyo | Dec 2019 | A1 |
| 20190392807 | Danjyo | Dec 2019 | A1 |
| 20200294485 | Tachibana | Sep 2020 | A1 |
| 20200312288 | Sato | Oct 2020 | A1 |
| 20210027753 | Danjyo | Jan 2021 | A1 |
| 20210193098 | Danjyo | Jun 2021 | A1 |
| 20210193114 | Danjyo | Jun 2021 | A1 |
| 20210295819 | Danjyo | Sep 2021 | A1 |
| 20220076651 | Danjyo | Mar 2022 | A1 |
| 20220076658 | Danjyo | Mar 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| 107430848 | Dec 2017 | CN |
| H02-269398 | Nov 1990 | JP |
| 04349497 | Dec 1992 | JP |
| H4-349497 | Dec 1992 | JP |
| H05-188953 | Jul 1993 | JP |
| 2000-276147 | Oct 2000 | JP |
| 2003-295873 | Oct 2003 | JP |
| 4735544 | Jul 2011 | JP |
| 2011-221085 | Nov 2011 | JP |
| 2012-083570 | Apr 2012 | JP |
| 2014010190 | Jan 2014 | JP |
| 2014-095856 | May 2014 | JP |
| 2018-54767 | Apr 2018 | JP |
| 2018-159831 | Oct 2018 | JP |
| 6610715 | Nov 2019 | JP |
| 2020-3816 | Jan 2020 | JP |
| 2021-099461 | Jul 2021 | JP |
| Entry |
|---|
| Japanese Office Action dated Nov. 2, 2021, in a counterpart Japanese patent application No. 2019-231928. (A machine translation (not reviewed for accuracy) attached.). |
| Japanese Office Action dated Nov. 2, 2021, in a counterpart Japanese patent application No. 2019-231927. (Cited in the related U.S. Appl. No. 17/129,653 and a machine translation (not reviewed for accuracy) attached.). |
| U.S. Appl. No. 17/129,653, filed Dec. 21, 2020. |
| U.S. Appl. No. 17/202,160, filed Mar. 15, 2021. |
| Japanese Office Action dated Jun. 20, 2023, in a counterpart Japanese patent application No. 2022-092637. (Cited in the related U.S. Appl. No. 17/129,653 and a machine translation (not reviewed for accuracy) attached.). |
| Number | Date | Country | |
|---|---|---|---|
| 20210193098 A1 | Jun 2021 | US |
