Patent Grant
10002604
This disclosure relates to a voice synthesis technology, and more particularly, relates to a real-time voice synthesis technology.
A voice synthesis technology is widespread in which a voice signal representative of a guidance voice in a voice guidance, a literary work reading voice, a song singing voice or the like is synthesized by electric signal processing by use of a plurality of kinds of synthesis information. For example, in the case of the singing voice synthesis, as the synthesis information, musical expression information is used such as information representative of the pitches and durations of the musical notes constituting a melody of a song which is the object of singing voice synthesis and information representative of phoneme sequences of the lyrics uttered in time with the musical notes. In the case of synthesis of a voice signal of a guidance voice in a voice guidance or a literary work reading voice, information representative of the phonemes of the guidance sentence or the sentence of the literary work and information representative of change of prosody such as intonation and accent are used as the synthesis information. Conventionally, for the voice synthesis of this kind, a so-called batch processing method has been common in which various kinds of synthesis information related to the entire voice of the object of synthesis are all inputted to a voice synthesizing apparatus in advance and a voice signal representative of the voice waveform of the entire voice of the synthesis object is generated in one batch based on those pieces of synthesis information. However, in recent years, a real-time voice synthesis technology has also been proposed (see, for example, JP-B-3879402).
An example of the real-time voice synthesis is a technology of synthesizing a singing voice by previously inputting information representative of the phoneme sequence of the lyrics of the entire song to a singing voice synthesizing apparatus and sequentially specifying the pitch and the like in uttering the lyrics by operating a keyboard resembling a piano keyboard. In recent years, it has also been proposed to perform singing voice synthesis in units of musical notes by letting the user sequentially input, for each musical note, musical note information representative of the pitch and phoneme sequence information representative of the phoneme sequence of the portion of the lyrics uttered in time with the musical note by use of a singing voice synthesis keyboard where a phoneme information input portion in which manipulating members for inputting the phonemes (consonants and vowels) constituting the phoneme sequence of the lyrics are arranged and a musical note information input portion resembling a piano keyboard are arranged side by side.
When information representative of the phoneme sequence of the lyrics of the entire song is previously stored in a singing voice synthesizing apparatus to perform real-time singing voice synthesis, a faltering unnatural singing voice as if the lyrics were uttered with a delay from the musical score is sometimes synthesized. The reason that such a falter occurs is as follows:
Likewise, in the real-time singing voice synthesis using a keyboard resembling a piano keyboard, as shown in
The present disclosure is made in view of the above-mentioned problem, and an object thereof is to provide a technology of enabling real-time synthesis of an unfaltering natural voice.
In order to achieve the above object, according to the present disclosure, there is provided a voice synthesizing method comprising:
a first receiving step of receiving first utterance control information generated by detecting a start of a manipulation on a manipulating member by a user;
a first synthesizing step of synthesizing, in response to a reception of the first utterance control information, a first voice corresponding to a first phoneme in a phoneme sequence of a voice to be synthesized to output the first voice;
a second receiving step of receiving second utterance control information generated by detecting a completion of the manipulation on the manipulating member or a manipulation on a different manipulating member; and
a second synthesizing step of synthesizing, in response to a reception of the second utterance control information, a second voice including at least the first phoneme and a succeeding phoneme being subsequent to the first phoneme of the voice to be synthesized to output the second voice.
As examples of voice output in response to the reception of the second utterance control information, the following are considered: a first example that a voice of the part succeeding the part of transition from the first phoneme to the succeeding phoneme in the phoneme sequence represented by the phoneme sequence information is synthesized and outputted; and a second example that a voice of repetitively uttering the transition part (or a voice of repetitively uttering the transition part with one or more than one silence in between) or a voice of continuously uttering the transition part is synthesized and outputted.
According to the above voice synthesizing method, the output of a voice of the part of transition from the silence to the first phoneme (for example, the part of transition from a silence to a consonant s in starting to sing “ [saita]” from silent state) is started in response to the start of a manipulation on the manipulating member to let the user provide an instruction to start voice utterance, so that the time lag between the start of the manipulation on the manipulating member and the start of utterance of the synthetic voice is substantially eliminated and an unfaltering voice can be synthesized in real time. Likewise, for the synthesis of the voice of a portion “
(ta)” of “
(saita)”, the output of the voice of the part of transition from the preceding phoneme (in this example, the vowel i) to the first phoneme (in this example, the consonant t) represented by the phoneme sequence information of the portion is started in response to the start of the manipulation on the manipulating member to let the user provide an instruction to start utterance, so that the time lag between the start of the manipulation on the manipulating member and the start of utterance of the synthetic voice is substantially eliminated and an unfaltering voice is synthesized. The output timing of the part of transition from the first phoneme to the succeeding phoneme (in the case of a portion of the lyrics constituted by a consonant and a vowel, the part of transition from the consonant to the vowel) can be adjusted by the completion of the manipulation on the manipulating member (for example, completely (full) depression of the manipulating member) or a manipulation on a different manipulating member, so that a natural singing voice accurately reproducing human singing characteristics can be synthesized. When the phoneme sequence information represents one phoneme (for example, a vowel), voice synthesis may be performed in response to the reception of the first utterance control information, or voice synthesis may be performed after the reception of the second utterance control information.
The above objects and advantages of the present disclosure will become more apparent by describing in detail preferred exemplary embodiments thereof with reference to the accompanying drawings, wherein:
Hereinafter, an embodiment of the present disclosure will be described.
(A: Embodiment)
The control portion 110 is, for example, a CPU (central processing unit). The control portion 110 operates according to a singing voice synthesis program stored in the storage portion 160, thereby functioning as the voice synthesis unit for synthesizing a singing voice based on the above-mentioned plurality of kinds of synthesis information. Details of the processing that the control portion 110 executes according to the singing voice synthesis program will be clarified later. While a CPU is used as the control portion 110 in the present embodiment, it is to be noted that a DSP (digital signal processor) may be used.
The manipulating portion 120 is the above-described singing voice synthesis keyboard, and has a phoneme information input portion and a musical note information input portion. By operating the manipulating portion 120, the user of the singing voice synthesizing apparatus 1 can specify a musical note included in a melody of the song which is the object of singing voice synthesis and the phoneme sequence of the portion of the lyrics uttered in time with the musical note. For example, when “ (sa)” of the lyrics is specified, of a plurality of manipulating members provided on the phoneme information input portion, a manipulating member corresponding to the consonant “s” and a manipulating member corresponding to the vowel “a” are successively depressed, and when “C4” is specified as the pitch of the musical note corresponding to the portion of the lyrics, of a plurality of manipulating members (keys) provided on the musical note information input portion, the key corresponding to the pitch is depressed to specify the start of the utterance thereof and the finger is moved away from the key to specify the end of the utterance. That is, the length of the time during which the key is depressed is the duration of the musical note. Moreover, by the speed of depression of the key corresponding to the musical note, the user can specify the intensity (velocity) of the voice when a portion of the lyrics is uttered in time with the musical note. As the arrangement that enables the specification of the velocity by the key depression speed, an arrangement in the related electronic keyboard instruments is adopted.
When an operation of specifying a phoneme sequence is performed, the phoneme information input portion (not shown in
For example, a two-make switch can be employed as the first sensor and the second sensor. One example of the two-make switch is disclosed in U.S. Pat. No. 5,883,327. In FIG. 1A of U.S. Pat. No. 5,883,327. contacts 9, 11 correspond to the first sensor and contacts 10, 12 correspond to the second sensor.
When detecting the start of depression of a key by the first sensor 121, the musical note information input portion of the manipulating portion 120 supplies the control portion 110 with a note-on event (MIDI [musical instrument digital interface] event) including pitch information (for example, the note number) representative of the pitch corresponding to the key as first utterance control information to provide an instruction to start utterance. When detecting by the second sensor 122 a full depression of the manipulating member the start of depression of which has been detected by the first sensor 121, the musical note information input portion supplies the control portion 110 with a note-on event including the pitch information corresponding to the key and the value of the velocity corresponding to the length of the time required from the detection of the start of depression by the first sensor 121 to the detection of the full depression by the second sensor 122, as second utterance control information. Then, when detecting the return from the completely depressed position by the second sensor 122, the musical note information input portion supplies the control portion 110 with third utterance control information to provide an instruction to stop utterance (in the present embodiment, note-off event). The information included in the second utterance control information is not limited to the information to specify the intensity of utterance (velocity); it may be information to specify the volume or may be both the velocity and the volume.
The display 130 is, for example, a liquid crystal display and a driving circuit thereof, and displays various images such as a menu image to prompt the use of the singing voice synthesizing apparatus 1 under the control of the control portion 110. The voice output portion 140 includes, as shown in
The external device I/F portion 150 is an aggregate of interfaces such as a USB (universal serial bus) interface and an audio interface for connecting other external devices to the singing voice synthesizing apparatus 1. While a case where the singing voice synthesis keyboard (the manipulating portion 120) and the voice output portion 140 are elements of the singing voice synthesizing apparatus 1 is described in the present embodiment, it is to be noted that the singing voice synthesis keyboard and the voice output portion 140 may be external devices connected to the external device I/F portion 150.
The storage portion 160 includes a non-volatile storage portion 162 and a volatile storage portion 164. The non-volatile storage portion 162 is formed of a non-volatile memory such as a ROM (read only memory), a flash memory or a hard disk, and the volatile storage portion 164 is formed of a volatile memory such as a RAM (random access memory). The volatile storage portion 164 is used by the control portion 110 as the work area for executing various programs. On the other hand, the non-volatile storage portion 162 previously stores, as shown in
The singing voice synthesis library 162a is a database storing fragment data representative of the voice waveforms of various phonemes and diphones (transition from a phoneme to a different phoneme [including silence]). The singing voice synthesis library 162a may be a database storing fragment data of triphones in addition to monophones and diphones or may be a database storing the stationary parts of the phonemes of the voice waveforms and parts of transition to other phonemes (transient parts). The singing voice synthesis program 162b is a program for causing the control portion 110 to execute singing voice synthesis using the singing voice synthesis library 162a. The control portion 110 operating according to the singing voice synthesis program 162b executes singing voice synthesis processing.
The singing voice synthesis processing is processing of synthesizing voice data representative of the voice waveform of a singing voice based on a plurality of kinds of synthesis information (the phoneme sequence information, the pitch information, the information representative of the velocity and volume of a voice, etc.) and outputting the voice data.
An explanation regarding an example of a singing voice synthesizing process will be described by referring to
Thereafter, at a step S203, it is determined that whether the control portion 110 receives the piece of second utterance control information. If the control portion 110 (second receiver) receives the piece of second utterance control information at the step S203, a process proceeds to a step S204, and then a second singing voice synthesis processing is started in response to a reception of the piece of second utterance control information by the control portion 110 (second synthesizer). If the control portion 110 has not received the piece of second utterance control information at the step S203, the control portion 110 waits for receiving the piece of second utterance control information. In this second singing voice synthesis processing, the control portion 110 reads from the singing voice synthesis library 162a the pieces of fragment data of the phonemes succeeding the part of transition from the first phoneme to the succeeding phoneme, synthesizes the voice waveform data of the part succeeding the transition part by combining the pieces of fragment data by performing signal processing to the pieces of fragment data of the phonemes such as processing of converting the pitch so that the pitch is matched with the one represented by the pitch information included in the first utterance control information and the adjustment of the attack depth (lessening at a rising waveform) according to the value of the velocity included in the piece of second utterance control information, and supplies the result voice waveform data to the voice output portion 140.
At a step S205, it is determined that whether the control portion 110 receives a piece of third utterance control information. If the control portion 110 receives the piece of third utterance control information at the step S205, in response to the reception of the third utterance control information, the control portion 110 ends the singing voice synthesis processing, and stops the output of the synthetic singing voice. If the control portion 110 has not received the piece of third utterance control information at the step S205, the control portion 110 waits for receiving the piece of third utterance control information.
For example, when a singing voice starting to sing “ (saita)” from silent state is synthesized, for the singing voice of a portion “
(sa)”, the output of the voice of the part of transition from a silence to the first phoneme (the consonant s) represented by the phoneme sequence information of the lyrics is started in response to the start of the manipulation on the manipulating member to provide an instruction to start utterance, and the output of the voice of the part succeeding the part of transition from the first phoneme to the succeeding phoneme (the vowel a) is started in response to the full depression of the manipulating member. This substantially eliminates the time lag between the start of the manipulation on the manipulating member and the start of utterance of the synthetic voice, which makes it possible to synthesize an unfaltering voice in real time. Likewise, for the singing voice of a portion “
(ta)” of “
(saita)”, the output of the voice of the part of transition from the preceding phoneme (in this example, the vowel i) to the first phoneme represented by the phoneme sequence information of the portion (in this example, the consonant t) is started in response to the start of the manipulation on the manipulating member to provide an instruction to start utterance, and the output of the voice of the part succeeding the part of transition from the first phoneme to the succeeding phoneme (the vowel a) is started in response to the full depression of the manipulating member. When the phoneme sequence information represents one vowel, singing voice synthesis may be started in response to the receptions of both the phoneme sequence information and the piece of first utterance control information by the control portion 110, or singing voice synthesis may be started after the reception of the piece of second utterance control information. In the latter mode, singing voice synthesis is performed with a voice intensity represented by the velocity included in the piece of second utterance control information, and in the former mode, singing voice synthesis is started with a predetermined default velocity and in response to the reception of the piece of second utterance control information, the velocity is changed so as to be a value corresponding to the velocity included in the piece of second utterance control information. Moreover, switching between the former mode and the latter mode may be made according to the user's selection.
When the first phoneme of the phoneme sequence represented by the phoneme sequence information is an unsustainable voice (for example, a plosive), the processing of repeating the output of the phoneme until the second utterance control information is received may be executed by the control portion 110, or the output of the phoneme is repeated with one or more than one silence in between so that the phoneme does not succeed such as repeating “the phoneme and a silence”, repeating “a silence, the phoneme and a silence” or repeating “a silence and the phoneme”. In a mode where an apparatus having a musical performance function in addition to the singing voice synthesis function is used as the singing voice synthesizing apparatus 1, when the first and the second utterance control information is inputted without any phoneme sequence information, instead of the singing voice synthesis output, the processing of outputting a musical performance sound by the musical performance function is executed by the control portion 110. Moreover, when no succeeding portion of the lyrics is inputted like when the portion succeeding the first portion “ (sa)” is not inputted in a case where a singing voice starting to sing with “
(saita)” from silent state is synthesized, the processing of synthesizing and outputting a voice of repetitively uttering the part of transition from the first phoneme (the consonant s) to the succeeding phoneme (the vowel a) in the phoneme sequence representing the portion of the lyric (or a voice of repetitively uttering the transition part with one or more than one silence in between) and a voice of continuously uttering the transition part may be executed by the control portion 110 in response to the full depression of the manipulating member to provide an instruction to start utterance. It is essential only that a voice including at least the part of transition from the first phoneme to the succeeding phoneme in the phoneme sequence represented by the phoneme sequence information is synthesized and outputted in response to the reception of the second utterance control information.
In the present embodiment, as shown in
Moreover, by using a sensor to detect that the user's finger has touched the manipulating member (for example, a capacitance sensor) as the first sensor 121 to detect the start of a manipulation on the manipulating member of the musical note information input portion, the synthesis of the voice of the part of transition from a silence or the phoneme of the preceding portion of the lyrics to the first phoneme in the phoneme sequence represented by the phoneme sequence information can be started before the manipulation on the manipulating member to specify the pitch is actually started, so that the delay until the start of the output of the synthetic singing voice can be further reduced. In this mode, the following may be performed: In addition to the sensor to detect that the user's finger has touched the manipulating member, a sensor to detect that the depression of the manipulating member has been started is provided, singing voice synthesis is started in response to the detection output of the former sensor and the output of the synthetic singing voice is started in response to the detection output of the latter sensor.
Moreover, in the present embodiment, the second utterance control information is outputted in response to the full depression of the manipulating member of the musical note information input portion, and the third utterance control information to provide an instruction to stop utterance is outputted in response to the return from the completely depressed position. However, the third utterance control information may be supplied to the control portion 110 in response to the detection, by the first sensor 121, of the return to the position before the start of depression. According to this mode, it is made possible to measure the time required for the return from the completely depressed position to the position before the start of depression and use the length of the time for the control of vanishment of the singing voice being uttered (control of utterance of the released part), so that the expressive power of the singing voice can be further improved by the user performing an operation such as slowly moving the finger from the fully depressed manipulating member. Moreover, it may be performed to detect, by the second sensor 122, that a force is applied to the manipulating member so as to be further depressed from the completely depressed position (or a different sensor to detect the magnitude of the force), supply the control portion 110 with utterance control information corresponding to the magnitude of the force and perform utterance control according to the utterance control information.
It may be performed to switch between an operation mode to output the utterance control information in twice as in the present embodiment and an operation mode to output utterance control information including information representative of the pitch and information representative of the velocity (or the volume) according to an instruction from the user in response to the full depression of the key like the related electronic keyboard instruments. Moreover, the following may be performed: The velocity included in the second utterance control information is not used for the singing voice synthesis and the second utterance control information is used only for identifying the output timing of the part of transition from a consonant to a vowel. In this case, it is unnecessary that the velocity be included in the second utterance control information, and it is also unnecessary that the adjustment of the attack depth or the like be executed by the control portion 110.
Next, an explanation regarding another example of a singing voice synthesizing process will be described. In the phoneme information input portion, during a period from a start time of manipulating on a manipulating member to specify a pitch to a time where the manipulating member is depressed to a completely depressed position of the manipulating member, if a manipulation on one or more different manipulating members to specify another pitch is started, the control portion 110 successively receives a plurality of pieces of first utterance control information generated by the manipulation. In this example, a synthesis processing (first singing voice synthesis processing) of the voice of the part of transition from a silence or the phoneme of the preceding portion of the lyrics to the first phoneme in the phoneme sequence represented by the phoneme sequence information is executed by the control portion 110 by using the earliest one piece selected from among the plurality of pieces of first utterance control information. Also, a synthesis (a second singing voice synthesis processing) of the voice including at least the part of transition from the first phoneme to the succeeding phoneme is executed by the control portion 110 by selecting a piece of second utterance control information corresponding to the earliest piece of first utterance control information (the piece of second utterance control information including information representative of the pitch the same as that included in the earliest piece of first utterance control information) from among one or a plurality of pieces of second utterance control information received after the first singing voice synthesis processing is executed. In this example, the control portion 110 does not accept the one or the plurality of pieces of first utterance control information subsequent to the earliest piece of first utterance control information until the second singing voice synthesis processing is executed. By the above processing, even if during the period from the start time of manipulating on the manipulating member to specify the pitch to the time when the manipulating member is depressed to the completely depressed position of the manipulating member, the manipulation on the different manipulating member to specify another pitch is started and then the plurality of pieces of first utterance control information are successively received, the singing voice synthesis processing is executed by using the earliest piece of first utterance control information from among the received first utterance control information.
For example, in a case that after a start to manipulate a manipulating member corresponding to a pitch “C3”, a manipulation to a different manipulating member corresponding to a pitch “D3” is started before the manipulating member corresponding to the pitch “C3” is completely depressed to a completely depressed position, the earliest piece of first utterance control information, that means, the piece of first utterance control information corresponding to the pitch “C3” is selected. Also, the piece of second utterance control information corresponding to the piece of selected first utterance control information is used for executing a singing voice synthesis processing. The piece of second utterance control information corresponds to the pitch “C3”.
Next, an explanation regarding the other example of a singing voice synthesizing process will be described by referring to
At a step S403, it is determined that whether (i) the control portion 110 receives the piece of first utterance control information, (ii) the control portion 110 receives the piece of second utterance control information, or (iii) the control portion 110 has not received both of the piece of first utterance control information and the piece of second utterance control information. If the control portion 110 receives the piece of first utterance control information at the step S403 (in a case of item (i) of step S403), a process is returned to the step S402, and then the control portion 110 performs a synthesis processing of the transition part from the silence or the phoneme of the preceding portion of the lyrics to the first phoneme in the phoneme sequence in response to the piece of first utterance control information received at the step S403. If the control portion 110 receives the piece of second utterance control information at the step S403 (in a case of item (ii) of step S403), a process proceeds to step S404, and then the control portion 110 performs a synthesis processing of a voice including at least a transition part from the first phoneme to a succeeding phoneme being subsequent to the first phoneme in response to the piece of second utterance control information received at the step S403.
If the control portion 110 has not received both of the piece of first utterance control information and the piece of second utterance control information at a step S403, the control portion 110 waits for receiving either the piece of first utterance control information or the piece of second utterance control information. An explanation of a process of a step S405 is omitted since the process of the step S405 is same as that of the step S205 in
By the above processes, the singing voice synthesis processing of the part of transition from a silence or the phoneme of the preceding portion of the lyrics to the first phoneme in the phoneme sequence represented by the phoneme sequence information can be executed by selecting the piece of first utterance control information (that is, the last piece of first utterance control information), which is received immediately before the reception of the piece of the second utterance control information, from among the plurality of pieces of first utterance control information which are successively received.
According to this configuration, even when a plurality of pieces of first utterance control information are successively acquired by a correction of mis-depression such as mis-touching, a singing voice can be synthesized with the corrected pitch. In a mode that the piece of second utterance control information, which is received first after the reception of one or more pieces of first utterance control information is received from the manipulating portion 120, is always adopted, it is unnecessary that the information representative of the pitch is included in the piece of second utterance control information.
For example, in a case that after a start to manipulate a manipulating member corresponding to a pitch “C3”, a manipulation to a different manipulating member corresponding to a pitch “D3” is started and then the different manipulating member is completely depressed to a completely depressed position and the control portion 110 receives a piece of second utterance control information corresponding to the different manipulating member before the manipulating member corresponding to the pitch “C3” is completely depressed to the completely depressed position, the piece of first utterance control information corresponding to the pitch “D3”, which is received immediately before the reception of the piece of second utterance control information, is selected. The piece of first utterance control information and the piece of second utterance control information corresponding to the pitch “D3” are used for executing a singing voice synthesis processing.
Moreover, when a plurality of utterance control information pairs each formed of the first and the second utterance control information including information representative of the same pitch which utterance control information pairs each correspond to a pitch different among utterance control information pairs are supplied from the manipulating portion 120 to the control portion 110, singing voice synthesis may be performed for each utterance control information pair (that is, synthesis of a plurality of kinds of singing voices with different pitches may be simultaneously performed in parallel). For example, when a manipulation on a manipulating member corresponding to a pitch “C3” and a manipulation on a different manipulating member corresponding to a pitch “D3” are conducted at substantially simultaneously, the singing voice syntheses executed in response to receptions of the piece of first utterance control information and the piece of second utterance control information are simultaneously performed for each of the pitch “C3” and the pitch “D3” in parallel. Therefore, the singing voice syntheses for the pitch “C3” and the pitch “D3” can be executed without faltering feeling.
(B: Modifications)
While an embodiment of the present disclosure have been described above, it is to be noted that the following modifications may be added to the embodiment:
(1) In the above-described embodiment, the first utterance control information is outputted by the manipulating portion 120 in response to the depression of the manipulating member to specify the pitch to a predetermined depth (or the detection of the user's finger touching on the manipulating member). However, the following may be performed: A sensor to detect that the user's finger has approached the manipulating member up to a distance shorter than a predetermined threshold value is used as the first sensor 121, and the first utterance control information is outputted by the manipulating portion 120 in response to the detection of the user's finger approaching the manipulating member up to the distance shorter than the predetermined threshold value by the sensor. In this case, in order to prevent the voice of the part of transition from a silence or the phoneme of the preceding portion of the lyrics to the first phoneme in the phoneme sequence represented by the phoneme sequence information from being continuously outputted without a limitation although the manipulating member is not operated in actuality, when neither the touching of the user's finger nor the depression (or the full depression) of the manipulating member is detected within a predetermined time from the output of the first utterance control information, fourth utterance control information to provide an instruction to stop the output of the voice of the transition part is outputted by the manipulating portion 120. Moreover, the following may be performed: A manipulating member to let the user provide an instruction to output the fourth utterance control information is provided on the manipulating portion 120 and the fourth utterance control information is outputted by the manipulating portion 120 in response to the detection of a manipulation on the manipulating member.
(2) In the above-described embodiment, a case is described in which the manipulating members to specify the pitch of the singing voice also assume the role of a manipulating member to let the user provide an instruction to start utterance, the first utterance control information is outputted in response to the start of a manipulation on the manipulating member (touching of the user's finger or depression to a predetermined depth) and the second utterance control information is outputted in response to the completion of the manipulation on the manipulating member (full depression of the manipulating member). However, it is to be noted that the role of outputting the second utterance control information may be assumed by a manipulating member different from the above-mentioned manipulating member (for example, a dial or a pedal for specifying the intensity or the volume of the utterance of the singing voice). Specifically, a foot-pedal-form manipulating member is provided on the manipulating portion 120 as the manipulating member to specify the intensity or the volume of the utterance of the singing voice, and the first utterance control information is outputted by the manipulating portion 120 in response to the detection of the start of a key operation on the musical note information input portion resembling a piano keyboard, whereas the second utterance control information is outputted by the manipulating portion 120 in response to the detection of the depression of the pedal-form manipulating member. Also in this mode, a voice corresponding to the transition from a silence or the phoneme of the preceding portion of the lyrics to the first phoneme in the phoneme sequence represented by the phoneme sequence information is outputted in response to the detection of the start of a key operation on the musical note information input portion resembling a piano keyboard, so that an unfaltering singing voice can be synthesized in real time with no time lag. Moreover, by adjusting the depression timing of the pedal-form manipulating member, the output timing of the voice of the part of transition from the first phoneme to the succeeding phoneme (for example, the part of transition from a consonant to a vowel) can be aligned with the timing of the musical note on the musical score, so that human singing characteristics can be accurately reproduced.
(3) While in the above-described embodiment, devices resembling an electronic keyboard instrument is used as an acquisition section for causing the singing voice synthesizing apparatus 1 to acquire the first and the second utterance control information (the musical note information input portion of the manipulating portion 120), devices resembling an electronic stringed instrument, an electronic wind instrument, an electronic percussion instrument or the like may be used as long as it resembles a MIDI-controlled electronic instrument. For example, when a device resembling an electronic stringed instrument such as an electronic guitar is used as the musical note information input portion of the manipulating portion 120, a sensor to detect that the user's finger or a pick has touched a string is provided as the first sensor 121, a sensor to detect that the user has started to pluck a string is provided as the second sensor 122, the first utterance control information is outputted in response to the detection output by the first sensor 121, and the second utterance control information is outputted in response to the detection output by the second sensor 122. In this case, the string assumes both the role of the manipulating member to let the user provide an instruction to start utterance and the role of the manipulating member to let the user specify the pitch, and further assume the role of the manipulating member to specify the velocity or the like. Also, the first utterance control information is received by the start of a manipulation (touching of the user's finger) on the manipulating member (string) to let the user provide an instruction to start voice utterance, and the second utterance control information is received by the completion of a manipulation (plucking by the user's finger or the like) on the manipulating member.
When a device resembling an electronic wind instrument is used as the musical note information input portion of the manipulating portion 120, a sensor to detect that the user's finger has touched a manipulating member resembling a piston or a key of a woodwind instrument is provided as the first sensor 121, a sensor to detect that the user has started to pipe is provided as the second sensor 122, the first utterance control information is outputted in response to the detection output by the first sensor 121, and the second utterance control information is outputted in response to the detection output by the second sensor 122. In this case, the manipulating member resembling a piston or a key of a woodwind instrument assumes the role of letting the user provide an instruction to start voice utterance and the role of letting the user specify the pitch, and a blowing mouth such as a mouthpiece assumes the role of the manipulating member to specify the velocity or the like. Also, the first utterance control information is received by the start of a manipulation (touching of the user's finger) on the manipulating member to let the user provide an instruction to start voice utterance (the manipulating member resembling a piston or a key of a woodwind instrument), and the second utterance control information is received by a manipulation (the start of piping) on the manipulating member (the blowing mouth such as a mouthpiece) different from the above-mentioned manipulating member. The second utterance control information may be outputted by the detection of the completion of the manipulation (full depression) of the manipulating member resembling a piston or a key of a woodwind instrument instead of outputting the second utterance control information by the detection of the start of piping on the blowing mouth such as a mouthpiece.
Moreover, when a device resembling an electronic percussion instrument is used as the musical note information input portion of the manipulating portion 120, a sensor to detect that a drumstick (or the user's hand or finger) has touched a beaten part is provided as the first sensor 121, a sensor to detect the completion of beating (for example, that the beating force has become maximum or that the beaten area of the beaten part has become maximum) is provided as the second sensor 122, the first utterance control information is outputted in response to the detection output by the first sensor 121, and the second utterance control information is outputted in response to the detection output by the second sensor 122. In this case, the beaten part assumes the role of the manipulating member to let the user provide an instruction to start utterance. Also, the first utterance control information is received by the start of the manipulation (touching of the user's finger or the like) on the manipulating member (beaten part) to let the user provide an instruction to start voice utterance, and the second utterance control information is received by the completion of the manipulation (that the beating force or the beaten area has become maximum) on the manipulating member. With the musical note information input portion resembling an electronic percussion instrument, there are cases where the pitch cannot be specified by an operation on the musical note information input portion. In this case, the musical note information representative of the musical notes constituting the melody of the song which is the object of singing voice synthesis (information representative of the pitch and the duration) is stored in the singing voice synthesizing apparatus 1, and the musical note information is successively read for use every time the first utterance control information is received. Moreover, it may be performed to divide the beaten part of the musical note information input portion resembling an electronic percussion instrument into a plurality of areas and associate each area with a different pitch to thereby enable pitch specification.
Moreover, the musical note information input portion is not limited to a MIDI-controlled one; it may be a general keyboard or a general touch panel to let the user input characters, symbols or numbers or may be a general input device such as a pointing device such as a mouse. When these general input devices are used as the musical note information input portion, the musical note information representative of the musical notes constituting the melody of the song which is the object of singing voice synthesis (information representative of the pitch and the duration) is stored in the singing voice synthesizing apparatus 1. Then, the first utterance control information is outputted by the manipulating portion 120 in response to the start of a manipulation on a manipulating member corresponding to a character, a symbol or a number, a touch panel, a mouse button, or the like, the second utterance control information is outputted by the manipulating portion 120 in response to the completion of the manipulation on the manipulating member, and the musical note information is successively read for use by the singing voice synthesizing apparatus 1 every time the first utterance control information is received.
It is essential only that the following mode be adopted: The first utterance control information is received in response to the start of a manipulation on the manipulating member to let the user provide an instruction to start utterance, the second utterance control information is received in response to the completion of a manipulation on the manipulating member (or a manipulation on a different manipulating member), a voice corresponding to the part of transition from a silence or the phoneme of the preceding portion of the lyrics to the first phoneme in the phoneme sequence represented by the phoneme sequence information is synthesized by use of a plurality of kinds of synthesis information in response to the acquisition of the first phoneme control information and outputted, and a voice including at least the part of transition from the first phoneme to the succeeding phoneme is synthesized by use of a plurality of kinds of synthesis information in response to the acquisition of the second utterance control information and outputted.
(4) In the above-described embodiment, a case is described in which the phoneme sequence information representative of the phoneme sequence of the portion of the lyrics uttered in time with a musical note is sequentially inputted for each musical note by an operation on the phoneme information input portion of the manipulating portion 120. However, the following may be performed: The phoneme sequence information related to the lyrics of the entire song which is the object of singing voice synthesis is previously stored in the non-volatile storage portion 162 of the singing voice synthesizing apparatus 1, the pitch and the like when each portion of the lyrics is uttered are sequentially specified for each musical note by an operation on the musical note input portion, and for each musical note, the phoneme sequence information corresponding to the musical note is read in response to the specification of the pitch and the like to synthesize a singing voice.
Moreover, in a case that voice synthesis is performed for each utterance control information pair when a plurality of utterance control information pairs each corresponding to a different pitch are supplied from the manipulating portion 120 to the control portion 110, the following may be performed: A plurality of kinds of phoneme sequence information representative of different portions of the lyrics are stored, and a singing voice of a different pitch and portion of the lyrics is synthesized by the control portion 110 for each utterance control information pair. For example, N (N is a natural number not less than 2) kinds of phoneme sequence information representative of different portions of the lyrics are sequenced and previously stored in the non-volatile storage portion 162, and when the number N of utterance control information pairs each including a different piece of pitch information is supplied from the manipulating portion 120 to the control portion 110, the processing of synthesizing the n-th (1≤n≤N) singing voice is executed by the control portion 110 by use of the first and the second utterance control information constituting the n-th phoneme sequence information and the n-th utterance control information pair (the input order of the first utterance control information is used as the input order of the utterance control information pairs). Moreover, it may be performed to predetermine the range of the pitch so as not to overlap one another for each of the number N of pieces of phoneme sequence information and for each piece of phoneme sequence information, perform voice synthesis by use of the utterance control information pair corresponding to the pitch belonging to the pitch range corresponding to the phoneme sequence information. For example, some split points are set in the pitch direction, and the pieces of phoneme sequence information are associated one-to-one with the ranges divided by the split points.
(5) In the above-described embodiment, the manipulating portion 120 that assumes the role of the acquisition section for causing the singing voice synthesizing apparatus 1 to acquire the first and the second utterance control information and a plurality of kinds of synthesis information and the voice output portion 140 for outputting a synthetic singing voice are incorporated in the singing voice synthesizing apparatus 1. However, a mode may be adopted that either one of the manipulating portion 120 and the voice output portion 140 or both of them are connected to the external device I/F portion 150 of the singing voice synthesizing apparatus 1. In the mode that the manipulating portion 120 is connected to the singing voice synthesizing apparatus 1 through the external device I/F portion 150, the external device I/F portion 150 assumes the role of the acquisition section.
An example of the mode in which both the manipulating portion 120 and the voice output portion 140 are connected to the external device I/F portion 150 is a mode in which an Ethernet (trademark) interface is used as the external device I/F portion 150, an electric communication line such as a LAN (local area network) or the Internet is connected to this external device I/F portion 150 and the manipulating portion 120 and the voice output portion 140 are connected to this electric communication line. According to this mode, it is possible to provide so-called cloud computing type singing voice synthesis service. Specifically, the phoneme sequence information inputted by operating various manipulating members provided on the manipulating portion 120 and the first and the second utterance control information are supplied to the singing voice synthesizing apparatus through the electric communication line, and the singing voice synthesizing apparatus executes singing voice synthesis processing based on the phoneme sequence information and the first and the second utterance control information supplied through the electric communication line. In this manner, the voice data of the synthetic singing voice synthesized by the singing voice synthesizing apparatus is supplied to the voice output portion 140 through the electric communication line, and a voice corresponding to the voice data is outputted from the voice output portion 140.
(6) In the above-described embodiment, the singing voice synthesis program 162b for causing the control portion 110 to execute the singing voice synthesis processing noticeably exhibiting the features of the present disclosure is previously stored in the non-volatile storage portion 162 of the singing voice synthesizing apparatus 1. However, this singing voice synthesis program 162b may be distributed in the form of being written on a computer-readable recording medium such as a CD-ROM (compact disk-read only memory) or may be distributed by a download through an electric communication line such as the Internet. This is because by causing a general computer such as a personal computer to execute the program distributed as described above, it is possible to cause the computer to function as the singing voice synthesizing apparatus 1 of the above-described embodiment. Moreover, it is to be noted that the present disclosure may be applied to a game program of a game including real-time singing voice synthesis processing as a part thereof. Specifically, the singing voice synthesis program included in the game program may be replaced with the singing voice synthesis program 162b. According to this mode, the expressive power of the singing voice synthesized as the game proceeds can be improved.
(7) In the above-described embodiment, an example of application of the present disclosure to a real-time singing voice synthesizing apparatus is described. However, the object of application of the present disclosure is not limited to the real-time singing voice synthesizing apparatus. For example, the present disclosure may be applied to a voice synthesizing apparatus that synthesizes a guidance voice in a voice guidance in real time or a voice synthesizing apparatus that synthesizes a voice of reading literary work such as a novel or a poem in real time. Moreover, the object of application of the present disclosure may be a toy having a singing voice synthesis function or a voice synthesis function (a toy incorporating a singing voice synthesizing apparatus or a voice synthesizing apparatus).
Here, the above embodiments are summarized as follows.
a first receiving step of receiving first utterance control information generated by detecting a start of a manipulation on a manipulating member by a user;
a first synthesizing step of synthesizing, in response to a reception of the first utterance control information, a first voice corresponding to a first phoneme in a phoneme sequence of a voice to be synthesized to output the first voice;
a second receiving step of receiving second utterance control information generated by detecting a completion of the manipulation on the manipulating member or a manipulation on a different manipulating member; and
a second synthesizing step of synthesizing, in response to a reception of the second utterance control information, a second voice including at least the first phoneme and a succeeding phoneme being subsequent to the first phoneme of the voice to be synthesized to output the second voice.
outputting third utterance control information to provide an instruction to stop an output of the first voice when the reception of the second utterance control information is not detected within a predetermined time from the output of the first utterance control information.
a third receiving step of receiving third utterance control information generated by detecting a completion of a manipulation on the manipulating member by the user, wherein the third utterance control information includes pitch information and a velocity or a volume;
a third synthesizing step of synthesizing, in response to a reception of the third utterance control information, a third voice to output the third voice; and
a switching step of switching between a first operation mode and a second operation mode,
wherein in the first operation mode, the first receiving step, the first synthesizing step, the second receiving step and the second synthesizing step are performed; and
wherein in the second operation mode, the third receiving step and the second synthesizing step are performed.
a first receiver configured to receive first utterance control information generated by detecting a start of a manipulation on a manipulating member by a user;
a first synthesizer configured to synthesize, in response to a reception of the first utterance control information, a first voice corresponding to a first phoneme in a phoneme sequence of a voice to be synthesized to output the first voice;
a second receiver configured to receive second utterance control information generated by detecting a completion of the manipulation on the manipulating member or a manipulation on a different manipulating member; and
a second synthesizer configured to synthesize, in response to a reception of the second utterance control information, a second voice including at least the first phoneme and a succeeding phoneme being subsequent to the first phoneme of the voice to be synthesized to output the second voice.
By the feature described in the above item (3), it is possible to synthesize an unfaltering natural voice in real time while appropriately specifying the pitch when a synthetic voice is uttered.
By the feature described in the above item (5), it is possible to synthesize an unfaltering natural voice in real time while appropriately specifying the velocity or volume when a synthetic voice is uttered in addition to the pitch.
By the feature described in the above item (6), synthetic voices with different pitches can be simultaneously synthesized in parallel.
Although the invention has been illustrated and described for the particular preferred embodiments, it is apparent to a person skilled in the art that various changes and modifications can be made on the basis of the teachings of the invention. It is apparent that such changes and modifications are within the spirit, scope, and intention of the invention as defined by the appended claims.
The present application is based on Japanese Patent Application No. 2012-250438 filed on Nov. 14, 2012, the contents of which are incorporated herein by reference.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2012-250438 | Nov 2012 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 4945557 | Kaneuchi | Jul 1990 | A |
| 5290964 | Hiyoshi et al. | Mar 1994 | A |
| 5311175 | Waldman | May 1994 | A |
| 5875427 | Yamazaki | Feb 1999 | A |
| 5883327 | Terada et al. | Mar 1999 | A |
| 6075196 | Fujiwara | Jun 2000 | A |
| 6163769 | Acero et al. | Dec 2000 | A |
| 6304846 | George | Oct 2001 | B1 |
| 7124084 | Kayama et al. | Oct 2006 | B2 |
| 20020166437 | Nishitani et al. | Nov 2002 | A1 |
| 20020184006 | Yoshioka | Dec 2002 | A1 |
| 20020184032 | Hisaminato | Dec 2002 | A1 |
| 20030004723 | Chihara | Jan 2003 | A1 |
| 20030009336 | Kenmochi | Jan 2003 | A1 |
| 20030009344 | Kayama | Jan 2003 | A1 |
| 20040186720 | Kemmochi | Sep 2004 | A1 |
| 20060085196 | Kayama et al. | Apr 2006 | A1 |
| 20060173676 | Kemmochi | Aug 2006 | A1 |
| 20060271367 | Hirabayashi et al. | Nov 2006 | A1 |
| 20070214947 | Nishibori et al. | Sep 2007 | A1 |
| 20120136661 | Fu et al. | May 2012 | A1 |
| 20120143600 | Iriyama | Jun 2012 | A1 |
| 20120166197 | Fu et al. | Jun 2012 | A1 |
| 20140006031 | Mizuguchi | Jan 2014 | A1 |
| 20140046667 | Yeom | Feb 2014 | A1 |
| Number | Date | Country |
|---|---|---|
| 1870130 | Nov 2006 | CN |
| 102479508 | May 2012 | CN |
| 102486921 | Jun 2012 | CN |
| 0 396 141 | Nov 1990 | EP |
| 1 675 101 | Jun 2006 | EP |
| 3879402 | Nov 2006 | JP |
| 2008-170592 | Jul 2008 | JP |
| Entry |
|---|
| European Search Report dated Mar. 11, 2014, for EP Application No. 13192421.9, ten pages. |
| Moog, B. (May 1, 1986). “Musical Instrument Digital Interface,” Journal of the Audio Engineering Society, vol. 34, No. 5, NY, USA, pp. 394-404. |
| Notification of Reasons for Refusal dated Feb. 10, 2015, for JP Application No. 2012-250438, with English translation, nine pages. |
| Chinese Search Report dated Mar. 2, 2016, for CN Application No. 201310572222.6, with English translation, four pages. |
| Notification of the First Office Action dated Mar. 2, 2016, for CN Application No. 201310572222.6, with English translation, seven pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20140136207 A1 | May 2014 | US |
