Diphone synthesis is one of the most popular methods used for creating a synthetic voice from recordings or samples of a particular person; it can capture a good deal of the acoustic quality of an individual, within some limits. The rationale for using a diphone, which is two adjacent half-phones, is that the “center” of a phonetic realization is the most stable region, whereas the transition from one “segment” to another contains the most interesting phenomena, and thus the hardest to model. The diphone, then, cuts the units at the points of relative stability, rather than at the volatile phone-phone transition, where so-called coarticulatory effects appear.
The invention herein disclosed presents an exemplary method and apparatus for diphone or concatenative synthesis when the computer system has insufficient or missing diphones.
In another embodiment of the invention, Source 110 is text with optional phonetic information. Phonetic Generator 120 is configured to convert the written text into the phonetic alphabet. Intonation Generator 125 is configured to generate pitch from the typed text and optional phonetic information. Together Phonetic Generator 120 and Intonation Generator 125 output a list of diphones corresponding to Source 110.
In each embodiment of the invention, Unit Selector 145 selects the best diphone (“hereinafter the selected diphone(s)”) from Diphone Database 150 which most closely matches the corresponding original diphone from Phonetic Generator 120 and Intonation Generator 125.
Natural sounding speech is created by Concatenator 160, by obtaining the diphones from Unit Selector 145 and concatenating them such that abrupt and unnatural transitions are minimized.
Although the invention admits the use of diphones in this disclosure, the invention is not limited in its use to diphones. Any unit of speech can be used.
In a second embodiment of the invention Source 110 is written text with or without phonetic descriptors. At alternative step 210, said text is obtained by Pronunciation Generator 120 and Intonation Generator 125, where Generator 120 and Intonation Generator 125 create a sequence of diphones representing said text.
At step 220, Unit Selector 145 determines which diphones from Diphone Database 150, i.e. the selected diphones, are the best matches to original diphones.
At step 230, Concatenator 160 combines the diphones into natural sounding speech.
At step 330, Concatenator 160 determines the stable regions of the first and second target diphones. The stable region is the portion of the waveform where the frequency is relatively uniform, i.e. there are few, if any, abrupt transitions. This tends to be the vowels portion of a diphone.
At Step 340, Concatenator 160 overlaps the waveforms of said first and second target diphones to provide a region to transition from the said first target diphone to the second target diphone while minimizing abrupt transitions. Overlapping waveforms is known to one skilled in the art of speech morphology.
At step 350, Concatenator 160 determines the quality of the match between the first and second target diphone collectively, with said first and second original diphone.
Each target diphone has an associated confidence score which represents the quality of the match between said target diphone and the corresponding original diphone. Should the confidence scores for said first target diphone and said second target diphone be 0.5 or lower, Concatenator 160 considers the diphone pair to be a good match, i.e. an easy concatenation. Should the confidence score for said first or second target diphone be above 0.5, Concatenator 160 considers said diphone pair to be a low quality match with the original first and second diphones.
At step 360, the Concatenator selects the time interval, i.e. a commencement location on the first target diphone and termination location on the second target diphone, in which to combine the first and second target diphones i.e. morph the two distinct diphones into natural sounding speech.
At step 370, Concatenator 160 morphs the first and second selected diphones.
For simplicity, although Waveform 410 is decomposed into its excitation function and filter function, Waveform 415 represents only the second formant of Waveform 420. Region 415a represents the stable region of Waveform 415.
Waveform 420 represents the waveform of the second diphone /or/. Region 420a represents the waveform of the /o/ portion of Waveform 420 and Region 420b represents the /r/ portion.
For simplicity, although Waveform 420 is decomposed into its excitation function and filter function, Waveform 425 only represents the second formant of Waveform 410. Region 425a represents the stable region of Waveform 425.
Region 430 represents the overlap of the stable regions between Waveform 415 and Waveform 425. This is the area where the morphing, or concatenation, occurs. Time index 440 represents the beginning of the first third of Region 425a, i.e. the overlapping stable area on Waveform 415 and Waveform 425. Time index 450 represents the end of the second third of Region 425a, i.e. the overlapping stable area on Waveform 415 and Waveform 425.
Region 460 represents the new morphed region between Diphone 410a, Diphone 410b, Diphone 420a and Diphone 420b, i.e. the /do/ and /or/ selected from Diphone Database 150.
For simplicity, although Waveform 510 is decomposed into its excitation function and filter function, Waveform 515 represents the second format of Waveform 510. Region 515a represents the stable region of Waveform 515.
Waveform 520 represents the waveform of the second diphone /or/. Region 520a represents the waveform of the /o/ portion of Waveform 520 and Region 520b represents the /r/ portion.
For simplicity, although Waveform 520 is decomposed into its excitation function and filter function, Waveform 525 represents the second formant of Waveform 520. Region 525a represents the stable region of Waveform 525.
Waveform 530 represents the overlap of the stable regions between Waveform 515 and Waveform 525. This is the area where the morphing, or concatenation, occurs. Time index 540 represents the beginning of Region 525a, i.e. the overlapping stable area on Waveform 515 and Waveform 525. Time index 550 represents the end of the second third of Region 525a, i.e. the overlapping stable area on Waveform 515 and Waveform 525.
Unlike Time Index 440, Time Index 550 occurs at the beginning of the stable region. Specifically, since Region 510b is not identical to the /o/ or /do/, Concatenator 160 diminishes the contribution of Region 510b.
Region 560 represents the new morphed region between Diphone 510a, Diphone 510b, Diphone 520a and Diphone 520b, i.e. the /du/ and /or/ selected from Diphone Database 150.
For simplicity, although Waveform 610 is decomposed into its excitation function and filter function, Waveform 615 represents the second formant of Waveform 610. Region 615a represents the stable region of Waveform 615.
Waveform 620 represents the waveform of the second diphone /ur/. Region 620a represents the waveform of the /u/ portion of Waveform 620 and Region 620b represents the /r/ portion.
For simplicity, although Waveform 620 is decomposed into its excitation function and filter function, Waveform 625 represents the second format of Waveform 620. Region 625a represents the stable region of Waveform 625.
Waveform 630 represents the overlap of the stable regions between Waveform 615 and Waveform 625. This is the area where the morphing, or concatenation, occurs. Time index 640 represents the beginning of the second third of Region 625a, i.e. the overlapping stable area on Waveform 615 and Waveform 625. Time index 650 represents the end of Region 625a.
Unlike Time Index 450 in
Region 660 represents the new morphed region between Diphone 610a, Diphone 610b, Diphone 620a and Diphone 620b, i.e. the /do/ and /ur/ selected from Diphone Database 150.
Number | Name | Date | Kind |
---|---|---|---|
5327521 | Savic | Jul 1994 | A |
7953600 | Hertz | May 2011 | B2 |
8594993 | Qian | Nov 2013 | B2 |
20020193994 | Kibre | Dec 2002 | A1 |
20030212555 | van Santen | Nov 2003 | A1 |
20040030555 | van Santen | Feb 2004 | A1 |
20040111266 | Coorman | Jun 2004 | A1 |
20050131679 | Gigi | Jun 2005 | A1 |
20120072224 | Khitrov | Mar 2012 | A1 |
Number | Date | Country | |
---|---|---|---|
20170162188 A1 | Jun 2017 | US |