The present invention relates generally to a system and method for increasing the dysarthria patients' speech conversion efficiency, and more particularly to a method which automatically generates personalized corpus text considering personalized language features.
At present, the performance of speech conversion systems is usually evaluated in the level of speech recognition using subjectivity lexicon recognition test. In order to verify the efficiency of the present speech conversion model, the subject needs to take a series of pronunciation tests in selecting-fitting process(may take several weeks), and currently there is no customized test based on the characteristics of the subject's speech sound, disease process, etc., for improving the test process. For example, the subject may record the pronunciations for thousands of words (or sentences) aimlessly in the test process, and then the conversion results of the present speech conversion model are applied to evaluate whether the subject needs to record more. In this situation, the subject easily to get annoyed by the long recording time and the unstable results. Furthermore, the test result would be with high uncertainty and errors because it affected by the subjective responses of the subject, such as stamina, emotion, age, linguistic competence and expressiveness, it is not satisfactory.
At present, there is no automatic generation method for common corpus texts considering personalized language features. In addition, there is no technology for real-time core corpus generation according to the phonetic feature processed by speech conversion system. Many popular speech signal processing systems (e.g. speech conversion) are designed based on deep learning architecture. However, for this type of signal processing architecture, representative training corpus is very important. The present methods mainly use mass speech data to try to attain the representative target, but the collection of mass speech data sometimes causes inconvenience to the users, as well as to the dysarthria patients. Therefore, for the users difficult to record much speech (e.g. dysarthria patients), it is very difficult for the patients to complete corpus recording. To solve the above problem, the present invention designs a real-time customized corpus text generation system in the concept of optimization theory (e.g. genetic algorithm), and proposes a system-user interactive mode to increase the training corpus recording efficiency. The invention reduces the corpus recording efficiency of users when using speech conversion system, so as to reduce the patients' difficulty in recording a lot of corpora when using the speech conversion (or other speech processing) system. In addition, the present invention generates new texts according to the deficiencies in current speech conversion system (e.g. poorly converted phonemes and tones, sentence time variation, etc.). These new texts enable the users to record correct training speech, so as to reduce the patients' recording load efficiently.
In view of this, the patent can convert unsound parts (e.g. phonemes, tone, etc.) based on speech conversion system, giving the users a direction of corpus recording, so that the efficiency of speech conversion system can be increased while the difficulty in phonetic transcription is reduced for users. This practice can enhance the usability and difficulty of the proposed system, so as to enhance the chance of success in the speech signal processing products based on deep learning.
The present invention relates to a system and method for increasing the dysarthria patients' speech conversion efficiency, including a text database module, including a corpus text database, storing a plurality of corpus candidate word lists; a model database module, including a tone model database which stores the tone models; an analysis model database which stores the analysis models; a model parameter database, storing a plurality of model parameters; a corpus generation module, connected to the text database module and the model database module, including a first corpus generation unit, generating an initial word list from the text database module; a second corpus generation unit, generating a kernel word list according to the text database module; a speech capture module, the speech of a normal articulator is recorded into a training corpus according to the initial word list or the kernel word list; the speech of an abnormal articulator is recorded into a sample corpus; a speech conversion module, connected to the speech capture module, including a matching unit, matching the training corpus and the sample corpus, marking an abnormally articulated and a correctly articulated sentence of the sample corpus; an analytic unit, the abnormal articulation is analyzed by a plurality of tone models and a plurality of analysis models to obtain an enhanced tone parameter, a model characteristic parameter is derived from the differences among the analysis models; an output module, connected to the speech conversion module, calculating a speech recognition accuracy and connected to an output equipment.
The present invention relates to a system and method for enhancing the dysarthria patients' speech conversion efficiency. The steps of the method are described below. S1. a corpus generation module extracts a plurality of corpus candidate word lists from a corpus text database of a text database module, a first corpus generation unit of the corpus generation module generates an initial word list according to the corpus candidate word lists; S2. a normal articulator records a training corpus through a speech capture module according to the initial word list, an abnormal articulator records an nth sample corpus through the speech capture module according to the initial word list, and the training corpus and the nth sample corpus are transmitted to a speech conversion module; S3. a matching unit of the speech conversion module matches the training corpus and the nth sample corpus, marking an abnormally articulated and a correctly articulated sentence of the nth sample corpus; an analytic unit, after the correct articulation and the unsoundly processed abnormal articulation are analyzed by a plurality of tone models and a plurality of analysis models, an nth enhanced tone parameter is obtained, and an nth model characteristic parameter is obtained according to the differences among the analysis models, and transmitted to the corpus generation module; S4. a second corpus generation unit of the corpus generation module generates an nth kernel word list according to the nth enhanced tone parameter and the nth model characteristic parameter, the abnormal articulator records a No. n+1 sample corpus according to the nth kernel word list, and the No. n+1 sample corpus is transmitted to the speech conversion module; S5. a matching unit of the speech conversion module matches the training corpus and the No. n+1 sample corpus, marking an abnormally articulated and a correctly articulated sentence of the No. n+1 sample corpus; an analytic unit analyzes the correct articulation and the unsoundly processed abnormal articulation through a plurality of tone models and a plurality of analysis models to obtain the No. n+1 enhanced tone parameter, the No. n+1 model characteristic parameter and the No. n+1 speech recognition accuracy.
Preferably, the corpus generation module can set up the articulation disorder type of the abnormal articulator; the enhanced tone parameter and the model characteristic parameter store the model parameters according to the articulation disorder type.
Preferably, the speech conversion module includes a natural language processing unit, performing sentence segmentation or word segmentation for the training corpus or the sample corpus according to the initial word list or the kernel word list of the corpus generation module 30.
Preferably, different texts can be the material of candidate word lists and sentences of this system.
Preferably, the speech recorded by this system is the algorithm development material of speech conversion systems (or hearing aids, artificial electronic ears, speech recognizers, etc.).
Preferably, this system converts unsound phonemes and tones and sentence time variation characteristic to generate new texts.
Preferably, through objective guide, the speech conversion system used in this system can be but not limited to speech recognizer, acoustoelectric characteristic analysis, phoneme and tone characteristics, STOI, PESQ, MCD, phoneme distribution relationship and so on. After evaluation, the processed unsound speech is quantized to the objective function of this system.
Preferably, the speech processing system of this system can improve the deficiencies (e.g. phonemes, tones, sound articulation, etc.) in the unsound abnormal articulation processed by the analytic unit.
Preferably, this system can execute core text generation according to the characteristics of the model (e.g. considering anterior and posterior phonetic features, with memory effectiveness).
Preferably, this system designs a real-time customized corpus text generation system in the concept of optimization theory (e.g. genetic algorithm), and proposes a system-user interactive mode to enhance the training corpus recording efficiency.
Preferably, this system generates the core text according to the model characteristics used by current converting system (e.g. considering time sequence, spectral space relation and attention model), so as to enhance the user's recording efficiency.
The present invention relates to a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the system is shown in
In the aforesaid embodiment, the analytic unit considers time sequence, spectral space relation, inflection characteristic and uses conversion model characteristic.
In the aforesaid embodiment, the enhanced tone parameter and the model characteristic parameter optimize the model parameters of the model parameter database, the optimized cost function includes the minimum mean square error and speech understanding oriented functions (STOI, SII, NCM, HASPI, ASR scores, etc.), speech quality oriented functions (PESQ, HASQI, SDR, etc.), and the model parameters in the model parameter database are updated after optimization.
A preferred embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, the tone models include using speech recognizer, acoustoelectric characteristic analysis, phoneme and tone characteristics, STOI, PESQ, MCD, phoneme distribution relation and so on.
A preferred embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, the analysis models include attention model, model with treatment of time, end to end learning model, natural language processing system and so on.
A preferred embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, the text database module 10 includes an articulation disorder text database, storing a plurality of articulation disorder candidate word lists.
An embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the corpus generation module 30 includes an articulation disorder type input setting of the abnormal articulator; the enhanced tone parameter and the model characteristic parameter store the model parameters according to the articulation disorder type.
A preferred embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, the speech conversion module 50 includes a natural language processing unit, executing sentence segmentation or word segmentation for the training corpus or the sample corpus according to the initial word list or the kernel word list of the corpus generation module 30.
An embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the corpus text database includes an expansion unit for increasing the content of the corpus text database, e.g. Academia Sinica colloquialism corpus, Academia Sinica Chinese corpus, NCCU spoken Chinese corpus, elementary school frequent words, Hanlin text dictionary and so on.
An embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the output equipment can be, but not limited to tabulating machine, display screen, speech, etc.
The present invention relates to a system and method for enhancing the dysarthria patients' speech conversion efficiency, as shown in
S1. A corpus generation module 30 extracts a plurality of corpus candidate word lists from a corpus text database of a text database module 10, a first corpus generation unit of the corpus generation module generates an initial word list according to the corpus candidate word lists;
S2. A normal articulator records a training corpus through a speech capture module 40 according to the initial word list, an abnormal articulator records an nth sample corpus through the speech capture module 40 according to the initial word list, and the training corpus and the nth sample corpus are transmitted to a speech conversion module 50;
S3. A matching unit of the speech conversion module 50 matches the training corpus and the nth sample corpus, marking an abnormally articulated and a correctly articulated sentence of the nth sample corpus; and an analytic unit analyzes the correct articulation and the processed unsound abnormal articulation through a plurality of tone models and a plurality of analysis models to obtain an nth enhanced tone parameter, an nth model characteristic parameter is obtained according to the differences among the analysis models, and transferred to the corpus generation module 30;
S4. A second corpus generation unit of the corpus generation module 30 generates an nth kernel word list according to the nth enhanced tone parameter and the nth model characteristic parameter, the abnormal articulator records a No. n+1 sample corpus according to the nth kernel word list, and the No. n+1 sample corpus is transferred to the speech conversion module 50;
S5. A matching unit of the speech conversion module 50 matches the training corpus and the No. n+1 sample corpus, marking an abnormally articulated and a correctly articulated sentence of the No. n+1 sample corpus; and an analytic unit analyzes the correct articulation and the processed unsound abnormal articulation through a plurality of tone models and a plurality of analysis models to obtain the No. n+1 enhanced tone parameter, the No. n+1 model characteristic parameter and the No. n+1 speech recognition accuracy.
Preferably, in the aforesaid embodiment, a termination condition for a speech recognition accuracy increment percentage can be preset in an input unit of the corpus generation module 30 before the process, when the speech recognition accuracy increment percentage reaches the termination condition, the speech conversion stops, the steps are described below.
S6. An output module judges whether the speech recognition accuracy increment percentage reaches the preset termination condition or not, if not, continue S4;
S7. When the speech recognition accuracy increment percentage reaches the preset termination condition, the dysarthria patient's speech conversion is completed, and the conversion result is exported by the output module.
A preferred embodiment of the present invention, the speech recognition accuracy computing equation is expressed as follows, represented by Word error rate (WER) and Character Error Rate (CER):
Sw is the number of words substituted, Dw is the number of words deleted, Iw is the number of words inserted, Nw=Sw+Dw+Cw.
(Note: Cw=number of correct words and number of correct tones.)
SC is the number of characters substituted, DC is the number of characters deleted, IC is the number of characters inserted, NC=SC+DC+CC.
(Note: CC=number of correct characters and number of correct tones.)
A preferred embodiment of the present invention, the termination condition computing equation is expressed as follows, when the WAcc and CAcc are larger than X %, or the number of iterations exceeds N and the accuracy is not increased anymore, the system is stopped. (Note: variables X and N can be determined by the user, X is assumed to be 90% and N is 10 in current embodiment.)
WA
CC(%)=(1−WER)*100 (3)
CA
CC(%)=(1−CER)*100 (4)
A preferred embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, a user inputs an articulation disorder region of the abnormal articulator in the input unit, the corpus generation module 30 extracts a plurality of articulation disorder candidate word lists corresponding to the articulation disorder region from an articulation disorder text database of the text database module 10 according to the articulation disorder region. The corpus generation module 30 generates the initial word list and the kernel word list according to the articulation disorder candidate word lists.
A preferred embodiment of the present invention, after evaluation, the processed unsound speech is quantized to an objective function of this system, the objective function of this system is the relation expressed as the minimization equation (5).
(Note: w1, w2 and w3 are the attention weights for adjusting initial, final and tone pattern (T). The Initial and initial are the target and estimated frequency of each initial respectively. The Final and final are the estimated frequency of each final. The T and t are the target and estimated frequency of each tone pattern respectively. The variable N is the estimated total number assessed, K is the number of tone patterns.)
An Chinese embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the forms of corpus delivered from the corpus generation unit include individual plurality of single word combinations (Table 1. The monosyllabic Mandarin word represented by traditional Chinese character and its Hanyu Pinyin.), a plurality of double word combinations (Table 2. The disyllabic Mandarin word represented by traditional Chinese characters and its Hanyu Pinyin.) and a plurality of phrase combinations (Table 3), or the mixture of the single word combinations, the double word combinations and the phrase combinations.
indicates data missing or illegible when filed
An embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the quantity of corpora of the training corpus can set multiple word combinations or sentence combinations as one training unit.
An embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the model parameters include the proportion or number of specific consonants, the proportion or number of specific vowels, the proportion or number of specific consonant-vowel combinations, and the proportion or number of specific ultrasonic band features.
An embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the initial word list covers all the vowels and consonants of language (e.g. if there are tones, those which are likely to be confused can be selected), covers the known tones which are likely to be confused in the language (e.g. similar manners and positions of articulation), and generates comparable materials. Shorter unit of organization of material is the priority (e.g. single word priority).
An Chinese embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the matching unit compares the phoneme recognitions before and after conversion, as shown in Table 4. (The matching unit is used to compare the phoneme recognition before and after conversion. Note that the “?” in the table represents the result of whether the speech recognizer recognizes the speech which was processed by the conversion system correctly or not.)
-“zhi1”
-“zhi1”
-“zhi1”
-“chi1”
-“zhi1”
-“chi1
-“shi1”
-“zhi1”
-“shi1”
-“zhi1”
-“zhi1”
-“zhi1”
-“chi1
-“zhi1”
-“zhi1”
-“shi1”
-“zhi1”
-“zhi1”
An Chinese embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the analytic unit expands the single word combination, double word combination and phrase combination sampling of examples in the same length for single word combination of unstable articulation, as shown in Table 5.
=>
=>
=>
=>
=>
=>
=>
=>
=>
=>
=>
In the aforesaid embodiment, if a speech recognition accuracy increment percentage is not reached after the single word combination of unstable articulation expands examples in the same length, the material unit with the error unit is expanded continuously, till the recognition result before conversion reaches or exceeds the speech recognition accuracy increment percentage.
An embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the speech conversion module 50 uses the Principle of Least Effort, for the articulation units which can be smoothly converted by the analytic unit, the speech samples of expansion training are generated automatically by the user's voice. For the articulation units which cannot be converted smoothly, new training materials are generated according to the aforesaid expansion length concept.
An Chinese embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, as shown in
An embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the enhanced tone parameter and the model characteristic parameter obtained by the analytic unit are stored in the model parameter database, they can be optimized with the existing model parameters. The optimized cost function includes minimum mean square error and speech understanding oriented functions (STOI, SII, NCM, HASPI, ASR scores, etc.), and speech quality oriented functions (PESQ, HASQI, SDR, etc.).
An embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein after the model parameters are optimized, the articulation disorder sentences of the articulation disorder candidate word lists of the articulation disorder text database corresponding to the articulation disorder type are adjusted.
An embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, wherein the corpus generation module 30 is shown in
An embodiment of the present invention, a system and method for enhancing the dysarthria patients' speech conversion efficiency, the process is shown in
S100˜S102. Such texts as candidate word lists and sentences are prepared for this system to choose, different texts can be used as the material of candidate word lists and sentences of this system.
S103. This system gives distribution objective based on target words to the core corpus text to generate initial word list (Wo).
S104. The user executes phonetic transcription based on initial word list (Wo), so as to obtain training corpus;
S105. The obtained training corpus is used as the training material of speech conversion (or other speech processing) system, so as to complete the model training
S106. The objective indicator includes speech recognizer, acoustoelectric characteristic analysis and phoneme and tone characteristics for evaluation.
S107. The unsound parts processed by current model are counted and converted into “enhanced tone parameter”, meanwhile S105 considers the model characteristic used in current speech conversion system (or other speech processing system), which is converted into “model characteristic parameter”.
S108˜S110. The “core corpus generation system” generates a word list (Wi) again according to the “enhanced tone parameter” and “model characteristic parameter”. In other words, this system can generate word list (Wi) again according to current unsound part processed by speech processing system and considering current model characteristic, and then the word list (Wi) is generated again, and the user reads the new training corpus again.
Repeat S104, the speech conversion (or other speech processing) system executes training again according to new training corpus, so as to enhance the effectiveness of system. The user optimizes the speech conversion system continuously according to S104 to S110, the system processing efficiency is improved continuously by user-system interdependent behavior pattern.
This system can more efficiently guide the patient to read appropriate training statements, the processing efficiency of speech conversion (or other speech processing) system is enhanced by each correct training statement acquisition process of the patient. To be more specific, the method of this patent can be used to generate an appropriate direction of speech acquisition, so as to increase the benefit of training corpus to current model, to reduce the use-cost of speech conversion (or other speech processing) system, and to increase the processing efficiency of outside test statements (unseen statements during training).
Although the present invention has been explained in relation to its preferred embodiment, it is to be understood that many other possible modifications and variations can be made without departing from the spirit and scope of the present invention as hereinafter claimed.
Number | Date | Country | Kind |
---|---|---|---|
110104509 | Feb 2021 | TW | national |