Patent Application
20230237928
The present disclosure relates to an apparatus and method for improving dysarthria and more specifically it relates to an apparatus and method for improving dysarthria that provides training to a person with dysarthria and display the visualized voice resulting from the training by receiving such voice.
In order to improve dysarthria caused by various causes such as brain damage, speech therapy is currently performed by a human based on logopedics. Speech therapies performed by humans is performed 2-3 times a week and evaluations of them can vary depending on the therapist, as a human performs it.
The present invention can add a game element so that a person with a dysarthria can perform training more focused. The present invention visualizes and shows the voice of a user with dysarthria in real time, so that the user can confirm his/her articulation in real time.
The present disclosure adds a game element to a training so that a person with dysarthria can perform the training whilst being more focused. The present disclosure visualizes and shows a voice of a user with dysarthria in real time, so that the user can confirm his/her articulation in real time.
According to one aspect of the present disclosure, a method of providing a language training to a user by a computing device comprising a processor and a memory, the method comprises, providing contents corresponding to the language training to a user terminal; receiving the user’s voice data from the user terminal; detecting a pitch and loudness of the user’s voice by analyzing the voice data; and generating a training evaluation by evaluating the user’s training for the contents corresponding to the language training based on the user’s voice data, further comprising determining a phoneme with poor pronunciation accuracy by analyzing the user’s voice data; and automatically generating and providing at least one of a vocabulary, a sentence, and a paragraph including the determined phoneme.
In one embodiment of the present disclosure, a method further comprises, after the detecting a pitch and a loudness of the user’s voice, measuring the user’s language level based on the detected user’s pitch and loudness; generating feedback in real time based on the measured language level of the user; updating contents representing the feedback corresponding to the language training; and transmitting the updated content in which the feedback appears to the user terminal in real time, so that the user can check the feedback in real time.
In one embodiment of the present disclosure, the contents corresponding to the language training is an image that includes an agent and an object, wherein the agent includes a first image, and the object includes a second image different from the first image; the generating a feedback includes generating the feedback so that the agent moves toward the object or moves away from the object in response to the detected loudness of the user’s voice.
In one embodiment of the present disclosure, the generating feedback includes generating a feedback where the agent moves towards a first direction facing the object in response to determining that the loudness of the detected user’s voice is greater than or equal to a selected threshold and the agent moves towards a second direction opposite to the first direction in response to determining the loudness of the detected user’s voice is less than the selected threshold.
In one embodiment of the present disclosure, the generating feedback includes removing the object overlapping with the agent from the contents in response to the agent overlapping with the object by moving towards the first direction.
In one embodiment of the present disclosure, the contents corresponding to the language training is an image that includes an agent and an object, wherein the agent includes a first image, and the object includes a second image different from the first image; and the generating of the feedback includes generating the feedback so that the agent moves in an upward or downward direction of the object in response to the pitch of the detected user’s voice.
In one embodiment of the present disclosure, the generating feedback includes generating a feedback where the agent moves towards the upward direction relative to the object in response to determining that the pitch of the detected user’s voice is greater than or equal to a selected threshold and moves towards the downward direction relative to the object in response to determining the loudness of the detected user’s voice is less than the selected threshold.
In one embodiment of the present disclosure, the contents corresponding to the language training is an image that includes an agent and an object, wherein the agent includes a first image, and the object includes a second and a third image different from the first image. The second image represents a first pitch and placed on a first position of the contents and the third image represents a second pitch different from the first pitch and placed on a second position of the contents that is different from the first position. The generating feedback includes placing the agent in line with the second image or the third image in response to the pitch of the detected user’s voice.
In one embodiment of the present disclosure, the contents corresponding to the language training may include a vocabulary of at least two syllables and an image of a human neck structure, and further comprises, after receiving the user’s voice data from the user terminal, determining whether the syllables of the user’s voice data corresponds to a syllable of the vocabulary of at least two syllables and changing the neck structure image in response to the correspondence between the user’s voice data and the syllables of the vocabulary of at least two-syllables.
In one embodiment of the present disclosure, the analyzing of the voice data to detect the pitch and loudness of the user’s voice includes obtaining a decibel value of the user’s voice and the measuring the user’s language level based on the detected user’s pitch and loudness. The measuring the user’s language level based on the detected user’s pitch and loudness includes acquiring at least one of the user’s sound length, beat accuracy, and breath holding time based on the decibel value.
In one embodiment of the present disclosure, the measuring the user’s language level based on the detected user’s pitch and loudness includes determining whether the pitch is maintained at a level greater than or equal to a threshold for a selected time based on the pitch.
In one embodiment of the present disclosure, the contents corresponding to the language training includes a sentence; and further comprises, after the receiving the user’s voice data from the user terminal, evaluating a pronunciation accuracy of the user by analyzing the voice data.
In one embodiment of the present disclosure, the evaluating a pronunciation accuracy of the user by analyzing the voice data includes measuring a pronunciation accuracy by converting voice data into a text and comparing it to a sentence included in contents corresponding to the language training and measuring a pronunciation accuracy through Deep learning.
In one embodiment of the present disclosure includes, after the providing the contents corresponding to the language training to the user terminal, receiving the user’s face image data from the user terminal, and detecting at least one of a user’s lip shape, cheek shape, and tongue movement by analyzing the face image data.
In one embodiment of the present disclosure, the contents corresponding to the language training includes contents for training the user’s breathing, vocalization, modulation, resonance, and prosody.
In one embodiment of the present disclosure includes, after detecting the pitch and loudness of the user’s voice, generating a training evaluation by evaluating the user’s training for content corresponding to the language training based on the user’s voice data, storing the training evaluation in the memory; and determining the language training to provide to the user based on the training evaluation
In one embodiment of the present disclosure, the generating the training evaluation by evaluating the user’s training includes analyzing the user’s voice data to determine a phoneme with poor pronunciation accuracy and automatically generating and providing at least one of a vocabulary, a sentence, and a paragraph including the determined phoneme.
The above methods of the present disclosure may be performed by a computing device comprising of a processor and a memory
According to another aspect of the present disclosure, a method of the present disclosure may be performed by a computing device comprising of a processor and a memory: including providing contents corresponding to the language training to a user terminal; receiving the user’s voice data and the pitch and decibels of the user’s voice collected based on a voice data from the user terminal; detecting a pitch and a loudness of the user’s voice by analyzing the voice data; and generating a training evaluation by evaluating the user’s training for the contents corresponding to the language training based on the user’s voice data, further comprising determining a phoneme with poor pronunciation accuracy by analyzing the user’s voice data and automatically generating and providing at least one of a vocabulary, a sentence, and a paragraph including the determined phoneme; and storing the training evaluation in the memory.
According to another aspect of the present disclosure, a method of which the computing device comprising a processor and a memory providing a language training to a user includes: providing first contents and second contents corresponding to the language training wherein the first contents including a first agent image and a first object image and the second contents including a second agent image and a second object image to a user terminal, wherein the first contents are configured such that the first agent image is movable in response to the pitch and loudness of the user’s voice; the second contents includes a first pitch image placed on a first position of the second contents, which represents a first pitch and a second pitch image that represents a second pitch and placed on a second position of the second contents different from the first position; and the second contents are configured such that the second agent image corresponds to the user’s pitch and is in line with the first pitch image or the second pitch image; receiving the user’s voice data; receiving a training evaluation of the user for each of the first contents and the second contents; preferentially providing any one of the first contents and the second contents to the user terminal based on the training evaluation; and storing the speech data and the training evaluation in the memory.
In one embodiment of the present disclosure includes providing third contents including at least one of a vocabulary, a sentence, and a paragraph to the user terminal; generating a training evaluation for the third content by analyzing the user’s voice data; and, based on the training evaluation for each of the first contents and the second contents and the training evaluation for the third contents, providing preferentially one of the first to third contents to the user terminal.
In one embodiment of the present disclosure, the generating a training evaluation for third contents includes determining a phoneme with poor pronunciation accuracy by analyzing the user’s voice data; and automatically generating at least one of a vocabulary, a sentence, and a paragraph that includes the determined phoneme.
Speech therapy can be performed as much as you want without time and space constraints. Personalized training can be provided. By visualizing and showing the voice of a user with dysarthria in real time, the training effect can be enhanced by allowing the user to check his or her articulation in real time.
Hereinafter, with reference to the accompanying drawings, the embodiments of the present disclosure will be described in detail so that those of ordinary skill in the art to which the present disclosure pertains can readily implement them. However, the present disclosure may be implemented in several different forms and is not limited to the embodiments described herein.
In order to clearly explain the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.
Throughout the specification, when a part “includes” or “comprises” a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.
It is to be understood that the techniques described in the present disclosure are not intended to be limited to specific embodiments, and include various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure.
The expression “configured to (or set to)” as used in this disclosure, depending on the context, can be used interchangeably with, for example, “suitable for”, “having the capacity to,” “designed to”, “adapted to”, “made to”, or “capable of”. The term “configured (or configured to)” is not necessarily means only “specifically designed to” hardware. Instead, in some circumstances, the expression “a device configured to” means that the device is “capable of” with other devices or components. For example, the phrases “a processor configured (or configured to perform) A, B, and C,” “a module configured (or configured to perform) A, B, and C”, means a dedicated processor (for example, it may mean an embedded processor) or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing corresponding operations by executing one or more software programs stored in a memory device.
The prior disclosures described in the present disclosure are incorporated herein by reference in their entirety, and it will be understood that the contents described in the prior disclosures is applied to the portions briefly described in the present disclosure by a person of ordinary skill in the art.
Hereinafter, a method and device for improving dysarthria according to an embodiment of the present disclosure will be described with reference to the drawings.
Referring to
In one embodiment of the disclosure, the terminal 100 may receive and store the personal information of the user 10 or transmit it to the server 200. The server 200 may store personal information of the user 10. The personal information may include biographical information and medical information of the user. For example, the personal information may be at least one of real name, gender, age (date of birth), phone number, and dysarthria related medical information. The terminal 100 may provide a questionnaire to the user 10, receive an answer, and store it or transmit it to the server 200. The questionnaire provided by the terminal 100 to the user 10 may include a questionnaire received from the server 200.
The server 200 may generate training based on the answer to the questionnaire or may provide pre-stored training to the user 10 through the terminal 100. In one embodiment of the disclosure of the disclosure, the training may be training for training at least one of breathing, vocalization, articulation, resonance, and prosody. The training is visualized and provided to the user 10. The user 10 may perform training through the terminal 100 or by articulating in response to the training provided by the terminal 100. The articulation of the user 10 may be transmitted to the server 200 in the form of voice data. The training will be described in detail in later part of the disclosure.
The server 200 analyzes the voice data of the user 10 and obtain at least one of, for example, a loudness (decibel), a pitch, a pronunciation accuracy, a sound length, a pitch change, a breath hold, a beat, or a reading speed of the user 10. A method of analyzing the voice data of the user 10 will be described in detail in later part of the disclosure.
The server 200 may provide feedback to the user 10 by using the result of analyzing the user 10 ’s voice data. In one embodiment of the disclosure, the server 200 may provide feedback to the user 10 in real time. For example, the server 200 may provide the user 10 with visualization of the state of at least one of the user 10 ‘s loudness (decibel), pitch, pronunciation accuracy, sound length, pitch change, breath hold, beat, or reading speed in real time. Feedback provided by the server 200 to the user 10 will be described in detail in later part of the disclosure. The server 200 may measure the user’s language level based on the analysis result. The server 200 may provide feedback to the user based on the user’s language level.
In one embodiment of the disclosure of the disclosure, the language level may be determined differently according to the user’s pitch or the user’s loudness. For example, when the user’s sound level or sound level is within a selected range, the language level may be set to normal. When the user’s loudness or pitch does not belong to the selected range, the language level may be set to a non-normal value.
The server 200 may provide the user 10 ’s voice data analysis result to the medical staff 20. The medical staff 20 may provide the diagnosis or opinion of the medical staff 20 to the server 200 based on the voice data analysis result. The server 200 may generate feedback to be provided to the user based on the diagnosis or opinion of the medical staff 20. The server 200 may provide the user 10 with a diagnosis or opinion of the medical staff 20 or feedback generated based thereon.
Training to improve dysarthria can be performed by the user 10 performing dysarthria training by vocalizing or articulating according to the training provided through the terminal 100 and visualized feedback on training for dysarthria is checked and the user 10 ’s vocalizations, articulations, etc. are controlled, all in real time.
A device for providing a method for improving dysarthria may include a server 200. The server 200 includes a communication module 210, a memory 220, a training unit 230, a feedback providing unit 240, and an analysis unit 250.
The communication module 210 may be configured to receive an input of the user 10, such as vocalization and articulation of the user 10, and to provide training and feedback to the user 10 from the server 200. Information input by the user 10 into the terminal 100 (e.g., vocalization and articulation of the user 10, feedback, etc.) may be transmitted to the server 200 through the communication module 21. The communication module 210 may receive voice data such as the user 10 ’s vocalization and articulation in real time. The voice data received in real time may be analyzed by the analysis unit 250.
For example, the communication method of the communication module 210 may use a network constructed according to standards including GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long) Term Evolution), LTE-A (Long Term Evolution-Advanced), etc.), WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), and 5G but is not limited thereto, and may include all transmission method standards to be developed in the future. It may include anything that can send and receive data through wired/wireless. Through the communication module 210, the script stored in the memory, visual information corresponding to the script, etc. may be updated.
The memory 220 is configured to store instructions that are executed by a processor (not shown). The memory 220 may be configured to store training, feedback, and analysis results provided by each of the training unit 230, the feedback providing unit 240, and the analysis unit 250.
In one embodiment of the disclosure, the memory 220 may include a computer-readable storage medium such as a data storage device that can be accessed by a computing device and provides persistent storage of data and executable instructions (e.g., software applications, programs, functions, etc.). Examples of the memory 220 include volatile and non-volatile memory, fixed and removable media devices, and any suitable memory device or electronic data store that maintains data for computing device access. The memory 220 may include various implementations of random-access memory (RAM), read-only memory (ROM), flash memory, and other types of storage media in various memory device configurations. The memory 220 may be configured to store executable software instructions (e.g., computer-executable instructions) executable with a processor or the same software application which may be implemented as a module.
In one embodiment of the disclosure, the training unit 230, the feedback providing unit 240, and the analysis unit 250 may be implemented by a processor and executable software instructions executable together with a processor stored in the memory 220. For example, the memory 220 may store instructions for performing the functions of the training unit 230, the feedback providing unit 240, and the analysis unit 250.
Training unit 230 may be configured to provide training to user 10. Training is an exercise to improve dysarthria, and may include at least one of non-verbal oral exercises, extended vocalization / loudness increase, pitch change training, resonance (velopharyngeal closure sound) training, syllable repetition training, and reading training. The training provided by the training unit 230 may be pre-stored in the memory 220.
In one embodiment of the disclosure, non-verbal oral exercises include exercises for strengthening the articulatory organs involved in speech production. For example, training for non-verbal oral exercise may provide an image guide for lip exercise, cheek-blowing exercise, and tongue exercise.
In one embodiment of the disclosure, the lip exercise may include a lip pulling exercise, a lip plucking exercise, and a lip pulling and plucking exercise. For example, a lip movement may include exercises to hold the lips in a “e” shape for 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, etc., hold the lips in a “o” shape for 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, etc., or repeating the lips between “e, o” shape 2 times, 3 times, 4 times, 5 times, etc. Ball inflating may include an exercise of inflating any one of both cheeks, the right cheek, and the left cheek, and maintaining it for a predetermined time, for example, 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, etc. The tongue exercise may include tongue sticking out, tongue raising, pushing the cheek with the tongue, moving the tongue side to side, moving the tongue following the shape of the lips, etc.
In one embodiment of the disclosure, the extended vocalization / loudness increase training includes extended vocalization, loudness reinforcement training for improving speech intelligibility. For example, the extended vocalization / loudness increase training may provide a suggested vocabulary and may provide training for the user 10 to follow the suggested vocabulary with a constant sound according to the target speech time and loudness.
In one embodiment of the disclosure, the suggested vocabulary may be provided in the form of a combination of a consonant and a vowel. For extended vocalization / loudness increase training, a target (e.g., loudness, vocalization time, etc.) may be set based on previous training contents. A goal may be provided to the user 10. Real-time analysis of extended vocalization / loudness increase training may be provided by the analysis unit 250 and the feedback providing unit 240 based on the user’s vocalization. The extended vocalization / loudness increase training may be training to identify the training result through the loudness, length, and pitch of the sound.
In one embodiment of the disclosure, the pitch change training includes training to improve the prosody and intelligibility of speech. The pitch-changing training includes a training that provides an increase in pitch, e.g., Do, Re, and Mi, or a descending pitch, e.g., Mi, Re, and Do, and training that verifies whether the user 10 changes the pitch in long and large manner. If the notes do not match, feedback can be provided to the user 10.
In one embodiment of the disclosure, the resonance training includes training to build the strength of the muscles that close the oropharynx (wind passage). For example, it includes a training that confirm that the user 10 makes a specific sound, e.g., “AK” with accurate pronunciation and hold a breath for a predetermined time, e.g., 1 second, 3 seconds, 5 seconds, 7 seconds, etc., while the back of the tongue is in contact with the uvula. This may be a training exercise for evaluating whether the user 10 makes a first sound and maintains the back of the tongue in a state in which the oropharynx is blocked with it for a certain period of time.
In one embodiment of the disclosure, the syllable repetition training may include training to loosen the muscles of the lips and tongue, improving modulation and intelligibility. For example, it includes training to repeat vocalizations of syllables made up of plosives, such as one, two, three, etc. syllables, in sync with a beat. For example, syllable repetition exercises can be provided at different rates. For example, the rate at which a syllable is presented may increase or decrease. The syllable repetition training may be a training to determine whether the suggested vocabulary is consistently pronounced. The syllable repetition training may be training to determine the loudness of the sound and whether it is repeated at a constant rate.
In one embodiment of the disclosure, the reading training may include training to improve speech intelligibility. For example, the reading training is a training in which a sentence or paragraph is provided, and the user 10 reads it in parts. This includes training in which sentences, paragraphs, etc. are presented to the user 10, and the user 10 reads it aloud several times in time in sync with the beat.
For vocabulary reading training, vocabularies of multiple syllables may be provided. A one-syllable vocabulary may be provided as a suggestive vocabulary with a beginning/final sound e.g., of Korean. A two- or three- syllable vocabulary may be provided as a suggestive vocabulary that includes a beginning, middle, and final sound e.g., of Korean. In this case, vocabularies due to phonological fluctuations may be excluded.
A detailed description of the training will be described later part of the disclosure in conjunction with the drawings.
The feedback providing unit 240 provides feedback to the user 10. In one embodiment of the disclosure, the feedback providing unit 240 may provide feedback to the user 10 in real time based on the analysis result of the voice data of the user 10 received in real time. Feedback may include a visualized image. The feedback may be configured to inform the user 10 whether the user 10 is performing well in the training. For example, the feedback may be an image or a text, configured to inform the user 10, comprising at least one of loudness, pitch, sound length, pitch change, breath holding time, time signature, reading speed, etc. of the user 10 ’s voice. A detailed description of the feedback is provided later part of the disclosure in conjunction with the drawings.
Analysis unit 250 is configured to analyze the voice data of the user 10 received by the server 200 in real time. The analysis unit 250 may measure a loudness (e.g., decibels) and a pitch (pitch) of the user 10 ’s voice based on the user 10 ’s voice data.
In one embodiment of the disclosure, the loudness of the user 10′s may be obtained using a signal-to-noise ratio (SNR). SNR refers to the ratio indicating how loud the voice is compared to the noise. A large SNR value means that the voice is larger than the noise, and 0 decibel can be construed that the voice and the noise are the same. For example, the intensity may be obtained using the root mean square (RMS) of the amplitude value in a part of the streaming voice. The SNR is calculated by 20*log to the intensity. And, according to the surrounding environment, a method of adding or subtracting a correction value to the SNR value is used to set the zero point. Since a method of obtaining the decibel magnitude using SNR is known in the prior art, further detailed description thereof will be omitted.
In one embodiment of the present disclosure, the pitch may be obtained through a change according to the frequency of the voice. For example, the frequency is calculated by obtaining the spectral data of an incoming voice. Spectral data can be obtained by converting speech data into a spectrogram. Spectrogram is an analysis method that is the basis of speech signal processing, which divides a continuously given speech signal into pieces of a certain length and then applies a Fourier transform to the pieces, and is a two-dimensional figure with its horizontal axis representing the time information of the piece and its vertical axis representing the size of the frequency component in decibel units. From the spectrogram, it is possible to obtain a pitch frequency indicating the height of a voice signal and a formant frequency in which frequency components are concentrated for each phoneme.
In order to reduce the leakage of the frequency band when sampling the spectral data, the Blackman-Harris type window of the Fast Fourier Transform (FFT) algorithm can be used. The frequency is obtained by normalizing the speech spectrum data. Normalizing includes obtaining maximum/minimum values of sampled data and selecting non-exciting values using a difference therebetween. Since this method is known in the prior art, further detailed description thereof will be omitted.
In one embodiment of the disclosure, the speech spectrum data may be analyzed using formants. Formant analysis can be used to measure pronunciation accuracy, similarity, and pitch change. Through formant analysis, specific frequencies for vowels and consonants can be known and can be used for evaluation with reference to them.
Analysis unit 250, based on the decibels and pitch, may obtain the user 10 ’s voice loudness, sound length, pitch change, breath hold, beat, etc. For example, based on the decibels, it is possible to acquire loudness, length of the sound, breath hold, beat, and change in the pitch based on the pitch of the user 10 ’s voice. In addition, the analysis unit 250 may be configured to obtain pronunciation accuracy using a Speech-to-Text or an artificial intelligence. In addition, the analysis unit 250 may obtain a reading speed of the user 10 by comparing the length of the suggested vocabulary or sentence spoken by the user 10 to a length of an exemplarily recorded suggested vocabulary and sentence.
In one embodiment of the disclosure, the analysis unit 250 may obtain the loudness, sound length, pitch, sound length, pitch change, pronunciation accuracy, breath holding time, beat accuracy, and reading speed using the following methods.
The loudness is obtained by checking whether the loudness is maintained greater than or equal to the threshold using the measured decibel value. The threshold for each step of training can be adjusted to ensure that the loudness is greater than or equal to the selected level. For example, it is evaluated by calculating the probability (%) of the number of times that the size is greater than or equal to the threshold by checking whether the size is greater than or equal to the threshold for a predetermined period of time for each training stage. It will be understood by those skilled in the art that the threshold is a selected value and can be set properly. The probability (%) of the number of times that the size is greater than or equal to the threshold can be used to determine the user’s language level. For example, if the probability is greater than or equal to the selected value, it can be construed that the user’s language level is normal or the goal of normal or training has been achieved.
Sound length can be evaluated using whether the sound is interrupted. For example, the sound length is obtained by using the measured decibel value and checking whether it is maintained at a level above the threshold for a certain period of time. The amount of time to be maintained for each step may vary. For example, it may be preset to step 1 (3 seconds), step 2 (5 seconds), step 3 (10 seconds), and step 4 (15 seconds). The analysis unit 250 may determine that there is a sound interruption when step 1, e.g., 3 seconds, is not maintained. If there is no sound interruption during step 1, the difficulty can be changed to step 2 in the next training. It will be understood that the time to be maintained at each step is optionally variable.
The pitch of a sound can be obtained by checking whether or not it occurs with a constant pitch. For example, the measured pitch value should be used to keep the pitch value within a threshold range. The pitch is evaluated by calculating the probability (%) of the number of times that it does not deviate from the threshold range by checking it a predetermined number of times during a predetermined time. Alternatively, the pitch can be obtained by checking that the measured pitch value and formant value are maintained for a time selected for each pitch, e.g., 1 second, 2 seconds, 3 seconds, 4 seconds. It can be evaluated by calculating the probability (%) of the number of times that the pitch value and the formant value for each pitch are maintained by checking them for a predetermined number of times during the selected time. Loudness measurement for resonance practice can be evaluated as a score when the decibel value is greater than or equal to the decibel value of a predetermined size using the average value of the decibel values measured when the first and second vocabularies of the suggested vocabulary are pronounced. For example, it can be evaluated by dividing the average decibel value into 0, not more than 20 dB, or 20, 35, 50, or 65 dB or more. The probability (%) of the number of times that the pitch value and the formant value for each pitch are maintained by checking them for a predetermined number of times during the selected time can be used to determine the user’s language level. For example, if the probability is greater than or equal to the selected value, it may be determined that the user’s language level is normal or that the goal of training has been achieved.
The sound length may be obtained based on whether the sound length is maintained for a certain period of time at a level greater than or equal to a threshold using the measured decibel value. Regarding each pitch, a selected amount of time greater than or equal to a threshold, e.g., 1 second, 2 seconds, 3 seconds, 4 seconds, and 5 seconds should be maintained. It is evaluated by calculating the probability (%) of the number of times that is maintained within the selected time, e.g., 1 second, 2 seconds, 3 seconds, 4 seconds, and 5 seconds. The probability of the maintained number of times can be used to determine the language level of the user. For example, if the probability is greater than or equal to the selected value, it may be determined that the user’s language level is normal or that the goal of training has been achieved.
Pronunciation accuracy is evaluated according to the accuracy by pronouncing a plurality of vocabularies (each consisting of a plurality of syllables). For example, 3 vocabularies (6 syllables) are pronounced and evaluated according to their accuracy. The pronunciation is evaluated according to a number of correct number of syllables out of 1 syllable, 2 syllables, 3 syllables, 4 syllables, 5 syllables or more. It is possible to check for the correct syllable by comparing the suggest vocabulary to formants. The number of correct answers can be used to determine the user’s language level. For example, when the number of correct answers is equal to or greater than a selected value, it may be determined that the user’s language level is normal or that the target of training has been achieved.
The breath hold time is evaluated by checking a case in which the decibel value measured between the pronunciation of the first vocabulary of the presented vocabulary and the pronunciation of the second vocabulary after the selected time, for example, 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, 8 seconds, 9 seconds, 10 seconds is greater than or equal to a threshold. It is evaluated by calculating the average of the number of suggested vocabularies by checking the case in which the breath holding time for which the magnitude less than the threshold is measured is longer than or equal to the selected time, for example 0, 1, 2, 3, 4, 5 seconds. For example, if 10 suggested vocabularies were practiced and the second vocabulary was pronounced 4 times in decibels greater than or equal to the threshold after the selected time, the training can be evaluated as 4 out of 5 points. The score can be used to determine the user’s language level. For example, if the score is greater than or equal to the selected value, it may be determined that the user’s language level is normal or that the goal of training has been achieved.
In one embodiment of the disclosure, the analysis of sentence and vocabulary reading training may be as follows. One example is to perform text similarity measurement (cosine similarity algorithm, Ravenstein’s distance algorithm, etc.) by comparing a speech file with the original text after text conversion (STT) using speech recognition, and another example is to use a recorded voice file to measure the pronunciation accuracy using Deep learning and includes a method of collecting data of correct and incorrect pronunciations of vocabularies, sentences, and paragraphs presented in exercises, and using each of the data for modeling then learned by Deep learning to measure the pronunciation accuracy.
Reading speed can be analyzed by comparing the total length of the recorded voice with the length of the presentation voice used for training.
Referring to
The apparatus 300 may include a communication module 310, a memory 320, an interface 325, a training unit 330, a feedback providing unit 340, and an analysis unit 350.
The communication module 310 is configured to be connected via wireless or wired to the apparatus 300 and an external device. The apparatus 300 may transmit information to or receive information from an external device (e.g., the server 200) through the communication module 310. In one embodiment of the disclosure, the information may be information to be provided to the medical staff 20 or information to be provided to the server 200, or information received from the medical staff 20 or information received from the server 200. The communication module 310 may be similar to or the same as the communication method of the communication module 210.
The memory 320, the training unit 330, the feedback providing unit 340, and the analysis unit 350 are substantially the same or similar to the memory 220, the training unit 230, the feedback providing unit 240, and the analysis unit 250, thus detailed descriptions thereof will be omitted.
Interface 325 is configured to receive voice information of the user 10, and provide training and feedback to the user 10. In one embodiment of the disclosure, the interface 325 may include at least one of all components that can communicate with the user 10, including a display, a touch screen, a microphone, a speaker, etc.
It will be understood by those of ordinary skill in the art that, in one embodiment of the disclosure, the function of a portion of at least any one of the memory 320, the training unit 330, the feedback unit 340, and the analysis unit 350 of the apparatus 300 can be implemented using the server 200′s memory 220, training unit 230, feedback unit 240, and analysis unit 250.
A method of providing a method for improving dysarthria may be provided to the user 10 through the terminal 100. In one embodiment of the disclosure, as shown in
In step 410, the server 200 provides training to the user 10. In one embodiment of the disclosure, the training unit 330 may provide training to the user 10, or a processor may be combined with the memory 320 to provide training to the user 10. The training is a training to improve dysarthria, and may include at least one of non-verbal oral exercise, extended vocalization / increase in loudness, pitch change training, resonance (oropharynx closure sound) training, syllable repetition training, and reading training.
In one embodiment of the disclosure, training may be provided based on the user 10 ’s existing training results. The status and existing training results of the user are stored in the memory 220 of the server 200. The training unit 230 may provide suitable training to the user 10 based on the state of the user 10 and an existing training result. For example, in the case of sound length training, the time to maintain the breath maintained for each stage is set differently, and the next stage of training can be provided after confirming that the previous stage has been passed. In the case of training including a plurality of steps, the training unit 230 may provide training of the next step after confirming that each step has been passed.
In response to the provided training, the user 10 generates sounds corresponding to voice data such as vocalization, articulation. In step 420, the server 200 receives the voice data of the user 10. The server 200 may receive the voice data of the user 10 through the communication module 210. The server 200 may receive voice data of the user 10 corresponding to the training in real time. In step 430, the analysis unit 350 analyzes the user’s voice data. In one embodiment of the disclosure, the analysis unit 250 may measure the loudness (e.g., decibels) and the pitch of the user 10 ’s voice based on the user 10 ’s voice data. The analysis unit 250 may acquire at least one of a loudness, a sound length, a pitch change, a breath hold, and a time signature of the user 10. The analysis unit 250 may obtain a loudness, a sound length, and a pitch for increasing the loudness of the extended vocalization. The analysis unit 250 may obtain a sound length and a pitch change for pitch change training. The analysis unit 250 may acquire pronunciation accuracy, breath holding time, and loudness for resonance practice. The analysis unit 250 may acquire pronunciation accuracy, beat accuracy, and loudness for syllable repetition practice. The analysis unit 250 may acquire pronunciation accuracy, reading speed, and loudness for training to read a vocabulary (e.g., 1, 2, 3 syllables). The analysis unit 250 may acquire pronunciation accuracy, reading speed, and loudness for training to read sentences and vocabulary with three or more word segments. The loudness, sound length, sound pitch, pitch change, pronunciation accuracy, breath holding time, beat accuracy, reading speed, etc. obtained by the analysis unit 250 are as described above, and thus detailed description thereof will be omitted.
In step 440, the feedback providing unit 240 generates feedback based on the voice data of the user 10 and the analysis result. The feedback may include a visualized image to inform the user 10 of the state of the user 10 ’s vocalization or articulation. Feedback may be provided based on the language level of the user 10. In step 450, the feedback providing unit 250 provides feedback to the user 10. The feedback providing unit 250 may provide feedback to the user 10 in real time. For example, the feedback providing unit 250 may be configured to maintain or notify the user 10 of a change in at least one of loudness, pitch, sound length, pitch change, pronunciation accuracy, breath holding time, beat accuracy, and reading speed of the user 10 ’s voice. Although not shown, the server 200 may store an analysis result of the voice data of the user 10. The analysis result may include a result performed by the user 10 in response to training. The analysis results may be referenced by the training unit 230 when providing the next training.
In one embodiment of the disclosure, the server 200 may be configured to correspond the user 10 ’s personal data with the user 10 ’s training content, analysis of the training, and feedback and store them on the memory 220. Accordingly, it is possible to provide personalized training, analysis, and feedback for each user 10. In one embodiment of disclosure, customized training may be provided by analyzing the part that the user 10 have deficiencies. For example, as a result of the analysis, training with a lower score or evaluation may be given as a top priority. The score or evaluation may be a score or evaluation that the user 10 inputs by oneself after each training, or it may be a score or evaluation evaluated by the server 200 according to a pre-stored criterion. For example, customized training may be provided based on the scores, or evaluations shown in
Referring to
Referring to
In one embodiment of the disclosure, at the time of the user 10 ’s vocalization, the agent 610 may advance (e.g., move towards the right side of the screen) in response to determining that the loudness is greater than or equal to a threshold. In response to determining that the loudness is less than the threshold, the agent 610 may move backward (e.g., move towards the left side of the screen). In response to determining that the sound level is greater than a threshold, the agent 610 may rise upwards on the screen, and the agent 610 may descend downward in response to determining that the sound level is less than a threshold. In one embodiment of the disclosure, at the time of the user 10 ’s vocalization, the agent 610 may move towards the object 620 in response to determining that the loudness is greater than or equal to a threshold. Here, the direction in which the agent 610 faces the object 620 may be referred to as a first direction. At the time of the user 10 ’s vocalization, the agent 610 may move in a direction opposite (or away from) the object 620 in response to determining that the loudness is less than the threshold. Here, a direction in which the agent 610 moves away from the object 620 or a direction opposite to the first direction may be referred to as a second direction. In response to determining that the pitch is greater than a threshold, the agent 610 rises to upward direction of the object 620, and in response to determining that the pitch is less than a threshold, the agent 610 descends in the downward direction of the object 620.
The method of measuring the loudness and pitch of the user 10’s voice has been described above, and detailed description thereof will be omitted. Accordingly, the server 200 can measure the loudness and pitch of the user 10′s voice in real time, and, by visualizing it based on the loudness and the pitch, provide feedback to the user 10 in real time through moving the agent 610.
Referring to
Referring to
Referring to
As described in
Referring to
Referring to
The human neck structure image 820 includes a visualized image for guiding the oropharyngeal closure, and the guide text 830 may provide the user 10 with a guide for training. The user 10 may perform training with reference to the agent image 810, the human neck structure image 820, and the guide text 830. In one embodiment of the disclosure, the vocabulary provided on the agent screen 810 may be a two-syllable vocabulary, and may consist of a vocabulary in which the back of the tongue touches the uvula of the user 10 when the first syllable is vocalized.
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
In one embodiment of the disclosure, the personal information of the user 10 and the training result of the user 10 may be stored in the server 200. Therefore, it is possible to provide customized training according to the previous training results for each user 10.
The apparatus and method described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may be implemented using one or more general purpose computers or special purpose computers, for example, a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable array (FPA), programmable logic unit (PLU), microprocessor, or a certain other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. Although, for the convenience of understanding, there are instances where one processing device is described as being used, a person of ordinary skill in the art will recognize that a processing device may include a plurality of processing elements and/or a plurality of types of processing elements. For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.
Software may include a computer program, code, instructions, or a combination of one or more of these, and configure a processing unit to behave as desired, or independently or collectively give instructions to the processing unit. The software and/or data may be permanently or temporarily embodied on a certain machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by or to provide instructions or data to the processor. The software may be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored in one or more computer-readable recording media.
The described embodiments of the present disclosure also allow certain tasks to be performed on a distributing computing environment performed by remote processing devices that are linked through a communications network. In the distributed computing environment, program modules may be located in both local and remote memory storage devices.
As described above, although the embodiments have been described with reference to the limited drawings, those of ordinary skill in the art may apply various technical modifications and variations to the above, based on them. Appropriate results can be achieved when, for example, the described techniques are performed in an order different from the described method, and/or the described components of a system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components or an equivalent may be substituted or exchanged to achieve an appropriate result.
Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2022-0010219 | Jan 2022 | KR | national |
