Patent Application
20240096347
Embodiments of the present disclosure relate to speech technology and, in particular, to a method and an apparatus for determining speech similarity, and a program product.
Many users choose to learn language online, for example, using language learning software to learn a certain language.
Many language learning software in the related art is provided with an analysis module functioned with similarity analysis of follow-up reading; and a user may read specified content, and software can analyze audio generated when the user reads the specified content, and determine similarity between the audio and a standard audio corresponding to the specified content, so that the user can get knowledge of the effect of follow-up reading.
However, the analysis module provided in the related art only can analyze one type of language typically, and adding similarity analysis function of another type of language for follow-up reading will result in a relatively large volume of the analysis module and relatively high requirements for a hardware device running the analysis module.
Embodiments of the present disclosure provide a method and an apparatus for determining speech similarity, a program product, for overcoming the problem in the related art of a large volume of a module which is functioned with similarity analysis of follow-up reading.
In a first aspect, an embodiment of the present disclosure provides a method for determining speech similarity based on speech interaction, including:
In a second aspect, an embodiment of the present disclosure provides a method for processing a data request instruction, applied to a server, and including:
In a third aspect, an embodiment of the present disclosure provides an apparatus for determining speech similarity, including:
In a fourth aspect, an embodiment of the present disclosure provides an apparatus for processing a data request instruction, disposed in a server, and including:
In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including:
In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having, stored thereon, a computer program, where the computer program is executed by a processor to implement the method for determining speech similarity based on speech interaction as described above in the first aspect or the method for processing a data request instruction as described above in the second aspect.
In a seventh aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, where the computer program, when being executed by a processor, implements the method for determining speech similarity based on speech interaction as described above in the first aspect or the method for processing a data request instruction as described above in the second aspect.
In an eighth aspect, an embodiment of the present disclosure provides a computer program, where the computer program, when being executed by a processor, implements the method for determining speech similarity based on speech interaction as described above in the first aspect or the method for processing a data request instruction as described above in the second aspect.
The embodiments provide a method and an apparatus for determining speech similarity, and a program product. The method includes: playing exemplary audio, and acquiring evaluation audio of a user, where the exemplary audio is audio of specified content that is read by using a specified language; acquiring a standard pronunciation feature corresponding to the exemplary audio, and extracting, from the evaluation audio, an evaluation pronunciation feature corresponding to the standard pronunciation feature, where the standard pronunciation feature is used to reflect a specific pronunciation of the specified content in the specified language; and determining a feature difference between the standard pronunciation feature and the evaluation pronunciation feature, and determining similarity between the evaluation audio and the exemplary audio according to the feature difference. In the scheme of the present application, the evaluation pronunciation feature corresponding to the standard pronunciation feature corresponding to the exemplary audio can be extracted from the evaluation audio, thereby achieving relatively small volume of a module functioned with similarity analysis of follow-up reading. Moreover, the standard pronunciation feature of the exemplary audio can reflect the specific pronunciation of the specified content in the specified language, so that the present scheme is able to provide a similarity analysis function for follow-up reading in plural categories of languages in case that there is a small computing volume.
In order to describe technical solutions in embodiments of the present disclosure or the related art more clearly, the accompanying drawings used in the description of the embodiments or the related art will be briefly described hereunder. Apparently, the drawings in the following description are intended for some embodiments of present disclosure. For persons of ordinary skill in the art, other drawings may be obtained based on these drawings without any creative effort.
In order to make objectives, technical solutions, and advantages of embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described hereunder clearly and comprehensively with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are only some embodiments of the present disclosure, rather than all the embodiments. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present disclosure without any creative effort shall fall within the protection scope of the present disclosure.
As shown in
The user can click a key 11 to control the user terminal to record follow-up audio 12. The user terminal can analyze the recorded audio 12 to determine similarity thereof with the exemplary audio, so that the user can get knowledge of the effect of follow-up reading.
However, in the scheme provided in the related art where an audio recorded during follow-up reading is analyzed to determine similarity, only audio in one type of language can be analyzed. For example, similarity analysis can be performed on only audio generated when the user performs follow-up reading by using Mandarin; and for another example, similarity analysis can be performed on only audio generated when the user performs follow-up reading by using English.
If, on the basis of the scheme provided in the related art, a similarity analysis function for follow-up reading in other categories of languages is directly added, volume of an analysis module achieving the entire function will be large, and thus relatively high requirements will be placed on a hardware device running the analysis module.
For example, if there is a need to analyze audio recorded in different dialects for follow-up reading to determine similarity thereof with the exemplary audio, volume of the analysis module will be relatively large.
In order to address the aforementioned technical problem, according to the scheme provided in the present application, when analyzing a recorded evaluation audio, only an evaluation pronunciation feature corresponding to a standard pronunciation feature corresponding to an exemplary audio is extracted from the evaluation audio, so that the volume of the module functioned with similarity analysis of follow-up reading can be relatively small. Further, the exemplary audio is audio of specified content that is read in a specified language, and the standard pronunciation feature is used to reflect a specific pronunciation of the specified content in the specified language, therefore, according to the scheme of the present application, similarity between the evaluation audio and the exemplary audio can be determined according to the standard pronunciation feature and the extracted evaluation pronunciation feature, moreover, such implementation can be applied to exemplary audio with different specified contents and different specified languages, thereby capable of providing a similarity analysis function for follow-up reading in plural categories of languages.
As shown in
Step 201, playing exemplary audio, and acquiring evaluation audio of a user, where the exemplary audio is audio of specified content that is read by using a specified language.
The method provided in the present application can be performed by an electronic device with computing power, and the electronic device may be, for example, a user terminal which may have a microphone. The user terminal may be, for example, a mobile phone, a tablet computer, or other device.
Specifically, the user terminal can play an exemplary audio, where the exemplary audio is audio of specified content that is read by using a specified language. For example, a section of literal content can be preset; and the literal content can be set as desired, which can be, for example, “Happy New Year”. The audio of such content that is read in a specified language can be pre-recorded. For example, audio of such content that is read in Cantonese can be pre-recorded. The language particularly used can also be set as desired.
Further, the exemplary audio is audio which is used to provide reference to the user, and hence it is also possible to pre-record pieces of reference audio for specified content that is read in a specified language, and then screen exemplary audio meeting the requirement therefrom. For example, reference audio of “Happy New Year” that is read in Cantonese can be recorded under different circumstances using different devices.
In practical application, after the user terminal finishes playing the exemplary audio, a microphone can also be turned on so as to acquire evaluation audio of the user.
In one embodiment, a key for triggering acquisition of the evaluation audio can be provided in an interface of the user terminal, and the user can click the key to trigger the user terminal to turn on the microphone and acquire the evaluation audio.
In another implementation, the user terminal may turn on the microphone after finishing playing the exemplary audio, and acquire the evaluation audio.
The user can perform follow-up reading after listening to the exemplary audio, and particularly a specified language may be used to read specified content, so that the user terminal can acquire the evaluation audio generated when the user performs follow-up reading for the exemplary audio.
In an optional implementation, the user may also operate the user terminal, and transmit a follow-up reading completion instruction to the user terminal. For example, a key indicative of follow-up reading completion can be displayed in the interface of the user terminal, and the user can click the key to transmit the follow-up reading completion instruction. In another implementation, the user can long press a preset key when performing follow-up reading, and release the key for the follow-up reading after the follow-up reading is completed, so that the follow-up reading completion instruction is transmitted to the user terminal.
In an implementation, the user terminal, when acquiring the evaluation audio, can also perform a detection thereto, so as to determine whether the user has finished the follow-up reading. For example, it can determined whether the user is still performing follow-up reading according to an energy value of the audio.
Step 202, acquiring a standard pronunciation feature corresponding to the exemplary audio, and extracting, from the evaluation audio, an evaluation pronunciation feature corresponding to the standard pronunciation feature, where the standard pronunciation feature is used to reflect a specific pronunciation of the specified content in the specified language.
Specifically, the user terminal can acquire a standard pronunciation feature corresponding to the exemplary audio.
Further, the standard pronunciation feature may be transmitted by a server to the user terminal, and the user terminal can store received standard pronunciation feature, and acquire the standard pronunciation feature when analyzing the evaluation audio. For example, when the user operates the user terminal to enable an application set with the method provided in the present application, the user terminal can interact with the server and request the server to transmit the exemplary audio and the standard pronunciation feature corresponding thereto to the user terminal.
In practical application, different pieces of exemplary audio also correspond to different standard pronunciation features, and the user terminal, when acquiring a standard pronunciation feature, can acquire the corresponding standard pronunciation feature according to an exemplary audio being played.
The standard pronunciation feature can be preset according to the specified content and the specified language, which can reflect a specific pronunciation of the specified content in the specified language.
Specifically, pieces of reference audio may be pre-recorded for the specified content that is read in the specified language, and any piece of the reference audio is used as the exemplary audio. Reference pronunciation features can be extracted for the each piece of reference, and the reference pronunciation features can characterize pronunciation features when the specified content is read in the specified language, and then these reference pronunciation features are fused to obtain the standard pronunciation feature corresponding to the exemplary audio, and the standard pronunciation feature has, fused therein, multiple reference pronunciation features, therefore, the standard pronunciation feature can characterize the specific pronunciation of the specified content in the specified language.
Further, the user terminal can also extract, from the evaluation audio, an evaluation pronunciation feature corresponding to the standard pronunciation feature. In this implementation, the evaluation pronunciation feature corresponding to the standard pronunciation feature can be extracted from the evaluation audio in a targeted way, without extracting all the features of the evaluation audio, thus the amount of data that needs to be processed can be reduced, and hardware requirements required when analyzing the evaluation audio can be lowered.
In an implementation, when extracting reference pronunciation features of each piece of reference audio, features of preset sampling points in each piece of reference audio can be collected, so that the standard pronunciation feature includes the features of these preset sampling points. When specified content is read in a specified language, the preset sampling points in the reference audio can be determined based on pronunciation positions characterized in that language.
In this implementation, the evaluation pronunciation feature corresponding to the standard pronunciation feature can be extracted from the evaluation audio according to the positions of the preset sampling points corresponding to the exemplary audio.
In another implementation, when extracting reference pronunciation features of each piece of reference audio, a preset category of features in each piece of reference audio can be collected, so that the standard pronunciation feature includes the preset category of features. When specified content is read in a specified language, the preset category of features in the reference audio can be determined based on features characterized in that language, which for example may be features representing a change of the overall tone, and which for another example may be features representing pronunciations of the entire text or part of the text.
In this implementation, the evaluation pronunciation feature corresponding to the standard pronunciation feature can be extracted from the evaluation audio according to the preset category of features corresponding to the exemplary audio.
Step 203, determining a feature difference between the standard pronunciation feature and the evaluation pronunciation feature, and determining similarity between the evaluation audio and the exemplary audio according to the feature difference.
The user terminal can also compare the standard pronunciation feature with the evaluation pronunciation feature, and then determine a feature difference between the standard pronunciation feature and the evaluation pronunciation feature. For example, a comparison can be made between a respective feature in the standard pronunciation feature and a respective feature in the evaluation pronunciation feature, and then the feature difference between the standard pronunciation feature and the evaluation pronunciation feature can be obtained.
For example, alignment processing can be performed between the standard pronunciation feature and the evaluation pronunciation feature, and the feature difference between the standard pronunciation feature and the evaluation pronunciation feature can be obtained through a comparison between a first feature in the standard pronunciation feature and a second feature in the evaluation pronunciation feature that are included in each alignment point.
Specifically, also, similarity between the evaluation audio and the exemplary audio can be determined according to the feature difference. For example, a feature distance between the standard pronunciation feature and the evaluation pronunciation feature can be determined according to the feature difference therebetween, and the feature distance can be used as the similarity between the evaluation audio and the exemplary audio.
In an optional implementation, the user terminal can also map the determined similarity to a score or evaluation content, and display the score, so that the user can get knowledge of the effect of follow-up reading. For example, a mapping relationship between similarity and a score or evaluation content can be set in advance, so that it is possible to determine the corresponding score or evaluation content according to the determined similarity.
The present application provides a method for determining speech similarity based on speech interaction, which includes: playing exemplary audio, and acquiring evaluation audio of a user, where the exemplary audio is audio of specified content that is read by using a specified language; acquiring a standard pronunciation feature corresponding to the exemplary audio, and extracting, from the evaluation audio, an evaluation pronunciation feature corresponding to the standard pronunciation feature, where the standard pronunciation feature is used to reflect a specific pronunciation of the specified content in the specified language; and determining a feature difference between the standard pronunciation feature and the evaluation pronunciation feature, and determining similarity between the evaluation audio and the exemplary audio according to the feature difference. In the scheme of the present application, the evaluation pronunciation feature corresponding to the standard pronunciation feature corresponding to the exemplary audio can be extracted from the evaluation audio, thereby achieving relatively small volume of a module functioned with similarity analysis of follow-up reading. Moreover, the standard pronunciation feature of the exemplary audio can reflect the specific pronunciation of the specified content in the specified language, so that the present scheme is able to provide a similarity analysis function for follow-up reading in plural categories of languages in case that there is a small computing volume.
As shown in
Step 301, transmitting, in response to a start instruction, a data request instruction to a server.
The method provided in the present application can be performed by an electronic device with computing power, and the electronic device may be, for example, a user terminal which may have a microphone. The user terminal may be, for example, a mobile phone, a tablet computer, or other device.
Specifically, a user may operate the user terminal and transmit a start instruction to the user terminal, where the start instruction is used to start a similarity analysis function for follow-up reading. For example, the similarity analysis function for follow-up reading can be provided in an application, as a prop in the application, and the application can be provided in the user terminal. Then, the user may operate the user terminal to start the application, and select the prop in the application with the similarity analysis function for follow-up reading, to transmit the start instruction to the user terminal.
Further, the user terminal can respond to the start instruction, and transmit a data request instruction to the server. The data request instruction is used to request data implementing the similarity analysis function for follow-up reading.
Step 302, receiving an encoder, exemplary audio, and a standard pronunciation feature corresponding to the exemplary audio.
In practical application, after receiving the data request instruction transmitted by the user terminal, the server can issue an encoder, an exemplary audio, and a standard pronunciation feature corresponding to the exemplary audio to the user terminal.
The server is pre-set with the encoder, the exemplary audio, and the standard pronunciation feature corresponding to the exemplary audio.
Specifically, the encoder transmitted by the server to the user terminal can be obtained through pre-training.
Further, speech recognition data can be used to train an initial model to obtain a speech recognition model. Then, the encoder in the speech recognition model is trained using pieces of audio in plural categories of languages, to obtain the encoder configured to extract a pronunciation feature.
In practical application, the speech recognition data can be audio data with text labels, and the speech recognition model obtained by training the speech recognition data can process a section of audio data, to obtain textual content corresponding to the audio data.
The speech recognition model includes an encoder, and the encoder can effectively extract text and pronunciation-related information, therefore, in the method according to the present application, the encoder capable of extracting a pronunciation feature can be obtained by training the encoder in the speech recognition model using audio data in plural categories of languages.
The audio data in plural categories of languages includes pieces of audio in plural categories of languages, and each piece of audio also has a language category label. For example, if a language used in a section of audio is Sichuan dialect, the language category label for the audio is a label representing Sichuan dialect. For another example, if a language used in a section of audio is Cantonese, the language category label for the audio is a label representing Cantonese. Training an encoder with audio data in plural categories of languages can improve differentiation by the encoder for pronunciation features in different categories of languages.
In order to further reduce hardware resources required by the method provided in the present application, the encoder may use a three-layer long short-term memory network, each layer of the network can be provided with 512 nodes.
Specifically, the server can be provided with several pieces of exemplary audio and standard feature information corresponding thereto, and transmit any one or more pieces of the pieces of exemplary audio to the user terminal, and issue the standard feature information corresponding to the exemplary audio.
Further, the standard pronunciation feature corresponding to the exemplary audio is obtained by fusing a plurality of reference pronunciation features, each reference pronunciation feature is obtained by using the encoder to extract a feature from each piece of reference audio, each piece of the reference audio is audio of the specified content that is read in the specified language, and the exemplary audio is any piece of audio of the reference audio.
In this implementation, multiple pieces of reference audio may be pre-recorded for the specified content that are read in the specified language. Then the encoder is used to perform feature extraction on each piece of reference audio to obtain a reference pronunciation feature corresponding thereto. Then respective reference pronunciation features are fused to obtain the standard pronunciation feature.
Since the reference audio is used to generate the standard pronunciation feature, the reference audio can be recorded by a user who use the specified language as a daily communication language, so that the standard pronunciation feature can accurately reflect the characteristic(s) of the specified language.
Step 303, playing the exemplary audio, and acquiring evaluation audio of a user, where the exemplary audio is audio of specified content that is read by using a specified language.
Specifically, after receiving exemplary audio, the user terminal can play any piece of the exemplary audio.
Step 303 is implemented using similar manners and principles as Step 201, which will not be described again.
In an implementation, the specified content corresponding to the exemplary audio can also be displayed in a user interface of the user terminal, which is used to prompt the user of contents that need for follow-up reading.
After finishing playing the exemplary audio, the user terminal can also play speech content for prompting the user to perform follow-up reading, e.g., “Please read after me”. In an implementation, after finishing playing the prompt content, the user terminal can also turn on a microphone to acquire evaluation audio of the user.
In an implementation, the user terminal can also be provided with a camera, so that a user image can be acquired and displayed in the user terminal. In an optional implementation, the user terminal can identify the user image, thereby determining whether the user has finished follow-up reading.
Step 304, acquiring the standard pronunciation feature corresponding to the exemplary audio, where the standard pronunciation feature is used to reflect a specific pronunciation of the specified content in the specified language.
Step 304 is implemented using similar manners and principles as Step 202 in terms of acquiring the standard pronunciation feature, which will not be described again.
Step 305, extracting, from the evaluation audio, an evaluation pronunciation feature corresponding to the standard pronunciation feature based on an encoder of a speech recognition model.
Further, the encoder of the speech recognition model transmitted by the server can be used by the user terminal, to extract, from the evaluation audio, an evaluation pronunciation feature corresponding to the standard pronunciation feature.
In practical application, the user terminal can input the evaluation audio into the encoder, to obtain the evaluation pronunciation feature corresponding to the standard pronunciation feature.
The encoder can distinguish pronunciation features in different categories of languages, and hence the encoder can extract, from the evaluation audio, the evaluation pronunciation feature corresponding to a language category. In addition, the standard pronunciation feature corresponding to the exemplary audio is also acquired using the encoder, therefore, by processing the evaluation audio using the same encoder, the evaluation pronunciation feature corresponding to the standard pronunciation feature can be obtained.
Specifically, the evaluation pronunciation feature corresponding to the standard pronunciation feature can be extracted by the encoder from the evaluation audio, and the evaluation pronunciation feature corresponding to the standard pronunciation feature can be extracted from the evaluation audio in a targeted way, without extracting all the features of the evaluation audio, thus the amount of data that needs to be processed can be reduced, and hardware requirements required when analyzing the evaluation audio can be lowered.
Step 306, determining a time wrapping function according to the standard pronunciation feature and the evaluation pronunciation feature.
Further, since audio is data with a timing sequence, the standard pronunciation feature corresponding to the exemplary audio also has a temporal attribute, and the evaluation pronunciation feature extracted from the evaluation data also has a temporal attribute. Therefore, a time wrapping function can be determined according to the standard pronunciation feature and the evaluation pronunciation feature, and the time wrapping function can represent a temporal correspondence between the standard pronunciation feature and the evaluation pronunciation feature.
In an implementation, a time wrapping function can be determined to non-linearly map a time axis of the evaluation pronunciation feature to a time axis of the standard pronunciation feature, so that the evaluation pronunciation feature and the standard pronunciation feature are aligned in terms of the time axes. The aligned standard pronunciation feature has a first feature corresponding to an alignment point, and the aligned evaluation pronunciation feature has a second feature corresponding to an alignment point. There is a certain feature difference between the first feature corresponding to each alignment point and the second feature corresponding to each alignment point, and the time wrapping function can fulfill a minimal sum of feature differences corresponding to respective alignment points.
In practical application, the time wrapping function meeting the above-described condition can be determined by the user terminal according to the standard pronunciation feature and the evaluation pronunciation feature.
Step 307, determining a plurality of combinations of alignment points according to the time wrapping function, the standard pronunciation feature and the evaluation pronunciation feature, where each combination of alignment points includes a standard feature point in the standard pronunciation feature and an evaluation feature point in the evaluation pronunciation feature.
After determining the time wrapping function, the user terminal can determine a plurality of combinations of alignment points based on the current time wrapping function, the standard pronunciation feature and the evaluation pronunciation feature, where each combination of alignment points includes a standard feature point in the standard pronunciation feature and an evaluation feature point in the evaluation pronunciation feature, and the standard feature point and the evaluation feature point in the combination of alignment points correspond to a same time point.
Step 308, determining, according to the standard feature point and the evaluation feature point included in each combination of alignment points, the feature difference corresponding to each combination of alignment points.
Specifically, for each combination of alignment points, the feature difference between the standard feature point and the evaluation feature point in the combination of alignment points can be determined. For example, a distance between the standard feature point and the evaluation feature point can be calculated as the feature difference of the combination of alignment points.
Step 309, determining the similarity between the evaluation audio and the exemplary audio according to the feature difference of each combination of alignment points.
Further, a sum of feature differences of respective combinations of alignment points may be used as the similarity between the evaluation audio and the exemplary audio. By means of feature point alignment followed by feature comparison, the feature difference between the evaluation audio and the exemplary audio can be accurately determined, thereby accurately determining the similarity therebetween.
As shown in
The user terminal inputs the evaluation audio 41 into an encoder 43, and the encoder 43 may output an evaluation pronunciation feature 44 of the evaluation audio 41 corresponding to the standard pronunciation feature 42. In an implementation, the evaluation audio 41 can be directly input into the encoder 43. In another implementation, the evaluation audio 41 also may be filtered first, and then the filtered audio is input into the encoder 43. For example, a Mel-filter bank can be used to process the evaluation audio 41.
The user terminal can also compare the standard pronunciation feature 42 with the evaluation pronunciation feature 44 to obtain similarity 45 between the evaluation audio 41 and the exemplary audio.
Step 310, acquiring a mapping function, and configuration information corresponding to the exemplary audio, where the configuration information is used to indicate a mapping relationship between a score and similarity which is between the evaluation audio and the exemplary audio.
Step 311, mapping the similarity between the evaluation audio and the exemplary audio to a score according to the mapping function and the configuration information corresponding to the exemplary audio.
In practical application, the user terminal can also acquire a mapping function, and configuration information corresponding to the exemplary audio.
In an optional implementation, the mapping function, and the configuration information corresponding to the exemplary audio may be issued by the server, so that the user terminal can acquire these pieces of information. For example, when transmitting the exemplary audio to the user terminal, the server can also issue the configuration information corresponding to the exemplary audio simultaneously, and can also issue the mapping function simultaneously.
The user terminal can store the received mapping function and the received configuration information corresponding to the exemplary audio, and acquire these pieces of information when mapping the similarity to a score.
If the server transmits pieces of exemplary audio to the user terminal, the server can also transmit configuration information corresponding to the each piece of exemplary audio to the user terminal.
Specifically, configuration information is used to indicate a mapping relationship between a score and similarity which is between the evaluation audio and the exemplary audio. For example, the configuration information can include several scores, and can also include a mapping relationship corresponding to the respective scores. Based on the configuration information, the mapping function can map the determined similarity to a score.
Further, the configuration information may also include a maximum score, similarity corresponding to the maximum score, a minimum score, and similarity corresponding to the minimum score. For example, the maximum score can be 100 and the minimum score can be 0.
In practical application, the mapping function may be a linear function, and the determined similarity can be mapped to a score through the linear function, the maximum score, the similarity corresponding to the maximum score, the minimum score and the similarity corresponding to the minimum score. By means of mapping similarity to a score through the linear function, the amount of data to be processed can be further reduced, thereby further lowering hardware requirements of the user terminal for performing the method provided in the present application.
The method provided in the present application is provided with configuration information corresponding to different pieces of exemplary audio, and the configuration information includes a maximum score and a minimum score. Maximum scores in respective pieces of configuration information may be set to a same value, such as 100; and minimum scores in respective pieces of configuration information may be set to a same value, such as 0, so that similarity can be mapped to a score range in a same scale according to the scheme provided in the present application.
Specifically, the user terminal can also display the determined score, so that the user can get knowledge of the effect of follow-up reading.
Further, the similarity corresponding to the maximum score is an average value of multiple pieces reference similarity, and each piece of reference similarity is similarity between each reference pronunciation feature and the standard pronunciation feature.
In practical application, a corresponding reference pronunciation feature can be extracted for each piece of reference audio, and the encoder can be used to extract the reference pronunciation feature of the reference audio. The reference similarity between each reference pronunciation feature and the standard pronunciation feature can be determined, for example, the reference similarity between each reference pronunciation feature and the standard pronunciation feature can be determined according to a dynamic time wrapping algorithm. An average value of the pieces of reference similarity is then determined as the similarity corresponding to the maximum score.
The similarity corresponding to the minimum score is an average value of multiple pieces of white noise similarity, each piece of white noise similarity is similarity between each white noise feature and the standard pronunciation feature, and each white noise feature is obtained by using the encoder to extract a feature from each piece of preset white noise audio.
Specifically, several pieces of white noise audio can also be prepared in advance to determine the similarity corresponding to the minimum score according to the several pieces of white noise audio. The encoder can be used to extract the white noise feature of each piece of white noise audio, and then the white noise similarity between each white noise feature and the standard pronunciation feature is determined, and then an average value of multiple pieces of white noise similarity is taken as the similarity corresponding to the minimum value.
As shown in
Step 501, receiving a data request instruction.
Step 502, transmitting, according to the data request instruction, an encoder based on a speech recognition model, an exemplary audio, and a standard pronunciation feature corresponding to the exemplary audio to a user terminal.
The exemplary audio is an audio of specified content that is read in a specified language, and the encoder is configured to extract, from an evaluation audio, an evaluation pronunciation feature corresponding to the standard pronunciation feature, where the standard pronunciation feature is used to reflect a specific pronunciation of the specified content in the specified language.
The method provided in the present application may be applied on the side of a server, and the server may provide data to the user terminal.
Specifically, the user terminal can transmit a data request instruction to the server based on a user operation. The server is provided with an encoder, exemplary audio, and a standard pronunciation feature corresponding to the exemplary audio as described in the embodiments shown in
As shown in
The apparatus for determining speech similarity based on speech interaction according to the present application is similar to the embodiments shown in
As shown in
In an implementation, the standard pronunciation feature corresponding to the exemplary audio is obtained by fusing a plurality of reference pronunciation features, each reference pronunciation feature is obtained by using the encoder to extract a feature from each piece of reference audio, a respective piece of reference audio is audio of the specified content that is read by using the specified language, and the exemplary audio is any piece of audio of the reference audio.
In an implementation, the analysis unit 630 includes:
In an implementation, the apparatus further includes a mapping unit 640 configured to:
In an implementation, the configuration information includes a maximum score, similarity corresponding to the maximum score, a minimum score, and similarity corresponding to the minimum score.
In an implementation, the similarity corresponding to the maximum score is an average value of multiple pieces of reference similarity, and each piece of reference similarity is similarity between each reference pronunciation feature and the standard pronunciation feature.
In an implementation, the similarity corresponding to the minimum score is an average value of multiple pieces of white noise similarity, each piece of white noise similarity is similarity between each white noise feature and the standard pronunciation feature, and each white noise feature is obtained by using the encoder to extract a feature from each piece of preset white noise audio.
In an implementation, the apparatus further includes a transceiving unit 650 which, before the acquiring unit 610 plays the exemplary audio, is configured to:
In an implementation, the speech recognition model is obtained by performing training on an initial model using speech recognition data; and
In an implementation, the encoder is a three-layer long short-term memory network.
The apparatus for determining speech similarity based on speech interaction according to the present application is similar to the embodiments shown in
As shown in
The apparatus for processing a data request instruction according to the present application is similar to the embodiments shown in
The present application further provides a computer program product, including a computer program, where the computer program, when being executed by a processor, implements the technical solution according to any one of method embodiments.
The present application further provides a computer program which, when being executed by a processor, implements the technical solution according to any one of method embodiments.
The apparatus provided in the embodiments can be used to perform the technical solutions in the foregoing method embodiments, and is implemented using similar principles and producing similar technical effects, which will not be described herein again in the embodiments.
Reference is made to
As shown in
Generally, the following apparatuses may be connected to the I/O interface 905: an input apparatus 906, including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, or a gyroscope; an output apparatus 907, including, for example, a liquid crystal display (LCD), a speaker, or a vibrator; a storage apparatus 908, including, for example, a magnetic tape, or a hard disk; and a communication apparatus 909. The communication apparatus 909 may enable the electronic device 900 to perform wireless or wire communication with other devices for data exchange. Although
In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication apparatus 909, or installed from the storage apparatus 908, or installed from the ROM 902. When the computer program is executed by the processing apparatus 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
It should be noted that, the above-mentioned computer readable medium in the present disclosure may be a computer readable signal medium or a computer readable storage medium or a combination of the both. The computer readable storage medium may be, for example, but not limited to, an electrical, a magnetic, an optical, an electromagnetic, an infrared, or a semiconductor system, apparatus or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM; or a flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction executive system, apparatus, or device. In the present disclosure, a computer readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and a computer readable program code is carried therein. This propagated data signal may adopt many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable media other than the computer readable storage medium, and the computer readable signal medium may send, propagate, or transmit the program used by or in combination with the instruction executive system, apparatus, or device. The program code contained on the computer readable medium may be transmitted by any suitable medium, including but not limited to: a wire, an optical cable, a radio frequency (RF), etc., or any suitable combination of the above.
The above-mentioned computer readable medium may be included in the above-mentioned electronic device; or it may exist alone without being assembled into the electronic device.
The above-mentioned computer readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device is caused to execute the method shown in above embodiments.
The computer program code used to perform operations of the present disclosure may be written in one or more programming languages or a combination thereof. The above-mentioned programming languages include object-oriented programming languages—such as Java, Smalltalk, C++, and also include conventional procedural programming languages—such as “C” language or similar programming languages. The program code may be executed entirely on a computer of a user, partly on a computer of a user, executed as an independent software package, partly executed on a computer of a user and partly executed on a remote computer, or entirely executed on a remote computer or a server. In a case where a remote computer is involved, the remote computer may be connected to the computer of the user through any kind of network—including a local area network (LAN) or a wide area network (WAN), or, it may be connected to an external computer (for example, use an Internet service provider to connect via the Internet).
The flowcharts and block diagrams in the drawings illustrate possible implementation architecture, functions, and operations of the system, method, and computer program product in accordance with the embodiments of the present disclosure. At this point, each block in the flowchart or the block diagram may represent a module, a program segment, or a part of code, and the module, the program segment, or the part of code contains one or more executable instructions for implementing a specified logical function. It should also be noted that, in some alternative implementations, the functions marked in the blocks may also occur in a different order from the order marked in the drawings. For example, two blocks shown one after another may actually be executed substantially in parallel, or sometimes may be executed in a reverse order, which depends on the functions involved. It should also be noted that, each block in the block diagram and/or flowchart, and a combination of the blocks in the block diagram and/or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.
The units involved in the embodiments described in the present disclosure may be implemented by means of software or hardware. Where a name of the unit does not constitute a limitation on the unit itself in some cases, for example, the acquiring unit can also be described as “a unit acquiring evaluation audio of a user.
The above functions described herein may be performed at least in part by one or more hardware logic components. For example, non-restrictively, exemplary types of hardware logic components that may be used include: a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on chip (SOC), a complex programmable logic device (CPLD), etc.
In the context of the present disclosure, a machine readable medium may be a tangible medium that may contain or store programs for use by or in combination with an instruction executive system, apparatus or device. The machine readable medium may be a machine readable signal medium or a machine readable storage medium. The machine readable medium may include, but is not limited to, an electronic, a magnetic, an optical, an electromagnetic, an infrared, or a semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the machine readable storage medium will include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or a flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
In a first aspect, according to one or more embodiments of the present disclosure, a method for determining speech similarity based on speech interaction is provided, including
According to one or more embodiments of the present disclosure, the extracting, from the evaluation audio, the evaluation pronunciation feature corresponding to the standard pronunciation feature includes:
According to one or more embodiments of the present disclosure, the standard pronunciation feature corresponding to the exemplary audio is obtained by fusing a plurality of reference pronunciation features, each reference pronunciation feature is obtained by using the encoder to extract a feature from each piece of reference audio, a respective piece of reference audio is audio of the specified content that is read by using the specified language, and the exemplary audio is any piece of audio of the reference audio.
According to one or more embodiments of the present disclosure, the determining the feature difference between the standard pronunciation feature and the evaluation pronunciation feature, and determining the similarity between the evaluation audio and the exemplary audio according to the feature difference includes:
According to one or more embodiments of the present disclosure, the method further includes:
According to one or more embodiments of the present disclosure, the configuration information includes a maximum score, similarity corresponding to the maximum score, a minimum score, and similarity corresponding to the minimum score.
According to one or more embodiments of the present disclosure, the similarity corresponding to the maximum score is an average value of multiple pieces of reference similarity, and each piece of reference similarity is similarity between each reference pronunciation feature and the standard pronunciation feature.
According to one or more embodiments of the present disclosure, the similarity corresponding to the minimum score is an average value of multiple pieces of white noise similarity, each piece of white noise similarity is similarity between each white noise feature and the standard pronunciation feature, and each white noise feature is obtained by using the encoder to extract a feature from each piece of preset white noise audio.
According to one or more embodiments of the present disclosure, before the playing the exemplary audio, the method further includes:
According to one or more embodiments of the present disclosure, the speech recognition model is obtained by performing training on an initial model using speech recognition data; and the encoder for extracting the pronunciation feature is obtained by performing training on the encoder in the speech recognition model using audio data in plural categories of languages.
According to one or more embodiments of the present disclosure, the encoder is a three-layer long short-term memory network.
In a second aspect, according to one or more embodiments of the present disclosure, a method for processing a data request instruction is provided, and the method is applied to a server and includes:
In a third aspect, according to one or more embodiments of the present disclosure, an apparatus for determining speech similarity is provided, including
According to one or more embodiments of the present disclosure, the feature extraction unit is specifically configured to:
According to one or more embodiments of the present disclosure, the standard pronunciation feature corresponding to the exemplary audio is obtained by fusing a plurality of reference pronunciation features, each reference pronunciation feature is obtained by using the encoder to extract a feature from each piece of reference audio, a respective piece of reference audio is audio of the specified content that is read by using the specified language, and the exemplary audio is any piece of audio of the reference audio.
According to one or more embodiments of the present disclosure, the analysis unit includes:
According to one or more embodiments of the present disclosure, the apparatus further includes a mapping unit configured to:
According to one or more embodiments of the present disclosure, the configuration information includes a maximum score, similarity corresponding to the maximum score, a minimum score, and similarity corresponding to the minimum score.
According to one or more embodiments of the present disclosure, the similarity corresponding to the maximum score is an average value of multiple pieces of reference similarity, and each piece of reference similarity is similarity between each reference pronunciation feature and the standard pronunciation feature.
According to one or more embodiments of the present disclosure, the similarity corresponding to the minimum score is an average value of multiple pieces of white noise similarity, each piece of white noise similarity is similarity between each white noise feature and the standard pronunciation feature, and each white noise feature is obtained by using the encoder to extract a feature from each piece of preset white noise audio.
According to one or more embodiments of the present disclosure, the apparatus further includes a transceiving unit which, before the acquiring unit plays the exemplary audio, is configured to:
According to one or more embodiments of the present disclosure, the speech recognition model is obtained by performing training on an initial model using speech recognition data; and the encoder for extracting the pronunciation feature is obtained by performing training on the encoder in the speech recognition model using audio data in plural categories of languages.
According to one or more embodiments of the present disclosure, the encoder is a three-layer long short-term memory network.
In a fourth aspect, according to one or more embodiments of the present disclosure, an apparatus for processing a data request instruction is provided. The apparatus is disposed in a server and includes:
In a fifth aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, including: at least one processor and a memory;
In a sixth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium in which a computer-executable instruction is stored is provided, where the computer-executable instruction, when being executed by a processor, implements the method for determining speech similarity based on speech interaction as described above in the first aspect and various possible designs of the first aspect, or the method for processing a data request instruction as described above in the second aspect and various possible designs of the second aspect.
In a seventh aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including: a computer program, where the computer program, when being executed by a processor, implements the method for determining speech similarity based on speech interaction as described above in the first aspect and various possible designs of the first aspect, or the method for processing a data request instruction as described above in the second aspect and various possible designs of the second aspect.
In an eighth aspect, according to one or more embodiments of the present disclosure, a computer program is provided. The computer program, when being executed by a processor, implements the method for determining speech similarity based on speech interaction as described above in the first aspect and various possible designs of the first aspect, or the method for processing a data request instruction as described above in the second aspect and various possible designs of the second aspect.
The above description is only preferred embodiments of the present disclosure and an illustration of the applied technical principles. Those skilled in the art should understand that, the disclosure scope involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but also covers other technical solutions formed by the arbitrary combination of the above technical features or their equivalent features without departing from the above disclosure concept, for example, a technical solution formed by replacing the above features with technical features with similar functions disclosed (but not limited to) in the present disclosure.
In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order or in a sequential order shown. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a single embodiment may also be implemented in combination in the single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub combination.
Although the subject matter has been described in a language specific to structural features and/or method logical actions, it should be understood that the subject matter defined in the appended claims is not limited to the specific features or actions described above. On the contrary, the specific features and actions described above are only exemplary forms for implementing the claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202110179824.X | Feb 2021 | CN | national |
This application is a national stage of International Application No. PCT/SG2022/050048, filed on Jan. 31, 2022, which claims priority to Chinese Patent Application No. 202110179824.X, filed on Feb. 7, 2021 to the China National Intellectual Property Administration, and entitled “Method and Apparatus for Determining Speech Similarity, and Program Product”. The disclosures of the aforementioned applications are hereby incorporated by reference in their entireties.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/SG2022/050048 | 1/31/2022 | WO |
