Patent Application
20200090682
Embodiments of the present disclosure relates to the field of speech information processing technologies, and in particular, to a voice activity detection method, a method for establishing a voice activity detection model, a computer device, and a storage medium.
With development of information technology, speech information is being widely used, and voice activity detection (VAD) is of a significant importance for speech information processing. The VAD is also referred to as speech endpoint detection and speech boundary detection. The VAD indicates detecting presence or absence of a speech in an environment including noises, and is generally used in a speech processing system, such as speech encoding and speech enhancement, to produce effects of, for example, reducing a speech encoding rate, saving communication bandwidth, reducing energy consumption of a computer device, and improving a recognition rate.
A conventional voice activity detection method is an energy-based voice activity detection method. According to this method, noise is considered to be stable, and energy of a speech portion is higher than the energy of a noise portion. Determining presence or absence of a voice activity is performed according to this method, and its main advantages may be good applicability to a situation with stable noise and relatively wide applicability. However, its disadvantages are poor effects in an environment with unstable noises, and is susceptible to erroneous judgments. Therefore, a problem of poor detection accuracy exists in the conventional voice activity detection method.
According to various embodiments in the present disclosure, a voice activity detection method, a method for establishing a voice activity detection model, a computer device, and a storage medium are provided.
According to an embodiment, there is provided a method for establishing a voice activity detection model, the method being performed by an execution device, the method including obtaining a training audio file and a target result of the training audio file; framing the training audio file to obtain an audio frame; extracting an audio feature of the audio frame, the audio feature comprising at least two types of features, and one of the at least two types of features comprising an energy; inputting the extracted audio feature as an input to a deep neural network model; performing information processing on the audio feature through a hidden layer of the deep neural network model, and outputting the processed audio feature through an output layer of the deep neural network model, to obtain a training result; determining a bias between the training result and the target result, and inputting the bias as an input to an error back propagation mechanism; and separately updating weights of the hidden layer until the deep neural network model reaches a preset condition, to obtain the voice activity detection model.
According to another embodiment, there is provided a computer device including at least one memory configured to store computer program code and at least one processor configured to access the computer program code and operate as instructed by the computer program code. The computer program code including file obtaining code configured to cause the at least one processor to obtain a training audio file; result obtaining code configured to cause the at least one processor to obtain a target result of the training audio file; framing code configured to cause the at least one processor to frame the training audio file to obtain an audio frame; extraction code configured to cause the at least one processor to extract an audio feature of the audio frame, the audio feature comprising at least two types of features, and one of the at least two types of features comprising an energy; inputting code configured to cause the at least one processor to input the audio feature as an input to a deep neural network model, and perform information processing on the audio feature through a hidden layer of the deep neural network model; outputting code configured to cause the at least one processor to output the processed audio feature through an output layer of the deep neural network model, to obtain a training result; and update and optimizing code configured to cause the at least one processor to determine a bias between the training result and the target result as an input to an error back propagation mechanism, and separately update weights of the hidden layer until the deep neural network model reaches a preset condition, to obtain a voice activity detection model.
According to another embodiment, there is provided a non-transitory computer-readable storage medium, storing executable instructions, the executable instructions capable of causing a computer to: obtain a training audio file and a target result of the training audio file; frame the training audio file to obtain an audio frame; extract an audio feature of the audio frame, the audio feature comprising at least two types of features, and one of the at least two types of features comprising an energy; input the audio feature as an input to a deep neural network model, performing information processing on the audio feature through a hidden layer of the deep neural network model, and output the processed audio feature through an output layer of the deep neural network model, to obtain a training result; and determine a bias between the training result and the target result as an input to an error back propagation mechanism, and separately update weights of the hidden layer until the deep neural network model reaches a preset condition, to obtain a voice activity detection model.
Details of one or more embodiments of the present disclosure are provided with reference to accompanying drawings. Other features, objectives, and advantages of the present disclosure will become more obvious with reference to the description herein, the accompanying drawings, and the claims.
To describe the technical solutions in the embodiments of the present disclosure more clearly, the following describes the embodiments with reference to the accompanying drawings. Apparently, the accompanying drawings in the following description show merely some embodiments, and a person of ordinary skill in the art may still derive other embodiments without creative efforts.
To make the objectives, technical solutions, and advantages of the present disclosure clearer and more comprehensible, the following describes the embodiments with reference to the accompanying drawings. It may be understood that the specific embodiments described herein are merely used to explain the present disclosure, but are not intended to limit the protection scope of the present disclosure.
The present disclosure may be applied to any scenario in which a voice activity needs to be detected.
A person skilled in the art may understand that, the structure shown in
As shown in
In step S110, the method may include obtaining a training audio file and a target result of the training audio file.
The execution device may obtain a training audio file from a training sample and a target result of the training audio file. The training sample may be obtained by performing voice detection on a set of training audio files in a manual recognition manner, and may be used as a target result of the training audio file. The training audio file is an audio file used for training, and may be an audio file in various application environment. The target result is a training result that is expected to be obtained from the training audio, but because the model may not be optimal during a training process, there may be a specific bias between a training result and the target result.
In step S120, the method may include framing the training audio file to obtain an audio frame, and extract an audio feature of the audio frame. The audio feature may include two types of features, such as energy and at least one feature different from the energy.
The execution device may frame an entire training audio file, and then, may perform voice activity detection on each audio frame. During a detection process, the execution device first needs to extract an audio feature of the audio frame. For example, the audio feature may further include at least one other feature, such as a zero-crossing rate, a mean value, and a variance. In a case where a plurality of features are included, the plurality of features may be classified as a plurality of features of a single type or a plurality of features not of a single type. Since different features have different interpretations of speech/non-speech, some features may be effective for stable noises, and some features may be effective for unstable noises. Further, one feature may express speech/non-speech more clearly than another feature and provide a better effect. Specifically, a plurality of features not of a single type may be more comprehensive than a plurality of features of a single type in terms of expressions of speech/non-speech, and therefore, may provide a better effect.
In step S130, the method may include using the audio feature as an input for a deep neural network model, performing information processing on the audio feature through a hidden layer of the deep neural network model, and outputting the audio feature through an output layer of the deep neural network model, to obtain a training result.
A framework of the deep neural network model, as shown in
For example, an information transferring rule may be represented as y=W●z+b, where z=f(yl−1) is a final output obtained after non-linear conversion, W and b are a weight and a bias of a hidden layer, respectively, yl−1 is a result of a linear product of a previous layer, f is a non-linear function, and a non-linear function of the hidden layer may be a sigmoid function, which is an S-shaped growth curve function.
In step S140, the method may use a bias between the training result and the target result as an input for an error back propagation mechanism, and separately update weights of the hidden layer until the deep neural network model reaches a preset condition, to obtain a voice activity detection model.
The bias may be a difference between the training result and the target result. The training result and the target result may both be represented by a mark. Each mark corresponds to an identification (ID) represented in a form of a value, and the bias may be represented by a difference between an ID corresponding to the training result and an ID corresponding to the target result. Before the deep neural network model reaches the preset condition, an error back propagation mechanism may be used to update weights of the hidden layers, one layer by one layer, from the output layer to the input layer. Further, the error back propagation mechanism may update the deep neural network model until the deep neural network model reaches the preset condition, and then, the deep neural network model may be used as the voice activity detection model. The error back propagation mechanism may be implemented by using an error back propagation algorithm. The preset condition may be a condition when the deep neural network is optimal. Whether the deep neural network is optimal may be determined by using a loss function. In the loss function, the training audio file in the training sample and the target result of the training audio file may be separately input into the deep neural network model, and the deep neural network becomes optimal when the accuracy of a preset consecutive quantity of times is unchanged. The loss function may be a maximum entropy function and a minimum mean square error function. When the weights of the hidden layers are updated, a stochastic gradient descent method may be used, and after a plurality of iterations, the deep neural network model may finally reach the preset condition. The following formula is used in the stochastic gradient descent method: Wt+1=Wt−ηΔWt, where η is a learning rate, Wt+1 is a weight of the (t+1)th hidden layer, Wt is a weight of the tth hidden layer, ΔWt is a result after a partial derivative of the weight is obtained in the loss function. In the maximum entropy loss function, a partial derivative of the weight w and the bias b in the loss function may be obtained, and the stochastic gradient descent method may be used to update the weight w and the bias b by iteration.
In the foregoing method for establishing a voice activity detection model, because the audio feature includes energy and at least one feature different from energy, the audio feature carries more information than that of a conventional detection manner. Accordingly, the deep neural network model has a strong learning capability, and a plurality of different features may be input at the same time to train the deep neural network model to obtain the voice activity detection model. As a result, when the voice activity detection is performed by using the voice activity detection model, the accuracy of the voice activity detection is high.
Further, according to an embodiment, the target result may include at least two speech categories or/and at least two noise categories. Compared with a conventional two-category modeling method in which only two categories, namely, a speech category and a noise category, are used, this embodiment uses a multi-category modeling method, and subdivides the speech category and the noise category into different categories to which the target result belongs. For example, the speech category may be subdivided into a first speech category SPK1, a second speech category SPK2, a third speech category SPK3, and so on. The noise category may be subdivided into a first noise category SIL1, a second noise category SIL2, a third noise category SIL3, and so on. Based on the subdivision of the speech category and the noise category into multiple categories, the multi-category modeling method may be trained more effectively than that of the two-category modeling method. In a detection process, a summation only needs to be performed on a same category, and then, final scores may be compared to obtain a final category. Accordingly, the result of the detection process may be more accurate.
According to an embodiment, referring to
In step S221, the method may include extracting independent audio features of each audio frame.
In step S222, the method may include fusing the respective independent audio features to obtain the fused audio feature.
Here, the independent audio feature indicates that the audio feature only includes an individual feature. In the deep neural network model, different types of independent audio features may be equal. A feature fusion method may be used to increase a quantity of dimensions of the audio feature. For example, in a case where a mean value is 40 dimensions, an energy feature increased by 10 dimensions becomes a 50-dimensional fused audio feature. Subsequently, the fused audio feature may be input into the deep neural network model.
Referring to
The operation of extracting an audio feature of each audio frame may include the following steps.
In step S323, the method may include extracting a single frame audio feature of each audio frame.
In step S324, the method may include using the single frame audio feature of the current frame and the single frame audio feature of the first preset quantity of frames before the current frame, and/or the single frame audio feature of the second preset quantity of frames after the current frame, as the extended frame audio feature of the current frame.
Specifically, the single frame audio feature indicates that the audio feature only includes a feature of one frame. The extended frame audio feature indicates an audio feature that includes features of a plurality of frames. The plurality of frames may specifically be a current frame and a first preset quantity of frames before the current frame and/or a second preset quantity of frames after the current frame. For example, assuming that the single frame audio feature is an N-dimensional audio feature, the first preset quantity of frames is m1, and the second quantity of frames is m2, a quantity of dimensions of the extended audio feature is N*(m1+1), N*(m2+1), or N*(m1+m2+1). Accordingly, each audio feature may be enabled by the extended audio feature to carry more information, and the accuracy of detection may be improved.
Since audio may be a short-time stable process, if the historical and future information can both be used to train the deep neural network model, an effect of the training will be better. For example, an independent audio feature of a first preset quantity of frames before a current time point (the current frame) may be historical information, and an independent audio feature of a second preset quantity of frames after the current time point may be future information. That is, the extended frame audio feature may include a single frame audio feature of a current frame, a single frame audio feature of a first preset quantity of frames before the current frame, and a single frame audio feature of a second preset quantity of frames after the current frame.
In step S325, the method may include separately using each audio frame as the current frame to obtain an extended audio feature of each audio frame.
As such, each audio feature may be extended so as to improve accuracy of audio activity detection.
Referring to
Referring to
In step S610, the method may include obtaining a to-be-detected audio file.
In step S620, the method may include framing the to-be-detected audio file to obtain an audio frame, and extracts an audio feature of the audio frame.
In step S630, the method may include obtaining the voice activity detection model that is obtained based on the foregoing method for establishing a voice activity detection model.
In step S640, the method may include inputting the audio feature into the voice activity detection model, to obtain a detection result.
According to an embodiment, it is unnecessary to obtain the target result or update the model according to the target result and the training result. It only needs to input the audio feature into the audio activity detection model and then, obtain a detection result.
Because the audio feature includes energy and at least one feature different from the energy, the audio feature may carry more information than that of a conventional detection manner. Accordingly, the deep neural network has a strong learning capability, and a plurality of different features may be input at the same time, to perform training to obtain the voice activity detection model. As a result, when the voice activity detection is performed by using the voice activity detection model, the accuracy of the voice activity detection is high.
As shown in
In step S842, the method may include inputting the audio feature into the voice activity detection model to obtain a frame detection result of each audio frame of the to-be-detected audio file.
In step S844, the method may include smoothing each frame detection result in the to-be-detected audio file, and obtaining a detection result of the to-be-detected audio file.
According to an embodiment, after detecting each audio frame in the to-be-detected audio file to obtain a frame detection result, the execution device may perform smoothing on the frame detection result of each audio frame in the to-be-detected audio file to correct an erroneous judgment in a detection process. During a judgment process, an obvious error may inevitably occur. For example, an outputted sequence of the deep neural network model may be SPK, SPK, SPK, SIL, and SPK. It is obvious that SIL is an erroneous judgment. After smoothing, a detection result is SPK, SPK, SPK, SPK, and SPK. Accordingly, the smoothed detection result provides more accurate detection result.
Still referring to
In step S850, the method may include determining a speech start point and a speech end point in the to-be-detected audio file in accordance with the detection result.
Further, the execution device may set a time threshold by using a sliding window. A speech start point may be found based on a length of a speech segment in the sliding window exceeding the time threshold. After the speech start point is found, a speech end point may be found based on a length of silence in the sliding window exceeding the threshold. Finally, the execution device may divide the audio into different sub-segments, such as a speech sub-segment and a noise sub-segment, according to the speech start point and the speech end point.
According to an embodiment, a computer device may be provided, and an internal structure of the computer device may be the structure shown in
Referring to
A file and result obtaining module 110 may be configured to obtain a training audio file and a target result of the training audio file.
A framing and feature extraction module 120 may be configured to frame the training audio file to obtain an audio frame, and extract an audio feature of the audio frame, the audio feature including at least two types of features, such as energy and at least one feature different from the energy.
The model inputting/outputting module 130 may be configured to use the audio feature as an input to a deep neural network model, perform information processing on the audio feature through a hidden layer of the deep neural network model, and output the audio feature through an output layer of the deep neural network model to obtain a training result.
The model updating and optimizing module 140 may be configured to use a bias between the training result and the target result, as an input to an error back propagation mechanism, and separately update weights of the hidden layer until the deep neural network model reaches a preset condition to obtain a voice activity detection model.
In the foregoing apparatus for establishing a voice activity detection model, because the audio feature includes energy and at least one feature different from the energy, the audio feature may carry more information than that of a conventional detection manner. Accordingly, the deep neural network model has a strong learning capability, and a plurality of different features may be input at the same time, to train the model to obtain the voice activity detection model. Therefore, when the voice activity detection is performed by using the voice activity detection model, the accuracy of the voice activity detection is high.
Further, the target result may include at least two speech categories or/and at least two noise categories.
Referring to
Still referring to
The framing and feature extraction module 220 may include a single frame feature extraction unit 223 configured to extract a single frame audio feature of each audio frame, an audio frame extension unit 224 configured to use the single frame audio feature of the current frame and the single frame audio feature of the first preset quantity of frames before the current frame, and/or the single frame audio feature of the second preset quantity of frames after the current frame, as the extended frame audio feature of the current frame, and the audio frame extension unit 224 further configured to separately use each audio frame as the current frame to obtain an extended audio feature of each audio frame.
According to an embodiment, a computer device may be provided, and an internal structure of the computer device may be the structure as shown in
Referring to
Because the audio feature includes energy and at least one feature different from energy, the audio feature carries more information than that of a conventional detection manner. Accordingly, the deep neural network has a strong learning capability, and a plurality of different features may be input at the same time, to perform training to obtain the voice activity detection model. As a result, when the voice activity detection is performed by using the voice activity detection model, the accuracy of the voice activity detection is high.
Referring to
According to an embodiment, the detection apparatus may further include a to-be-detected audio segmentation module 450 configured to determine a speech start point and a speech end point in the to-be-detected audio file according to the detection result.
The apparatus for establishing a voice activity detection model and the voice activity detection apparatus provided herein may be implemented in a form of a computer program. The computer program may run on the computer device as shown in
Using an apparatus for establishing a voice activity detection model as an example, referring to
This application further provides a computer device corresponding to the foregoing method and apparatus for establishing a voice activity detection model, including a memory and a processor. The memory stores a computer program, and the computer program causes the processor to perform the following operations: obtaining a training audio file and a target result of the training audio file; framing the training audio file to obtain an audio frame, and extracting an audio feature from the audio frame, the audio feature including at least two types of features including energy and another type other than the energy; using the audio feature as an input to a deep neural network model, performing information processing on the audio feature through a hidden layer of the deep neural network model, and outputting the processed audio feature through an output layer of the deep neural network model, to obtain a training result; and using a bias between the training result and the target result as an input to an error back propagation mechanism, and separately updating weights of the hidden layer until the deep neural network model reaches a preset condition, to obtain a voice activity detection model.
The target result may include at least two speech categories and/or at least two noise categories.
According to an embodiment, the audio feature may be a fused audio feature in which the fused audio feature includes at least two independent audio features, and the independent audio features include energy and at least one of a zero-crossing rate, a mean value, and a variance. The operation of extracting an audio feature of each audio frame may include extracting independent audio features of each audio frame; and fusing the respective independent audio features to obtain the fused audio feature.
According to an embodiment, the audio feature may be an extended frame audio feature in which the extended frame audio feature includes a single frame audio feature of a current frame and a single frame audio feature of a first preset quantity of frames before the current frame, and/or a single frame audio feature of a second preset quantity of frames after the current frame. The operation of extracting an audio feature of each audio frame may include extracting the single frame audio feature of each audio frame; using the single frame audio feature of the current frame and the single frame audio feature of the first preset quantity of frames before the current frame or/and the single frame audio feature of the second preset quantity of frames after the current frame as the extended frame audio feature of the current frame; and separately using each audio frame as the current frame to obtain an extended audio feature of each audio frame.
The present disclosure further provides a computer device corresponding to the foregoing voice activity detection method and apparatus, including a memory and a processor. The memory stores a computer program, and the computer program causes the processor to obtain a to-be-detected audio file; frame the to-be-detected audio file to obtain an audio frame, and extract an audio feature of the audio frame; obtain the voice activity detection model that is obtained based on the foregoing method for establishing a voice activity detection model; and input the audio feature into the voice activity detection model to obtain a detection result.
Furthermore, the operation of inputting the audio feature into the voice activity detection model, to obtain a detection result, may include inputting the audio feature into the voice activity detection module, to obtain a frame detection result of each audio frame of the to-be-detected audio file; and smoothing each frame detection result in the to-be-detected audio file, and obtaining a detection result of the to-be-detected audio file.
After obtaining a detection result of the to-be-detected audio file, the computer device may be further configured to determine a speech start point and a speech end point in the to-be-detected audio file according to the detection result.
According to an embodiment, a non-volatile storage medium corresponding to the foregoing method and apparatus for establishing a voice activity detection model may be implemented. One or more non-volatile storage mediums storing a computer program may be provided, and may be configured to perform: obtaining a training audio file and a target result of the training audio file; framing the training audio file to obtain an audio frame, and extracting an audio feature of the audio frame, the audio feature including at least two types of features including energy and another type other than the energy; using the audio feature as an input to a deep neural network model, performing information processing on the audio feature through a hidden layer of the deep neural network model, and outputting the audio feature through an output layer of the deep neural network model, to obtain a training result; and using a bias between the training result and the target result as input of an error back propagation mechanism, and separately updating weights of the hidden layer until the deep neural network model reaches a preset condition, to obtain a voice activity detection model.
In addition, the target result may include at least two speech categories and/or at least two noise categories.
According to an embodiment, the audio feature may be a fused audio feature in which the fused audio feature may include at least two independent audio features, and the independent audio features may include energy and at least one of a zero-crossing rate, a mean value, and a variance. Further, the operation of extracting an audio feature of each audio frame may include extracting independent audio features of each audio frame; and fusing the respective independent audio features to obtain the fused audio feature.
According to an embodiment, the audio feature may be an extended frame audio feature, the extended frame audio feature may include a single frame audio feature of a current frame and a single frame audio feature of a first preset quantity of frames before the current frame, and/or a single frame audio feature of a second preset quantity of frames after the current frame. The operation of extracting an audio feature of each audio frame may include extracting the single frame audio feature of each audio frame; and using the single frame audio feature of the current frame and the single frame audio feature of the first preset quantity of frames before the current frame, and/or the single frame audio feature of the second preset quantity of frames after the current frame, as the extended frame audio feature of the current frame; and separately using each audio frame as the current frame to obtain an extended audio feature of each audio frame.
Furthermore, a non-volatile storage medium corresponding to the foregoing voice activity detection method and apparatus may be implemented. One or more non-volatile storage mediums storing a computer program may be provided, and the computer program may cause one or more processors to: obtain a to-be-detected audio file; frame the to-be-detected audio file to obtain an audio frame, and extract an audio feature from the audio frame; obtain the voice activity detection model that is obtained based on the foregoing method for establishing a voice activity detection model; and input the audio feature into the voice activity detection model, to obtain a detection result.
In addition, the operation of inputting the audio feature into the voice activity detection model, to obtain a detection result, may include inputting the audio feature into the voice activity detection module to obtain a frame detection result of each audio frame of the to-be-detected audio file; and smoothing each frame detection result in the to-be-detected audio file, and obtaining a detection result of the to-be-detected audio file.
Further still, after the operation of obtaining a detection result of the to-be-detected audio file, the operation of obtaining may further include determining a speech start point and a speech end point in the to-be-detected audio file according to the detection result.
A person of ordinary skill in the art may understand that all or some of the processes of the methods herein may be implemented by a computer program instructing a hardware. The program may be stored in a non-transitory computer-readable storage medium. When the program operates, the methods described herein may be performed. Any reference to the memory, storage, database, or another medium used in the respective embodiments provided in the present disclosure may include a non-volatile and/or volatile memory. The non-volatile memory may include a read only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM) or an external cache memory. A RAM may be available in a variety of forms, such as a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDRSDRAM), an enhanced SDRAM (ESDRAM), a synchronized link (Synchlink) DRAM (SLDRAM), a memory bus (Rambus) direct RAM (RDRAM), a direct memory bus dynamic RAM (DRDRAM), and a memory bus dynamic RAM (RDRAM).
Technical features of the embodiments described herein may be randomly combined. To make description concise, not all combinations of the technical features of the embodiments are described. However, as long as combinations of these technical features do not contradict each other, it is to be considered that the combinations all fall within the scope of the present disclosure.
The embodiments described herein only show several implementations of the present disclosure, but they are not to be construed as a limitation to the patent scope of the present disclosure. A person of ordinary skill in the art may make various changes and improvements without departing from the ideas of the present disclosure, which shall all fall within the protection scope of the present disclosure.
| Number | Date | Country | Kind |
|---|---|---|---|
| 201710824269.5 | Sep 2017 | CN | national |
This application is a continuation of International Patent Application No. PCT/CN2018/102982 filed on Aug. 29, 2018, which claims priority from Chinese Patent Application No. 201710824269.5, entitled “VOICE ACTIVITY DETECTION, METHOD FOR ESTABLISHING MODEL THEREOF, APPARATUS, DEVICE, and STORAGE MEDIUM” and filed in the Chinese Patent Office on Sep. 13, 2017, which are incorporated herein by reference in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/CN2018/102982 | Aug 2018 | US |
| Child | 16678001 | US |
