Patent Grant
11676573
This disclosure relates to using contextual features in expressive end-to-end speech synthesis systems.
Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. For instance, neural networks may convert input text to output speech. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.
Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from a previous time step in computing an output at a current time step. An example of a recurrent neural network is a long short term (LSTM) neural network that includes one or more LSTM memory blocks. Each LSTM memory block can include one or more cells that each include an input gate, a forget gate, and an output gate that allows the cell to store previous states for the cell, e.g., for use in generating a current activation or to be provided to other components of the LSTM neural network.
One aspect of the disclosure provides a system for generating an output audio signal of expressive speech of current input text. The system includes a context encoder, a text-prediction network in communication with the context encoder, and a text-to-speech (TTS) model in communication with the text-prediction network. The context encoder is configured to receive one or more context features associated with current input text to be synthesized into expressive speech, and process the one or more context features to generate a context embedding associated with the current input text. Each context feature is derived from a text source of the current input text. The text-prediction network is configured to receive the current input text from the text source, receive the context embedding associated with the current input text front the context encoder, and process the current input text and the context embedding associated with the current input text to predict, as output, a style embedding for the current input text. The text source includes sequences of text to be synthesized into expressive speech and the style embedding specifies a specific prosody and/or style for synthesizing the current input text into expressive speech. The TTS model is configured to receive the current input text from the text source, receive the style embedding predicted by the text-predication network, and process the current input text and the style embedding to generate an output audio signal of expressive speech of the current input text. The output audio signal has the specific prosody and/or style specified by the style embedding.
Implementations of the disclosure may include one or more of the following optional features. In some implementations, the one or more context features associated with the current input text comprise at least one of: the current input text, previous text from the text source that precedes the current input text; previous speech synthesized from the previous text; upcoming text from the text source that follows the current input text; or a previous style embedding predicted by the text-prediction network based on the previous text and a previous context embedding associated with the previous text. In some examples, the text source includes a text document and the one or more context features associated with the current input text include at least one of: a title of the text document, a title of a chapter in the text document; a title of a section in the text document; a headline in the text document; one or more bullet points in the text document; entities from a concept graph extracted from the text document; or one or more structured answer representations extracted from the text document.
In other examples, the text source includes a dialogue transcript and the current input text corresponds to a current turn in the dialogue transcript. In these examples, the one or more context features associated with the current input text include at least one of previous text in the dialogue transcript that corresponds to a previous turn in the dialogue transcript, or upcoming text in the dialogue transcript that corresponds to a next turn in the dialogue transcript.
The text source may also include a query-response system in which the current input text corresponds to a response to a current query received at the query-response system. Here, the one or more context features associated with the current input text may include at least one of text associated with the current query or text associated with a sequence of queries received at the query response-system, or audio features associated with the current query or audio features associated with the sequence of queries received at the query response-system. The sequence of queries may include the current query and one or more queries preceding the current query.
In some implementations, the TTS model includes an encoder neural network, a concatenator, and an attention-based decoder recurrent neural network. The encoder neural network is configured to receive the current input text from the text source and process the current input text to generate a respective encoded sequence of the current input text. The concatenator is configured to receive the respective encoded sequence of the current input text from the encoder neural network, receive the style embedding predicted by the textual-prediction network and generate a concatenation between the respective encoded sequence of the current input text and the style embedding. The attention-based decoder recurrent neural network is configured to receive a sequence of decoder inputs, and for each decoder input in the sequence, process the corresponding decoder input and the concatenation between the respective encoded sequence of the current input text and the style embedding to generate r frames of the output audio signal, wherein r comprises an integer greater than one.
In the implementations when the TTS model includes the encoder neural network, the encoder neural network may include an encoder pre-net neural network and an encoder CBHG neural network. The encoder pre-net neural network configured to receive a respective embedding of each character in a sequence of characters of the current input text, and for each character, process the respective embedding to generate a respective transformed embedding of the character. The encoder CBHG neural network is configured to receive the transformed embeddings generated by the encoder pre-net neural network, and process the transformed embeddings to generate the respective encoded sequence of the current input text. In some configurations, the encoder CBHG neural network includes a bank of 1-D convolutional filters, followed by a highway network, and followed by a bidirectional recurrent neural network.
In some configurations, the text-prediction network includes a time-aggregating gated recurrent unit (GRU) recurrent neural network (RNN) and one or more fully-connected layers. The GRU RNN is configured to receive the context embedding associated with the current input text and an encoded sequence of the current input text, and generate a fixed-length feature vector by processing the context embedding and the encoded sequence. The one or more fully-connected layers are configured to predict the style embedding by processing the fixed-length feature vector. In these configurations, the one or more fully-connected layers may include one or more hidden fully-connected layers using ReLU activations and an output layer that uses tanh activation to emit the predicted style embedding.
The context model, the text-prediction model, and the TTS model may be trained jointly. Alternatively, a two-step training procedure may train the TTS model during a first step of the training procedure, and separately train the context model and the text-prediction model jointly during a second step of the training procedure.
Another aspect of the disclosure provides a method for generating an output audio signal of expressive speech of current input text. The method includes receiving, at data processing hardware, current input text from a text source. The current input text is to be synthesized into expressive speech by a text-to-speech (TTS) model. The method also includes generating, by the data processing hardware, using a context model, a context embedding associated with current input text by processing one or more context features derived from the text source. The method also includes predicting, by the data processing hardware, using a text-prediction network, a style embedding for the current input text by processing the current input text and the context embedding associated with the current input text. The style embedding specifies a specific prosody and/or style for synthesizing the current input text into expressive speech. The method also includes generating, by the data processing hardware, using the TTS model, the output audio signal of expressive speech of the current input text by processing the style embedding and the current input text. The output audio signal has the specific prosody and/or style specified by the style embedding.
This aspect may include one or more of the following optional features. In some implementations, the one or more context features associated with the current input text comprise at least one of: the current input text; previous text from the text source that precedes the current input text; previous speech synthesized from the previous text; upcoming text from the text source that follows the current input text; or a previous style embedding predicted by the text-prediction network based on the previous text and a previous context embedding associated with the previous text. In some examples, the text source includes a text document and the one or more context features associated with the current input text include at least one of a title of the text document, a title of a chapter in the text document; a title of a section in the text document, a headline in the text document, one or more bullet points in the text document, entities from a concept graph extracted from the text document; or one or more structured answer representations extracted from the text document.
In other examples, the text source includes a dialogue transcript and the current input text corresponds to a current turn in the dialogue transcript. In these examples, the one or more context features associated with the current input text include at least one of previous text in the dialogue transcript that corresponds to a previous turn in the dialogue transcript, or upcoming text in the dialogue transcript that corresponds to a next turn in the dialogue transcript.
The text source may also include a query-response system in which the current input text corresponds to a response to a current query received at the query-response system. Here, the one or more context features associated with the current input text may include at least one of text associated with the current query or text associated with a sequence of queries received at the query response-system, or audio features associated with the current query or audio features associated with the sequence of queries received at the query response-system. The sequence of queries may include the current query and one or more queries preceding the current query.
In some implementations, generating the output audio signal includes: receiving, at an encoder neural network of the text-to-speech model, the current input text from the text source; generating, using the encoder neural network, a respective encoded sequence of the current input text, generating, using a concatenator of the text-to-speech model, a concatenation between the respective encoded sequence of the current input text and the style embedding; receiving, at an attention-based decoder recurrent neural network of the text-to-speech model, a sequence of decoder inputs; and for each decoder input in the sequence of decoder inputs, processing, using the attention-based decoder recurrent neural network, the corresponding decoder input and the concatenation between the respective encoded sequence of the current input text and the style embedding to generate r frames of the output audio signal, wherein r includes an integer greater than one. In these implementations, generating the respective encoded sequence of the current input text includes receiving, at an encoder pre-net neural network of the encoder neural network, a respective embedding of each character in a sequence of characters of the current input text, for each character in the sequence of characters, processing, using the encoder pre-net neural network, the respective embedding to generate a respective transformed embedding of the character; and generating, using an encoder CBHG neural network of the encoder neural network, respective encoded sequence of the current input text by processing the transformed embeddings. In some configurations, the encoder CBHG neural network includes a bank of 1-D convolutional filters, followed by a highway network, and followed by a bidirectional recurrent neural network.
In some examples, predicting the style embedding for the current input text includes: generating, using a time-aggregating gated recurrent unit (GRU) recurrent neural network (RNN) of the text-prediction model, a fixed-length feature vector by processing the context embedding associated with the current input text and an encoded sequence of the current input text; and predicting, using one or more fully-connected layers of the text-prediction model that follow the GRU-RNN, the style embedding by processing the fixed-length feature vector. The one or more fully-connected layers may include one or more hidden fully-connected layers using ReLU activations and an output layer that uses tanh activation to emit the predicted style embedding.
The context model, the text-prediction model, and the TTS model may be trained jointly. Alternatively, a two-step training procedure may train the TTS model during a first step of the training procedure, and separately train the context model and the text-prediction model jointly during a second step of the training procedure.
The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.
Like reference symbols in the various drawings indicate like elements.
The synthesis of realistic human speech is an underdetermined problem in that a same text input has an infinite number of reasonable spoken realizations. While end-to-end neural network-based approaches are advancing to match human performance for short assistant-like utterances, neural network models are sometimes viewed as less interpretable or controllable than more conventional models that include multiple processing steps each operating on refined linguistic or phonetic representations.
A major challenge for text-to-speech (TTS) systems is developing models for producing a natural-sounding speaking style for a given piece of input text. Particularly, some of the factors that contribute to the challenge for producing natural-sounding speech include high audio fidelity, correct pronunciation, and acceptable prosody and style, whereby “prosody” generally refers to low-level characteristics such as pitch, stress, breaks, and rhythm. Prosody impacts “style”, which refers to higher-level characteristics of speech such as emotional valence and arousal. As such, prosody and style are difficult to model because they encompass information not specified in the text to be synthesized, and allow the synthesized speech to be spoken in an infinite number of ways. Simply put, text is underspecified in that information about style and prosody is not available, leaving mapping from text to speech a one-to-many problem.
While providing high-level style labels (e.g., conveying emotion) or low-level annotations (e.g., syllabic stress markers, speed controls, pitch tracks, etc.) as inputs to a synthesizer may improve the modeling of prosody and style, there are a number of drawbacks to these approaches. Namely, explicit labels are difficult to define with precisions, costly to acquire, noisy in nature, and do not guarantee a correlation with perceptual quality by a listener. Moreover, explicit label inputs for modeling prosody and style are often derived from hand-tuned heuristics or separately trained models. In addition, the context from which these inputs were derived from is usually lost.
Generally, TTS systems generate speech by synthesizing a single sentence or paragraph at a time. As a result, when context from which a piece of text is drawn is not accessible, the natural expressivity of the resulting synthesized speech is limited. It is particularly challenging to convey a wide-range of speaking styles when synthesizing speech from long-form expressive datasets of text, such as audiobooks. For instance, simply collapsing a wide-range of different voice characteristics into a single, averaged model of prosodic style results in synthesized speech having a specific speaking style that may not accurately reflect an appropriate emotional valence and arousal that the text is meant to convey. In an example, applying a single, averaged model of prosodic style for synthesizing speech for an audiobook will not adequately represent all of the speaking styles needed to convey different emotions, such as emotional transitions from a happy chapter in the audiobook to a following sad chapter in the audiobook. Similarly, audiobooks may contain character voices with significant style variation. In these examples, using the averaged model of prosodic style will produce monotonous-sounding speech that does not convey emotional transitions or the variation of style between different character voices. While providing reference audio that conveys a target prosodic style for the speech to be synthesized or manually-selecting weights to select the target prosodic style at inference time may effectively disentangle factors of different speaking styles, these approaches are trained on supervised learning models and are not ideal for synthesizing speech from such long-form expressive datasets of input text (e.g., audiobooks).
Implementations herein are directed toward exemplary architectures configured to apply prosodic style embeddings as “virtual” speaking style labels for use in an end-to-end text-to-speech (TTS) model for producing synthesized speech from an input text sequence. As will become apparent, these exemplary architectures can be trained using unsupervised models to learn and predict stylistic renderings from context derived from the input text sequence alone, requiring neither explicit labels during training nor other auxiliary inputs at inference. As such, these implementations are able to capture speaker-independent factors of variation, including speaking style and background noise, from text alone.
Implementations herein are further directed toward a context-prediction system configured to receive additional context features as conditional inputs for predicting stylistic renderings for a current input text sequence. Here, the input text sequence and each context feature may serve as context for predicting a suitable stylistic rendering of the speech synthesized front the input text sequence. The context features may include word embeddings, sentence embeddings, and/or speech tags (e.g., noun, verb, adjective, etc.) As used herein, available context features can include, without limitation, previous/past text, upcoming/future text, and previous-past audio. To put another away, context features may be derived from a text source of the current input text to be synthesized. Additional sources of context features can be obtained from a document structure containing the text to be synthesized, such as title, chapter title, section title, headline, bullet points, etc. In some examples, concepts relating to entities from a concept graph (e.g., Wikipedia) and/or a structured answer representation are sources contextual features. Moreover, in a digital assistant setting, audio/text features derived from a query (or sequence of queries) may be used as contextual features when synthesizing a response, while text of a previous and/or next “turn” in a dialogue may be derived as contextual features for synthesizing corresponding dialogue. Additionally or alternatively, characters and objects (e.g., emoji's) present within a virtual environment may also be sources of contextual features for predicting stylistic renderings for a current input text sequence.
Referring to
To process the input text 104, the subsystem 102 is configured to interact with an end-to-end text-to-speech model 150 that includes a sequence-to-sequence recurrent neural network 106 (hereafter “seq2seq network 106”), a post-processing neural network 108, and a waveform synthesizer 110.
After the subsystem 102 receives input text 104 that includes a sequence of characters in a particular natural language, the subsystem 102 provides the sequence of characters as input to the seq2seq network 106. The seq2seq network 106 is configured to receive the sequence of characters from the subsystem 102 and to process the sequence of characters to generate a spectrogram of a verbal utterance of the sequence of characters in the particular natural language.
In particular, the seq2seq network 106 processes the sequence of characters using (i) an encoder neural network 112, which includes an encoder pre-net neural network 114 and an encoder CBHG neural network 116, and (ii) an attention-based decoder recurrent neural network 118. CBHG is an acronym for Convolutions, Filter Banks and Highway layers, Gated Recurrent Units. Each character in the sequence of characters can be represented as a one-hot vector and embedded into a continuous vector. That is, the subsystem 102 can represent each character in the sequence as a one-hot vector and then generate an embedding, i.e., a vector or other ordered collection of numeric values, of the character before providing the sequence as input to the seq2seq network 106.
The encoder pre-net neural network 114 is configured to receive a respective embedding of each character in the sequence and process the respective embedding of each character to generate a transformed embedding of the character. For example, the encoder pre-net neural network 114 can apply a set of non-linear transformations to each embedding to generate a transformed embedding. In some cases, the encoder pre-net neural network 114 includes a bottleneck neural network layer with dropout to increase convergence speed and improve generalization capability of the system during training.
The encoder CBHG neural network 116 is configured to receive the transformed embeddings from the encoder pre-net neural network 206 and process the transformed embeddings to generate encoded representations of the sequence of characters. The encoder CBHG neural network 112 includes a CBHG neural network 200 (
The attention-based decoder recurrent neural network 118 (herein referred to as “the decoder neural network 118”) is configured to receive a sequence of decoder inputs. For each decoder input in the sequence, the decoder neural network 118 is configured to process the decoder input and the encoded representations generated by the encoder CBHG neural network 116 to generate multiple frames of the spectrogram of the sequence of characters. That is, instead of generating (predicting) one frame at each decoder step, the decoder neural network 118 generates r frames of the spectrogram, with r being an integer greater than one. In many cases, there is no overlap between sets of r frames.
In particular, at decoder step t, at least the last frame of the r frames generated at decoder step t−1 is fed as input to the decoder neural network 118 at decoder step t+1 In some implementations, all of the r frames generated at the decoder step t−1 are fed as input to the decoder neural network 118 at the decoder step t+1. The decoder input for the first decoder step can be an all-zero frame (i.e. a <GO> frame). Attention over the encoded representations is applied to all decoder steps, e.g., using a conventional attention mechanism. The decoder neural network 118 may use a fully connected neural network layer with a linear activation to simultaneously predict r frames at a given decoder step. For example, to predict 5 frames, each frame being an 80-D (80-Dimension) vector, the decoder neural network 118 uses the fully connected neural network layer with the linear activation to predict a 400-D vector and to reshape the 400-D vector to obtain the 5 frames.
By generating r frames at each time step, the decoder neural network 118 divides the total number of decoder steps by r, thus reducing model size, training time, and inference time. Additionally, this technique substantially increases convergence speed, i.e., because it results in a much faster (and more stable) alignment between frames and encoded representations as learned by the attention mechanism. This is because neighboring speech frames are correlated and each character usually corresponds to multiple frames. Emitting multiple frames at a time step allows the decoder neural network 118 to leverage this quality to quickly learn how to, i.e., be trained to, efficiently attend to the encoded representations during training.
The decoder neural network 118 may include one or more gated recurrent unit neural network layers. To speed up convergence, the decoder neural network 118 may include one or more vertical residual connections. In some implementations, the spectrogram is a compressed spectrogram such as a mel-scale spectrogram. Using a compressed spectrogram instead of, for instance, a raw spectrogram may reduce redundancy, thereby reducing the computation required during training and inference.
The post-processing neural network 108 is configured to receive the compressed spectrogram and process the compressed spectrogram to generate a waveform synthesizer input. To process the compressed spectrogram, the post-processing neural network 108 includes the CBMG neural network 200 (
In some implementations, the post-processing neural network 108 and the sequence-to-sequence recurrent neural network 106 are trained jointly. That is, during training, the system 100 (or an external system) trains the post-processing neural network 108 and the seq2seq network 106 on the same training dataset using the same neural network training technique, e.g., a gradient descent-based training technique. More specifically, the system 100 (or an external system) can backpropagate an estimate of a gradient of a loss function to jointly adjust the current values of all network parameters of the post-processing neural network 108 and the seq2seq network 106. Unlike conventional systems that have components that need to be separately trained or pre-trained and thus each component's errors can compound, systems that have the post-processing neural network 108 and seq2seq network 106 that are jointly trained are more robust (e.g., they have smaller errors and can be trained from scratch). These advantages enable the training of the end-to-end text-to-speech model 150 on a very large amount of rich, expressive yet often noisy data found in the real world.
The waveform synthesizer 110 is configured to receive the waveform synthesizer input, and process the waveform synthesizer input to generate a waveform of the verbal utterance of the input sequence of character s in the particular natural language. In some implementations, the waveform synthesizer is a Griffin-Lim synthesizer. In some other implementations, the waveform synthesizer is a vocoder. In some other implementations, the waveform synthesizer is a trainable spectrogram to waveform inverter. After the waveform synthesizer 110 generates the waveform, the subsystem 102 can generate speech 120 using the waveform and provide the generated speech 120 for playback, e.g., on a user device, or provide the generated waveform to another system to allow the other system to generate and play back the speech. In some examples, a WaveNet neural vocoder replaces the waveform synthesizer 110. A WaveNet neural vocoder may provide different audio fidelity of synthesized speech in comparison to synthesized speech produced by the waveform synthesizer 110.
The 1-D convolutional subnetwork 208 may include a bank of 1-D convolutional filters 204 followed by a max pooling along time layer with a stride of one 206. The bank of 1-D convolutional filters 204 may include K sets of 1-D convolutional filters, in which the k-th set includes Ck filters each having a convolution width of k. The 1-D convolutional subnetwork 208 is configured to receive an input sequence 202, for example, transformed embeddings of a sequence of characters that are generated by an encoder pre-net neural network 114 (
After the 1-D convolutional subnetwork 208 generates the subnetwork outputs, the residual connection 210 is configured to combine the subnetwork outputs with the original input sequence 202 to generate convolution outputs. The highway network 212 and the bidirectional recurrent neural network 214 are then configured to process the convolution outputs to generate encoded representations of the sequence of characters. In particular, the highway network 212 is configured to process the convolution outputs to generate high-level feature representations of the sequence of characters. In some implementations, the highway network includes one or more fully-connected neural network layers.
The bidirectional recurrent neural network 214 is configured to process the high-level feature representations to generate sequential feature representations of the sequence of characters. A sequential feature representation represents a local structure of the sequence of characters around a particular character. A sequential feature representation may include a sequence of feature vectors. In some implementations, the bidirectional recurrent neural network is a gated recurrent unit neural network.
During training, one or more of the convolutional filters of the 1-D convolutional subnetwork 208 can be trained using batch normalization method, which is described in detail in S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv: 1502.03167, 2015. In some implementations, one or more convolutional fillers in the CBHG neural network 200 are non-causal convolutional filters, i.e., convolutional filters that, at a given time step T, can convolve with surrounding inputs in both directions (e.g., T−1, T−2 and T+1, T+2, . . . etc.). In contrast, a causal convolutional filter can only convolve with previous inputs ( . . . T−1, T−2, etc.). In some other implementations, all convolutional filters in the CBHG neural network 200 are non-causal convolutional filters. The use of non-causal convolutional filters, batch normalization, residual connections, and max pooling along time layer with stride one improves the generalization capability of the CBHG neural network 200 on the input sequence and thus enables the text-to-speech conversion system to generate high-quality speech.
At operation 302, the method 300 includes the system receiving a sequence of characters in a particular natural language, and at operation 304, the method 300 includes the system providing the sequence of characters as input to a sequence-to-sequence (seq2seq) recurrent neural network 106 to obtain as output a spectrogram of a verbal utterance of the sequence of characters in the particular natural language. In some implementations, the spectrogram is a compressed spectrogram, e.g., a mel-scale spectrogram. In particular, the seq2seq recurrent neural network 106 processes the sequence of characters to generate a respective encoded representation of each of the characters in the sequence using an encoder neural network 112 that includes an encoder pre-net neural network 114 and an encoder CBHG neural network 116.
More specifically, each character in the sequence of characters can be represented as a one-hot vector and embedded into a continuous vector. The encoder pre-net neural network 114 receives a respective embedding of each character in the sequence and processes the respective embedding of each character in the sequence to generate a transformed embedding of the character. For example, the encoder pre-net neural network 114 can apply a set of non-linear transformations to each embedding to generate a transformed embedding. The encoder CBHG neural network 116 then receives the transformed embeddings from the encoder pre-net neural network 114 and processes the transformed embeddings to generate the encoded representations of the sequence of characters.
To generate a spectrogram of a verbal utterance of the sequence of characters, the seq2seq recurrent neural network 106 processes the encoded representations using an attention-based decoder recurrent neural network 118. In particular, the attention-based decoder recurrent neural network 118 receives a sequence of decoder inputs. The first decoder input in the sequence is a predetermined initial frame. For each decoder input in the sequence, the attention-based decoder recurrent neural network 118 processes the decoder input and the encoded representations to generate r frames of the spectrogram, in which r is an integer greater than one. One or more of the generated r frames can be used as the next decoder input in the sequence. In other words, each other decoder input in the sequence is one or more of the r frames generated by processing a decoder input that precedes the decoder input in the sequence.
The output of the attention-based decoder recurrent neural network thus includes multiple sets of frames that form the spectrogram, in which each set includes r frames. In many cases, there is no overlap between sets of r frames. By generating r frames at a time, the total number of decoder steps performed by the attention-based decoder recurrent neural network is reduced by a factor of r, thus reducing training and inference time. This technique also helps to increase convergence speed and learning rate of the attention-based decoder recurrent neural network and the system in general.
At operation 306, the method 300 includes generating speech using the spectrogram of the verbal utterance of the sequence of characters in the particular natural language. In some implementations, when the spectrogram is a compressed spectrogram, the system can generate a waveform from the compressed spectrogram and generate speech using the waveform.
At operation 308, the method 300 includes providing the generated speech for playback. For example, the method 300 may provide the generated speech for playback by transmitting the generated speech from the system to a user device (e.g., audio speaker) over a network for playback.
In the example shown, the reference audio signal 402 may be represented as spectrogram slices having a length LR and dimension DR. The spectrogram slices associated with the reference audio signal 402 may be indicative of a Mel-warped spectrum. In the example shown, the reference encoder 400 includes a six-layer convolutional layer network 404 with each layer including 3×3 filters with 2×2 stride, SAME padding, and ReLU activation. Batch normalization is applied to every layer and the number of filters in each layer doubles at half the rate of downsampling: 32, 32, 64, 128, 128. A recurrent neural network 410 with a single 128-width Gated Recurrent Unit (GRU-RNN) layer receives the output 406 from the last convolutional layer and outputs a 128-dimensional output 412 applied to a fully connected layer 420 followed by an activation function 430 that outputs the predicted prosody embedding PE 450. The recurrent neural network 410 may include other types of bidirectional recurrent neural networks.
The choice of activation function 430 (e.g., a softmax or tanh) in reference encoder 400 may constrain the information contained in the prosody embedding PE 450 and help facilitate learning by controlling the magnitude of the prosody embedding PE 450. Moreover, the choice of the length LR and the dimension DR of the reference audio signal 402 input to the reference encoder 400 impacts different aspects of prosody learned by the encoder 400. For instance, a pitch track representation may not permit modeling of prominence in some language since the encoder does not contain energy information, while a Mel Frequency Cepstral Coefficient (MFCC) representation may, at least to some degree depending on the number of coefficients trained, prevent the encoder 400 from modeling intonation.
While the prosody embedding PE 450 output from the reference encoder 400 can be used in a multitude of different TTS architectures for producing synthesized speech, a seed signal (e.g., reference audio signal 402) is required for producing the prosody embedding PE 450 at inference time. For instance, the seed signal could be a “Say it like this” reference audio signal 402. Alternatively, to convey synthesized speech with an intended prosody/style, some TTS architectures can be adapted to use a manual style embedding selection at inference time instead of using the reference encoder 400 to output a prosody embedding PE 450 from a seed signal. Referring to
During training, the text-prediction system 500 of
In some implementations, the style token layer 510 includes the style token layer disclosed by “Style Tokens. Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthsis”, arXiv preprint arXiv: 1803.09017, Mar. 23, 2018, the contents of which are incorporated by reference in their entirety. The style token layer 510 includes a style attention module 512 configured to learn in an unsupervised manner, during training, a convex combination of trainable style tokens 514, 514a-n that represent the prosody embedding PE 450 output from the reference encoder 400. Here, the style token layer 510 uses the prosody embedding PE 450 as a query vector to the attention module 512 configured to learn a similarity measure between the prosody embedding and each style token 514 in a bank of randomly initialized style token 514, 514a-n. The style tokens 514 (also referred to as ‘style embeddings’) may include corresponding embeddings shared across ail training sequences. Thus, the attention module 512 outputs a set of combination weights 516, 516a-n that represent the contribution of each style token 514 to the encoded prosody embedding PE 450. The attention module 512 may determine the combination weights 516 by normalizing the style tokens 514 via a softmax activation. The resulting style embedding SE 550 output from the style token layer 510 corresponds to the weighted sum of the style tokens 514. Each style token 514 may include a dimensionality that matches a dimensionality of a state of the text encoder 502. While the examples show the style token layer 510 including five (5) style tokens 514, the style token layer 510 may include any number of style tokens 514. In some examples, ten (10) style tokens 514 is selected to provide a rich variety of prosodic dimensions in the training data.
In some configurations, the style token layer 510 is trained jointly with the TTS model 650 and the text-prediction model 520. In other configurations, the style token layer 510 and the TTS model 650 are trained separately, while the style token layer 510 and the text-prediction model 520 are trained jointly.
With continued reference to
The text-prediction networks 520a, 520b provide two text-prediction pathways for predicting style embeddings 550 during inference based on input text 502. Each of these networks 520a, 520b may be trained jointly by using operators configured to stop gradient flow. Referring to
With continued reference to
Referring to
In some implementations, the text-prediction model 520 determine an L1 loss between the predicted style embedding SE 550P and the target style embedding SE 550T output front the style token layer 510. Thereafter, the model 520b may be configured to stop gradient flow to prevent backpropagation of any text prediction error through the style token layer 510. Moreover, the cross-entropy loss can be added to the final loss of the TTS model 650 during training.
With continued reference to
Generally, the context model 610 is configured to receive and process the one or more context features 602 to generate a context embedding 612 associated with the current input text 502. The current input text 502 refers to a sequence of characters to be synthesized into expressive speech 680. The current input text 502 could be a single sentence in some examples, while in other examples, the current input text 502 includes a paragraph. The sequence of characters in the current input text 502 and the resulting synthesized expressive speech 680 of the current input text 502 are associated with a particular language. Moreover, each context features 602 may be derived from a text source 800 (
The text-prediction model 520 may include the text-prediction models 520 described above with reference to
In some examples, the TTS model 650 is configured to receive the current input text 502 (e.g., from the text source 800), receive the style embedding SE 550P predicted by the text-prediction model 520, and process the input text 502 and the style embedding SE 550P to generate the output audio signal 670 of expressive speech of the current input text 502. Here, the output audio signal 670 has a specific prosody and style specified by the style embedding SE 550.
The TTS model 650 includes the encoder 652, a concatenator 654, an attention module 656, the decoder 658, and a synthesizer 475. In some implementations, the TTS model 650 includes the TTS model 150 of
The input text 502 may include phoneme inputs produced by a text normalization front-end and lexicon since prosody is being addressed, rather than the model's ability to learn pronunciation front graphemes. However, the input text 502 may additionally or alternatively include grapheme inputs. The attention model 656 and the decoder 658 may collectively include the attention-based decoder recurrent neural network 118 (
During the training stage,
The context features 602 are derived from the text source 800 of the current input text 502, wherein the context model 610 is configured to generate a context embedding 612 associated with the current input text 502 by processing the context features 602 and feed the context embedding 612 to the text-prediction model 520. For instance, in the above example, the context features 602 are derived from the text document, and may include, without limitation, the current input text 502 (Tt) to be synthesized, previous text (Tt−1) from the text source that precedes the current input text, previous speech synthesized 680 (e.g., previous output audio signal 670 (yt−1) from the previous text, upcoming text (Tt+1) from the text source that follows the current input text, a previous style embedding predicted by the text-prediction network 520 based on the previous text and a previous context embedding associated with the previous text. Additionally, the one or more context features 602 derived from the text document may include at least one of: a title of the text document; a title of a chapter in the text document; a title of a section in the text document, a headline in the text document; one or more bullet points in the text document, entities from a concept graph extracted from the text document, or one or more structured answer representations extracted from the text document. In some examples, the context features 602 associated with text (e.g., current input text, previous text, upcoming text, etc.) include features extracted from the text that may include, without limitation, vowel-level embeddings, word-level embeddings, sentence-level embeddings, paragraph-level embeddings, and/or speech tags (e.g., noun, verb, adjective, etc.) for each word.
Additionally, the text-prediction model 520 receives each training sample of input text 502 corresponding to the transcript of the reference audio signal 402 and the corresponding context embedding 612 generated for each training sample of input text 502 for predicting combination weights (CW) 516P (
For text documents, context features 602 may include monologue context such as previous text (e.g., N sentences prior to the current text 502), previous audio 670 corresponding to the previous text, upcoming text (e.g., N sentences after the current text 502) For instance, previous text describing a sad event can help predict a style embedding for synthesizing expressive speech of current text that conveys a prosody/style indicative of sad emotion. Context features 602 may also be derived from document structure such as title, chapter title, section title, a headline, bullet points, etc. Text documents may also include concepts such as entities from a concept graph (e.g., a Wikipedia entry) that may be extracted as context features 602.
For a query-response system (e.g., question and answering), the context features 602 may include audio/text, features from a spoken query or text features from a textual query which the current text 502 corresponds to a transcript of a response to be synthesized into expressive speech. The context features 602 may include the audio/text features from a sequence of queries that leads to a current response. Additionally or alternatively, the context features 602 may be extracted from a structured answer representation of the response used by a digital assistant. For a dialogue transcript (turn taking), the context features 602 may include previous text features of a previous “turn” in a dialogue and/or upcoming text features of a next “turn” in the dialogue. A text source 800 corresponding to a virtual environment may provide context features 802 corresponding to any characters and/or objects present in the virtual environment.
Referring back to the inference stage of
During a second step of the two-step training procedure, the decoupled context module 710 uses the target style embedding SE(t) produced by the pre-trained style encoder 750 for each time step (t) for predicting a corresponding style embedding SE(t). As with the full contextual TTS network 700a of
By decoupling the context module 710 from the TTS model 650, the networks 700c, 700d each provide a good training efficiency, wherein the ability to track long-term context is only available in network 700c. Additionally, decoupling the TTS model 650 permits using the TTS model 650 for both a context mode (as described in
A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an “application,” an “app,” or a “program.” Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.
The non-transitory memory may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. The non-transitory memory may be volatile and/or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot, programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.
The computing device 900 includes data processing hardware (e.g., a processor) 910, memory 920, a storage device 930, a high-speed interface/controller 940 connecting to the memory 920 and high-speed expansion ports 950, and a low speed interface/controller 960 connecting to a low speed bus 970 and a storage device 930. The computing device 900 may provide (via execution on the data processing hardware 910) the text-to speech conversion system 100, the TTS models 150, 650, the reference encoder 400, the deterministic reference encoder 400, the context model 610, and the text-prediction model 520. Each of the components 910, 920, 930, 940, 950, and 960, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 910 can process instructions for execution within the computing device 900, including instructions stored in the memory 920 or on the storage device 930 to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display 980 coupled to high speed interface 940. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 900 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).
The memory 920 stores information non-transitorily within the computing device 900. The memory 920 may be a computer-readable medium, a volatile memory unites), or non-volatile memory unit(s). The non-transitory memory 920 may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device 900. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.
The storage device 930 is capable of providing mass storage for the computing device 900. In some implementations, the storage device 930 is a computer-readable medium. In various different implementations, the storage device 930 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 920, the storage device 930, or memory on processor 910.
The high speed controller 940 manages bandwidth-intensive operations for the computing device 900, while the low speed controller 960 manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller 940 is coupled to the memory 920, the display 980 (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports 950, which may accept various expansion cards (not shown). In some implementations, the low-speed controller 960 is coupled to the storage device 930 and a low-speed expansion port 990. The low-speed expansion port 990, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.
The computing device 900 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 900a or multiple times in a group of such servers 900a, as a laptop computer 900b, or as part of a rack server system 900c.
At operation 1004, the method 1000 includes generating, by the data processing hardware 910, using a context model 610, a context embedding 612 associated with the current input text 502 by processing one or more context features 602 derived from the text source 800. At operation 1006, the method 1000 includes predicting, by the data processing hardware 910, using a text-prediction network (also referred to as “text-prediction model) 520, a style embedding 550 for the current input text 502 by processing the current input text 502 and the context embedding 612 associated with the current input text 502. Notably, the style embedding 550 predicted by the text-prediction network 520 specifies a specific prosody and/or style for synthesizing the current input text 502 into expressive speech 680. The style embedding 550 may be predicted by either one of the text-prediction network 520a of
At operation 1008, the method 1000 also includes generating, by the data processing hardware 910, using the TTS model 650, the output audio signal 670 of expressive speech 680 of the current input text 502 by processing the style embedding 550 and the current input text 502. Here, the output audio signal 670 has the specific prosody and/or style specified by the style embedding 550. As discussed above, the ITS model 650 (or other system downstream from the model 650) uses a synthesizer 675 to synthesize the resulting expressive speech 680. Thus, the expressive speech 680 refers to synthesized speech.
Various implementations of the systems and techniques described herein can be realized in digital electronic and/or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer readable medium, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.
The processes and logic flows described in this specification can be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks: and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user and optionally a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user, for example, by sending web pages to a web browser on a users client device in response to requests received from the web browser.
A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.
This U.S. patent application claims priority under 35 U.S.C. 119(e) to U.S. Provisional Application 62/882,511, filed on Aug. 3, 2019. The disclosure of this prior application is considered part of the disclosure of this application and is hereby incorporated by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 10475438 | Chicote | Nov 2019 | B1 |
| 20150186359 | Fructuoso | Jul 2015 | A1 |
| 20150364128 | Zhao | Dec 2015 | A1 |
| 20170092259 | Jeon | Mar 2017 | A1 |
| 20180096677 | Pollet | Apr 2018 | A1 |
| 20190122651 | Arik | Apr 2019 | A1 |
| 20200082807 | Kim | Mar 2020 | A1 |
| 20200306641 | Kolen | Oct 2020 | A1 |
| Entry |
|---|
| Wang, Yuxuan, et al. “Tacotron: A fully end-to-end text-to-speech synthesis model.” arXiv preprint arXiv: 1703.10135 164 (2017). (Year: 2017). |
| International Search Report and Written Opinion for the related Application No. PCT/US2020/042416. |
| Daisy Stanton et al: “Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis”, arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Aug. 4, 2018 (Aug. 4, 2018). |
| Yuxuan Wang et al.: “Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-end Speech Synthesis”, arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Mar. 24, 2018 (Mar. 24, 2018). |
| Rj Skerry-Ryan et al.: “Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron”, arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Mar. 24, 2018 (Mar. 24, 2018). |
| Haohan Guo et al : “Conversational End-to-End TTS for Voice Agent”, arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, May 21, 2020 (May 21, 2020), XP081676072, the whole document. |
| Number | Date | Country | |
|---|---|---|---|
| 20210035551 A1 | Feb 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62882511 | Aug 2019 | US |
