Patent Application
20230274100
Text translation into different languages is an important area that supports a wide variety of different tools and applications. The quality of results can differ widely depending on the specific languages, models used in translations, and how such models are trained. Certain models may be trained using zero-shot or few-shot approaches in limited translation instances but may not be sufficiently effective in multilingual translations or other text rewriting situations. For instance, it can be challenging to properly convey a particular style or level of formality in translations across various languages. It is also possible for “accidental translations” to occur, in which a model provides a correct answer but in the wrong language. There may also be significant resource tradeoffs between using larger models versus incurring excessive memory costs. These and other issues can adversely affect multilingual translation in conventional approaches.
Aspects of the technology provide a model-based approach for multilingual text rewriting that is applicable across many languages and across different styles that may include various formality levels or other attributes of the text. The approach enables performance of attribute-controlled machine translation (e.g., dialect-aware translation, formality-controlled translations), as well as the ability to adapt text to a given domain or locale. One aspect involves leveraging a large pretrained multilingual model and fine-tuning it to perform general-purpose multilingual attribute transfer.
The techniques and tools described herein establish that few-shot approaches can naturally extend to a multilingual setting. Multilingual models can endow text with additional attributes in one language using exemplars of that attribute from another language, according to a zero-shot style transfer. The model presented is capable of manipulating both language and textual attributes jointly. This approach can support zero-shot formality-sensitive translation, with no labeled data in the target language. There are many computer-based applications in which this technology can be greatly beneficial, including translation services, healthcare apps, video streaming services, videoconferencing, creative writing apps, etc.
According to one aspect, a system is configured for a multilingual text rewriting model. The system comprises memory configured to store a set of text exemplars in a source language and a set of rewritten texts in a plurality of languages different from the source language, and one or more processing elements operatively coupled to the memory. The one or more processing elements implement a multilingual text rewriter as a neural network having a number of modules including: a corruption module configured to generate a corrupted version of an input text sequence based on the set of text exemplars in the source language; an encoder module comprising an encoder neural network configured to receive the corrupted version of the input text sequence and to generate a set of encoded representations of the corrupted version of the input text sequence; a style extractor module configured to extract a set of style vector representations associated with the input text sequence; and a decoder module comprising a decoder neural network configured to receive the set of encoded representations and the set of style vector representations and to output the set of rewritten texts in the plurality of languages. Each style vector representation of the set is added element-wise to one of the set of encoded representations. A set of model weights is shared by the encoder module and the style extractor module, and a unique token is appended to the input text sequence for style extraction instead of mean-pooling all of the style vector representations in the set. The system is configured to provide rewritten text in selected ones of the plurality of languages according to a change in a least a sentiment or a formality of the input text sequence.
In one example, the encoder module, style extractor module and the decoder module are configured as transformer stacks initialized from a common pretrained language model. Alternatively or additionally, both the encoder module and the decoder module are attention-based neural network modules. Alternatively or additionally, the corruption module employs a corruption function C for a given pair of non-overlapping spans (s1, s2) of the input text sequence, so that the model is capable of reconstructing span s2 from C(s2) and Vs1, where Vs1 is an attribute vector of span s1. Here, the corruption function C may be as follow: C:=ƒ(⋅, source language, −Vs1). Alternatively or additionally, the style extractor module may include a set of style extractor elements including a first subset configured to operate on different style exemplars and a second subset configured to operate on the input text sequence prior to corruption by the corruption module.
During training, the corruption module may be configured to employ at least one of token-level corruption or style-aware back-translation corruption. Alternatively or additionally, the set of model weights may be initialized with the weights of a pretrained text-to-text model. Alternatively or additionally, the model weights for the style extractor module may not be tied to the weights of the encoder module during training.
The system may be configured to extract pairs of random non-overlapping spans of tokens from each line of text in a given text exemplar, and configured to use a first-occurring span in the text as an exemplar of attributes of a second span in the text. Alternatively or additionally, a cross-lingual learning signal can be added to a training objective for training the model. Alternatively or additionally, the encoder module may be configured to use negation of a true exemplar vector associated with the input text sequence as an attribute vector in a forward pass operation. Alternatively or additionally, the decoder module may be configured to receive a set of stochastic tuning ranges that provide conditioning for the decoder module.
In an example, the set of style vector representations corresponds to a set of attributes associated with the input text sequence, and the system is configured to input a set of exemplars illustrating defined attributes and to extract corresponding attribute vectors for use in rewriting the input text sequence into the plurality of languages according to the defined attributes. In this case, the system may be configured to form an attribute delta vector including a scale factor, in which the attribute delta vector is added to the set of encoded representations before processing by the decoder module.
According to another aspect, a computer-implemented method for providing multilingual text rewriting according to a machine learning model is provided. The method comprises: obtaining an input text sequence based on a set of text exemplars in a source language; generating, by a corruption module, a corrupted version of the input text sequence; receiving, by an encoder neural network of an encoder module, the corrupted version of the input text sequence; generating, by the encoder module, a set of encoded representations of the corrupted version of the input text sequence; extracting, by a style extractor module, a set of style vector representations associated with the input text sequence; receiving, by a decoder neural network of a decoder module, the set of encoded representations and the set of style vector representations, in which each style vector representation of the set is added element-wise to one of the set of encoded representations; outputting, by the decoder module, a set of rewritten texts in a plurality of languages different from the source language; and storing the rewritten text in selected ones of the plurality of languages according to a change in a least a sentiment or a formality of the input text sequence. A set of model weights is shared by the encoder module and the style extractor module, and a unique token is appended to the input text sequence for style extraction instead of mean-pooling all of the style vector representations in the set.
In one example, generating the corrupted version of the input text sequence is performed according to a corruption function C for a given pair of non-overlapping spans (s1, s2) of the input text sequence, so that the model is capable of reconstructing span s2 from C(s2) and Vs1, where Vs1 is an attribute vector of span s1. Alternatively or additionally to any of the above, the method further comprising: extracting pairs of random non-overlapping spans of tokens from each line of text in a given text exemplar; and using a first-occurring span in the text as an exemplar of attributes of a second span in the text. Alternatively or additionally, the method further comprises adding a cross-lingual learning signal to a training objective for training the model. Alternatively or additionally, the method further comprises applying a set of stochastic tuning ranges to selectively condition the decoder module. In another example, the set of style vector representations corresponds to a set of attributes associated with the input text sequence. Here, outputting the set of rewritten texts includes generating one or more versions of the input text sequence in selected ones of the plurality of languages according to the set of attributes.
The technology provides an encoder-decoder architectural approach with attribute extraction to train rewriter models that can be used in “universal” textual rewriting across many different languages. A cross-lingual learning signal is incorporated into the training approach. Certain training processes do not employ any exemplars. This approach enables not just straight translation, but also the ability to create new sentences with different attributes.
Aspects of the technology are used to test the conjecture that large models trained on massive multilingual corpora should be able to “identify” a textual attribute (e.g., formality) given just a few exemplars in one language (e.g., English), and “apply” the underlying concept to a generative task in another language in a zero-shot fashion (e.g., formality transfer in Chinese). This can be tested by leveraging a large pretrained multilingual model and fine-tuning it to perform general-purpose multilingual attribute transfer.
Note that in the following discussion, the terms “attributes” and “styles” are used interchangeably. The model can go beyond traditional tasks associated with style transfer, for example extending to translation (language transfer) and style-conditioned translation. The term “exemplars” is used to denote inputs show-casing a particular attribute.
The machine translation rewrite model discussed herein may employ a neural network such as a convolutional neural network (CNN) or a recurrent neural network (RNN), e.g., a bidirectional long short-term memory (Bi-LSTM) RNN. Alternatively or additionally, the model may employ a self-attention architecture. This may include an approach such as the Transformer neural network encoder-decoder architecture. An exemplary Transformer-type architecture is shown in
System 200 of
System 200 can perform any of a variety of tasks that require processing sequential inputs to generate sequential outputs. System 200 includes an attention-based sequence transduction neural network 206, which in turn includes an encoder neural network 208 and a decoder neural network 210. The encoder neural network 208 is configured to receive the input sequence 202 and generate a respective encoded representation of each of the network inputs in the input sequence. Generally, an encoded representation is a vector or other ordered collection of numeric values. The decoder neural network 210 is then configured to use the encoded representations of the network inputs to generate the output sequence 204. Generally, both the encoder 208 and the decoder 210 are attention-based. In some cases, neither the encoder nor the decoder includes any convolutional layers or any recurrent layers. The encoder neural network 208 includes an embedding layer (input embedding) 212 and a sequence of one or more encoder subnetworks 214. The encoder neural 208 network may N encoder subnetworks 214.
The embedding layer 212 is configured, for each network input in the input sequence, to map the network input to a numeric representation of the network input in an embedding space, e.g., into a vector in the embedding space. The embedding layer 212 then provides the numeric representations of the network inputs to the first subnetwork in the sequence of encoder subnetworks 214. The embedding layer 212 may be configured to map each network input to an embedded representation of the network input and then combine, e.g., sum or average, the embedded representation of the network input with a positional embedding of the input position of the network input in the input order to generate a combined embedded representation of the network input. In some cases, the positional embeddings are learned. As used herein, “learned” means that an operation or a value has been adjusted during the training of the sequence transduction neural network 206. In other cases, the positional embeddings may be fixed and are different for each position.
The combined embedded representation is then used as the numeric representation of the network input. Each of the encoder subnetworks 214 is configured to receive a respective encoder subnetwork input for each of the plurality of input positions and to generate a respective subnetwork output for each of the plurality of input positions. The encoder subnetwork outputs generated by the last encoder subnetwork in the sequence are then used as the encoded representations of the network inputs. For the first encoder subnetwork in the sequence, the encoder subnetwork input is the numeric representations generated by the embedding layer 212, and, for each encoder subnetwork other than the first encoder subnetwork in the sequence, the encoder subnetwork input is the encoder subnetwork output of the preceding encoder subnetwork in the sequence.
Each encoder subnetwork 214 includes an encoder self-attention sub-layer 216. The encoder self-attention sub-layer 216 is configured to receive the subnetwork input for each of the plurality of input positions and, for each particular input position in the input order, apply an attention mechanism over the encoder subnetwork inputs at the input positions using one or more queries derived from the encoder subnetwork input at the particular input position to generate a respective output for the particular input position. In some cases, the attention mechanism is a multi-head attention mechanism as shown. In some implementations, each of the encoder subnetworks 214 may also include a residual connection layer that combines the outputs of the encoder self-attention sub-layer with the inputs to the encoder self-attention sub-layer to generate an encoder self-attention residual output and a layer normalization layer that applies layer normalization to the encoder self-attention residual output. These two layers are collectively referred to as an “Add & Norm” operation in
Some or all of the encoder subnetworks can also include a position-wise feed-forward layer 218 that is configured to operate on each position in the input sequence separately. In particular, for each input position, the feed-forward layer 218 is configured receive an input at the input position and apply a sequence of transformations to the input at the input position to generate an output for the input position. The inputs received by the position-wise feed-forward layer 218 can be the outputs of the layer normalization layer when the residual and layer normalization layers are included or the outputs of the encoder self-attention sub-layer 216 when the residual and layer normalization layers are not included. The transformations applied by the layer 218 will generally be the same for each input position (but different feed-forward layers in different subnetworks may apply different transformations).
In cases where an encoder subnetwork 214 includes a position-wise feed-forward layer 218 as shown, the encoder subnetwork can also include a residual connection layer that combines the outputs of the position-wise feed-forward layer with the inputs to the position-wise feed-forward layer to generate an encoder position-wise residual output and a layer normalization layer that applies layer normalization to the encoder position-wise residual output. As noted above, these two layers are also collectively referred to as an “Add & Norm” operation. The outputs of this layer normalization layer can then be used as the outputs of the encoder subnetwork 214.
Once the encoder neural network 208 has generated the encoded representations, the decoder neural network 210 is configured to generate the output sequence in an auto-regressive manner. That is, the decoder neural network 210 generates the output sequence, by at each of a plurality of generation time steps, generating a network output for a corresponding output position conditioned on (i) the encoded representations and (ii) network outputs at output positions preceding the output position in the output order. In particular, for a given output position, the decoder neural network generates an output that defines a probability distribution over possible network outputs at the given output position. The decoder neural network can then select a network output for the output position by sampling from the probability distribution or by selecting the network output with the highest probability.
Because the decoder neural network 210 is auto-regressive, at each generation time step, the decoder network 210 operates on the network outputs that have already been generated before the generation time step, i.e., the network outputs at output positions preceding the corresponding output position in the output order. In some implementations, to ensure this is the case during both inference and training, at each generation time step the decoder neural network 210 shifts the already generated network outputs right by one output order position (i.e., introduces a one position offset into the already generated network output sequence) and (as will be described in more detail below) masks certain operations so that positions can only attend to positions up to and including that position in the output sequence (and not subsequent positions). While the remainder of the description below describes that, when generating a given output at a given output position, various components of the decoder 210 operate on data at output positions preceding the given output positions (and not on data at any other output positions), it will be understood that this type of conditioning can be effectively implemented using shifting.
The decoder neural network 210 includes an embedding layer (output embedding) 220, a sequence of decoder subnetworks 222, a linear layer 224, and a softmax layer 226. In particular, the decoder neural network can include N decoder subnetworks 222. However, while the example of
In some implementations, the embedding layer 220 is configured to map each network output to an embedded representation of the network output and combine the embedded representation of the network output with a positional embedding of the output position of the network output in the output order to generate a combined embedded representation of the network output. The combined embedded representation is then used as the numeric representation of the network output. The embedding layer 220 generates the combined embedded representation in the same manner as described above with reference to the embedding layer 212.
Each decoder subnetwork 222 is configured to, at each generation time step, receive a respective decoder subnetwork input for each of the plurality of output positions preceding the corresponding output position and to generate a respective decoder subnetwork output for each of the plurality of output positions preceding the corresponding output position (or equivalently, when the output sequence has been shifted right, each network output at a position up to and including the current output position). In particular, each decoder subnetwork 222 includes two different attention sub-layers: a decoder self-attention sub-layer 228 and an encoder-decoder attention sub-layer 230. Each decoder self-attention sub-layer 228 is configured to, at each generation time step, receive an input for each output position preceding the corresponding output position and, for each of the particular output positions, apply an attention mechanism over the inputs at the output positions preceding the corresponding position using one or more queries derived from the input at the particular output position to generate a updated representation for the particular output position. That is, the decoder self-attention sub-layer 228 applies an attention mechanism that is masked so that it does not attend over or otherwise process any data that is not at a position preceding the current output position in the output sequence.
Each encoder-decoder attention sub-layer 230, on the other hand, is configured to, at each generation time step, receive an input for each output position preceding the corresponding output position and, for each of the output positions, apply an attention mechanism over the encoded representations at the input positions using one or more queries derived from the input for the output position to generate an updated representation for the output position. Thus, the encoder-decoder attention sub-layer 230 applies attention over encoded representations while the decoder self-attention sub-layer 228 applies attention over inputs at output positions.
In the example of
Some or all of the decoder subnetwork 170 also include a position-wise feed-forward layer 232 that is configured to operate in a similar manner as the position-wise feed-forward layer 218 from the encoder 208. In particular, the layer 232 is configured to, at each generation time step: for each output position preceding the corresponding output position: receive an input at the output position, and apply a sequence of transformations to the input at the output position to generate an output for the output position. The inputs received by the position-wise feed-forward layer 232 can be the outputs of the layer normalization layer (following the last attention sub-layer in the subnetwork 222) when the residual and layer normalization layers are included or the outputs of the last attention sub-layer in the subnetwork 222 when the residual and layer normalization layers are not included. In cases where a decoder subnetwork 222 includes a position-wise feed-forward layer 232, the decoder subnetwork can also include a residual connection layer that combines the outputs of the position-wise feed-forward layer with the inputs to the position-wise feed-forward layer to generate a decoder position-wise residual output and a layer normalization layer that applies layer normalization to the decoder position-wise residual output. These two layers are also collectively referred to as an “Add & Norm” operation. The outputs of this layer normalization layer can then be used as the outputs of the decoder subnetwork 222.
At each generation time step, the linear layer 224 applies a learned linear transformation to the output of the last decoder subnetwork 222 in order to project the output of the last decoder subnetwork 222 into the appropriate space for processing by the softmax layer 226. The softmax layer 226 then applies a softmax function over the outputs of the linear layer 224 to generate the probability distribution (output probabilities) 234 over the possible network outputs at the generation time step. The decoder 210 can then select a network output from the possible network outputs using the probability distribution.
One baseline configuration that can be employed with aspects of the technology employs an English-language few-shot style transfer model encoder decoder Transformer, which is equipped with an additional “style extractor” that is a variation of the encoder neural network 208 discussed above. This configuration removes the need for training labels, and offers a single model that can target an unrestricted set of style attributes. Aspects of this approach are discussed in detail in “TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling”, by Riley et al., Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, © 2021, the entire disclosure of which is incorporated herein by reference.
Here, the style extractor takes exemplars of a style as input and extracts a fixed-width “style vector” by mean-pooling the output embeddings across tokens. These style vectors (and combinations thereof) can then be added element-wise to the encoded representations of a given input before being passed to the decoder in order to induce targeted style changes.
The training configuration 300 shows an architecture for few-shot style transfer. Encoder 302, decoder 304 and style extractor 306 are transformer stacks initialized from a pretrained language model such as T5, which is trained using teacher forcing. During training, the model reconstructs a corrupted input provided by corruption block 308, conditioned on a fixed-width “style vector” extracted from the preceding sentence. The corruption block 308 may employ two types of corruption: token-level and style-aware back-translation. In the former, a noise function is applied independently to each token, with a probability of either dropping a token, replacing it with one of the examples in the current batch at the same position or keeping it as is. In the latter, the model is used in inference mode to change the style of the input to match the style of a random sentence in the batch, and the resulting output is used as a corrupted version of the input. The corruption block 308 corrupts the input by (i) dropping, (ii) replacing, and/or (iii) shuffling tokens, which may be performed in that order. The training occurs in an unsupervised fashion, so the model can handle arbitrary styles at inference time. To compensate for the lack of training labels, this architecture relies on the intuition that style is a “slow-moving feature”, consistent across long spans of text.
One or more stochastic tuning ranges 310 can be used to provide extra conditioning for the decoder 304, and enable fine-grained control of inference. This can include different “add” and “delete” rates. By way of example, for every input/output pair during training, the system can calculate the proportions of tokens that were added and deleted. The “add rate” is the proportion of output tokens absent from the input, and the “delete rate” is the proportion of input tokens absent from the output. These rates can be provided to the decoder as ranges covering but not necessarily centered on the “true” rates. For instance, each range width may be sampled uniformly from [0,1], and uniformly sample the “alignment” of the true rate within the range. In this approach, the final ranges are clipped to [0,1], and a vector containing the upper and lower bound of each range is prepended to the encoder hidden state sequence. This approach provides more flexibility at inference time, so the system can enforce tight or loose constraints on each rate.
To incorporate the style vector into the rest of the model, it is added to each of the final encoder hidden states. The weights of the model may be initialized with those of a pretrained (e.g., T5) model. Both the style extractor and the encoder can be initialized from the pretrained encoder, but the weights need not be tied during training.
As shown in the example inference configuration 320 of
In order to make the model robust and applicable to “universal” text rewriting, one aspect of the technology includes not separating the attribute extraction module (e.g., style extractor 306 in
While one approach that is focused on a particular language employs a sentence-splitting algorithm that is based on detecting English punctuation, it has been found that such an approach can lead to problems for some languages. In particular, sentence-splitting during preprocessing of non-English text may discard any data that does not include ASCII punctuation. For languages like Chinese and Thai, which do not typically use ASCII punctuation, this would filter out most well-formed text, leaving mainly text in the wrong language, or a non-natural language (e.g., Javascript code). This could adversely affect system operation, so a more language-agnostic approach has been adopted. Here, the system extracts pairs of random non-overlapping spans of tokens from each line of text, and uses the first-occurring span in the text as an exemplar of the attributes of the second span. This approach allows the system to retain all data, is independent of language, and still exploits the “slow-moving feature” intuition discussed above.
In one scenario, the system starts with an mT5 checkpoint as the initial model, for instance the XL variant. While it is possible to use style-aware back-translation, according to one aspect of the technology an additional cross-lingual learning signal is added to the training objective by forcing the model to not only perform an attribute transfer procedure but also to translate the sentence to a baseline language such as English. English is particularly suitable due to the availability of parallel data between English and many other languages. To make this objective more similar to the cycle-consistency found in traditional back-translation, the system can use the negation of the “true” exemplar vector associated with the input (as opposed to leveraging a random exemplar) as the attribute vector in the forward pass. More explicitly, let ƒ be the function which takes a triplet of (text, language, attribute vector), then proceed to pass the text through the encoder, add the attribute vector element-wise to the encoded representations, then decodes the result into the given language through sampling. For a given pair (s1, s2) of non-overlapping spans, one can define the corruption function C as follow:
C:=ƒ(⋅, English, −Vs1)
where Vs1 is the attribute vector of the first span, and the model is tasked with reconstructing s2 from C(s2) and Vs1. The intuition behind the negative exemplar is that it is desirable for the model to learn that the changes induced by an attribute vector should be undone when applying the same procedure with the negative of this attribute vector. Note that the system can perform the decoding operation using sampling, e.g., with a temperature of 1.5.
In addition to the cross-lingual treatment of the style-aware back-translation objective, the approach also includes translation as another cross-lingual learning objective by leveraging English-centric translation data for some language pairs. For this task, exemplars are not used.
There are multiple ways one could use the attribute vectors to perform rewriting. Suppose the system is provided with sets of exemplars illustrating attributes A and B, which are used to extract attribute vectors VA and VB, respectively. Given an input x with attribute A, the system should rewrite it to exhibit attribute B. One inference strategy is to first extract the attribute vector Vx of x, then form the following attribute delta vector:
V
A,B,x
:=V
x+λ(VB−VA)
In this case, λ is a scale factor, which may be chosen by the user. The resulting vector VA,B,x then gets added to the encoded representation of x before being passed to the decoder. For within-language tasks, where the output should be in the same language as the input, the system can use this inference strategy.
However, in certain cross-language experiments, it was found the model was more reluctant to change languages when using this strategy. For this reason, according to one aspect of the technology the system does not include the vector Vx in the computation of VA,B,x for cross-language tasks, and instead uses the following equation:
V
A,B,x:=λ(VB−VA)
This section describes certain experiments using the multilingual architecture. First is a discussion of the selection of languages and data sources considered, as well as data preprocessing steps. Next, the performance of the models is measured in the challenging tasks of low-resource and unsupervised machine translation which is viewed as rewriting text in a different language. After this experiment, a multilingual variant of the sentiment transfer task is introduced, which allows for exploration of the interaction between multilinguality and style transfer. Finally, consider the problem of multiple attribute rewriting through two cross-lingual sentence rewriting tasks and zero-shot formality-aware machine translation task for the specific language pair English-Spanish.
According to the experimental setup, the system drew monolingual data from mC4, the same dataset used to train mT5. The “-Latin” languages were removed due to their small data size. Parallel data for 46 of these languages with English was leveraged, by taking all language pairs with more than 500,000 lines of parallel data from the OPUS 100 (dataset, with the exception of Hebrew, Croatian, Bosnian (because their language codes in OPUS 100 were not present in the language codes of mC4), and Japanese. Japanese was excluded as it appears in the sentiment evaluation, and it was desirable to test the model's ability to perform zero-shot attribute transfer in languages where no parallel data is available. As Japanese has a unique script and no genealogical relation with other languages under consideration, it poses a particularly challenging case. The two-character language codes for the utilized languages are as follows: ar, bg, bn, ca, cs, da, de, el, es, et, eu, fa, fi, fr, gl, hi, hu, id, is, it, ko, lt, lv, mg, ink, ml, ms, mt, nl, no, pl, pt, ro, ru, si, sk, sl, sq, sr, sv, th, tr, uk, ur, vi, and zh.
Both parallel and monolingual datasets were used for training the rewriter models. For each batch, a random selection was made to select to sample from either the monolingual or parallel datasets (with equal likelihood), and then uniformly sample from all available datasets in the chosen category.
The monolingual data was preprocessed into input-exemplar pairs, using the approach described above. In addition, any pair was discarded where either element is shorter than five tokens. In the case of parallel data, no exemplars were leveraged, and instead used a vector of zeros. For both data sources, any training example with either input, exemplar or target (in the case of parallel data) longer than 64 tokens was discarded. Language-specific tokens were pre-pended to the input to prime the model for translations. Instead of introducing new tokens, the tokens associated to the target language names in English were reused.
The models were trained in JAX (discussed in “JAX: composable transformations of Python+NumPy programs” by Bradbury et al., 2018). The Adafactor optimizer was used (discussed in “Adafactor: Adaptive learning rates with sublinear memory cost”, in Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596-4604). The accumulated optimizer states were reused from the original mT5 training. A constant learning rate of 1e-3 was used and trained for 25,000 steps, using batches of 512 inputs. Both the Jax and Adafactor references are incorporated herein by reference.
For machine translation, the system utilized BLEU scores (as discussed in “Bleu: a method for automatic evaluation of machine translation”, 2002, in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318). The scores were computed through the sacreBLEU5 library (see “A call for clarity in reporting BLEU scores”, 2018, in Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191). The output was tokenized for the Indic languages using the tokenizer provided by the Indic-NLP library (see “Indic NLP Library, A unified approach to NLP for Indian languages by Anoop Kunchukuttan, 2020). For decoding, a beam search was used, with a beam size of 5 for machine translation, within-language sentiment transfer and formality-aware machine translation, and beam size of 1 for cross-language sentiment transfer. Each of the identified references is incorporated herein by reference.
Aspects of the technology introduce several baselines, which follow a similar training recipe as the general rewriter model discussed herein, but with some features removed. The baselines include no parallel data (-para); no language tokens (-lang tokens); no exemplars in training (-exemplars); and no back-translation (-BT). The “-para” setting tests whether the additional supervision signal obtained from the parallel data is necessary or even useful, and the “-lang tokens” setting addresses a similar question regarding language tokens. The “-exemplars” setting tests whether we suffer any degradation in translation quality by introducing the ability to leverage exemplars at inference time. Finally, given the expensive nature of back-translation, the “-BT” setting tests whether this objective is truly necessary.
Parallel data only accounts for a fraction of the languages available in mC4, yet there is interest in rewriting across all languages in mC4, potentially changing languages and attributes at the same time. To ensure the model is capable of such cross-lingual tasks, the translation quality is inspected on a selection of low-resource languages. Aspects of the technology study both languages with parallel data as well as those without it, a setting referred to as “unsupervised machine translation by language transfer” or “multilingual unsupervised machine translation”. In the latter setting, there is a consideration of the translation quality for low-resource English-centric pairs where the associated low-resource languages have no parallel data at all, with English or otherwise.
According to one set of tests, Turkish, Gujarati, and Kazakh were selected as low-resource languages, using the available newstest test sets from WMT, using newstest2018 for Turkish and newstest2019 for Gujarati and Kazakh. Of these three languages, in one example the model only sees parallel data for the language pair English-Turkish.
Table 1 compares the instant Universal Rewriter model with the bilingual unsupervised translation models and large multilingual unsupervised translation models of certain prior approaches and their analogous supervised variants. To control the output language in the baseline with no language tokens, during testing the system leveraged 214 (16,384) examples of monolingual data for each language with λ=0.5, chosen from {0.5, 1.0, 1.5} by looking at devset performance. For testing, this data was randomly drawn from the newscrawl datasets.
Notably, the rewriter model attains comparable performance with the external baselines, despite not being directly trained for unsupervised translation. It is interesting to note that the baseline with no parallel data collapses, failing to produce the correct language and giving low BLEU scores. A similar pattern for the baseline is seen without language tokens (-lang tokens) when translating into languages without parallel data. On the other hand, the remaining baselines that leverage parallel data manage to attain reasonable scores (even for language pairs without parallel data), justifying the inclusion of the parallel data. Finally, it is observed that removing the ability to control text attributes via exemplars (-exemplars) confers no meaningful benefit for translation quality. This finding supports the view that a single general model can be well-suited to adjust both language and other textual attributes.
Next, turning to style transfer in a multilingual setting, it is demonstrated for the first time the possibility of performing attribute transfer in one language using only exemplars from another language. Specifically, it is observed that Universal Rewriter model discussed herein can transfer sentiment in French, German and Japanese (as shown in Table 2 of
It is noted that previous work on multilingual style transfer has been limited to a less ambitious setting where (i) models are trained for transferring one specific attribute, (ii) a large corpus of labels is provided at training time, and (iii) the labels cover all target languages. By contrast, the current approach is attribute-agnostic (one model can transfer arbitrary attributes), few-shot (only using labels at inference time), and zero-shot across languages.
A specific multilingual style transfer evaluation was developed for testing, crafted from an existing dataset for multilingual classification. In particular, this evaluation leverages the multilingual Amazon reviews dataset, which was presented by Prettenhofer and Stein. In “Cross-language text classification using structural correspondence learning”, 2010, in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1118-1127 (incorporated herein by reference). This dataset consists of Amazon reviews in English, French, German and Japanese with review ratings in the range 1-5. For testing, the approach treated reviews rated 4 or 5 as positive, and reviews rated 1 or 2 as negative, and dropped reviews rated 3. mBERT was fine-tuned (the multilingual variant of BERT) on the remaining reviews and treated it as an oracle sentiment classifier, splitting the reviews into training and development sets. This model achieved 80.1% accuracy on this development set.
The Universal Rewriter model was compared with various ablation models on the task of transferring negative to positive sentiment. For evaluation, examples longer than 61 tokens were removed, samples were disregarded that disagreed with the oracle classifier, only keeping those which it assigned above 80% probability of being the correct class. Prepending two tokens to every example was done to control the language, and one end-of-sentence token. This gave a maximum of 64 tokens, which is the maximum length that the models were trained on. Development and test sets were constructed from the remaining examples, consisting of 800 examples, 200 from each language, equally balanced in positive and negative reviews.
For this experiment, there were three questions of interest: (1) Are English exemplars sufficient to perform sentiment transfer in non-English languages? (2) Is the Universal Rewriter model able to simultaneously transfer sentiment and language (multiple attribute rewriting? And (3) Is the use of parallel data necessary to achieve either of the above goals? To study the first question, the system leverage twenty (total) handcrafted English exemplars, exhibiting positive and negative sentiment for all experiments.
To address the third question, the testing purposely excluded English-Japanese parallel data. Given Japanese's unique script, it was believed performance on Japanese sentiment transfer and translation is a reasonable proxy for the low-resource languages within mC4 for which there may be difficulties obtaining parallel data. Since model behavior depends on the inference-time parameter λ, results are shown in chart 700 of
Whether the model is capable of rewriting multiple attributes in one pass can be evaluated by assessing its ability to perform sentiment transfer and translation simultaneously. For such testing, the same exemplars, development and test sets were reused as in the previous experiment. Here, instead of simply transferring sentiment, the system also varies the target language to produce translations into each other language. In this cross-lingual setting, defining content preservation is more challenging, since self-BLEU is no longer a meaningful measure as the input and target could be of potentially very different languages. Given the strong performance of the universal rewriter model on translation as shown above, “neutral” translations were generated using λ=0 and treating those as gold labels. Then self-BLEU was redefined to be the BLEU score computed using these predictions as references. As λ increases one can expect to see more changes relative to the neutral translations, but also an increase in transfer accuracy.
Three levels of supervision were considered: (1) unsupervised translation for the language pairs Japanese↔{English, French, German}, (2) zero-shot translation for the language pairs French↔German, which lack parallel data but both languages have parallel data with English, and (3) supervised translation for the language pairs {French, German}↔English, which have parallel data. Note that for the zero-shot (and some of the unsupervised) pairs, exemplars are in a different language from both the input and output, which acts as an additional test of the cross-linguality of the model.
As a more practical test of the model's ability to transfer multiple attributes in one pass, formality-sensitive translation can also be evaluated. Such a task requires the model to translate input text into another language, while simultaneously controlling the formality level of the output text. This type of control is useful when the desired formality level is known ahead of time, and may not match that of the source text—for example, when providing translations for second language learners, or for use within an assistive agent.
For this test, assume labels are only available in English, and in limited quantity. Specifically, 10 English exemplars were constructed (total) of formal and informal language, as shown in
Tables 3A-C of
For a more thorough examination, focus on the language pair English-Spanish. Here, in one test 50 sentences were first randomly sampled (in which sentences containing toxic words or ungrammatical text, as well as those longer than 64 tokens or shorter than 3 tokens were manually removed), containing 2nd person pronouns from the English News-Crawl datasets. Next, informal, neutral, and formal translations were generated of these 50 sentences, yielding a total of 150 sentences. Then bilingual speakers were asked to assess the formality and quality of the translations. Speakers rated formality on a scale of 0-4 (using the labels: Very informal, Informal, Neutral, Formal, and Very formal), and the quality on a scale of 0-6 (with the labels: No meaning preserved, Some meaning preserved, Most meaning preserved and few grammar mistakes, and Perfect meaning and grammar for the values 0, 2, 4, and 6 respectively).
To encourage high-quality translations, beam search was used with a beam size of 5. λ=2:3, 0, and 1.5 to generate the informal, neutral, and formal translations respectively. Items with an asterisk are statistically significantly different from the analogous result for the neutral. The average formality and translation quality scores are presented in Table 4 of
It can be seen that on average, the universal rewriting model is successful in increasing perceived formality when moving from informal to neutral to formal translations. While the formal and informal outputs exhibit a slight degradation in translation quality (as compared to neutral outputs), on average raters still judge that most of the meaning of the input is preserved. Given the relatively small number of sentences used in this experiment (3 sets of 50 translations), statistical significance tests performed using paired bootstrap resampling (as described by Philipp Koehn (2004) in “Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing), pages 388-395, incorporated herein by reference) with 1000 bootstrapped datasets. The average formality scores for the informal (resp. formal) translations were verified as being lower (resp. higher) than the scores associated with the neutral translations with at least 95% statistical significance. Here, “resp. lower” or “resp. higher” mean that the formality scores for the stylized-translations were either higher (when making them formal) or lower (when making them informal) than the base translation. This demonstrates that the model is able to control the formality of the generated text.
An additional evaluation of formality was performed by assessing whether the model successfully uses the T-V distinction in Spanish to convey formality (described by Brown and Gilman (1960), “The pronouns of power and solidarity”, in Thomas A. Sebeok, editor, Style in Language, pages 253-276, incorporated herein by reference). This involved manually inspecting each translation to assess whether the model used tu (informal) or usted (formal), either explicitly or implicitly through verb conjugations. The results are presented in Table 5 of
It can be seen that for the informal translations, the model almost exclusively uses tu. By contrast, when generating formal translations, the outputs are more varied, with the model often avoiding second person forms entirely. For example, the rewriter's formal translation for the sentence “This will allow you to turn on certain features: is “Esto permitirá activar determinadas funciones”, which translates to “This allows one to activate certain features.” Nevertheless, it can be seen that the automatic evaluation shows the same trend observed in the human evaluation.
Overall, these results demonstrate that the multilingual (universal) rewriting model is capable of zero-shot transfer of the (broadly construed) notion of formality provided by English exemplars onto other languages.
In view of the above discussion, an overall approach for training the multilingual text rewriting model can include obtaining one or more text strings for every language to be supported, extracting 2 sub-spans of text and tokenizing them. The process can randomly replace tokens to corrupt them. The spans are passed through the encoder to reuse mT5 model (or equivalent model). Untouched spans are passed through the encoder to extract the style. Note that the style is effectively an attribute, which can encompass any one (or more) of: formality, positive v, negative, dialect, author style, magazine style, food items, movie items, or even something that could be changed about a sentence while leaving other attributes the same. The style vector is added to the outputs of encoder model, and that combination is input to the decoder model (in which or more stochastic tuning ranges provide extra conditioning). The resultant multilingual model can handle not only sentences, but even random spans of text. It consumes exemplars in one language and extracts results in a different language,
TPU, GPU, CPU or other computing architectures can be employed to implement aspects of the technology in accordance with the features disclosed herein. One example computing architecture is shown in
As shown in
The processors may be any conventional processors, such as commercially available GPUs, CPUs, TPUs, etc. Alternatively, each processor may be a dedicated device such as an ASIC or other hardware-based processor. Although
The data, such as source text files or text segment and/or translated output text in multiple languages, may be operated on by the system to train one or more models. This can include filtering or curating the input dataset(s). The trained models may be used on textual input to provide translated text to one or more users, for instance users of computers 1312 and/or 1314.
Such computers may include all of the components normally used in connection with a computing device such as the processor and memory described above as well as a user interface subsystem for receiving input from a user and presenting information to the user (e.g., text, imagery and/or other graphical elements). The user interface subsystem may include one or more user inputs (e.g., at least one front (user) facing camera, a mouse, keyboard, touch screen and/or microphone) and one or more display devices (e.g., a monitor having a screen or any other electrical device that is operable to display information (e.g., text, imagery and/or other graphical elements). Other output devices, such as speaker(s) may also provide information to users.
The user-related computing devices (e.g., 1312-1314) may communicate with a back-end computing system (e.g., server 1302) via one or more networks, such as network 1310. The network 1310, and intervening nodes, may include various configurations and protocols including short range communication protocols such as Bluetooth™, Bluetooth LE™, the Internet, World Wide Web, intranets, virtual private networks, wide area networks, local networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi and HTTP, and various combinations of the foregoing. Such communication may be facilitated by any device capable of transmitting data to and from other computing devices, such as modems and wireless interfaces.
In one example, computing device 1302 may include one or more server computing devices having a plurality of computing devices, e.g., a load balanced server farm or cloud computing system, that exchange information with different nodes of a network for the purpose of receiving, processing and transmitting the data to and from other computing devices. For instance, computing device 1302 may include one or more server computing devices that are capable of communicating with any of the computing devices 1312-1314 via the network 1310.
The universal rewriter models discussed herein are extremely robust and are suitable for use with a number of computer-focused applications. This can include translation services from one language to another (or many), healthcare applications that support patients who may be most comfortable communicating in their native language, video streaming services that provide subtitles in a number of selectable languages, and videoconferencing services that may support a real-time closed-captioning feature for the users.
Although the technology herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present technology. It is therefore to be understood that numerous modifications may be made to the illustrative embodiments and that other arrangements may be devised without departing from the spirit and scope of the present technology as defined by the appended claims.
