Patent Grant
11694042
The present disclosure relates generally to systems and methods for computer learning that can provide improved computer performance, features, and uses. More particularly, the present disclosure relates to systems and methods for classification.
Recent neural network models have achieved remarkable performance on sentiment classification in English and other languages. However, their success heavily depends on the availability of a large amount of labeled data or parallel corpus. In reality, some low-resource languages or applications have limited labeled data or are even without any labels or parallel corpus, which may hinder training a robust and accurate classifier.
To build classification models (such as sentiment classification models) for low-resource languages, recent researchers developed cross-lingual text classification (CLTC) models (see Ruochen Xu and Yiming Yang, “Cross-lingual distillation for text classification,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1415-1425, Vancouver, Canada (2017) (hereinafter, Xu and Yang (2017)) and Akiko Eriguchi, Melvin Johnson, Orhan Firat, Hideto Kazawa, and Wolfgang Macherey, “Zero-Shot Cross-Lingual Classification Using Multilingual Neural Machine Translation,” Technical report, arXiv:1809.04686 (2018) (hereinafter, Eriguchi et al. (2018)), which transfers knowledge from a resource-rich (source) language to a low-resource (target) language. A core of those models is to learn a shared language-invariant feature space that is indicative of classification for both languages. Therefore, a model trained from the source language can be applied to the target language. Based on how the shared feature space is learned, there are typically three categories, namely word-level alignments, sentence-level alignments, and document-level alignments. Those models can well capture the semantic similarity between two languages. They, however, require parallel resources such as a bilingual dictionary, parallel sentences, and parallel Wikipedia articles. Such a limitation may prevent these models from being applicable in languages without any parallel resources.
Recently, there have been several attempts at developing “zero-resource” models. Most notably, Yftah Ziser and Roi Reichart, “Deep pivot-based modeling for cross-language cross-domain transfer with minimal guidance,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 238-249, Brussels, Belgium (2018) (hereinafter, Ziser and Reichart (2018)) proposed a cross-lingual & cross-domain (CLCD) model that builds on pivot-based learning and bilingual word embedding. Although CLCD does not directly need labeled data or parallel corpus, it requires bilingual word embeddings (BWEs) that requires thousands of translated words as a supervised signal. Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Q. Weinberger, “Adversarial deep averaging networks for cross-lingual sentiment classification,” Trans. Assoc. Comput. Linguistics, 6:557-570 (2018) (hereinafter, Chen et al. (2018)) developed an adversarial deep averaging network to learn latent sentence representations for classification, but it had an implicit dependency on BWEs that required pretraining on a large bilingual parallel corpus. Zhuang Chen and Tieyun Qian, “Transfer capsule network for aspect level sentiment classification,” in Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL), pages 547-556, Florence, Italy (2019) (hereinafter, Chen and Qian (2019)) extended the cross-lingual model in Chen et al. (2018) to multiple source languages by using the unsupervised BWEs (Guillaume Lample, Alexis Conneau, Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou, “Word translation without parallel data,” in Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, Canada (2018b) (hereinafter, Lample et al. (2018b))) and by adding an individual feature extractor for each source language, which eliminated the dependency on a parallel corpus. Nevertheless, their model is very sensitive to the quality of BWEs and performs poorly on distant language pairs such as English-Japanese, as illustrated in their experimental study.
In parallel, cross-lingual language models (LMs) trained from raw Wikipedia texts, such as multilingual BERT (Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171-4186, Minneapolis, Minn. (2019)) (hereinafter, Devlin et al. (2019)) and XLM (Alexis Conneau and Guillaume Lample, “Cross-lingual language model pretraining,” in Advances in Neural Information Processing Systems (NeurIPS), pages 7057-7067, Vancouver, Canada (2019)) (hereinafter, Conneau and Lample (2019)), have been prevalent in solving zero-shot classification problems (Shijie Wu and Mark Dredze, “Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833-844, Hong Kong, China (2019)) (hereinafter, Wu and Dredze (2019)). Those models use the BERT-style Transformer architecture (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing (NIPS), pages 6000-6010, Long Beach, Calif. (2017)) (hereinafter, Vaswani et al. (2017)) simultaneously trained from multiple languages to construct a sentence encoder, and fine-tune the encoder and a classifier on labeled training data from the source language, which is illustrated in
Accordingly, what is needed are systems and methods that provide improved cross-lingual classification.
References will be made to embodiments of the disclosure, examples of which may be illustrated in the accompanying figures. These figures are intended to be illustrative, not limiting. Although the disclosure is generally described in the context of these embodiments, it should be understood that it is not intended to limit the scope of the disclosure to these particular embodiments. Items in the figures may not be to scale.
In the following description, for purposes of explanation, specific details are set forth in order to provide an understanding of the disclosure. It will be apparent, however, to one skilled in the art that the disclosure can be practiced without these details. Furthermore, one skilled in the art will recognize that embodiments of the present disclosure, described below, may be implemented in a variety of ways, such as a process, an apparatus, a system, a device, or a method on a tangible computer-readable medium.
Components, or modules, shown in diagrams are illustrative of exemplary embodiments of the disclosure and are meant to avoid obscuring the disclosure. It shall also be understood that throughout this discussion that components may be described as separate functional units, which may comprise sub-units, but those skilled in the art will recognize that various components, or portions thereof, may be divided into separate components or may be integrated together, including, for example, being in a single system or component. It should be noted that functions or operations discussed herein may be implemented as components. Components may be implemented in software, hardware, or a combination thereof.
Furthermore, connections between components or systems within the figures are not intended to be limited to direct connections. Rather, data between these components may be modified, re-formatted, or otherwise changed by intermediary components. Also, additional or fewer connections may be used. It shall also be noted that the terms “coupled,” “connected,” “communicatively coupled,” “interfacing,” “interface,” or any of their derivatives shall be understood to include direct connections, indirect connections through one or more intermediary devices, and wireless connections. It shall also be noted that any communication, such as a signal, response, reply, acknowledgement, message, query, etc., may comprise one or more exchanges of information.
Reference in the specification to “one or more embodiments,” “preferred embodiment,” “an embodiment,” “embodiments,” or the like means that a particular feature, structure, characteristic, or function described in connection with the embodiment is included in at least one embodiment of the disclosure and may be in more than one embodiment. Also, the appearances of the above-noted phrases in various places in the specification are not necessarily all referring to the same embodiment or embodiments.
The use of certain terms in various places in the specification is for illustration and should not be construed as limiting. A service, function, or resource is not limited to a single service, function, or resource; usage of these terms may refer to a grouping of related services, functions, or resources, which may be distributed or aggregated. The terms “include,” “including,” “comprise,” and “comprising” shall be understood to be open terms and any lists the follow are examples and not meant to be limited to the listed items. A “layer” may comprise one or more operations. The words “optimal,” “optimize,” “optimization,” and the like refer to an improvement of an outcome or a process and do not require that the specified outcome or process has achieved an “optimal” or peak state. The use of memory, database, information base, data store, tables, hardware, cache, and the like may be used herein to refer to system component or components into which information may be entered or otherwise recorded.
In one or more embodiments, a stop condition may include: (1) a set number of iterations have been performed; (2) an amount of processing time has been reached; (3) convergence (e.g., the difference between consecutive iterations is less than a first threshold value); (4) divergence (e.g., the performance deteriorates); (5) an acceptable outcome has been reached; and (6) all of the data has been processed.
One skilled in the art shall recognize that: (1) certain steps may optionally be performed; (2) steps may not be limited to the specific order set forth herein; (3) certain steps may be performed in different orders; and (4) certain steps may be done concurrently.
Any headings used herein are for organizational purposes only and shall not be used to limit the scope of the description or the claims. Each reference/document mentioned in this patent document is incorporated by reference herein in its entirety.
It shall also be noted that although embodiments described herein may be within the context of sentiment classification, aspects of the present disclosure are not so limited. Accordingly, the aspects of the present disclosure may be applied or adapted for use in other contexts.
It shall be noted that any experiments and results provided herein are provided by way of illustration and were performed under specific conditions using a specific embodiment or embodiments; accordingly, neither these experiments nor their results shall be used to limit the scope of the disclosure of the current patent document.
Recent neural network models have achieved impressive performance on sentiment classification in English as well as other languages. Their success heavily depends on the availability of a large amount of labeled data or parallel corpus. In this patent document, embodiments are presented that can handle even the extreme scenario of cross-lingual sentiment classification, in which the low-resource language does not have any labels or parallel corpus.
Presented herein are embodiments of a classification model (which may be referred to generally, for convenience, as multi-view encoder-classifier (MVEC)), in an unsupervised setting, in which there is monolingual corpora from two languages and labels in the source language. Different from previous language model (LM)-based fine-tuning approaches that adjust parameters solely based on the classification error of training data, embodiments utilize the encoder-decoder network from unsupervised machine translation (UMT) to regularize and refine the shared latent space. In one or more embodiments, the transformer-based encoder regularized by a language discriminator learns shared but more refined language-invariant representations, which are effective for both reconstructing sentences from two languages by the decoder and generating multi-view feature representations for classification from input documents. In one or more embodiments, two views from the encoder are constructed: (i) the encoded sentences in the source language; (ii) the encoded translations of the source sentences in the target language.
In one or more embodiments, a MVEC implementation may be partially initialized by pretrained LMs but further fine-tuned to align sentences from two languages better, accurately predict labeled data in the source language, and encourage consensus between the predictions from the two views. Embodiments of the full model may be trained in an end-to-end manner to update parameters for the encoder-decoder, the language discriminator, and the classifier at each iteration. Some of the contributions include but are not limited to the following:
Embodiments of an unsupervised sentiment classification model without any labels or parallel resource requirements for the target language are presented. By designing a multi-view classifier and integrating it with pretrained LMs and UMT, a model (MVEC) embodiment may be built on a more refined latent space that is robust to language shift with better model interpretation compared to previous zero-shot classification works.
An embodiment was extensively evaluated using five (5) language pairs involving eleven (11) sentiment classification tasks. The tested full model embodiment outperformed state-of-the-art unsupervised fine-tuning approaches and partially supervised approaches using cross-Lingual resources in 8/11 tasks. Therefore, the results provide a strong lower bound performance on what future semi-supervised or supervised approaches are expected to produce.
CLTC aims to learn a universal classifier that can be applied to languages with limited labeled data, which is naturally applicable for sentiment analysis. Traditional supervised methods utilize cross-lingual tools, such as machine translation systems, and train a classifier on the source language. The latest models used parallel corpus either to learn a bilingual document representation or to conduct cross-lingual model distillation.
In the unsupervised setting, Chen et al. (2018) learned language-invariant latent cross-lingual representations with adversarial training. Ziser and Reichart (2018) used pivot-based learning and structure-aware deep neural network (DNN) to transfer knowledge to low-resourced languages. In both papers, however, they have an implicit dependency on BWEs, which requires a bilingual dictionary to train. Chen and Qian (2019) was the first fully unsupervised approach using the unsupervised BWEs (Lample et al. (2018b)) and multi-source languages with adversarial training.
In contrast, one or more model embodiments herein are a multi-view classification model that is seamlessly integrated with pretrained LMs (e.g., Conneau and Lample (2019)) and an encoder-decoder such as from UMT (e.g., Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato, “Unsupervised machine translation using monolingual corpora only,” in Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, Canada (2018) (hereinafter, Lample et al. (2018a)) with adversarial training. Hence, a more fine-tuned latent space is learned to better capture document-level semantics and generate multiple views to represent the input.
UMT does not rely on any parallel corpus to perform translation. At the word-level, Lample et al. (2018b) claim to have built a bilingual dictionary between two languages by aligning monolingual word embeddings in an unsupervised way. At the sentence and document level, Lample et al. (2018a) proposed a UMT model by learning an autoencoder that can reconstruct two languages under both within-domain and cross-domain settings. Lample et al. (2018c) (Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc′Aurelio Ranzato, “Phrase-based & neural unsupervised machine translation,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5039-5049, Brussels, Belgium (2018c)) extended Lample et al. (2018a) with a phrase-based approach. Since an aim of embodiments is to learn more refined language-invariant representations for classification, in one or more embodiments, the encoder from a UMT system may be employed to generate multiple views of the input and enable knowledge transfer.
A task of multi-view transfer learning is to simultaneously learn multiple representations and transfer the learned knowledge from source domains to target domains, which have fewer training samples. Generally, data from different views contains complementary information and multi-view learning exploits the consistency from multiple views.
Fu et al. (2015) (Yanwei Fu, Timothy M. Hospedales, Tao Xiang, and Shaogang Gong, “Transductive multi-view zero-shot learning,” in IEEE Trans. Pattern Anal. Mach. Intell., 37(11):2332-2345 (2015)) and Zhang et al. (2019) (Qingheng Zhang, Zequn Sun, Wei Hu, Muhao Chen, Lingbing Guo, and Yuzhong Qu, “Multi-view knowledge graph embedding for entity alignment,” in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI), pages 5429-5435, Macao, China (2019) both exploit the complementarity of multiple semantic representations with semantic space alignment. Unlike these approaches, one or more embodiments herein use an encoder-decoder framework to generate multiple views for input from the source language and enforce a consensus between their predictions. Furthermore, in one or more embodiments, a language discriminator may be introduced to encourage the encoder to generate language-invariant representations from the input.
In this section, general workflows of model embodiments are presented, including the details of each component and training method embodiments.
Given monolingual text data {Dsrc, Dtgt} from both the source and target language with a subset of labeled samples {DsrcL, ysrcL} in the source language where ysrcL is a vector of class labels and DsrcL⊂Dsrc, a task aims to build a universal classification model f(X; θ)→y parameterized by θ that can be directly applicable to unlabeled data in the target language, where X is an input document from any language and y is its class label. Note that in one or more embodiments it may be assumed that two languages share the same class types.
In one or more embodiments, a multi-view encoder-classifier (MVEC) comprises: an encoder-decoder and a classifier. In one or more alternative embodiments, a MVEC embodiment also comprises a language discriminator.
Motivated by the success of unsupervised machine translation (UMT) and reconstruction regularization by an autoencoder, embodiments adopt an encoder-decoder framework 215 and introduce a self-reconstruction loss within one language and back-translation reconstruction loss across languages together with the normal loss from classification of the classifier 225. For simplicity, self-reconstruction loss is denoted as “within-domain loss” and back-translation reconstruction loss is denoted as “cross-domain loss” herein.
Although the encoder from the UMT may generate a latent representation for input documents (in which a “document” may comprise one or more words or sentences), there may still be a semantic gap between the source and target language. Embodiments may enrich the encoder-decoder framework 215 with a language discriminator 220 that can produce fine-tuned latent representations to align latent representations from two languages better. Such representations are useful to train a language-invariant classifier (e.g., classifier 225) that is robust to the shift in languages.
As illustrated in the embodiment depicted in
In one or more embodiments, under the unsupervised setting, an MVEC embodiment observes unlabeled monolingual corpora from two languages and some labeled documents in the source language. In one or more embodiments, the unlabeled monolingual data is normally sampled from the application domain, i.e., unlabeled product reviews or social media posts, which is used in both adopting pretrained LMs in the target domain and training UMT. As shown in
Lines 310 lines indicate the message flow within the source language 320, and lines 315 indicate the message flow of the target language 325, according to embodiments of the present disclosure. Lines 305 indicate the message flows from the encoder 330 to the text classifier 345, according to embodiments of the present disclosure. In one or more embodiments, the encoder 330 and the decoder 335 share the same parameters between the two languages.
Let x(l)=(x1(l), x2(l), x3(l), . . . , xn(l)) denote the input document of n words from a particular language l, where l∈{src, tgt}. In one or more embodiment, the encoder is a neural network eθ
Given the latent representation (or encoded representation) Z(l) as the input, the decoder dθ
It is more likely for the encoder-decoder to merely memorize every input word one-by-one if there are no imposed constraints. To improve the robustness of encoder-decoder, embodiments adopt Denoising Autoencoders (DAE) (Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol, “Extracting and composing robust features with denoising autoencoders,” in Proceedings of the Twenty-Fifth International Conference on Machine Learning (ICML), pages 1096-1103, Helsinki, Finland (2008)) (hereinafter, Vincent et al. (2008)), which recovers input from its corrupted version.
There are at least three ways to inject noise into an input document including shuffle, dropout, and replacement by special words. In one or more embodiments, every word is dropped and replaced with probabilities of pd and pb, respectively, and the input document is slightly shuffled by implementing random permutation a on the input document, where pd and pb may be viewed as hyper-parameters for controlling noise levels. In one or more embodiments, the permutation a satisfies the condition |σ(i)−i|≤k, ∀i∈{1, . . . , n}, where n is the length of input document and k is another hyper-parameter.
In one or more embodiments, the noise model is applied to unlabeled data used for training the encoder-decoder and the discriminator, while labeled data is kept in its originality for all components training. In one or more embodiments, G(.) is used to denote a stochastic noise model, which takes input document x(l) and generates G(x(l)) as a randomly sampled noisy version of x(l).
To incorporate the encoder-decoder as regularization components, embodiments consider both within-domain and cross-domain objective functions. The first objective function (within domain) aims to reconstruct a document from a noisy version of itself within a language, whereas the second objective function (cross-domain) targets to teach the model to translate an input document across languages.
Specifically, given a language l∈{src, tgt}, an embodiment of the within-domain objective function may be written as:
Rwd(θed,l)=x˜D
where θed=[θenc, θdec], {circumflex over (x)}˜d(e(G(x))) is a reconstruction of the corrupted version of x sampled from the monolingual dataset Dl, and Δ is the sum of token-level cross-entropy loss to measure discrepancy between two sequences.
Given the encoded document (i.e., the sequence of hidden or latent states that are output by the encoder), a decoder (e.g., decoder 335 in
Given the input document in its original language and the reconstructed document in the same language, a within-domain regularization can be computed (415). As noted above, the within-domain regularization is related to reconstruction error of reconstructing the text document into its original language using the encoder and the decoder and may be computed as discussed above with respect to Equation (1).
Similar to teaching the encoder-decoder framework to reconstruct an input text document in one language, embodiments consider teaching the encoder-decoder to reconstruct x in one language from a translation of x in the other language, leading to the cross-domain objective function. An embodiment of the cross-domain objective function may be written as:
where (l1, l2)∈{(src, tgt), (tgt, src)} and T(.) is the current UMT model applied to input document x from language l1 to language l2.
Given the input document in its original language before translation and the reconstructed document in the same language (i.e., back-translating), a cross-domain regularization may be computed (515). As noted above, the cross-domain regularization is related to reconstruction error of reconstructing the text document into its original language using the encoder and the decoder and may be computed as discussed above with respect to Equation (2).
3.4. Language Discriminator Embodiments
Cross-lingual classifiers work well when their input produced by the encoder is language-invariant. Thus, embodiments of the encoder map input documents from both languages into a shared feature space independent of languages. In one or more embodiments, to achieve this goal, a language discriminator (e.g., language discriminator 340 in
LD(θD|θenc)=−(l,x
where θD denotes parameters of the discriminator, (l, x(l)) corresponds to language and document pairs uniformly sampled from monolingual datasets (e.g., source language dataset 320 and target language dataset 325 in
Also, in one or more embodiments, the encoder is trained to “fool” the discriminator:
with lj=l1 if li=l2 and vice versa.
Described above are how embodiments obtain a language-invariant latent space to encode two languages. However, such approaches may not be sufficient to generalize well across languages if one simply trains a classifier on the encoder's output for the source language. One key difference between prior approaches, such as Chen et al. (2018), and embodiments herein is that one or more embodiments herein use UMT, which can generate multiple views for the input labeled documents from the source language. Embodiments may thereby benefit from multi-view learning's superior generalization capability over single-view learning.
In one or more embodiments, the encoder may be used to generate two view representations for text documents from training documents: the first view representation comprises an encoded representation of the text document using the text document in its original language as an input into the encoder, and the second view representation comprises an encoded back-translation representation of the text document using the text document translated from its original language into another language as an input into the encoder. Particularly, in terms of the classifier, two views of the input may be considered: (i) the encoded labeled documents from the source language; and (ii) the encoded back-translations of the source documents from the target language.
In one or more embodiments, a learning objective is to train the classifier to match predicted document labels with ground truth from the source language and to encourage the two predictive distributions on the two views to be as similar as possible. In one or more embodiments, the following objective function may be used:
where (x, y)˜{DscrL, ysrcL}, DKL (.∥.) is KL Divergence to measure the difference between two distributions, y is the class label of input document x, θc are parameters of the classifier, and Pθ
In one or more embodiments, the final objective function at one iteration of the learning process is to minimize the following loss function:
Lall=LC+λwd×(Rwd
where λwd, λcd, λadv are the hyper-parameters to trade-off among within-domain loss, the cross-domain loss, and the adversarial loss, respectively.
Methodology 1 (below) provides a detailed procedure of an embodiment method. Inputs to the process include a dataset of documents in the source language (Dsrc), a subset of this dataset form a labeled dataset (ysrcL) comprising, for each text document of the subset, a label associated with text document, and a dataset of documents in the target language (Dtgt).
Model embodiments may use an initial translation machine T(0), which provides a translation from one language to another, which translations are used for generating second views of text documents that are used for calculating the cross-domain loss in Eq. (2) and classifier loss in Eq. (5). To accelerate the training, T(0) may be initialized by pretraining a transformer-based UMT, which has the same encoder-decoder architecture as the model embodiment, on monolingual text. After pretraining, the pretrained encoder-decoder network may be used to initialize the model embodiment, and start training the classifier and the discriminator. Meanwhile, in one or more embodiments, the encoder and the decoder are refined on the monolingual datasets and labeled data from the source language.
In one or more embodiments, during each training step, the optimization iterates from updating θD in Eq. (3) to updating θed and θC in Eq. (6). Note that if a batch of documents drawn from monolingual data are all unlabeled, then updating classifier parameters is suspended and only the parameters of the language discriminator and encoder-decoder are updated.
In one or more embodiments, updating θed and θC may occur at the end of each batch, and the updated encoder and decoder of the multi-view encoder-classifier training system may be used to update the encoder and the decoder of the UMT, which may occur at the end of each epoch, although it may also occur more frequently.
This method may continue until a stop condition is reach, such as a maximum number of epochs has been reached (as illustrated in the embodiment of Methodology 1, below), although other stop conditions may additionally or alternatively be used.
The selected documents and their translations are input (615) into a multi-view encoder-classifier system that comprises an encoder that converts a text document into a latent representation; and a classifier that receives the latent representation of a document as an input and outputs a label for the document.
In one or more embodiments, during each batch, a language discriminator loss is determined and the language discriminator is updated (620). Also, losses and regularization terms are determined for updating (625) the classifier, the encoder, and the decoder. As noted previously, the parameters of the language discriminator may be updated using Eq. (3), and the parameters of the classifier, the encoder, and the decoder may be updated using Eq. (6).
For example, in one or more embodiments, for documents input in their original language, a within-domain regularization related to reconstruction error of reconstructing the text document into its original language using the encoder and a decoder that decodes the latent representation from the encoder into a reconstruction of the text document is determined. For text documents translated into the other language and input into the system, a cross-domain regularization related to reconstruction error of reconstructing the text document into its original language using the translated version of the text document as the input to the encoder is determined. For text documents from the labeled dataset, a multi-view classification loss that comprises: (1) a classification loss component related to the text classifier's ability to correctly predict the labels for the text documents and (2) a view consensus component related to a comparison of the text classifier's label probabilities obtained using the latent representations of the text documents in the first language and the classification label probabilities obtained using the latent representations of the text document translated into the second language is determined.
Also, for at least some of the text documents regardless of dataset source, an adversarial encoder loss is determined for training the encoder to generate a latent representation for the text document that increases difficulty for a language discriminator, which receives the latent representation and outputs a prediction of whether the text document from which the latent representation was generated was in the first language or the second language, to correctly predict the language of the text document from which the latent representation was generated. And, a language discriminator loss is determined for training the language discriminator to correctly predict the language of the text document from which the latent representation was generated.
Given the losses and regularizations, the language discriminator loss is used to update parameters of the language discriminator, and the multi-view classification loss, the within-domain regularization, and the cross-domain regularization are used to update parameters of the encoder, the decoder, and the classifier.
In one or more embodiments, if another batch of documents is to be processed (630), the methodology returns to step 605; otherwise, the process may check (635) if a stop condition has been reached. If a stop condition has not been reached, the process may update (640) the encoder and decoder of a translator, if one was used, and return to step 605 (e.g., to perform another epoch). However, if a stop condition has been reached, the trained MVEC comprising a trained encoder and a trained classifier are output (645).
In one or more embodiments, a trained multi-view encoder-classifier system includes an encoder and a classifier. The encoder, which may comprise one or more neural network layers, receives a text document written in a target language and converts the text document into an encoded representation. The classifier, which may comprise one or more neural network layers, receives the encoded representation and outputs a class label for the text document.
In one or more embodiments, the encoder and the classifier were trained as part of a multi-view encoder-classifier training system by performing steps comprising: using the encoder to generate two view representations for text documents from training documents, the first view representation comprising an encoded representation of the text document using the text document in its original language as an input into the encoder, and the second view representation comprising an encoded back-translation representation of the text document using the text document translated from its original language into another language as an input into the encoder.
Given the different encoded views, a loss is used to update at least the encoder and the text classifier. In one or more embodiments, the loss comprises: a within-domain regularization related to reconstruction error of reconstructing the text document using the first view representation and a decoder, and a cross-domain regularization related to reconstruction error of reconstructing the text document using the second view representation the decoder. In one or more embodiments, the loss also includes, for text documents that have associated ground truth labels, a multi-view classification loss comprising a classification component related to the text classifier's predictions of the labels for those text documents and a consensus component related to reducing difference between the text classifier's predictive distribution of the first view representations and the text classifier's predictive distribution of the second view representations. In one or more embodiments, the consensus component involves using KL Divergence to measure a difference between the text classifier's predictive distribution of the first view representations and the text classifier's predictive distribution of the second view representations.
In one or more embodiments, the multi-view encoder-classifier training system also includes a language discriminator that receives the encoded representations of the text documents and predicts a language of the text document, and training the encoder and the text classifier of the multi-view encoder-classifier system further includes using the language discriminator in adversarial training with the encoder to encourage the encoder to generate encoded representations that are invariant from the language of the text document.
In one or more embodiments, an unsupervised machine translator, comprising an encoder and a decoder, is used to translate text documents from the source language to the target language and from the target language to the source language. Also, the encoder and the decoder of the unsupervised machine translator may share parameters with the encoder and the decoder of the multi-view encoder-classifier system, and the encoder and the decoder of the unsupervised machine translator may be updated using updated parameters of the encoder and the decoder of the multi-view encoder-classifier training system.
Experiments were conducted on cross-lingual multiclass and binary sentiment classification using five language pairs involving 11 tasks. More specifically, English is always the source language, and the target languages are French, German, Japanese, Chinese, and Arabic, respectively.
It shall be noted that these experiments and results are provided by way of illustration and were performed under specific conditions using a specific embodiment or embodiments; accordingly, neither these experiments nor their results shall be used to limit the scope of the disclosure of the current patent document.
Dataset 1 (French, German, Japanese). This is a multilingual sentiment classification dataset in four languages, including English (en), French (fr), German (de), and Japanese (ja), covering three products. For each product in each language, there are 2000 documents in each of the training and test sets. Each document contains a title, a category label, a review, and a 5-point scale star rating. Multi-class ratings were converted to binary ratings by thresholding at 3-point. For each product, since the test set in English is not used, the English training and test sets were combined and 20% (800) documents were randomly sampled as the validation set to tune hyper-parameters, and the remaining 3200 samples were used for training. For each target language, the original 2000 test samples were used for comparison with previous methods. Unlike Chen et al. (2018) and Chen and Qian (2019) that used labeled data in the target language for model selection, the labels of reviews in the target language were only used for testing. There are 105 k, 58 k, 317 k, 300 k unlabeled reviews for English, French, German, and Japanese, respectively, which can be used as monolingual data to train the encoder-decoder of the tested MVEC model embodiment.
Dataset 2 (Chinese). This dataset comprises two groups or sources: (i) Group A: 700 k reviews in English with five classes, and (ii) Group B: 170 k reviews in Chinese segmented and annotated with five classes. All of the first group of reviews were split into a training set with 650 k reviews and validation set with 50 k reviews. The 650 k review contents are also served as the monolingual training data for English. For Chinese review data, 150 k reviews were sampled as the monolingual training set. The rest 20 k reviews were treated as the test set.
Dataset 3 (Arabic). An Arabic sentiment dataset comprising over 1100 documents annotated with three labels (negative, neutral, positive) was used. The dataset was split into half as training and the other half as testing. Since validation data in the target language was not needed to tune the model, 1000 documents were randomly sampled as test data. For English resource, Group A of Dataset 2 reviews were used and follow the same split as the Chinese case but converted 5 level reviews into 3 levels (i.e., 1 & 2→“negative,” 3→“neutral,” and, 4 & 5→“positive’). Also, 161 k sentences were randomly sample from a United Nations Corpus Arab subset as unlabeled monolingual data for model training.
For French, German and Japanese, binary classification was performed. For Chinese and Arabic, multi-class classification was performed.
Data Preprocessing. Monolingual data of each language was extracted and tokenized using Moses (Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, and et al. 2007. Moses: Open-source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL), Prague, Czech Republic). Then, the neural machine translation was used for rare words with subword units, named fastBPE (Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Berlin, Germany) in three steps. In detail, BPE code was collected from the pretrained XLM-100 models, then applied to all tokenized data and used to extract the training vocabulary. To constrain the tested model embodiment's size, the top 60 k most frequent subword units in the training set were kept. Finally, monolingual data and labeled data was binarized for model training, validation, and testing.
Pretraining Details. As mentioned earlier, the tested model embodiment uses an initial translation machine to compute reconstruction loss and classifier loss. Pretrained language models were leveraged to initialize a transformer-based UMT, such as described in (Lample et al., 2018a), and it was trained on text. In particular, 10 million sentences were sampled from each language pairs and an XLM library was used to train a UMT for 200K steps. The resulting encoder-decoder were used to initialize the test model embodiment.
Regarding word embedding initialization, the embeddings obtained from the 1st layer of pretrained language models were used, which has demonstrated better cross-lingual performance in a number of evaluation metrics over MUSE.
Training Details. In the experiment, both encoder and decoder are 6-layer transformers with 8-head self-attention. Both subword embedding and hidden state dimension were set to 1024 and greedy decoding was used to generate a sequence of tokens. The encoder-decoder and classifier were trained using an Adam optimizer with a learning rate of 10−5 and a mini-batch size of 32. The hidden dimension was set to 128 for both classifier and discriminator. For parameters of denoising auto-encoder, pd=0.1, pb=0.2 and k=3. Finally, a grid search was performed for hyper-parameters on {0.5, 1, 2, 4, 8} and λwd and λcd were set to 1 and λavd was set to 4. To prevent gradient explosion, the gradient L2 norm was clipped by 5.0. The approach was implemented in PaddlePaddle and all experiments were conducted on an NVIDIA Tesla M40 (24 GB) GPU.
Competing Methods. The tested embodiment was compared with several recently published results. Due to the space limit, several representative baselines are introduced: LR+MT translated the bag of words from target language to source language via machine translation and then built a logistic regression model. BWE baselines rely on Bilingual Word Embeddings (BWEs), wherein 1-to-1 indicates that it was only transferring from English, while 3-to-1 means the training data from all other three languages. CLDFA (Ruochen Xu and Yiming Yang. 2017. Cross-lingual distillation for text classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1415-1425, Vancouver, Canada) (hereinafter, Xu and Yang (2017)) was built on model distillation on parallel corpora with adversarial feature adaptation technique. PBLM (Yftah Ziser and Roi Reichart. 2018. Deep pivot-based modeling for cross-language cross-domain transfer with minimal guidance. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 238-249, Brussels, Belgium) (hereinafter, Ziser and Reichart (2018)) used bilingual word embeddings and pivot-based language modeling for cross-domain & cross-lingual classification. MBERT (Devlin et al., 2019) and XLM-FT (Conneau and Lample (2019)) directly fine-tuned a single layer classifier based on pretrained LM multilingual BERT and XLM.
In TABLE 1 (see
A first observation from TABLE 1 is that the test model embodiment and the fine-tuned multilingual LM MBERT (Devlin et al. (2019)) and XLM-FT (Conneau and Lample (2019)) outperformed all previous methods including the methods with cross-lingual resources for 8/9 tasks by a large margin, which indicates the huge benefit from pretrained LMs in the zero-shot setting. Compared with MBERT and XLM-FT, the test model embodiment obtains better performance when the target language is more similar to the source language, for example, German and French, and one task in Japanese.
52.54
43.36
TABLE 2 shows the comparison between the test method embodiment and a few other published results, including ADAN (Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Q. Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Trans. Assoc. Comput. Linguistics, 6:557-570) (hereinafter, Chen et al. (2018))) and mSDA (Minmin Chen, Zhixiang Eddie Xu, Kilian Q. Weinberger, and Fei Sha. 2012. Marginalized denoising autoencoders for domain adaptation. In Proceedings of the 29th International Conference on Machine Learning (ICML), Edinburgh, UK) (hereinafter, Chen et al. (2012))) for Chinese and Arabic languages in multi-class setting. Similarly, the model embodiment obtains slightly better accuracy in Chinese. Overall, built on top of the pretrained LMs and UMT, a full model embodiment achieves the state-of-the-art performance on 8/11 sentiment classification tasks, especially when the target language is more similar to the source language.
Moreover, the effectiveness of encoder-decoder-based regularization in reducing the language shift in the shared latent space is illustrated. Intuitively, if the fine-tuned latent space is less sensitive to the language shift, the performance on validation sets and test sets should be highly correlated during training. In
From
To understand the effect of different components in model embodiments on the overall performance, an ablation study was conducted, as reported in TABLE 3. Clearly, the encoder-decoder trained either by the within-domain objective or cross-domain objective is the important. For Dataset 1 data in three languages (German, French, Japanese), the model without cross-domain loss obtains prediction accuracy of 83.22%, 82.40%, and 72.05%, which gets decreased by 5%-7% compared with the full model embodiment. The performance is also significantly degraded when the adversarial training component is removed because the distribution of latent document representations is not similar between two languages. The two-views consensus component also has a significant effect on the performance, with a performance drop up to 5 points for en-jp. Such a result verifies, as asserted herein, that cross-lingual model benefits from training on multiple views of the input.
88.61
88.62
78.67
43.36
49.70
To further explore the effectiveness of embodiments, the encoder's output and the last layer before softmax was visualized for 10 randomly sampled Dataset 1 reviews in English and their translations in French using a machine translation.
As seen in the lower-left panel (1015) of
Presented herein are embodiments of a cross-lingual multi-view encoder-classifier (MVEC) that require neither labeled data in the target language nor cross-lingual resources with the source language. In one or more embodiments, built upon pretrained language models, embodiments utilize the encoder-decoder component with a language discriminator from an unsupervised machine translation system to learn a language-invariant feature space. The approach presented herein departs from previous models that could only make use of the shared language-invariant features or depend on parallel resources. By constructing the fine-tuned latent feature space and two views of input from the encoder-decoder of UMT, embodiments significantly outperform previous methods for 8/11 zero-shot sentiment classification tasks.
In one or more embodiments, aspects of the present patent document may be directed to, may include, or may be implemented on one or more information handling systems (or computing systems). An information handling system/computing system may include any instrumentality or aggregate of instrumentalities operable to compute, calculate, determine, classify, process, transmit, receive, retrieve, originate, route, switch, store, display, communicate, manifest, detect, record, reproduce, handle, or utilize any form of information, intelligence, or data. For example, a computing system may be or may include a personal computer (e.g., laptop), tablet computer, mobile device (e.g., personal digital assistant (PDA), smart phone, phablet, tablet, etc.), smart watch, server (e.g., blade server or rack server), a network storage device, camera, or any other suitable device and may vary in size, shape, performance, functionality, and price. The computing system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, read only memory (ROM), and/or other types of memory. Additional components of the computing system may include one or more disk drives, one or more network ports for communicating with external devices as well as various input and output (I/O) devices, such as a keyboard, mouse, stylus, touchscreen and/or video display. The computing system may also include one or more buses operable to transmit communications between the various hardware components.
As illustrated in
A number of controllers and peripheral devices may also be provided, as shown in
In the illustrated system, all major system components may connect to a bus 1116, which may represent more than one physical bus. However, various system components may or may not be in physical proximity to one another. For example, input data and/or output data may be remotely transmitted from one physical location to another. In addition, programs that implement various aspects of the disclosure may be accessed from a remote location (e.g., a server) over a network. Such data and/or programs may be conveyed through any of a variety of machine-readable medium including, for example: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices that are specially configured to store or to store and execute program code, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, other non-volatile memory (NVM) devices (such as 3D XPoint-based devices), and ROM and RAM devices.
Aspects of the present disclosure may be encoded upon one or more non-transitory computer-readable media with instructions for one or more processors or processing units to cause steps to be performed. It shall be noted that the one or more non-transitory computer-readable media shall include volatile and/or non-volatile memory. It shall be noted that alternative implementations are possible, including a hardware implementation or a software/hardware implementation. Hardware-implemented functions may be realized using ASIC(s), programmable arrays, digital signal processing circuitry, or the like. Accordingly, the “means” terms in any claims are intended to cover both software and hardware implementations. Similarly, the term “computer-readable medium or media” as used herein includes software and/or hardware having a program of instructions embodied thereon, or a combination thereof. With these implementation alternatives in mind, it is to be understood that the figures and accompanying description provide the functional information one skilled in the art would require to write program code (i.e., software) and/or to fabricate circuits (i.e., hardware) to perform the processing required.
It shall be noted that embodiments of the present disclosure may further relate to computer products with a non-transitory, tangible computer-readable medium that have computer code thereon for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind known or available to those having skill in the relevant arts. Examples of tangible computer-readable media include, for example: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices that are specially configured to store or to store and execute program code, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, other non-volatile memory (NVM) devices (such as 3D XPoint-based devices), and ROM and RAM devices. Examples of computer code include machine code, such as produced by a compiler, and files containing higher level code that are executed by a computer using an interpreter. Embodiments of the present disclosure may be implemented in whole or in part as machine-executable instructions that may be in program modules that are executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In distributed computing environments, program modules may be physically located in settings that are local, remote, or both.
One skilled in the art will recognize no computing system or programming language is critical to the practice of the present disclosure. One skilled in the art will also recognize that a number of the elements described above may be physically and/or functionally separated into modules and/or sub-modules or combined together.
It will be appreciated to those skilled in the art that the preceding examples and embodiments are exemplary and not limiting to the scope of the present disclosure. It is intended that all permutations, enhancements, equivalents, combinations, and improvements thereto that are apparent to those skilled in the art upon a reading of the specification and a study of the drawings are included within the true spirit and scope of the present disclosure. It shall also be noted that elements of any claims may be arranged differently including having multiple dependencies, configurations, and combinations.
This patent application is related to and claims priority benefit under 35 USC § 119(e) to co-pending and commonly-owned U.S. Pat. App. No. 63/039,967, filed on 16 Jun. 2020, entitled “CROSS-LINGUAL UNSUPERVISED SENTIMENT CLASSIFICATION WITH MULTI-VIEW TRANSFER LEARNING,” and listing Hongliang Fei and Ping Li as inventors, which patent document is incorporated by reference herein in its entirety and for all purposes.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20050071152 | Morimoto | Mar 2005 | A1 |
| 20110258229 | Ni | Oct 2011 | A1 |
| 20210390270 | Fei | Dec 2021 | A1 |
| Entry |
|---|
| Zhou et al., “Cross-lingual sentiment classification with stacked autoencoders”. Knowl Inf Syst 47, 27-44 (Year: 2016). |
| Xin et al., “A Robust Self-Learning Framework for Cross-Lingual Text Classification”. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6306-6310, Hong Kong, China (Year: 2019). |
| The Extended European Search Report dated Nov. 5, 2021, In European Application No. EP21178520. (9pgs). |
| Jeremy Jordan ,“Introduction to autoencoders,” 2018, [online], [Retrieved Oct. 26, 2021] Retrieved from Internet: <URL:https://www.jeremyjordan.me/autoencoders/> [retrieved on Oct. 26, 2021] (17pgs). |
| Lample et al., “Cross-lingual Language Model Pretraining,” arXiv preprint arXiv: 1901.07291, 2019. (10pgs). |
| Fei et al.,“Cross-Lingual Unsupervised Sentiment Classification with Multi-View Transfer Learning,” In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. (13pgs). |
| Sabour et al.,“Dynamic routing between capsules,” arXiv preprint arXiv:1710.09829, 2017. (11 pgs). |
| Sennrich et al.,“Neural machine translation of rare words with subword units,” arXiv preprint arXiv:1508.07909, 2016. (11 pgs). |
| Smith et al.,“Offline bilingual word vectors, orthogonal transformations and the inverted softmax,” arXiv preprint arXiv:1702.03859, 2017. (10pgs). |
| Vaswani et al.,“Attention is all you need,” arXiv preprint arXiv:1706.03762, 2017. (15pgs). |
| Vincent et al.,“Extracting and composing robust features with denoising autoencoders,” In Proceedings of the Twenty-Fifth International Conference on Machine Learning (ICML), 2008. (8pgs). |
| Wu et al.,“Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT,” In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on NaturalLanguage Processing (EMNLP-IJCNLP), 2019. (12pgs). |
| Xiao et al.,“Semi-supervised representation learning for cross-lingual text classification,” In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013. (11 pgs). |
| Andrade et al.,“Cross-lingual text classification using topic-dependent word probabilities,” In Proceedings of the 2015 Conference of the North American Chapter of the ACL, 2015. (6pgs). |
| N′uria Bel et al.,“Cross-lingual text categorization,” In Proceedings of the 7th European Conference on Research and Advanced Technology for Digital Libraries, 2003.(14pgs). |
| M. Chen et al.,“Marginalized denoising autoencoders for domain adaptation,” In Proceedings of the 29th International Conference on Machine Learning (ICML), 2012. (8pgs). |
| X. Chen et al.,“Adversarial deep averaging networks for cross-lingual sentiment classification,” arXiv preprint arXiv:1606.01614, 2018. (14pgs). |
| Z. Chen et al.,“Transfer capsule network for aspect level sentiment classification,” In Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL), 2019. (10pgs). |
| Conneau et al.,“Cross-lingual language model pretraining,” arXiv preprint arXiv:1901.07291, 2019. (10pgs). |
| Conneau et al.,“Very deep convolutional networks for text classification,” arXiv preprint arXiv:1606.01781, 2017. (10pgs). |
| Devlin et al.,“BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2019. (16pgs). |
| K. Xu et al.,“Towards a universal sentiment classifier in multiple languages,” In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017. (10 pgs). |
| R. Xu et al.,“Cross-lingual distillation for text classification,” In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2017.(11 pgs. |
| Q. Zhang et al.,“Multi-view knowledge graph embedding for entity alignment,” arXiv preprint arXiv:1906.02390, 2019. (7pgs). |
| X. Zhang et al.,“Character-level convolutional networks for text classification,” arXiv preprint arXiv:1509.01626, 2015. (9pgs). |
| Zhao et al.,“Multi-view learning overview: Recent progress and new challenges,” Inf. Fusion, 38:43-54, 2017. (52 pgs). |
| Zhou et al.,“Cross-lingual sentiment classification with bilingual document representation learning,” In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), 2016. (10 pgs). |
| Ziemski et al.,“The United Nations Parallel Corpus v1.0,” In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC), 2016. (5pgs). |
| Ziser et al.,“Deep pivot-based modeling for cross-language cross-domain transfer with minimal guidance,”. |
| Lample et al.,“Unsupervised machine translation using monolingual corpora only,” arXiv preprint arXiv:1711.00043, 2018. (14pgs). |
| Lample et al.,“Word translation without parallel data,” arXiv preprint arXiv:1710.04087, 2018. (14pgs). |
| Lample et al.,“Phrase-based & neural unsupervised machine translation,” arXiv preprint arXiv:1804.07755, 2018. (14pgs). |
| Li et al.,“A survey of multi-view representation learning,” arXiv preprint arXiv:1610.01206, 2018. (20pgs). |
| Lin et al.,“An empirical study on sentiment classification of chinese review using word embedding,” arXiv preprint arXiv:1511.01665, 2015. (9pgs). |
| Mohammad et al.,“How translation alters sentiment,” Journal of Artificial Intelligence Research 55, 95-130, 2016. (37pgs). |
| Pham et al.,“Learning distributed representations for multilingual text sequences,” In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, 2015. (7pgs). |
| Prettenhofer et al.,“Cross-language text classification using structural correspondence learning,” In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), 2010. (10pgs). |
| Dong et al.,“A robust selflearning framework for cross-lingual text classification,” In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019. (5pgs). |
| Duh et al.,“Is machine translation ripe for cross-lingual sentiment classification?” In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011. (5pgs). |
| Eriguchi et al.,“Zero-shot cross-lingual classification using multilingual neural machine translation,” arXiv preprint arXiv:1809.04686, 2018. (8pgs). |
| Fu et al.,“Transductive multi-view zero-shot learning,” arXiv preprint arXiv:1501.04560, 2015. (17pgs). |
| He et al.,“An interactive multi-task learning network for end-to-end aspect-based sentiment analysis,” arXiv preprint arXiv: 1906.06906, 2019. (12pgs). |
| Keung et al.,“Adversarial learning with contextual embeddings for zero-resource cross-lingual classification and NER,” arXiv preprint arXiv:1909.00153, 2019. (6pgs). |
| Kingma et al.,“Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2015. (15pgs). |
| Koehn et al.,“Moses: Open source toolkit for statistical machine translation,” In Proc. of the 45th Annual Meeting of the ACL, 2007. (4pgs). |
| In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. (12pgs). |
| Zou et al.,“Bilingual Word Embeddings for Phrase-Based Machine Translation,” In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013. (6 pgs). |
| Number | Date | Country | |
|---|---|---|---|
| 20210390270 A1 | Dec 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| 63039967 | Jun 2020 | US |
