REPRESENTATION LEARNING OF CROSS-LANGUAGE TEXTS

BACKGROUND

Natural Language Understanding (NLU) is a technology that uses natural language to communicate with computers, which aims to enable computers to understand and use a natural language to achieve communication between humans and computers, thereby replacing humans to perform various tasks related to the natural language, e.g., a machine reading comprehension task, a classification task, a question answering task, etc. A NLU task may be performed through a trained machine learning model. A performance of the machine learning model to perform the NLU task depends on a large amount of reliable training data. For resource-rich languages such as English, there are large-scale human-labeled training data for some NLU tasks. Therefore, these NLU tasks have outstanding performance for the resource-rich languages.

SUMMARY

This Summary is provided to introduce a selection of concepts that are further described below in the Detailed Description. It is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Embodiments of the present disclosure propose a method and apparatus for representation learning of cross-language texts. A source language text and a target language text may be obtained. An initial joint representation of the source language text and the target language text may be generated. Relations among a plurality of words in the source language text and the target language text may be identified. A joint representation of the source language text and the target language text may be generated based on the initial joint representation and the relations. The joint representation may be projected to at least a target language representation corresponding to the target language text.

It should be noted that the above one or more aspects comprise the features hereinafter fully described and particularly pointed out in the claims. The following description and the drawings set forth in detail certain illustrative features of the one or more aspects. These features are only indicative of the various ways in which the principles of various aspects may be employed, and this disclosure is intended to include all such aspects and their equivalents.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed aspects will hereinafter be described in conjunction with the appended drawings that are provided to illustrate and not to limit the disclosed aspects.

FIG. 1 is a schematic diagram illustrating an exemplary source language text, an exemplary target language text and exemplary relations among them according to an embodiment of the present disclosure.

FIG. 2 illustrates an exemplary process for constructing a graph corresponding to a source language text and a target language text according to an embodiment of the present disclosure.

FIG. 3 illustrates an exemplary graph according to an embodiment of the present disclosure.

FIG. 4 illustrates an exemplary process for representation learning of cross-language texts according to an embodiment of the present disclosure.

FIG. 5 illustrates an exemplary process for updating a previous joint representation according to an embodiment of the present disclosure.

FIG. 6 illustrates an exemplary process for obtaining current attention information according to an embodiment of the present disclosure.

FIG. 7 illustrates an exemplary process for pre-training a representation obtaining model according to an embodiment of the present disclosure.

FIG. 8 illustrates exemplary graphs adopting masking strategies according to embodiments of the present disclosure.

FIG. 9 is a flowchart of an exemplary method for representation learning of cross-language texts according to an embodiment of the present disclosure.

FIG. 10 illustrates an exemplary apparatus for representation learning of cross-language texts according to an embodiment of the present disclosure.

FIG. 11 illustrates an exemplary apparatus for representation learning of cross-language texts according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure will now be discussed with reference to several exemplary implementations. It is to be understood that these implementations are discussed only for enabling those skilled in the art to better understand and thus implement the embodiments of the present disclosure, rather than suggesting any limitations on the scope of the present disclosure.

It is desired to extend NLU tasks such as a machine reading comprehension task, a classification task, a question answering task, etc., to resource-scarce languages, such as German, Spanish, French, etc. However, for the resource-scarce languages, there is little or no reliable training data, which restricts the performance of machine learning models when performing the NLU tasks for resource-scarce languages. The problem of lack of training data for the resource-scarce languages may be solved through cross-lingual NLU. The cross-language NLU may transfer knowledge acquired from texts in resource-rich languages to texts in resource-scarce languages with the aid of machine translation, to help enhance the understanding of texts in the resource-scarce languages, thereby may obtaining more accurate representations of the texts in the resource-scarce languages, and further improving the performance of the NLU task for the texts in the resource-scarce languages. Herein, a resource-rich language that provides knowledge may be referred to as a source language, and a resource-scarce language that learns knowledge may be referred to as a target language. Accordingly, a text in a source language may be referred to as a source language text, and a text in a target language may be referred to as a target language text. There are some existing cross-lingual NLU methods. For example, training data in the source language for a specific NLU task may be translated into training data in the target language, and the translated training data in the target language may be used to train a machine learning model. When the trained machine learning model is actually deployed, a representation of the target language text may be generated and the specific NLU task may be performed with the generated representation.

Embodiments of the present disclosure propose an improved method for obtaining a representation of a target language text through cross-lingual NLU. Herein, a representation may refer to a collection of information that is generated based on raw data and in a form that is conducive to be processed by a machine learning model. A source language text and a target language text may be obtained, a joint representation of the source language text and the target language text may be generated based on relations among a plurality of words in the source language text and the target language text, and the joint representation may be projected to a representation corresponding to the source language text and/or a representation corresponding to the target language text. Herein, a word may broadly refer to a basic language unit that constitutes texts in different languages, and relations among words may refer to associations among the words based on predetermined criteria. Relations among a plurality of words in the source language text and the target language text may include, e.g., alignment relations among words in the source language text and words in the target language text. Two words that have an alignment relation may usually have similar semantic meaning. Considering the alignment relation when generating a joint representation of the source language text and the target language text may facilitate to understand the semantics of each word in the source language text and the target language text. In addition, the relations among the plurality of words in the source language text and the target language text may include dependency relations related to a syntax structure of each text, e.g., dependency relations among a plurality of words in the source language text and dependency relations among a plurality of words in the target language text. The correct alignment relations between the source language text and the target language text may be found with the aid of the dependency relations. Considering the alignment relations and dependency relations when generating the representations of the source language text and the target language text may enhance the knowledge transfer between the source language text and the target language text, to generate a better joint representation of the source language text and the target language text, thereby obtaining a more accurate representation corresponding to the source language text and/or a more accurate representation corresponding to the target language text. Through the method according to the embodiments of the present disclosure, both the representation of the source language text and the representation of the target language text may be improved, thus this method may also be referred to as a representation learning method of cross-language texts. The representation of the source language text and/or the representation of the target language text may be further used to perform various NLU tasks. When performing tasks with the source language representation and/or target language representation, a more accurate representation may facilitate to obtain a more accurate result.

In an aspect, an embodiment of the present disclosure proposes to explicitly model a plurality of words in a source language text and a target language text and relations among the plurality of words though constructing a graph corresponding to the source language text and the target language text, thereby semantic associations among these words may be better captured. In the graph, the words in the source language text and the target language text may be represented through nodes, and the relations among the words may be represented through edges among the nodes. The edges among the nodes may correspond to the relations among the words. For example, when a relation between two words is an alignment relation, an edge between two nodes corresponding to the two words may be an alignment edge; and when a relation between two words is a dependency relation, an edge between two nodes corresponding to the two words may be a dependency edge. Herein, a node having an edge with a current node may be referred to as a neighbor node of the current node. For example, a node having an alignment edge with a current node may be an alignment neighbor node of the current node, and a node having a dependency edge with a current node may be a dependency neighbor node of the current node. Since the graph corresponding to the source language text and the target language text constructed according to the embodiments of the present disclosure integrates the dependency relations related to the syntax structure and the alignment relations, the graph may also be referred to as a Syntax-Enhanced and Alignment-Aware graph, abbreviated as a SA graph.

In another aspect, an embodiment of the present disclosure proposes to obtain a representation of at least one neighbor node of each node corresponding to each word in a source language text and a target language text when obtaining a representation of the source language text and a joint representation of the target language text, and update a representation of a node with the obtained representation of the at least one neighbor node, wherein representations of various node may be combined into the joint representation. The neighbor nodes may be identified from the SA graph. In this way, intrinsic semantic associations among neighboring nodes may be propagated via the edge connections in the SA graph, which facilitates to generate a more accurate representation of each node.

In yet another aspect, an embodiments of the present disclosure proposes that when updating a representation of a node with a representation of a neighbor node, if the neighbor node is a neighbor node with an alignment edge with the node, the representation of the node may be updated based on a semantic difference between the neighbor node and the node. For example, when the semantic difference between the neighbor node and the node is large, e.g., when there is an alignment error, a weight of the representation of the neighbor node may be reduced when the representation of the node is being updated, thereby, the impact caused by the alignment error may be alleviated. In addition, when the neighbor node is a neighbor node with a dependency edge with the node, the representation of the node may be updated based on an importance of the neighbor node relative to the node. For example, if the neighbor node is important relative to the node, when the representation of the node is being updated, a greater weight corresponding to the representation of the neighbor node may be given, so that more consideration may be given to the representation of the neighbor node when the representation of the node is being updated.

In still another aspect, an embodiment of the present disclosure proposes to obtain a representation of a source language text and a representation of a target language text through a model based on deep learning, and proposes to a pre-training strategy to guide the model to be able to learn a text representation from semantic associations such as alignment relations embedded in between a source language text and a target language text, dependency relations among a plurality of words in the source language text, and the dependency relations among a plurality of words in the target language text. Herein, a model used to obtain a representation of a source language text and a representation of a target language text may be referred to as a representation obtaining model. In a pre-training strategy, a representation obtaining model may be pre-trained through masking one or more nodes in one node set of a source language node set corresponding to the source language text sample and a target language node set corresponding to the target language text sample; and for each node, recovering the node with at least the representation of a alignment neighbor node of the node. The pre-training strategy may guide the representation obtaining model to learn text representations from semantic associations embedded in alignment relations. In another pre-training strategy, the representation obtaining model may be pre-trained through masking one or more node pairs having alignment edges in a source language node set corresponding to a source language text sample and a target language node set corresponding to a target language text sample; and for each node, recovering the node with a representation of a dependency neighbor node of the node. The pre-training strategy may guide the representation obtaining model to learn the text representation from semantic associations embedded in dependency relations.

FIG. 1 is a schematic diagram 100 illustrating an exemplary source language text, an exemplary target language text and exemplary relations among them according to an embodiment of the present disclosure. A source language text 102 may be a text “We should protect the environment” is English, which may include a set of source language words, i.e., “we”, “should”, “protect”, “the”, and “environment”. Herein, a word included in a source language text in a source language may be referred to as a source language word. A target language text 104 may be a text “Wir sollten die Umwelt schützen” in German, which may include a set of target language words, i.e., “wir”, “sollten”, “die”, “Umwelt” and “schützen”. Herein, a word included in a target language text in a target language may be referred to as a target language word. The source language text 102 may be obtained through translating the target language text 104. Alternatively, the target language text 104 may be obtained through translating the source language text 102.

According to an embodiment of the present disclosure, relations among a plurality of words in the source language text 102 and the target language text 104 may be identified.

Alignment relations between the source language words in the source language text 102 and the target language words in the target language text 104 may be identified. In an embodiment, if a source language word and a target language word have similar semantic meaning, there is an alignment relation between the source language word and the target language word. The alignment relation may be identified through a known alignment identification technology such as GIZA++. In the schematic diagram 100, the alignment relations between the source language words in the source language text 102 and the target language words in the target language text 104 are shown by solid lines. For example, there may be an alignment relation between the source language word “we” and the target language word “wir”, there may be an alignment relation between the source language word “should” and the target language word “sollten”, there may be an alignment relation between the source language word “protect” and the target language word “schützen”, there may be an alignment relation between the source language word “the” and the target language word “die”, and there may be an alignment relation between the source language word “environment” and the target language word “Umwelt”.

In addition, dependency relations among a set of words in the same language text may be identified, such as the dependency relations among a set of source language words, the dependency relations among a set of target language words, etc. In an embodiment, a syntax tree corresponding to a text may be constructed based on a syntax structure of the text, and it may be determined that there is a dependency relation between two words based on an association between the two words in the syntax tree. The dependency relation may be identified through a known dependency identification technology such as Stanza. The schematic diagram 100 shows a syntax tree 106 corresponding to the source language text 102, and a syntax tree 108 corresponding to the target language text 104. In the syntax trees 106 and 108, the dependency relations among the source language words in the source language text 102 and the dependency relations among the target language words in the target language text 104 are shown by dotted lines, respectively. For example, there may be dependency relations between the source language word “protect” and the source language words “we”, “should” and “environment”, there may be a dependency relation between the source language word “environment” and the source language word “the”; and there may be dependency relations between the target language word “schützen” and the target language words “Umwelt”, “sollten” and “wir”, there may be a dependency relation between the target language word “Umwelt” and the target language word “die”.

It should be appreciated that the criteria and methods for identifying the relations among the plurality of words in the source language text and the target language text described above in conjunction with FIG. 1 are only exemplary. According to actual application requirements, the relations among the words may also be identified through other criteria or methods.

According to an embodiment of the present disclosure, a plurality of words in a source language text and a target language text and relations among the plurality of words may be explicitly modeled though constructing a graph corresponding to the source language text and the target language text, thereby semantic associations among these words may be better captured. FIG. 2 illustrates an exemplary process 200 for constructing a graph corresponding to a source language text and a target language text according to an embodiment of the present disclosure.

At 210, a set of source language words in a source language text and a set of target language words in a target language text may be set as a plurality of nodes.

At 220, for every two nodes in the plurality of nodes, it may be determined whether there is a relation between two words corresponding to the two nodes. As described above in conjunction with FIG. 1, the relations may include an alignment relation between a source language words in the set of source language words and a target language word in the set of target language words. If the source language word and the target language word have similar semantic meaning, there is a relation between the source language word and the target language word, and the relation is the alignment relation. In addition, the relations may further include dependency relations among the set of source language words and dependency relations among the set of target language words. A syntax tree corresponding to a text may be constructed based on a syntax structure of the text, and it may be determined that there is a relation between two words based on association between the two words in the syntax tree, and that the relation is a dependency relation.

At 230, in response to determining that there is a relation between two words, an edge between two nodes corresponding to the two words that corresponds to the relation may be determined. For example, when there is an alignment relation between two words, it may be determined that there is an alignment edge between two nodes corresponding to the two words. In addition, when there is a dependency relation between two words, it may be determined that there is a dependency edge between two nodes corresponding to the two words.

The steps 220 and 230 may be performed for every two nodes in the plurality of nodes corresponding to the set of source language words in the source language text and the set of target language words in the target language text. At 240, a set of edges among the plurality of nodes may be obtained. Therefore, through the exemplary process from the step 220 to the step 240 in the process 200, the set of edges among the plurality of nodes may be determined based on the relations among the plurality of words in the source language text and the target language text.

At 250, the plurality of nodes and the obtained set of edges may be combined into a graph corresponding to the source language text and the target language text. As described above, since the graph corresponding to the source language text and the target language text constructed according to the embodiments of the present disclosure integrates the dependency relations related to the syntax structure and the alignment relations, the graph may also be referred to as a Syntax-Enhanced and Alignment-Aware graph, abbreviated as a SA graph.

It should be appreciated that the process 200 in FIG. 2 is only an example of a process for constructing an SA graph corresponding to a source language text and a target language text based on relations among a plurality of words in the source language text and the target language text. According to actual application requirements, the process for constructing a SA graph may comprise any other steps, and may comprise more or fewer steps.

FIG. 3 illustrates an exemplary SA graph 300 according to an embodiment of the present disclosure. The SA graph 300 may be, e.g., a SA graph constructed through the process 200 in FIG. 2 and corresponding to the source language text 102 and the target language text 104 in FIG. 1.

The nodes 302 to 310 may respectively correspond to the source language words “protect”, “we”, “should”, “environment” and “the” in the source language text 102. Herein, a node corresponding to a source language word in a source language text may be referred to as a source language node. That is, the nodes 302 to 310 may be referred to as source language nodes. The node 312 to the node 320 may respectively correspond to the target language words “wir”, “sollten”, “Umwelt”, “die”, and “schützen” in the target language text 104. Herein, a node corresponding to a target language word in a target language text may be referred to as a target language node. That is, the node 312 to the node 320 may be referred to as target language nodes.

In the graph 300, an alignment edge between each source language node in the source language nodes 302 to 310 and a corresponding target language node in the target language nodes 312 to 320 is shown by solid lines. For example, there may be an alignment edge between the source language node 302 and the target language node 320, there may be an alignment edge between the source language node 304 and the target language node 312, there may be an alignment edge between the source language node 306 and the target language node 314, there may be an alignment edge between the source language node 308 and the target language node 316, and there may be an alignment edge between the source language node 310 and the target language node 318.

In addition, in the graph 300, dependency edges among the source language nodes 302 to 310 and dependency edges among the target language nodes 312 to 320 are shown by dotted lines. For example, there may be dependency edges between the source language node 302 and the source language nodes 304 to 308, and there may be a dependency edge between the source language node 308 and the source language node 310; and there may be dependency edges between the target language node 320 and the target language nodes 312 to 316, and there may be a dependency edge between the target language node 316 and the target language node 318.

Each node may have one or more neighbor nodes. Herein, a node having an edge with a current node may be referred to as a neighbor node of the current node. As described above, edges may include an alignment edge and a dependency edge. Accordingly, a node having an alignment edge with a current node may be an alignment neighbor node of the current node, and a node having a dependency edge with a current node may be a dependency neighbor node of the current node. For example, the node 302 may be an alignment neighbor node of the node 320, and the node 314 may be a dependency neighbor node of the node 320.

It should be appreciated that the SA graph 300 shown in FIG. 3 is only an example of a constructed SA graph corresponding to a source language text and a target language text. According to actual application requirements, a SA graph corresponding to a source language text and a target language text may have any other structure and may include other elements.

FIG. 4 illustrates an exemplary process 400 for representation learning of cross-language texts according to an embodiment of the present disclosure. For example, a source language representation 452 of a source language text 402 and a target language representation 462 of a target language text 404 may be obtained through a representation obtaining model 410. Herein, a representation corresponding to a source language text obtained through a representation obtaining model may be referred to as a source language representation, and a representation corresponding to a target language text obtained though a representation obtaining model may be referred to as a target language representation.

Firstly, the source language text 402 S and the target language text 404 T may be obtained. The target language text T may be given, and then the source language text S may be obtained through translating the target language text T. Alternatively, the source language text S be given, and then the target language text T may be obtained through through translating the source language text S. The source language text S and the target language text T may include tokens of different lengths. A length of the source language text S and a length of the target language text T may be unified as l though operations such as padding or truncation. The source language text S (S∈ custom-character ) and the target language text T (T∈) with a uniform length may be input in parallel into the representation obtaining model 410, where d is a dimension of a token embedding vector.

The source language text S may be input into a transformer layer 420 in the representation obtaining model 410. The transformer layer 420 may generate an initial source language representation 422 A₀^sof the source language text S. The target language text T may be input into a transformer layer 430 in the representation obtaining model 410. The transformer layer 430 may generate an initial target language representation 432 A₀^tof the target language text T. The above processing may be expressed by the following formulas:

A
₀
^s=Transformer(S) (1)

A
₀
^t=Transformer(T) (2)

The initial source language representation A₀^sand the initial target language representation A₀^tmay be combined, e.g., cascaded, to an initial joint representation 434 [A₀^s; A₀^t] ([A₀^s; A₀^t] ∈ custom-character ). The initial joint representation [A₀^s; A₀^t] may be provided to a set of improved transformer layers 440. The set of improved transformer layers 440 may generate a joint representation 442 [A^s; A^t] of the source language text S and the target language text T based on the initial joint representation [A₀^s; A₀^t] and relations among a plurality of words in the source language text S and the target language text T. In an implementation, a SA graph 452 corresponding to the source language text S and the target language text T may be constructed based on the relations among a plurality of words in the source language text S and the target language text T though a graph construction module 450. The graph constructing module 450 may construct the SA graph 452 through e.g., the process 200 in FIG. 2. Further, the set of improved transformer layers 440 may update the initial joint representation [A₀^s; A₀^t] to a joint representation [A^s; A^t] based on the graph 452.

The update operation may be performed iteratively through the set of improved transformer layers 440, to update the initial joint representation [A₀^s; A₀^t] to the joint representation [A^s; A^t]. The set of improved transformer layers 440 may include, e.g., N improved transformer layers 440-1, 440-2, . . . , 440-N having a same model structure. An improved converter layer 440-n (n∈[1, N]) may update a previous joint representation [A_n-1^s; A_n-1^t] output by a previous improved transformer layer 440-n-1 to a current joint representation [A_n^s; A_n^t], as shown in the following formula:

[A_n^s; A_n^t]=Transformer_sa([A_n-1^s; A_n-1^t]) (3)

An exemplary process for updating a previous joint representation will be described later in conjunction with FIG. 5. After the joint representation [A^s; A^t] is obtained, the joint representation [A^s; A^t] may be projected to the source language representation 452 A^scorresponding to the source language text S through a transformer layer 450. In addition, the joint representation [A^s; A^t] may be projected to the target language representation 462 A^tcorresponding to the target language text T through a transformer layer 460.

The source language representation A^sor the target language representation A^tmay be further used to perform various NLU tasks, such as a machine reading comprehension task, a classification task, a question answering task, etc. Taking the target language representation A^tbeing used to perform the machine reading comprehension task as an example, the task may find an answer fragment for a specific question from a given passage. The target language representation A^tmay be provided to two separate linear layers. After being processed through each linear layer, a softmax operation may be performed respectively, to generate final prediction results of an answer segment, i.e., a start position prediction p_s^t∈ custom-character and an end position prediction p_e^t∈ for the answer segment, as shown in the following formulas:

p
_s
^t=softmax(A^t·u_s+b_s) (4)

p
_e
^t=softmax(A^t·u_e+b_e) (5)

wherein u_sand u_e∈ custom-character , as well as b_sand b_e∈ are trainable model parameters. For example, when training for the machine reading comprehension task, the parameters u_s, u_e, b_s, and b_emay be learned through minimizing a standard cross-entropy loss as shown in the following formula:

$\begin{matrix} ℒ^{t} = - \frac{1}{ 𝒟 } \sum_{i = 1}^{ 𝒟 } (𝒴_{s, i}^{t} \cdot \log (p_{s, i}^{t}) + 𝒴_{e, i}^{t} \cdot \log (p_{e, i}^{t})) & (6) \end{matrix}$

wherein ∥ custom-character ∥ is the number of training samples, and ∈ are ground truth labels for the start position and the end position of the i-th training sample, respectively.

It should be appreciated that the process 400 in FIG. 4 is only an example of a process for representation learning of cross-language texts. According to actual application requirements, the process for representation learning of cross-language texts may comprise any other steps, and may comprise more or fewer steps. In addition, it should be appreciated that although in the above description, the source language representation or the target language representation is used to perform the machine reading comprehension task, the source language representation or the target language representation generated according to the embodiments of the present disclosure may also be used to perform various other NLU tasks, such as a classification task, a question answering task, etc.

FIG. 5 illustrates an exemplary process 500 for updating a previous joint representation according to an embodiment of the present disclosure. For example, a previous joint representation 504 corresponding to a source language text and a target language text may be updated to a current joint representation 512 corresponding to the source language text and the target language text through an improved transformer layer 510. The improved transformer layer 510 may, e.g., correspond to any transformer layer 440-n in the improved transformer layers 440-1, 440-2, . . . , 440-N in FIG. 4. For example, current attention information 522 corresponding to the source language text and the target language text may be obtained based on a SA graph 502 and a previous joint representation 504 corresponding to the source language text and the target language text, and the previous joint representation 504 may be updated to the current joint representation 512 corresponding to the source language text and the target language text based on the current attention information 522.

The improved transformer layer 510 may include an attention learning module 520. The attention learning module 520 may obtain current attention information 522 based on the SA graph 502 and the previous joint representation 504. The current attention information 522 may be, e.g., an attention matrix G∈ custom-character , and an element G_i,jtherein may be an attention score indicating an attention applied by a word i to a word j. An exemplary process for obtaining current attention information will be described later in conjunction with FIG. 6. The attention score G_i,jmay include information about an alignment relation and/or a dependency relation between the word i and the word j.

The current attention information 522 G may be provided to a multi-head attention layer 530. The multi-head attention layer 530 may firstly obtain a triplet for each head through applying a linear transformation to the previous joint representation 504. Taking the m-th head head_mas an example, the multi-head attention layer 530 may firstly obtain a triplet for the head_mthrough applying linear transformation W_m^q, W_m^kand W_m^v(W_m^q, W_m^kand W_m^v∈ custom-character ) to the previous joint representation 504, i.e., query Q_m, key K_mand value V_m(Q_m, K_mand V_m∈), wherein d_his a dimension of head_m, and W_m^q, W_m^kand W_m^vare trainable model parameters. Then, the head_mmay perform an attention operation, as shown in the following formula:

$\begin{matrix} {head}_{m} = softmax (\frac{Q_{m} K_{m}^{T}}{\sqrt{d_{k}}} + G) V_{m} & (7) \end{matrix}$

The output of each head may be cascaded. Subsequently, after being processed by an addition normalization layer 540, a feed forward layer 550, and an addition normalization layer 560, the current joint representation 512 may be obtained.

It should be appreciated that the process 500 in FIG. 5 is only an example of a process for updating a previous joint representation. According to the actual application requirements, the process for updating a previous joint representation may comprise any other steps, and may comprise more or fewer steps. In addition, the specific order or hierarchy of the steps in the process 500 is only exemplary, and the process for updating a previous joint representation may be performed in an order different from the described one.

FIG. 6 illustrates an exemplary process 600 for obtaining current attention information according to an embodiment of the present disclosure. The process 600 may be performed by, e.g., the attention learning module 520 in FIG. 5. In the process 600, current attention information corresponding to a source language text and a target language text may be obtained based on a SA graph corresponding to the source language text and the target language text and a previous joint representation of the source language text and the target language text.

At 610, a previous word representation corresponding to each word may be obtained from a previous joint representation. A previous word representation corresponding to a word i may be denoted as h_i.

Subsequently, a current word representation of each word may be obtained based at least on a SA graph. At 620, one or more neighbor nodes of a node corresponding to the word may be identified from the SA graph. One or more neighbor nodes of a node i corresponding to the word i may be denoted as custom-character (i). Adescribed above, neighbor nodes (i) may include, e.g., an alignment neighbor node, a dependency neighbor node, etc. The alignment neighbor node of the node i may be denoted as (i), and the dependency neighbor node of the node i may be denoted as (i).

At 630, the previous word representation may be updated to a current word representation based at least on one or more representations of the one or more neighbor nodes. Continuing to take the word i as an example, the previous word representation h_imay be updated to a current word representation f_ibased at least on representations of the neighbor nodes custom-character (i), as shown in the following formula:

f
_i= custom-character (h_i, (i)) (8)

wherein custom-character _a(⋅,⋅) is a function that uses the representations of the neighbor nodes (i) to update the representation h_iof the node i.

The neighbor nodes custom-character (i) may include, e.g., one or more alignment neighbor nodes _a(i). The previous word representation h_imay be updated, based at least on one or more representations of one or more alignment neighbor nodes _a(i), to a current word representation f_i^awhich based on alignment relations, as shown in the following formula:

f
_i
^a= custom-character _a(h_i, _a(i)) (9)

wherein custom-character _a(⋅,⋅) is a function that uses representations of the alignment neighbor nodes _a(i) to update the representation h_iof the node i.

The alignment neighbor nodes custom-character _a(i) are identified based on alignment relations between the words corresponding to the nodes. Word alignment is a challenging task. The existing alignment methods may have alignment errors. An embodiment of the present disclosure proposes to update the previous word representation h_ibased on a semantic difference between an alignment neighbor node custom-character _a(i) and the node i. For example, the impact of alignment errors for updating the current word representation f_i^amay be alleviated through a gating mechanism. In the gating mechanism, when a semantic difference between an alignment neighbor node _a(i) and the node i is large, e.g., when there is an alignment error, a weight of a representation of the alignment node custom-character _a(i) may be reduced when the representation of the node i is being updated, thereby, the impact caused by the alignment error may be alleviated. When there are a plurality of alignment neighbor nodes, the plurality of alignment neighbor nodes may be taken as a whole to determine the semantic difference with the node i. A gating coefficient g_i∈ custom-character reflecting a semantic difference between an alignment neighbor node _a(i) and a node i may be adopted to control a weight of a representation of the alignment neighbor node _a(i) in updating the representation of a word i. The gating coefficient gi may be obtained, e.g., through the following formula:

g
_i=σ(V₁·h_i+W₁·h_j) (10)

where h_j=avg{h_j|h_j∈ custom-character _a(i)} is an average value of the representations of the alignment neighbor nodes _a(i), and o is a sigmoid function. When there is an alignment neighbor node, h_jmay be a representation of the alignment neighbor node. Subsequently, the current word representation f_i^aof the word i may be generated based on h_jand the gating coefficient g_i, as shown in the following formula:

f
_i
^a=(1−g_i)⊙(V₂·h_i)+g_i⊙(W₂·h_j) (11)

wherein ⊙ may represent an element-wise multiplication operation. When the semantic difference between the alignment neighbor node custom-character _a(i) and the node i is large, the gating coefficient g_iwill be small or even close to zero, so that the weight of the representation of the alignment neighbor node _a(i) in updating the representation of the word i may be reduced to alleviate the impact caused by alignment error.

The neighbor nodes custom-character (i) may also include, e.g., one or more dependency neighbor nodes _s(i). The previous word representation h_imay be updated, based on one or more representations of one or more dependency neighbor nodes _s(i), to a current word representation f_i^swhich is based on dependency relations, as shown in the following formula:

f
_i
^s= custom-character _s(h_i, _s(i)) (12)

wherein custom-character _s(⋅,⋅) is a function that uses representations of the dependency neighbor node _s(i) to update the representation h_iof the node i.

According to an embodiment of the present disclosure, the representation of the node i may be updated based on an importance of a dependency neighbor node custom-character _s(i) relative to the node i. When there are a plurality of dependency neighbor nodes, the representation of the node i may be updated based on an importance of each dependency neighbor node relative to the node i. For example, if a dependency neighbor node is important relative to the node i, when the representation of the node i is being updated, a greater weight corresponding to the representation of the dependency neighbor node may be given, so that more consideration may be given to the representation of the dependency neighbor node when the representation of the node i is being updated. An importance a_iuof a dependency neighbor node u∈ custom-character _s(i) in the dependency neighbor nodes _s(i) relative to the node i may be calculated, e.g., through the following formula:

$\begin{matrix} α_{i u} = \frac{\exp (LR (W_{4} [h_{i}; h_{u}]))}{\sum_{k \in 𝒩_{S} (i)} \exp (LR (W_{4} [h_{i}; h_{k}]))} & (13) \end{matrix}$

wherein LR may be the Leaky ReLU activation function, and W₄∈ custom-character is a trainable model parameter.

Subsequently, a current word representation f_i^sbased on the dependency relation of the word i may be generated based on one or more representations of one or more dependency neighbor nodes custom-character _a(i) and an importance a_iucorresponding to each dependency neighbor node, as shown in the following formula:

f
_i
^s=σ(Σ(a_iuW₃h_u,∀u∈ custom-character (i))) (14)

After obtaining the current word representation f_i^abased on the alignment relations and the current word representation f_i^sbased on the dependency relations of the word i, the current word representation f_iof the word i may be generated based on the two current word representations. In an implementation, the current word representation f_imay be calculated as an average value of a sum of the current word representation f_i^aand the current word representation f_i^s, as shown in the following formula:

$\begin{matrix} f_{i} = \frac{1}{2} (f_{i}^{a} + f_{i}^{s}) & (15) \end{matrix}$

At 640, a current attention score corresponding to two words may be calculated based on two current word representations corresponding to the two words. For example, a current attention score G_i,jcorresponding to the word i and the word j may be calculated based on a current word representation f_icorresponding to the word i and a current word representation f_jcorresponding to the word j, as shown in the following formula:

G
_i,j=(W_att·f_i+b_att)·(W_att·f_j+b_att) (16)

wherein W_att∈ custom-character and b_att∈ are trainable model parameters.

The steps 610-640 may be performed for every two words in a set of source language words in the source language text and a set of target language words in the target language text. At 650, a set of current attention scores may be obtained, and each current attention score corresponds to two words in the set of source language words and the set of target language words.

At 660, the set of current attention scores may be combined into current attention information, i.e., an attention matrix G.

It should be appreciated that the process 600 in FIG. 6 is only an example of a process for obtaining current attention information. According to the actual application requirements, the process for obtaining current attention information may comprise any other steps, and may comprise more or fewer steps. For example, the current word representation f_iof the word i may be calculated through performing other combination operations on the current word representation f_i^aand the current word representation f_a^sof the word i, e.g., a weighted sum operation, a maximum pooling operation, etc. In addition, the specific order or hierarchy of the steps in the process 600 is only exemplary, and the process for obtaining current attention information may be performed in an order different from the described one.

The exemplary process for obtaining the source language representation of the source language text and the target language representation of the target language text according to the embodiments of the present disclosure is described above in conjunction with FIGS. 4 to 6. After the initial joint representation of the source language text and the target language text is obtained, the initial joint representation may be updated to the joint representation though iteratively performing the update operation. The update operation may comprise e.g., obtaining the current attention information based on the SA graph corresponding to the source language text and the target language text and the previous joint representation; and updating the previous joint representation to the current joint representation based on the current attention information. When calculating the current attention information, the alignment relations between the words in the source language text and the words in the target language text are considered. Two words that have an alignment relation may usually have similar semantic meaning. Considering the alignment relations when generating the joint representation of the source language text and the target language text may facilitate to understand semantics of each word in the source language text and the target language text. In addition, the relations among a plurality of words in the source language text and the target language text may include dependency relations related to a syntax structure of each text, e.g., dependency relations among a plurality of words in the source language text and dependency relations among a plurality of words in the target language text. Correct alignment relations between the source language text and the target language text may be found with the aid of the dependency relations. Considering the alignment relations and the dependency relations when generating the representation of the source language text and the target language text may enhance the knowledge transfer between the source language text and the target language text, to generate a more accurate joint representation of the source language text and the target language text. The joint representation may be projected to the source language representation corresponding to the source language text and/or the target language representation corresponding to the target language text. When performing a task with the source language representation and/or target language representation, a more accurate representation may facilitate to obtain a more accurate result.

According to an embodiment of the present disclosure, in order to enhance the ability of a representation obtaining model, such as the representation obtaining model 410 in FIG. 4, to generate representations, the representation obtaining model may be pre-trained with a training dataset. The training dataset may include one or more sample pairs, and each sample pair may include a source language text sample and a corresponding target language text sample. FIG. 7 illustrates an exemplary process 700 for pre-training a representation obtaining model according to an embodiment of the present disclosure. The representation obtaining model may be, e.g., the representation obtaining model 410 in FIG. 4.

At 702, a source language text sample and a target language text sample may be obtained. In an embodiment, the target language text sample may be obtained firstly, and then the source language text sample may be obtained through translating the target language text sample. Alternatively, the source language text sample be obtained firstly, and then the target language text sample may be obtained through translating the source language text sample.

At 704, a SA graph corresponding to the source language text sample and the target language text sample may be constructed. The SA graph corresponding to the source language text sample and the target language text sample may be constructed through, e.g., the process 200 of FIG. 2. The constructed SA graph may include a source language node set corresponding to the source language text sample and a target language node set corresponding to the target language text sample.

After the SA graph is constructed, according to the embodiments of the present disclosure, one or more nodes in the source language node set and/or the target language node set may be masked.

At 706a, only one or more nodes in one node set corresponding to one sample may be masked. For example, one or more source language nodes in the source language node set may be masked without masking any target language node in the target language node set, or one or more target language nodes in the target language node set may be masked without masking any source language node in the source language node set. This masking strategy may also be referred to as a single language masking strategy.

FIG. 8 illustrates exemplary SA graphs 800a and 800b adopting masking strategies according to embodiments of the present disclosure. The SA graphs 800a and 800b may correspond to the SA graph 300 in FIG. 3. Gray nodes in the SA graphs 800a and 800b may be masked nodes. The SA graph 800a may adopt e.g., the single language masking strategy. For example, a node 802 and a node 804 may be masked. The two nodes are in the same node set. In addition, for the node 804, since the node 802 is also masked, a dependency edge between the two nodes is cut off.

Alternatively, at 706b, one or more node pairs having alignment edges in the source language node set and the target language node set may be masked. For example, one or more source language nodes and one or more target language nodes having alignment edges with the one or more source language nodes may be masked. In other words, whenever a node is masked, a node aligned with it should also be masked. This masking strategy may also be referred to as a cross-language masking strategy. The SA graph 800b in FIG. 8 may adopt, e.g., the cross-language masking strategy. For example, the node 804 and a node 806 may be masked. The two nodes are in different node sets, and there is an alignment edge between the two nodes. In addition, for the node 804, since the node 806 is also masked, an alignment edge between the two nodes is cut off.

In the pre-training phase, the masking operation at 706a that adopts the single-language masking strategy may be performed in one part of the phase, and the masking operation at 706b that adopts the cross-language masking strategy may be performed in another part of the phase. For example, for a plurality of sample pairs included in the training dataset, the operation at 706a may be performed for a part of the sample pairs, and the operation at 706b may be performed for another part of the sample pairs.

After the masking operation at 706a or the masking operation at 706b is performed, at 708, a source language representation of the source language text sample and a target language representation of the target language text sample may be generated. The source language representation and the target language representation may be generated, e.g., through the representation obtaining model 410 in FIG. 4.

After the source language representation and the target language representation are generated, each masked node may be recovered based on representations of neighbor nodes of the masked node and a representation of the masked node, recovery losses corresponding to the masked node may be calculated based on recovery results, and the representation obtaining module may be pre-trained through minimizing the recovery losses. The above process will be described through taking a node i, which is one of masked nodes, as an example.

At 710, representations of one or more neighbor nodes custom-character (i) of the node i may be obtained. In the case where the masking operation at 706a is performed, the neighbor nodes (i) may include only alignment neighbor nodes (i), or include alignment neighbor nodes _a(i) and dependency neighbor nodes _s(i) of the node i. In the case where the masking operation at 706b is performed, the neighbor nodes custom-character (i) may only include dependency neighbor nodes _s(i) of the node i. A representation corresponding to each neighbor node in the neighbor nodes (i) may be obtained from a joint representation of the source language text sample and the target language text sample.

At 712, the node i may be recovered based on one or more representations of the one or more neighbor nodes custom-character (i), to obtain a first recovery result. For example, the one or more representations of the one or more neighbor nodes (i) may be input into a linear classifier. The linear classifier may output a probability predicted based on the representation for each word in an entire vocabulary, wherein the predicted probability for a word i corresponding to the node i may be denoted as P(i| custom-character (i)).

At 714, a first recovery loss custom-character _SA(i) corresponding to the node i may be calculated based on the first recovery result. In an implementation, the first recovery loss _SA(i) may be calculated through a cross-entropy function, as shown in the following formula:

custom-character
_SA(i)=−log P(i|(i)) (17)

The node i may be recovered through the representation of the node i. At 720, a representation h_iof the node i may be obtained. The representation h_icorresponding to the node i may be obtained from a joint representation of the source language text sample and the target language text sample.

At 722, the node i is recovered based on the representation h_iof the node i, to obtain a second recovery result. For example, the representation of the node i may be input into a linear classifier. The linear classifier may output a probability based on the representation for each word in an entire vocabulary predicted, wherein the predicted probability for a word i corresponding to the node i may be denoted as P(i|h_i).

At 724, a second recovery loss custom-character _TLM(i) corresponding to the node i may be calculated based on the second recovery result. In an implementation, the second recovery loss _TLM(i) may be calculated through a cross-entropy function, as shown in the following formula:

custom-character
_TLM(i)=−log P(i|h_i) (18)

After obtaining the first recovery loss custom-character _SA(i) and the second recovery loss _TLM(i) corresponding to the node i, at 730, a total recovery loss (i) may be calculated based on the first recovery loss _SA(i) and the second recovery loss _TLM(i). In an implementation, the total recovery loss (i) may be calculated as a sum of the first recovery loss custom-character _SA(i) and the second recovery loss _TLM(i), as shown in the following formula:

custom-character (i)=_SA(i)+_TLM(i) (19)

At 732, a representation obtaining model may be pre-trained through minimizing the total recovery loss custom-character (i).

In the case of adopting the single language masking strategy, since alignment neighbor nodes aligned with a masked node in another language node set are not masked, the representation obtaining model may at least recover the masked node through the representations of the alignment neighbor nodes. That is, this masking strategy may encourage the representation obtaining model to explore semantic associations through alignment relations, thereby learning text representations from semantic associations embedded in the alignment relations. In the case of adopting the cross-language masking strategy, since nodes aligned with a current node are also masked and alignment edges are cut off, the representation obtaining model is forced to use representations of dependency neighbor nodes to recover the current node, which may guide the representation obtaining model to learn text representations from semantic associations embedded in dependency relations.

It should be appreciated that the process 700 in FIG. 7 is only an example of a process for pre-training a representation obtaining model. According to the actual application requirements, the process for pre-training a representation obtaining model may comprise any other steps, and may comprise more or fewer steps. For example, although it is generally described in the above discussion that the masking operation at 706a may be performed in a part of the pre-training phase, and the masking operation at 706b may be performed in another part of the pre-training phase, according to actual application requirements, one of the masking operation at 706a and the masking operation at 706b may be enhanced, e.g., the masking operation at 706a may be performed in most of the pre-training phases, or the masking operation at 706b may be performed in most of the pre-training phases. In addition, the specific order or hierarchy of the steps in the process 700 is only exemplary, and the process for pre-training the representation obtaining model may be performed in an order different from the described one.

FIG. 9 is a flowchart of an exemplary method 900 for representation learning of cross-language texts according to an embodiment of the present disclosure.

At 910, a source language text and a target language text may be obtained.

At 920, an initial joint representation of the source language text and the target language text may be generated.

At 930, relations among a plurality of words in the source language text and the target language text may be identified.

At 940, a joint representation of the source language text and the target language text may be generated based on the initial joint representation and the relations.

At 950, the joint representation may be projected to at least a target language representation corresponding to the target language text.

In an implementation, the source language text may be obtained through translating the target language text, or the target language text may be obtained through translating the source language text.

In an implementation, the generating an initial joint representation may comprise: generating an initial source language representation of the source language text and an initial target language representation of the target language text; and combining the initial source language representation and the initial target language representation into the initial joint representation.

In an implementation, the source language text may include a set of source language words. The target language text may include a set of target language words. The identifying relations may comprise: identifying alignment relations between the set of source language words and the set of target language words; and/or identifying dependency relations among the set of source language words and dependency relations among the set of target language words.

In an implementation, the method 900 may further comprise: construct a graph corresponding to the source language text and the target language text based on the relations. The generating a joint representation may comprise: updating the initial joint representation to the joint representation based on the graph.

The constructing a graph may comprise: setting a set of source language words in the source language text and a set of target language words in the target language text as a plurality of nodes; determining a set of edges among the plurality of nodes based on the relations; and combining the plurality of nodes and the set of edges into the graph.

The determining a set of edges may comprise, for every two nodes in the plurality of nodes: determining whether there is a relation between two words corresponding to the two nodes; and in response to determining that there is a relation between the two words, determining an edge between the two nodes that corresponds to the relation.

The initial joint representation may be updated through iteratively performing an update operation. The update operation may comprise: obtaining current attention information based on the graph and a previous joint representation; and updating the previous joint representation to a current joint representation based on the current attention information.

The obtaining current attention information may comprise: calculating a current attention score corresponding to every two words in the source language text and the target language text based on the graph, to obtain a set of current attention scores; and combining the set of current attention scores into the current attention information.

The calculating a current attention score may comprise: obtaining a current word representation of each of the two words based at least on the graph; and calculating the current attention score based on two current word representations corresponding to the two words.

The obtaining a current word representation may comprise: obtaining a previous word representation corresponding to the word from the previous joint representation; identifying at least one neighbor node of a node corresponding to the word from the graph; and updating the previous word representation to the current word representation based at least on a representation of the at least one neighbor node.

The at least one neighbor node may include at least one alignment neighbor node having an alignment edge with the node. The updating the previous word representation may be further based on a semantic difference between the at least one alignment neighbor node and the node.

The at least one neighbor node may include at least one dependency neighbor node having a dependency edge with the node. The updating the previous word representation may be further based on an importance of the at least one dependency neighbor node relative to the node.

In an implementation, the joint representation may be generated through a representation obtaining model. The pre-training of the representation obtaining model may comprise at least: masking one or more nodes in only one node set of a source language node set corresponding to a source language text sample and a target language node set corresponding to a target language text sample; and for each node of the one or more nodes, recovering the node with at least a representation of at least one alignment neighbor node of the node.

In an implementation, the joint representation may be generated through a representation obtaining model. The pre-training of the representation obtaining model may comprise at least: masking one or more node pairs having alignment edges in a source language node set corresponding to a source language text sample and a target language node set corresponding to a target language text sample; and for each node of the one or more node pairs, recovering the node with a representation of at least one dependency neighbor node of the node.

The pre-training of the representation obtaining model may further comprise: recovering the node with a representation of the node.

In an implementation, the method 900 may further comprise: projecting the joint representation to a source language representation corresponding to the source language text.

It should be appreciated that the method 900 may further comprise any step/process for representation learning of cross-language texts according to the embodiments of the present disclosure described above.

FIG. 10 illustrates an exemplary apparatus 1000 for representation learning of cross-language texts according to an embodiment of the present disclosure.

The apparatus 1000 may comprise: a text obtaining module 1010, for obtaining a source language text and a target language text; an initial joint representation generating module 1020, for generating an initial joint representation of the source language text and the target language text; a relation identifying module 1030, for identifying relations among a plurality of words in the source language text and the target language text; a joint representation generating module 1040, for generating a joint representation of the source language text and the target language text based on the initial joint representation and the relations; and a projecting module 1050, for projecting the joint representation to at least a target language representation corresponding to the target language text. Moreover, the apparatus 1000 may further comprise any other modules configured for representation learning of cross-language texts according to the embodiments of the present disclosure described above.

FIG. 11 illustrates an exemplary apparatus 1100 for representation learning of cross-language texts according to an embodiment of the present disclosure.

The apparatus 1100 may comprise at least one processor 1110 and a memory 1120 storing computer-executable instructions. The computer-executable instructions, when executed, may cause the at least one processor 1110 to: obtain a source language text and a target language text, generate an initial joint representation of the source language text and the target language text, identify relations among a plurality of words in the source language text and the target language text, generate a joint representation of the source language text and the target language text based on the initial joint representation and the relations, and project the joint representation to at least a target language representation corresponding to the target language text.

In an implementation, the computer-executable instructions, when executed, may further cause the at least one processor 1110 to: construct a graph corresponding to the source language text and the target language text based on the relations. The generating a joint representation may comprise: updating the initial joint representation to the joint representation based on the graph.

It should be appreciated that the processor 1110 may further perform any other step/process of the methods for representation learning of cross-language texts according to the embodiments of the present disclosure described above.

The embodiments of the present disclosure propose a computer program product for representation learning of cross-language texts, comprising a computer program that is executed by at least one processor for: obtaining a source language text and a target language text; generating an initial joint representation of the source language text and the target language text; identifying relations among a plurality of words in the source language text and the target language text; generating a joint representation of the source language text and the target language text based on the initial joint representation and the relations; and projecting the joint representation to at least a target language representation corresponding to the target language text. In addition, the computer programs may further be performed for implementing any other step/process of the methods for representation learning of cross-language texts according to the embodiments of the present disclosure described above.

The embodiments of the present disclosure may be embodied in non-transitory computer-readable medium. The non-transitory computer readable medium may comprise instructions that, when executed, cause one or more processors to perform any operation of a method for representation learning of cross-language texts according to embodiments of the present disclosure as described above.

It should be appreciated that all the operations in the methods described above are merely exemplary, and the present disclosure is not limited to any operations in the methods or sequence orders of these operations, and should cover all other equivalents under the same or similar concepts. In addition, the articles “a” and “an” as used in this description and appended claims, unless otherwise specified or clear from the context that they are for the singular form, should generally be interpreted as meaning “one” or “one or more.”

It should also be appreciated that all the modules in the apparatuses described above may be implemented in various approaches. These modules may be implemented as hardware, software, or a combination thereof. Moreover, any of these modules may be further functionally divided into sub-modules or combined together.

Processors have been described in connection with various apparatuses and methods. These processors may be implemented using electronic hardware, computer software, or any combination thereof. Whether such processors are implemented as hardware or software will depend upon the particular application and overall design constraints imposed on the system. By way of example, a processor, any portion of a processor, or any combination of processors presented in the present disclosure may be implemented with a microprocessor, microcontroller, digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic device (PLD), a state machine, gated logic, discrete hardware circuits, and other suitable processing components configured to perform the various functions described throughout the present disclosure. The functions of a processor, any portion of a processor, or any combination of processors presented in this disclosure may be implemented with software executed by a microprocessor, a microcontroller, a DSP, or other suitable platforms.

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, threads of execution, procedures, functions, etc. The software may reside on a computer-readable medium. A computer-readable medium may include, e.g., memory, the memory may be e.g., a magnetic storage device (e.g., hard disk, floppy disk, magnetic strip), an optical disk, a smart card, a flash memory device, random access memory (RAM), read only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), a register, or a removable disk. Although a memory is shown separate from a processor in the various aspects presented throughout the present disclosure, the memory may be internal to the processor, e.g., a cache or register.

The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents to the elements of the various aspects described throughout the present disclosure that are known or later come to be known to those of ordinary skilled in the art are expressly incorporated herein and encompassed by the claims.

REPRESENTATION LEARNING OF CROSS-LANGUAGE TEXTS

Information

Publication Number

Date Filed

Date Published

Inventors

Original Assignees

CPC

International Classifications

Abstract

Description

Claims

Priority Claims (1)

PCT Information