Patent Application
20200125944
The subject technology relates generally to transferring a neural model from one language to a neural model in a second language. More specifically, the subject technology relates to transferring a neural model from one language to a neural model in a second language with representation projection as weak supervision.
Currently natural language processing is heavily Anglo-centric, while the demand for models that work in languages other than English is greater than ever. Yet, the task of transferring a model from one language to another can be expensive: in terms of factors such as annotation costs, engineering time, and effort.
Current research in Natural Language Processing (NLP) and Deep Learning has yielded systems that can achieve human parity in several key research areas such as speech recognition and machine translation. That is, these systems perform at or above the same level as humans. However, much of this research revolves around models, methods, and datasets that are Anglo-centric.
It is estimated that only about 350 million people are native English speakers, while another 500 million to 1 billion speak it as a second language. This accounts for at most 20% of the world's population. With language technologies making inroads into the digital lives of people, what is needed are NLP applications that can understand the other 80% of the world. However, building such systems from scratch can be expensive, time-consuming, and technically challenging.
According to one aspect of the present technology, a method for cross-lingual neural model transfer may include training a first neural model of a first language having a plurality of layers on annotated data of the first language based on a labeled loss function, where the training of the first neural model includes defining and updating parameters of each of the layers of the first neural model; and training a second neural model of a second language having a plurality of layers on parallel data between the first language and the second language based on an unlabeled loss function, wherein the training of the second neural model includes copying all but the lowest level layer of the first neural model and defining and updating parameters of the lowest level layer of the second neural model.
The training may be a 2-stage training process, where the first model is completely trained prior to the training of the second model, or alternatively both the first model and the second model may be jointly trained after an initial training of the first model in a joint training process.
The following description and the annexed drawings set forth in detail certain illustrative aspects of the claimed subject matter. These aspects are indicative, however, of but a few of the various ways in which the principles of the innovation may be employed and the claimed subject matter is intended to include all such aspects and their equivalents. Other advantages and novel features of the claimed subject matter will become apparent from the following detailed description of the innovation when considered in conjunction with the drawings.
Non-limiting and non-exhaustive examples are described with reference to the following Figures.
In the following detailed description, references are made to the accompanying drawings that form a part hereof, and in which are shown by way of illustrations specific embodiments or examples. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. Embodiments may be practiced as methods, systems, or devices. Accordingly, embodiments may take the form of a hardware implementation, an entirely software implementation, or an implementation combining software and hardware aspects. The following detailed description is therefore not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and their equivalents.
One reason that building NLP systems from scratch is expensive, time consuming, and technically challenging is that performant NLP models often rely on vast amounts of high quality annotated data, which comes at the cost of annotator time, effort, and money. Annotated data is some language artifact (e.g., any text) that is annotated with some additional artifact. For example, a text may be reviewed for a criterion and a label or annotation may be added to the text based on that criterion. By way of example, the criterion may be sentiment, and the label or annotation may include positive sentiment or negative sentiment.
Other exemplary criteria include style classification, where the label may include whether the artifact is formal or informal; intent understanding, where the label may include prediction of intent of the artifact selected from a plurality of predetermined intents (e.g., scheduling an event, requesting information, or providing an update); message routing, where the label may include prediction of a primary recipient among a plurality of recipients; task duration, where the label may include prediction of the duration of an event; or structured content recognition, where the label may include prediction of a category of the artifact (e.g, sorting emails into categories such as flight itineraries, shipping notifications, or hotel reservations).
Given the great cost of building systems from scratch, much of the research community's efforts towards building tools for other languages have relied on transferring existing English models to other languages.
Previous efforts to transfer English models to other languages have relied on Machine Translation (MT) to translate training data or test data from English to a target language. Other efforts have additionally considered utilizing bilingual dictionaries to directly transfer features.
Building state-of-the-art MT systems requires expertise and vast amounts of training data, which is expensive. Meanwhile, bilingual dictionaries can be equally expensive to build if done manually, or contain significant noise if induced automatically.
Other research includes the study of the transferability of neural network components in the context of image recognition. This research illustrates a technical problem in the conventional art, i.e., that higher layers of the network tend to be more specialized and domain specific, and therefore less generalizable.
However, the technical solution according to embodiments of the present disclosure includes a framework for cross-lingual transfer that is the opposite: specifically, the higher layers of a network are shared between models of different languages while maintaining separate language specific embeddings (i.e., parameters of the lower layers of the network). By sharing the higher layers of the network, accurate models can be generated in a multitude of languages, without reliance on MT, bilingual dictionaries, or annotated data in the language of the model.
Sharing information across domains also pertains to Multi-Task Learning. Work in this area can be broadly separated into two approaches: hard parameter sharing and soft parameter sharing. In hard parameter sharing, models share a common architecture with some task-specific layers, while in soft parameter sharing, tasks have their own sets of parameters that are constrained by some shared penalty.
Prior research including label projection, feature projection, and weak supervision differs from embodiments of the present disclosure, which are drawn to a neural framework that integrates task featurization, model learning, and cross-lingual transfer in a joint schema, while also being flexible enough to accommodate a wide range of target applications.
In solving the technical problems faced by the conventional art, embodiments of a general framework of the present disclosure can easily and effectively transfer neural models from one language to other languages. The framework in one aspect relies on task representations as a form of weak supervision, and is model and task agnostic. In general, neural networks include a series of nodes arranged in layers including an input layer and a prediction layer. The portion of the neural network between the input layer and the prediction layer may include one or more layers that transform inputs into representations. Each layer after the input layer trains on the preceding layer, and therefore each layer increases in feature complexity and abstraction. Task representations capture an abstract depiction of a prediction problem, and are embodied as a layer in a neural network model preceding the prediction layer. By utilizing the disclosed framework, many existing neural architectures can be ported to other languages with minimal effort.
The only requirements for transferring a neural model according to embodiments of the present disclosure are parallel data and a loss defined over task representations.
A framework according to embodiments of the present disclosure can reduce both monetary and computational costs by forgoing reliance on machine translations or bilingual dictionaries, while accurately capturing semantically rich and meaningful representations across various languages. By eliminating any reliance on or interaction with a translation means, the framework can reduce the number of instructions handled by the processor, thereby increasing system speed, conserving memory, and reducing power consumption.
It is with respect to these and other general considerations that embodiments of the present disclosure are described below. Also, although relatively specific problems have been discussed, it should be understood that the embodiments should not be limited to solving the specific problems identified above.
Hereinbelow, a framework according to embodiments is described which can transfer an existing neural model of a first language to a second language with minimal cost and effort.
Specifically, the framework: (i) is model and task agnostic, and thus applicable to a wide range of new and existing neural architectures; (ii) needs only a parallel corpus and does not need target language training data, a translation system, or a bilingual dictionary; and (iii) has the sole modelling requirement of defining a loss over task representations, thereby greatly reducing the engineering effort, monetary costs, and computational costs involved in transferring a model from one language to another.
Embodiments are especially useful when a high quality MT system is not available for a target language or specialized domain. Conventionally, an MT system, bilingual dictionary, or a pivot lexicon are required to transfer a model from one language to another; however, according to embodiments, none of these is required to accurately predict outcomes at a rate that is on par, or even exceeds, conventional solutions.
A framework for transferring a neural model from a first language to a second language according to embodiments is described in greater detail. For the sake of example, embodiments are shown and described where the first language is English and the second language is French. Of course, the present technology is not limited hereto, and it should be understood that the only limitation of the first language and the second language is that they are not the same dialect of the same language.
The training portion 101 depicts both English neural model 200 and French neural model 300. As illustrated in
Labeled data is data that has been supplemented with contextual information, typically directly by humans, and may also be referred to as annotated data.
So long as the parallel data is aligned between languages, it may be aligned at any level including character level, word level, sentence level, paragraph level, or otherwise.
According to an example embodiment as shown in
The English neural model 200 may be a neural NLP model, and may include three distinct components: an embedding layer 201, a task-appropriate model architecture 202, and a prediction layer 203.
In more detail, the English neural NLP model 200 includes a first layer that is an embedding layer 201, which converts linguistic units w (characters, words, sentences, paragraphs, pseudo paragraphs, etc.) into mathematical representations of the linguistic units w. The mathematical representations may preferably be a dense representation of vectors that include mostly non-zero values, or alternatively a sparse representations of vectors that include many zero values.
The third layer is a prediction layer 203 used to produce a probability distribution over a space of output labels. According to an example embodiment, the prediction layer 203 may include a softmax function.
Between the prediction layer 203 and the embedding layer 201 is a task-appropriate model architecture 202.
As the framework 100 is both model and task agnostic, the structure of the task-appropriate model architecture 202 may include any number of layers and any number of parameters. That is, the task-appropriate model architecture 202 is what tailors the model to a particular task or application and the configuration of the network and number of layers does not affect the application of the general framework.
Therefore, for simplicity, the task-appropriate model architecture 202 is depicted as including an x-layer network 202a (where x is a non-zero integer number of layers) and a task representation layer 202b as the layer immediately preceding the prediction layer 203.
As shown in
As shown in
According to an embodiment, the RNN may comprise for example Gated Recurrent Units (GRUs). However it should be understood that the disclosure is not limited in this regard, and the RNN may also be a long short-term memory network (LSTM) or other network.
Model transfers according to embodiments rely on two features. First, the task-appropriate architecture and the prediction layer are shared across languages. Second, all of the information required to make a successful prediction is contained in the task representation layer.
As shown in
An indication of successful model transfer is that the French model and the English model predict the same thing when considering parallel data. That is, the content of the prediction is irrelevant, but rather the success of the model transfer is based on the sameness of the prediction of the French model and English model. The content of the prediction can be an actual label when the approach is label projection. Alternatively, representation projection can be utilized where the aim is to produce the same task representation in both languages. Representation projection is a softer form of weak supervision when compared to supervision based on label projection and is the preferred projection according to an embodiment.
To better illustrate a framework in accordance with embodiments, consider a task T and labeled data DL={xi,yi)|0≤i≤N}, where xi are English inputs, yi are outputs that take on K possible values such that each xi is annotated with a value yi, and N is the number of linguistic units included in the labeled data DL. Without loss of generality assume that the inputs xi={ei1, . . . , eij} are sequences of English words. Moreover, a parallel data set DP={(ej,fj)|0≤j≤M}, where ej={ej1, . . . , eji} and fj={fj1, . . . , fji} are parallel English and French linguistic units respectively, and M is the number of linguistic unit pairs included in the parallel data DP.
English embeddings included in the English embedding layer 201 may be denoted as U={{right arrow over (ul)}|∀ei∈VE}, such that there is a vector {right arrow over (ul)} for every word in an English vocabulary VE. The English vocabulary includes all words found in the inputs xi. French embeddings included in the French embedding layer 301 may be denoted as V={{right arrow over (vl)}|∀fi∈VF}, such that there is a vector {right arrow over (vl)} for every word in a French vocabulary VF.
In the case of a shared model architecture, the dimensions d of vectors {right arrow over (ul)} and {right arrow over (vl)} must be the same. A mapping of the English sequence ej={ej1, . . . , ejm} to a sequence of vectors is denoted as and a mapping of the French sequence fj={fj1, . . . , fjn} to a sequence of vectors is denoted as
. An x-layer model 202b is denoted as μ with parameters θμ that takes as input, a sequence of embeddings, and yields a task representation. Specifically, for an English input xi, the task representation is denoted as:
R
x
T=μ(;θμ) (1)
Finally, a prediction layer 203 is denoted as n with parameters θπ that yields a probability distribution over the K output variables:
where πk is the kth neuron of the layer, and the shorthand is used to denote P(ŷl=k). The framework according to an embodiment then optimizes two losses.
Labeled Loss:
Given that the model includes labeled English data DL as an input, the following loss is optimized for the combined network:
L
D
=Σi=1NΣk=1KΔL(,yi) (3),
where ΔL is a loss function defined between and the response variable yi. For example, in a binary case ΔL might be a cross-entropy loss, though it should be understood that this is by way of example only, and the framework is not limited thereto.
Unlabeled Loss:
The English task representations generated by the model are used as weak supervision on the parallel data for the French side. Specifically:
L
D
=Σj=1MΔP(Re
where ΔP is a loss function between task representations yielded on parallel inputs. Since task representations are vectors, the mean-squared error between them, for example, might be an appropriate loss, though the framework is not limited thereto.
Then jointly, the final optimization is given by L=LD
In contrast to conventional frameworks, in a framework in accordance with an embodiment there is no requirement for MT, since neither training nor test data is ever translated. Nor are any other resources, such as a pivot lexicon or bilingual dictionary, used. The only requirement is parallel data and the definition of a loss function LD
With well-defined loss functions ΔL and ΔP, training consists of back-propagating errors through the network and updating the parameters of the model.
As shown in
A labeled loss LD
in step S303. In this context “*” denotes an optimized value for the arg max function in step S303.
After training the first model 200, the embeddings U of the first model and the shared model parameters θμ and θπ are frozen in step S304.
An unlabeled loss LD
in step S306. That is, in the second stage of the 2-stage training, only the second embeddings V of the second model are updated on the parallel data.
In step S307, the first embeddings U of the embedding layer 201 of the first model 200 are replaced with the second embeddings V of the embedding layer 301 of the second model 300. This combined model is the updated second model 300. Therefore, the updated second model 300 includes parameters V*, θμ, θπ.
As shown in
A labeled loss LD
in step S403.
An unlabeled loss LD
L. L is a weighted combination of labeled loss and unlabeled loss and is given by L=LD
In joint training, when processing the parallel data DP parameters of both the first model 200 and the second model 300 are updated in step S404.
In step S406, the first embeddings U of the embedding layer 201 of the first model 200 are replaced with the second embeddings V of the embedding layer 301 of the second model 300. This combined model is the updated second model 300. Therefore, the updated second model 300 includes parameters V*, θ*μ, θ*π.
To better illustrate a general framework according to an embodiment, in the following demonstrative example, a sentiment classifier is transferred from one language to another language.
In this example, the sentiment classifier predicts whether a language artifact is positive or negative. According to embodiments, the only necessary steps are to define the model architecture μ and the two loss functions LD
Given the binary nature of the prediction task, the prediction layer may be given as a sigmoid layer with one output neuron that computes the probability of a positive label: =σ(θπT·Rx
L
D
=−Σi=1N(yi log +(1−yi)log(1−
)) (5).
On the parallel side, the unlabeled loss may be a mean-squared error loss:
where dT is the dimension of the task representation RT, and RT(i) denotes its ith dimension.
While the above example defines loss functions for a binary system, it should be understood that other loss functions can be defined for other systems, and systems may have any number of possible outputs.
So as to demonstrate task representation as weak supervision, Table 1 shows several sentiment bearing English words, and their nearest French neighbors (by vector cosine distance on their respective embeddings) in the joint model according to an embodiment.
It can be seen in Table 1 above that the positive (or negative) sentiment terms in English have definitions that are similar to the nearest neighbor positive (or negative) terms in French. While the nearest neighbor terms in French are not necessarily direct translations or even synonyms, the sentiment prediction task does not require translations; it is sufficient to identify words that echo the same sentiment. Thus, a framework for model transfer in accordance with embodiments is able to identify sentiment similarity across languages, without direct supervision and only using a weak fuzzy signal from representation projection.
While the framework does not require MT, MT may be utilized according to an embodiment.
For example, training-time Translation (TrnT) may be utilized, which translates training data from a first language into another language, and then trains a sentiment model in that language. Test-time Translation (TstT) may be utilized, which trains a sentiment model in a first language and uses the trained sentiment model to classify language artifacts that are translated into the first language at test time.
Thus a framework according to an embodiment, while capable even without a translation engine, may optionally be used in combination with a translator.
The framework can be applied to multi-modal (instead of multi-lingual) transfer. That is, models may be transferred between different modes, including language, images, videos, audio clips, etc. For example, sentiment understanding may be transferred to images, without the need for explicit image annotations. In such a multi-modal transfer, annotated data may include labeled sentiment data in a first language. Parallel data may include images with captions in the first language. Once the framework is trained on the annotated and parallel data, the framework can predict the sentiment of images with no captions.
The computer system may be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. The computer system may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.
The components of computer system may include, but are not limited to, a server 500, one or more processors or processing units 510, and a system memory 520. Processor 510 may include software module that performs the methods described herein. The module may be programmed into the integrated circuits of processor 510, or loaded from memory 520, or network (not shown) or combinations thereof.
The computer system may include a variety of computer system readable media. Such media may be any available media that is accessible by computer system, and it may include both volatile and non-volatile media, removable and non-removable media.
Volatile memory may include random access memory (RAM) and/or cache memory or others. Other removable/non-removable, volatile/non-volatile computer system storage media may include a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided.
As will be appreciated by one skilled in the art, aspects of the framework may be embodied as a system, method, or computer program product. Accordingly, aspects of the disclosed technology may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the disclosed technology may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the disclosed technology may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages, a scripting language such as Perl, VBS or similar languages, and/or functional languages such as Lisp and ML and logic-oriented languages such as Prolog. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
Aspects of the disclosed technology are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The computer program product may comprise all the respective features enabling the implementation of the methodology described herein, and which, when loaded in a computer system, is able to carry out the methods. Computer program, software program, program, or software, in the present context means any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following: (a) conversion to another language, code or notation; and/or (b) reproduction in a different material form.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements, if any, in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the disclosed technology has been presented for purposes of illustration and description, but is not intended to be exhaustive or limiting. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The embodiment was chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various embodiments with various modifications as are suited to the particular use contemplated.
Various aspects of the present disclosure may be embodied as a program, software, or computer instructions embodied in a computer or machine usable or readable medium, which causes the computer or machine to perform the steps of the method when executed on the computer, processor, and/or machine. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform various functionalities and methods described in the present disclosure is also provided.
The system and method of the present disclosure may be implemented and run on a general-purpose computer or special-purpose computer system. The terms “computer system” and “computer network” as may be used in the present application may include a variety of combinations of fixed and/or portable computer hardware, software, peripherals, and storage devices. The computer system may include a plurality of individual components that are networked or otherwise linked to perform collaboratively, or may include one or more stand-alone components. The hardware and software components of the computer system of the present application may include and may be included within fixed and portable devices such as desktop, laptop, and/or server. A module may be a component of a device, software, program, or system that implements some “functionality”, which can be embodied as software, hardware, firmware, electronic circuitry, or etc.
Although specific embodiments have been described, it will be understood by those of skill in the art that there are other embodiments that are equivalent to the described embodiments. Accordingly, it is to be understood that the disclosure is not to be limited by the specific illustrated embodiments, but only by the scope of the appended claims.
