The present application claims the benefit of priority under 35 U.S.C. § 119 to Russian Patent Application No. 2018146352 filed Dec. 25, 2018, the disclosure of which is incorporated by reference herein.
The present disclosure is generally related to computer systems, and is more specifically related to systems and methods for neural network training utilizing specialized loss functions.
“Neural network” herein shall refer to a computational model vaguely inspired by the biological neural networks that constitute human brains. The neural network itself is not an algorithm, but rather a framework for many different machine learning algorithms to work together and process complex data inputs. Such a system would “learn” to perform tasks by considering examples, generally without being programmed with any task-specific rules.
In accordance with one or more aspects of the present disclosure, an example method of neural network training utilizing loss functions reflecting neighbor token dependencies may comprise: receiving a training dataset comprising a plurality of labeled tokens; determining, by a neural network, a first tag associated with a current token processed by the neural network, a second tag associated with a previous token which has been processed by the neural network before processing the current token, and a third tag associated with a next token to be processed by the neural network after processing the current token; computing, for the training dataset, a value of a loss function reflecting a first loss value, a second loss value, and a third loss value, wherein the first loss value is represented by a first difference of the first tag and a first label associated with the current token by the training dataset, wherein the second loss value is represented by a second difference of the second tag and a second label associated with the previous token by the training dataset, and wherein the third loss value is represented by a third difference of the third tag and a third label associated with the next token by the training dataset; and adjusting a parameter of the neural network based on the value of the loss function.
In accordance with one or more aspects of the present disclosure, another example method of neural network training utilizing loss functions reflecting neighbor token dependencies may comprise: receiving a training dataset comprising a plurality of labeled natural language words, wherein each label identifies a part of speech (POS) associated with a respective word; determining, by a neural network, a first tag identifying a first POS associated with a current word processed by the neural network, a second tag identifying a second POS associated with a previous word which has been processed by the neural network before processing the current word, and a third tag identifying a third POS associated with a next word to be processed by the neural network after processing the current word; computing, for the training dataset, a value of a loss function reflecting a first loss value, a second loss value, and a third loss value, wherein the first loss value is represented by a first difference of the first tag and a first label associated with the current word by the training dataset, wherein the second loss value is represented by a second difference of the second tag and a second label associated with the previous word by the training dataset, and wherein the third loss value is represented by a third difference of the third tag and a third label associated with the next word by the training dataset; and adjusting a parameter of the neural network based on the value of the loss function.
In accordance with one or more aspects of the present disclosure, an example computer-readable non-transitory storage medium may comprise executable instructions that, when executed by a computer system, cause the computer system to: receive a training dataset comprising a plurality of labeled natural language words, wherein each label identifies a part of speech (POS) associated with a respective word; determine, by a neural network, a first tag identifying a first POS associated with a current word processed by the neural network, a second tag identifying a second POS associated with a previous word which has been processed by the neural network before processing the current word, and a third tag identifying a third POS associated with a next word to be processed by the neural network after processing the current word; compute, for the training dataset, a value of a loss function reflecting a first loss value, a second loss value, and a third loss value, wherein the first loss value is represented by a first difference of the first tag and a first label associated with the current word by the training dataset, wherein the second loss value is represented by a second difference of the second tag and a second label associated with the previous word by the training dataset, and wherein the third loss value is represented by a third difference of the third tag and a third label associated with the next word by the training dataset; and adjust a parameter of the neural network based on the value of the loss function.
The present disclosure is illustrated by way of examples, and not by way of limitation, and may be more fully understood with references to the following detailed description when considered in connection with the figures, in which:
Described herein are methods and systems for neural network training utilizing loss functions reflecting neighbor token dependencies (e.g., relationships between the tokens of the input sequence being processed by the neural network). Neural networks trained by the methods described herein may be utilized for sequence labeling, i.e., processing an input sequence of tokens and associating each token with a label of a predetermined set of labels. The sequence labeling task may be defined as follows: producing, for an input sequence of tokens w1, . . . , wn=: W1n, a corresponding sequence of tags t1, . . . , tn=: tn1; ti ∈T, where T denotes a set of possible tags.
An example of the sequence labeling task is part-of-speech (POS) tagging, such that a neural network would process a natural language text and assign a POS-identifying tag to each work of the natural language text. “Part of speech” herein shall refer to a category of words. Words that are assigned to the same part of speech generally exhibit similar morphological attributes (e.g., similar inflection patterns). Commonly listed English parts of speech are noun, verb, adjective, adverb, pronoun, preposition, conjunction, interjection, and article. In certain implementations, POS tagging refers to assigning, to each work of the natural language text, a tag identifying a set of grammatical, morphological, and/or semantic attributes.
Accordingly, the POS labeling task may be defined as follows: producing, for an input natural language text represented by a sequence of words w1, . . . , wn=: w1n, a corresponding sequence of tags t1, . . . , tn=: tn1; ti ∈T, where T denotes a set of defined parts of speech. In particular, a neural network trained by the methods described herein is capable of resolving the homonymy, i.e., utilizing the context (relationships between words) for distinguishing between identical words having different meanings.
In other examples, the tags produced by neural networks trained by the methods described herein may identify various other grammatical and/or morphological attributes of words of natural language texts processed by the neural networks. Each tag may be represented by a tuple of grammatical and/or morphological associated with a natural language word. These tags may be utilized for performing a wide range of natural language processing tasks, e.g., for performing syntactic and/or semantic analysis of natural language texts, machine translation, named entity recognition, etc.
A neural network includes multiple connected nodes called “artificial neurons,” which loosely simulate the neurons in a human brain. Each connection, like the synapses in the human brain, can transmit a signal from one artificial neuron to another. An artificial neuron that receives a signal would process it and then transmit the transformed signal to other additional artificial neurons. In common neural network implementations, the output of each artificial neuron is computed by a function of a linear combination of its inputs. The connections between artificial neurons are called “edges.” Edge weights, which increase or attenuate the signals being transmitted through respective edges, are defined at the network training stage based on a training dataset that includes a plurality of labeled inputs (i.e., inputs with known classification). In an illustrative example, all the edge weights are initialized to random or predetermined values. For every input in the training dataset, the neural network is activated. The observed output of the neural network is compared with the desired output specified by the training data set, and the error is propagated back to the previous layers of the neural network, in which the weights are adjusted accordingly. This process may be repeated until the output error falls below a predetermined threshold.
The methods described herein utilize recurrent neural networks, which are capable of maintaining the network state reflecting the information about the inputs which have been processed by the network, thus allowing the network to use their internal state for processing subsequent inputs. However, common neural networks are susceptible to the gradient attenuation effect, which renders a network practically incapable of processing long input sequences (such as input sequences of more than five tokens).
The gradient attenuation effect may be avoided by utilizing long short-term memory (LSTM) layers, which are utilizing a gating mechanism allowing the network to choose, for performing the next layer of processing, between the own state and the input. Since the LSTM neural networks exhibit very low gradient attenuation, such networks are capable of processing longer input sequences (such as input sequences of tens of tokens).
However, a common LSTM neural network would only yield information about one of the two generally available contexts (left or right) of a given word. Accordingly, the systems and methods of the present disclosure utilize bi-directional LSTM networks (BiLSTM). A BiLSTM outputs a concatenation of the forward and backward passes of an ordinary LSTM.
The feature extraction layer 110 is employed for producing feature vectors representing the input tokens 120A-120N, which are sequentially fed to the feature extraction layer 110. In certain implementations, each feature vector may be represented by a corresponding embedding, i.e., a vector of real numbers which may be produced, e.g., by a neural network implementing a mathematical transformation from a space with one dimension per word to a continuous vector space with a much lower dimension. In an illustrative example, the feature extraction layer 110 may utilized a predetermined set of embeddings, which may be pre-built on a large corpus of natural language texts. Accordingly, word embeddings carry the semantic information and at least some morphological information about the words, such that the words which are utilized in similar contexts, as well as synonyms, would be assigned feature vectors which are located close to each other in the feature space.
The BiLSTM layer 130 processes the feature vectors produced by the feature extraction layer 110 and yields a set of vectors, such that each vector encodes information about a corresponding input tokens and its context. The prediction layer 140, which may be implemented as a feed-forward network, processes the set of vectors produced by the BiLSTM layer 130 and for each vector yields a tag of a predetermined set of tags 150A-150N (e.g., a tag indicative of a POS of the corresponding input token).
The network training may involve processing, by the neural network, a training dataset that may include one or more input sequences with classification tags assigned to each token (e.g., a corpus of natural language texts with part of speeches assigned to each word). A value of a loss function may be computed based on the observed output of the neural network (i.e., the tag produced by the neural network for a given token) and the desired output specified by the training dataset for the same token. The error reflected by the loss function may be propagated back to the previous layers of the neural network, in which the edge weights and/or other network parameters may be adjusted accordingly in order to minimize the loss function. This process may be repeated until the value of the loss function would stabilize in the vicinity of a predetermined value or fall below a predetermined threshold.
Accordingly, the baseline neural network 100 may be enhanced by adding two secondary outputs which would, in addition to the tag yields by the prediction layer 140 for the current token, yield a tag associated with the previous token and a tag associated with the next token, as schematically illustrated by
Therefore, the neural networks and training methods described herein represent significant improvements over various common systems and methods. In particular, employing loss functions that are specifically aimed at training the neural network to recognize neighbor token dependencies (e.g., relationships between the neighboring tokens of the input sequences) yields significant improvement of the overall quality and efficiency of sequence labeling methods. Various aspects of the above referenced methods and systems are described in details herein below by way of examples, rather than by way of limitation.
As noted herein above, the baseline neural network 100 may process word embeddings, which are built in such a way such that the words which are utilized in similar contexts, as well as synonyms, would be assigned feature vectors which are located close to each other in the feature space. However, a word embedding matrix, in which every dictionary word is mapped to a vector in the feature space, while having an enormous size, would still be unable to produce an embedding corresponding to a word which is not found in the dictionary. Furthermore, the relatively large size of a word embedding vector is explained by the fact that the vector carries the semantic information about the initial word, while such information may not always be useful on the labeling task (e.g., POS tagging) to be performed by the neural network. Accordingly, the neural networks implemented in accordance with one or more aspects of the present disclosure are designed to process inputs which, in addition to word-level embeddings, may include character-level embeddings and grammeme-level embeddings.
The character-level embeddings do not rely on a dictionary, but rather view each input token as a sequence of characters. A vector may be assigned to a given input token, e.g., by processing an input sequence of tokens (e.g., a natural language text represented by a sequence of words) by a neural network (such as an LSTM network and/or a fully-connected network). In certain implementations, the input tokens may be truncated by a predetermined size (e.g., 12 characters). Character-level embeddings carry grammatical and/or morphological information about the input tokens.
The grammeme-level embedding of a given word may be produced by a neural network that, for each input word, would construct a vector each element of which is related to a specific grammatical attribute of the word (e.g., reflects a probability of the input word to be associated with the specific grammatical attribute). The neural network may apply an additional dense layer to the intermediate representation of the word, such that the resulting vector produced by the neural network would represent not only individual grammatical attributes, but also certain interactions between them.
In the illustrative example of
The feature extraction layer 210 produces the grammeme embeddings 212 (e.g., by processing the input tokens by an LSTM network and/or a fully-connected network) and character-level embeddings 214 (e.g., by processing the input tokens by another LSTM network and/or a fully-connected network). The gramemme embedding 212 are then fed to the dense layer 216, the output of which is concatenated with the character-level embedding 214 and is fed to the dense layer 218. A dense layer performs a transformation in which every input is connected to every output by a linear transformation characterized by a weight value, which may be followed by a non-linear activation function (e.g., ReLU, Softmax, etc.).
It should be noted that in various other implementations, the neural networks implementing the methods described herein may process input vectors representing any combinations (e.g., concatenations) of word-level embeddings, character-level embeddings, and/or grammeme-level embeddings.
Referring again to
Conversely, in the training mode, two auxiliary prediction pipelines of the prediction layer 230 may be utilized, such that the first auxiliary prediction pipeline that includes the dense layers 232 and 233 receives its input from the backward LSTM 224 and produces the tag 242 associated with the previous token 204; the second auxiliary prediction pipeline that includes the dense layers 236 and 237 receives its input from the forward LSTM 226 and produces the tag 248 associated with the next token 206. The loss function may be computed which takes into account the differences between the respective predicted tags and the tags specified by the training dataset for the current, previous, and next tokens. Thus, the two auxiliary prediction pipeline of the prediction layer 230 are only utilized in the network training mode.
In an illustrative example, the loss function may be represented as a weighted sum reflecting the differences between the respective predicted tags and the tags specified by the training dataset for the current, previous, and next tokens:
L=w1d(Tprev,T′prev)+w2d(Tcur,T′cur)+w3d(Tnext,T′next)
where L is the value of the loss function,
d is the distance metric in the tag space,
w1, w2, and w3 are the weight coefficients,
Tprev is the tag produced by the neural network for the previous token,
Tprev is the tag associated with the previous token by the training dataset,
Tcur is the tag produced by the neural network for the current token,
Tcur is the tag associated with the current token by the training dataset,
Tnext is the tag produced by the neural network for the next token, and
Tnext is the tag associated with the next token by the training dataset.
The network training may involve processing, by the neural network, a training dataset that may include one or more input sequences with classification tags assigned to each token (e.g., a corpus of natural language texts with part of speeches assigned to each word). A value of a loss function may be computed based on the observed output of the neural network (i.e., the tag produced by the neural network for a given token) and the desired output specified by the training dataset for the same token. The error reflected by the loss function may be propagated back to the previous layers of the neural network, in which the edge weights and/or other network parameters may be adjusted accordingly in order to minimize the loss function. This process may be repeated until the value of the loss function would stabilize in the vicinity of a predetermined value or fall below a predetermined threshold.
As noted herein above, utilizing the loss function based on the three tags would force the neural network to recognize neighbor token dependencies (e.g., relationships between the neighboring tokens of the input sequences) and would thus yield a significant improvement of the overall quality and efficiency of sequence labeling methods.
At block 310, a computer system implementing the method may receive a training dataset comprising a plurality of labeled tokens (e.g., a natural language text in which each word is labeled by a tag identifying a grammatical attribute of the word, such as a POS associated with the word).
At block 320, the computer system may determine, by a neural network, the first tag associated with the current token processed by the neural network, the second tag associated with the previous token which has been processed by the neural network before processing the current token, and the third tag associated with the next token to be processed by the neural network after processing the current token. In an illustrative example, the tags may represent respective grammatical attributes (such as POS) associated with the tokens. The neural network may include a feature extraction layer, a bi-directional long-short term memory (BiLSTM) layer, and a prediction layer, such that the BiLSTM layer further includes a BiLSTM, a backward LSTM and a forward LSTM, and the outputs of the backward LSTM and the forward LSTM are fed to the BiLSTM, as described in more detail herein above.
At block 330, the computer system may compute, for the training dataset, a value of a loss function reflecting the differences between the respective computed tags and corresponding labels specified by the training dataset. In an illustrative example, the loss function may be represented by a weighted sum of the difference of the computed tag for the current token and the label associated with the current token by the training dataset, the difference of the computed tag for the previous token and the label associated with the previous token by the training dataset, and the difference of the computed tag for the next token and the label associated with the next token by the training dataset, as described in more detail herein above.
At block 340, the computer system may adjust, based on the computed value of the loss function, one or more parameters of the neural network which undergoes the training. In an illustrative example, the error reflected by the loss function value is back-propagated starting from the last layer of the neural network, and the weights and/or other network parameters are adjusted in order to minimize the loss function.
The process described by blocks 320-340 may be repeated until the value of the loss function would stabilize in a vicinity of a certain value or fall below a predetermined threshold or fall below a predetermined threshold.
At block 350, the computer system may employ the trained neural network for performing a sequence labeling task, such as a natural language processing task (e.g., POS tagging) of one or more input natural language texts, and the method may terminate.
At block 410, a computer system implementing the method may receive an input dataset comprising a plurality of tokens (e.g., a natural language text comprising a plurality of words).
At block 420, the computer system may employ a neural network (e.g., a neural network having the architecture of the neural network 200 of
At block 430, the computer system may processes the feature vectors produced by the feature extraction layer and yield a set of vectors, such that each vector encodes information about a corresponding input tokens and its context, as described in more detail herein above.
At block 440, the computer system may process the set of information encoding vectors and for each vector may yield a tag of a predetermined set of tags (e.g., a tag indicative of a grammatical attributed of the corresponding input token). Upon completing the operations of method 450, the method may terminate.
Exemplary computer system 500 includes a processor 502, a main memory 504 (e.g., read-only memory (ROM) or dynamic random access memory (DRAM)), and a data storage device 518, which communicate with each other via a bus 530.
Processor 502 may be represented by one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, processor 502 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. Processor 502 may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. Processor 502 is configured to execute instructions 526 for performing the methods described herein.
Computer system 500 may further include a network interface device 522, a video display unit 510, a character input device 512 (e.g., a keyboard), and a touch screen input device 514.
Data storage device 518 may include a computer-readable storage medium 524 on which is stored one or more sets of instructions 526 embodying any one or more of the methods or functions described herein. Instructions 526 may also reside, completely or at least partially, within main memory 504 and/or within processor 502 during execution thereof by computer system 500, main memory 504 and processor 502 also constituting computer-readable storage media. Instructions 526 may further be transmitted or received over network 516 via network interface device 522.
In an illustrative example, instructions 526 may include instructions of method 300 of neural network training utilizing loss functions reflecting neighbor token dependencies, implemented in accordance with one or more aspects of the present disclosure. In another illustrative example, instructions 526 may include instructions of method 400 of neural-network-based sequence labeling, implemented in accordance with one or more aspects of the present disclosure. While computer-readable storage medium 524 is shown in the example of
The methods, components, and features described herein may be implemented by discrete hardware components or may be integrated in the functionality of other hardware components such as ASICS, FPGAs, DSPs or similar devices. In addition, the methods, components, and features may be implemented by firmware modules or functional circuitry within hardware devices. Further, the methods, components, and features may be implemented in any combination of hardware devices and software components, or only in software.
In the foregoing description, numerous details are set forth. It will be apparent, however, to one of ordinary skill in the art having the benefit of this disclosure, that the present disclosure may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present disclosure.
Some portions of the detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, graphemes, characters, terms, numbers, or the like.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “determining”, “computing”, “calculating”, “obtaining”, “identifying,” “modifying” or the like, refer to the actions and processes of a computer system, or similar electronic computer system, that manipulates and transforms data represented as physical (e.g., electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions.
It is to be understood that the above description is intended to be illustrative, and not restrictive. Various other implementations will be apparent to those of skill in the art upon reading and understanding the above description. The scope of the disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.
Number | Date | Country | Kind |
---|---|---|---|
RU2018146352 | Dec 2018 | RU | national |
Number | Name | Date | Kind |
---|---|---|---|
9659248 | Barbosa et al. | May 2017 | B1 |
11321538 | Fontecilla | May 2022 | B1 |
20170270100 | Audhkhasi et al. | Sep 2017 | A1 |
20170308790 | Nogueira dos Santos et al. | Oct 2017 | A1 |
20180121788 | Hashimoto | May 2018 | A1 |
20180121799 | Hashimoto | May 2018 | A1 |
20180189604 | Zhang et al. | Jul 2018 | A1 |
20180225281 | Song | Aug 2018 | A1 |
20180260379 | Yu | Sep 2018 | A1 |
20180268023 | Korpusik | Sep 2018 | A1 |
20190073351 | Zhang | Mar 2019 | A1 |
20190130218 | Albright | May 2019 | A1 |
20190228073 | Chatterjee | Jul 2019 | A1 |
20190228099 | Bajaj | Jul 2019 | A1 |
20200285951 | Kim | Sep 2020 | A1 |
20210279551 | Hashimoto | Sep 2021 | A1 |
Number | Date | Country |
---|---|---|
0327266 | Aug 1995 | EP |
2630427 | Sep 2017 | RU |
2641447 | Jan 2018 | RU |
2665273 | Aug 2018 | RU |
2017083746 | May 2017 | WO |
Entry |
---|
Patel, Raj Nath, et al., “Recurrent Neural Network based Part-of-Speech Tagger for Code-Mixed Social Media Text”, https://arxiv.org/pdf/1611.04989, 7 pages, Nov. 16, 2016. |
Plank, Barbara, “Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss”, https://arxiv.org/pdf/1604.05529, 7 pages, Jul. 21, 2016. |
Wang, Tian and Cho, Kyunghyun, “Larger-Context Language Modelling with Recurrent Neural Network”, 11 pages, Aug. 7-12, 2016. |
Kashuap, Dinesh Kumar and Josan, Gurpreet Singh, “Prediction of Part of Speech Tags for Punjabi using support vector machines—Semantic Scholar”, https://pdfs.semanticscholar.org/.../dceb4d5daf9b5a31ad6c5 . . . , 3 pages, retrieved from the internet on Apr. 10, 2019. |
Kurimo, Mikko, “Neural network language model (NNLM)—MyCourses”, https://mycourses.aalto.fi/mod/resource/view.php?id.., 61 pages, retrieved from the internet on Apr. 10, 2019. |
Tu, Lifu et al., “Learning to Embed Words in Context for Syntactic Tasks”, ttic.uchicago.edu/˜lifu/papers/tu+ etal.repl4nlp17.pdf, 11 pages. |
“Neural Language Models—Language modeling and sequence tagging”, https://www.coursera.org/.../neural-language-models-bAJan, 2 pages, retrieved from the internet on Apr. 10, 2019. |
“Primer on Neural Network Models for Natural Language Processing”, https://machinelearningmastery.com/primer-neural-network-models-natural-language-proc, 19 pages, Sep. 15, 2017. |
“Learning Character-level Representations for Part-of-Speech Tagging”, proceedings.mlr.press/v32/santos14.pdf, 9 pages, retrieved from the internet on Apr. 10, 2019. |
“Deep Learning, NLP, and Representations—colah's blog”, colah.github.io/posts/2014-07-NLP-RNNs-Representations/, 13 pages, Jul. 7, 2014. |
The MIT Press Journals, “Neural Network Methods for Natural Language Processing”, https://www.mitpressjoumals.org/doi/full/10.1162/COLI_r_0031, 9 pages, retrieved from the internet on Apr. 10, 2019. |
Zheng, Xiaoqing, et al., “Learning Context-Specific Word/Character Embeddings”, https://aaai.org/ocs/index.php/AAAI/AAAI17/..../14266, 7 pages, 2017. |
Anastasyev, Daniil et al., “Part-of-Speech Tagging with Rich Language Description”, Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog 2018”, 14 pages, http://www.dialog-21.ru/media/3895/anastasyevdgetal.pdf. |
Anastasyev, Daniil et al., “Improving Part-of-Speech Tagging via Multi-task Learning and Character-level Word Representations”, Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog 2017”, May 31-Jun. 2, 2018, http://www.dialog-21.ru/media/4282/anastasyevdg.pdf, 12 pages. |
Sorokin, A.A., “Improving Neural Morphological Tagging Using Language Models”, Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog 2018” (2018), 14 pages, May 30-Jun. 2, 2018, http://www.dialog-21.ru/media/4530/sorokinaa.pdf. |
Sorokin, et al., “MorphoRuEval—2017: An Evaluation Track for the Automatic Morphological Analysis Methods for Russian”, Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialog 2017”, May 31-Jun. 3, 2017, 17 pages, http://www.dialog-21.ru/media/3951/sorokinaetal.pdf. |
Toutanova, Kristina et al., “Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network”, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (2003), 8 pages, https://nlp.stanford.edu/pubs/tagging.pdf. |
Tsuboi, Yuta, “Neural Networks Leverage Corpus-wide Information for Part-of-speech Tagging”, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Oct. 25-29, 2014, https://www.aclweb.org/anthology/D14-1101, 13 pages. |
Anastasyev D.G., Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialogue 2018”, “Improving Part-Of-Speech Tagging Via Multi-Task Learning And Character-Level Word Representations”, 14 pages, May 30-Jun. 2, 2018. |
Number | Date | Country | |
---|---|---|---|
20200202211 A1 | Jun 2020 | US |