SEQUENCE MODELING VIA SEGMENTATIONS

Abstract

In neural-network-based approaches to sequence modeling, an output sequence may be modeled via segmentations, the probability of the output sequence being constructed as a sum of products of output-segment probabilities, taken over all valid output-sequence segmentations. A set of artificial neural networks may model the distribution of the output-sequence probability with a recurrent neural network modeling the distributions of the individual output-segment probabilities, optionally in conjunction with a second recurrent neural network modeling concatenations of output segments. In various embodiments, this approach is applied to neural phrase-based machine translation.

Description

BACKGROUND

In recent years, a lot of research in machine learning has focused on modeling sequence data, and especially on sequence-to-sequence transduction, that is, the prediction of an output sequence from a given input sequence. Sequence modeling is relevant for many tasks related to natural-language processing, including, for example, machine translation, speech recognition, handwriting recognition, and language modeling; however, it also has applications in other disciplines, e.g., for protein or DNA structure prediction. Real-world sequence modeling tasks present a number of challenges-generally differing between types of tasks. For example, handwriting and speech recognition involve the prediction of a sequence of labels (from a set of possible labels, such as a set of letters or phonemes) from noisy, a-priori unsegmented input data (e.g., image data capturing a continuous pen-stroke sequence, or audio data, respectively). In machine translation from text in a source language to text in a target language, the input comes practically noise-free and pre-segmented into words; however, with human languages, the word order is often not preserved in the translation, and even the number of words can differ between input and output sequences, such that a naïve, sequential mapping from input to output elements may yield unsatisfactory results.

A variety of neural-network-based approaches to sequence modeling have been developed. The various approaches differ in their respective capabilities and limitations; some of them are tailored to a specific task or class of tasks and therefore meet some of the above-noted challenges better than others. One approach, known as connectionist temporal classification (CTC), uses recurrent neural networks to map an unsegmented input sequence to an output sequence of labels that is no longer than the input sequence; CTC has had some success in speech recognition. Follow-up research on CTC has led to a sequence transducer that allows for output sequences of any length and furthermore models sequential dependencies between input and output and within the output sequence. The sequence transducer extends the range of applications to tasks such as text-to-speech (where the output sequence is generally longer than the input sequence), but it conditions the prediction of the output sequence on the entire input sequence. Yet another approach involves “online” sequence-to-sequence modeling with partial conditioning to allow for incremental predictions as an input stream arrives, albeit at the cost of some approximation in the alignment between input and output sequences.

SUMMARY

This summary section is provided to introduce aspects of embodiments in a simplified form, with further explanation of the embodiments following in the detailed description. This summary section is not intended to identify essential or required features of the claimed subject matter, and the particular combination and order of elements listed in this summary section is not intended to provide limitation to the elements of the claimed subject matter.

Disclosed herein is a neural-network-based approach to sequence modeling (e.g., for use in speech recognition, machine translation, or other natural-language-processing tasks) in which the output sequence is explicitly treated as broken up into sub-sequences herein referred to as “segments.” This approach is based on the observation that many types of sequences—including, in particular, natural-language sequences—admit of segmental structure, and on the assumption that such segmental structure can be leveraged to make better output predictions for a given input. Sentences, for instances, can be structured into phrases each including one or more words (the words constituting the elements of the sequence, and the phrases constituting the segments), and the meaning of the individual words can be informed by the context of the phrases in which they appear.

In contrast to many prior approaches to sequence modeling (including some approaches that model sequences via segments), the instant approach does not require a segmented input sequence, and, indeed, not even sequence input. The instant approach models the probability of an output sequence as the product of the probabilities of the output segments for a given segmentation, summed over all valid segmentations. For non-sequence input, all output-segment probabilities are conditioned on that input. For sequence input, each input element of the input sequence is deemed to emit exactly one output segment, empty segments being permitted. In this manner, the instant approach allows for output sequences of any length relative to the input sequence. Further, it discovers the segmentation of the input sequence implicitly, rather than taking it as input information.

In accordance with various embodiments, probability distributions of the output segments are modeled with recurrent neural networks all sharing the same network parameters (or, put differently, with multiple instances of the same recurrent neural network). These networks may be conditioned not only on the respective input (element), but also on the concatenation of preceding output segments, which may be modeled with a separate recurrent neural network. The set of neural networks may be trained by backward propagation or errors, using dynamic programming techniques to reduce the computational complexity. For example, in some embodiments, during the forward phase, the probability of the output sequence is computed from recursively defined forward and backward propagations that depend on the individual segment probabilities. Further, to reduce the computational cost of computing the segment probabilities themselves, as well as their gradients as needed for the backward phase, the length of the segments may be limited to a fixed maximum value, and computations for longer segments may be reused for shorter segments contained in the longer segments. Once the neural network has been trained, an output sequence may be predicted from a given input, using a beam search algorithm. In some embodiments, a separate beam search is performed for each output segment to compute possible extensions of partial output sequences constructed from preceding segments, and if different segmentations result in multiple instances of the same partial output sequence, those are merged. Beneficially, in accordance with various embodiments, the beam search algorithm allows for incremental predictions of the output sequence.

In various embodiments, the segmentation-based sequence modeling approach described herein is applied to the problem of machine translation of text or speech from one human language to another. In this context, the assumption of monotonic alignment between the input and output sequences, as is implicit in modeling each input element as emitting a respective output segment, generally does not hold. To address this issue, the neural-network architecture is expanded with an additional layer that locally reorders elements of a sequence representing the source-language input, and then passing an encoding of the reordered sequence on as input to the neural networks generating the output sequence.

Accordingly, in one aspect, a method involves constructing an output-sequence probability as a sum, taken over all valid output-sequence segmentations, of products of output-segment probabilities; modeling a distribution of the output-sequence probability with a set of one or more artificial neural networks, and using one or more hardware processors to train the set of artificial neural networks. The set of artificial neural networks models the distributions of the output-segment probabilities with respective instances of a first recurrent neural network having an associated softmax layer, and, in training the neural networks, a dynamic programming algorithm is used to recursively compute the output-sequence probability from the output-segment probabilities. In some embodiments, the output-segment probabilities depend on respective concatenations of preceding output segments, and the set of artificial neural networks models the concatenations with a second recurrent neural network.

Computing the output-sequence probability may include recursively computing forward and backward probabilities for two-way output-sequence partitionings. Further, the set of artificial neural networks may be trained using backward propagation of errors, and, in computing the output-segment probabilities during a forward propagation phase and in computing gradients of the output-segment probabilities used during a backward propagation phase, contributions computed for longer output segments may be reused during computations for shorter output segments contained in the respective longer output segments. In some embodiments, in computing the output-segment probabilities, an output-segment length is limited to a specified maximum value.

The method may further include using the one or more hardware processors to perform a beam search algorithm to determine an output sequence for a given input based on the trained set of artificial neural networks. The input may be an input sequence, and the beam search algorithm may include, for each element of the input sequence, performing a left-to-right beam search and thereafter merging any identical partial candidate output sequences obtained for multiple respective segmentations of the output sequence.

In some embodiments, the output-sequence probability is constructed for non-sequence input, and empty segments are not permitted in output sequences. In other embodiments, the output-sequence probability is constructed for an input sequence, and the output sequence is modeled as monotonically aligned with the input sequence and as having a number of segments equal to a number of elements in the input sequence, empty segments being permitted. The input sequence may represent a human-language sequence in a first language and the output sequence may represent a human-language sequence in a second language that corresponds to a translation from the first language. The method may further include training a second set of neural networks that generates the input sequence from the human-language sequence in the first language. The second set of neural networks may include a network layer that locally reorders elements of a sequence of embedded representations of elements of the human-language sequence in the first language; an output element of that network layer may be computed as a non-linear transformation of a weighted liner combination of elements of the sequence of embedded representations within an input window, each element being weighted in the weighted linear combination by gate values that depend on all elements within the input window. The second set of neural networks may further include a word-embedding layer that generates the embedded representations of the elements of the human-language sequence in the first language by word embedding, and/or a bidirectional recurrent neural network that generates the input sequence from the reordered sequence of embedded representations.

In another aspect, one or more machine-readable media store data defining a trained set of (one or more) artificial neural networks modeling an output-sequence probability, and instructions for execution by one or more hardware processors that, when executed, cause the one or more hardware processors to perform operations for creating an output sequence from an input. The output-sequence probability is constructed as a sum of products of output-segment probabilities taken over all valid output-sequence segmentations, and the set of artificial neural networks models the distributions of the output-segment probabilities with respective instances of a recurrent neural network having an associated softmax layer. The operations performed by the one or more hardware processors include using a beam search algorithm to determine the output sequence for the input based on the trained set of artificial neural networks. The input may be an input sequence, and the beam search algorithm may include, for each element of the input sequence, performing a left-to-right beam search and thereafter merging any identical partial candidate output sequences obtained for multiple respective segmentations of the output sequence. In some embodiments, the set of artificial neural networks is trained based on pairs of input and output sequences that represent human-language sequences in first and second languages, the output sequences corresponding to translations from the first language into the second language. The data may further define a second set of artificial neural networks modeling generation of the input sequences from human-language sequences in the first language, the second set of neural networks comprising a local reordering layer for locally reordering an embedded representation of the human-language sequence in the first language.

In yet another aspect, a system includes one or more hardware processors and one or more machine-readable media storing data defining a set of (one or more) artificial neural networks modeling an output-sequence probability and instructions for execution by the one or more hardware processors, wherein the output probability is constructed as a sum of products of output-segment probabilities taken over all valid output-sequence segmentations, the set of one or more artificial neural networks modeling the distributions of the output-segment probabilities with respective instances of a recurrent neural network having an associated softmax layer, and wherein the instructions, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations to train the set of one or more artificial neural networks, using a dynamic programming algorithm to recursively compute the output-sequence probability from the output-segment probabilities.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing will be more readily understood from the following detailed description of various embodiments, in particular, when taken in conjunction with the accompanying drawings.

FIG. 1 is a flowchart providing an overview of neural-network-based methods for modeling sequences via segmentations, in accordance with various embodiment.

FIG. 2A is a conceptual network diagram illustrating an example mapping between non-sequence input and a segmented output sequence, in accordance with various embodiments.

FIG. 2B is a conceptual network diagram illustrating an example mapping between an input sequence and a segmented output sequence, in accordance with various embodiments.

FIG. 2C is conceptual network diagram illustrating example dependencies between output segments, in accordance with various embodiments.

FIG. 3A is a schematic of a recurrent neural network for modeling segment probabilities, in accordance with various embodiments.

FIG. 3B is a schematic of a recurrent neural network for modeling segment interdependencies, in accordance with various embodiments.

FIG. 4 is a flowchart illustrating a dynamic programming method for training a set of artificial neural networks, in accordance with various embodiments.

FIG. 5A is a conceptual diagram illustrating the efficient computation of example segment probabilities in accordance with various embodiments.

FIG. 5B is a conceptual diagram illustrating the efficient computation of example segment probability gradients in accordance with various embodiments.

FIG. 6 is a flowchart illustrating a beam search algorithm for computing an output sequence from a given input using a trained set of artificial neural networks, in accordance with various embodiments.

FIG. 7 is a flow chart of an example sequence-transduction method that incorporates sequence modeling via segmentations, in accordance with various embodiments.

FIG. 8 is a flowchart illustrating an example neural-network architecture for phrase-based machine translation, in accordance with various embodiments.

FIG. 9 is a text example illustrating local reordering in accordance with various embodiments.

FIG. 10A is a conceptual network diagram illustrating an example local reordering layer in accordance with various embodiments.

FIG. 10B is a conceptual network diagram illustrating an example result of local reordering in accordance with various embodiments.

FIG. 11 is an example grid of gate values, illustrating the behavior of the local reordering layer in accordance with various embodiments.

FIG. 12 is a block diagram illustrating an example system to implement neural networks for modeling sequences via segmentations, in accordance with various embodiment.

FIG. 13 is a block diagram illustrating an example computational system accordance with various embodiments.

DETAILED DESCRIPTION

Segmental structure is a common pattern in many types of sequences, especially human-language sequences. Speech and text, for instance, can be broken up into phonemes (distinct sounds) and the letter sequences representing phonemes (e.g., digraphs like the English “th” or “sh” or “ng”), respectively, and at a larger scale into words and phrases. To illustrate the phrase structure of sentences, for instance, consider the sentence “Machine learning is part of artificial intelligence,” which can be meaningfully segmented into “[Machine learning] [is] [part of] [artificial intelligence]” (where individual segments are enclosed in square brackets). As another example, to illustrate the phoneme structure of words, consider the word “thought,” which can be broken up into “[th][ou][ght].” As will be apparent from these examples, segmental structure strongly effects the statistics governing sequences, both within segments by rendering some sub-sequences (corresponding to possible segments) more likely than others, and between segments by imposing rules and restrictions (such as phonotactics rules and rules of grammar) on the permissible sequences of segments. Modeling sequences via segmentations, taking dependencies within and across segments into account, provides a way to capture such sequential statistics to improve output predictions. Yet, many machine-learning-based approaches to modeling sequences have not taken the segmental structure of output sequences into account.

FIG. 1 is a flowchart providing an overview of a neural-network-based method 100 for modeling sequences via segmentations, in accordance with various embodiment. The goal of the method 100 is to determine an output sequence, or a probability distribution over possible output sequences, from a given input. The output sequence is herein denoted by y_1:T, where T is the length of the sequence, and the individual elements y_t (t=1 . . . T) are numbers or vectors each uniquely representing, or mapping onto, one of the labels within the output “alphabet” or “vocabulary” (herein used synonymously), i.e., the set of possible labels, or onto an empty output. Depending on the task at hand, the output vocabulary may be a set of characters (e.g., all alphanumeric characters), a set of words in a given language, or some other specified set of labels. The input is in some instances a single fixed-length vector x. In other embodiments, the input is itself a sequence, herein denoted by x_1:T′, where T′ is the length of the sequence, and the individual elements x_t (t=1 . . . T′) are fixed-length vectors. The probability of the output sequence for a given input is denoted by p(y_1:T|x) or p(y_1:T|x_1:T′) for non-sequence and sequence input, respectively.

In accordance herewith, the probability of the output sequence is modeled via segmentations of y_1:T. Since the segmentation of the output sequence is not known in advance, the probability of the output sequence is modeled as the sum of the probabilities of all valid segmentations, and the probability of each segmentation, in turn, is modeled as the product of the probabilities of the individual segments. Accordingly, the method 100 involves constructing the probability of the output sequence as the sum, taken over all valid segmentations, of products of the output-segment probabilities (act 102). Denoting by , the set of all valid segmentations of y_1:T and by a_1:τ(a)∈_y an individual segmentation within the set that consists of r(a) segments, the concatenation of a_1:T(a) constitutes the sequence y_1:T: π(a_1:T(a))=y_1:T, where π(⋅) is the concatenation operator. For example, for an output sequence of length T=5, one possible segmentation into τ(a)=3 segments would be a_1:T(a)={{y₁, $}, {y₂, y₃, $}, {y₄, y₅, $}}, where the dollar symbol $ denotes the end of a segment. In the concatenation operator π(⋅), the dollar symbol $ will be ignored. Note that, while the number of distinct segments for a length-T sequence is (T²), the number of distinct segmentations, |_y|, is exponentially large.

For non-sequence input x, the probability of each segment a_t (t=1 . . . τ(a)) in the output sequence y_1:T depends on that input x as well as the concatenation of all previous segments, π(a_1:t-1). Thus, the probability of the output sequence is defined as:

$p\left(y_{1\text{:}T}x\right)\stackrel{△}{=}y_{}p\left(a_{1\text{:}\tau \left(a\right)}x\right)=y_{}\prod _{t=1}^{\tau \left(a\right)}p\left(a_tx,\pi \left(a_{1\text{:}t-1}\right)\right).$

To illustrate, FIG. 2A shows, in a conceptual network diagram 200, an example mapping between non-sequence input 202 and a segmented output sequence 204. As shown, the segments {y₁, $} and {y₂, y₃, $} each take x as input. (The concatenation of the preceding segments is omitted from this depiction.) Furthermore, within each segment, the output elements depend on the respective preceding output elements; for instance, within the second segment, element y₃ depends on element y₂. In the case of non-sequence input, empty segments (those containing only $) are not permitted in the output sequence.

For sequence input, each element x_t (t=1 . . . T′) of the input sequence is modeled to emit exactly one segment a_t of the output sequence, and, accordingly, each segmentation consists of exactly T′ segments. In this scenario, empty output segments (a_t={$}) are permitted, such that the number of non-empty segments may be less than the number of elements in the input sequence. Since each segment may have zero, one, or more elements, this mapping places no constraints on the relative lengths of the input and output sequences (i.e., the output sequence may be shorter, longer, or equal in length to the input sequence). The mapping does, however, imply monotonic alignment between the input and output sequences, meaning that the relative order of the input elements x_t and the output segments a_t they emit is preserved. The probability of each output segment a_t depends on the respective input element x_t as well as the concatenation of all previous segments, π(a_1:t-1). The resulting probability of the output sequence is given by:

$p\left(y_{1\text{:}T}x_{1\text{:}T^{\prime }}\right)\stackrel{△}{=}p\left(a_{1\text{:}\tau \left(a\right)}x_{1\text{:}T^{\prime }}\right)=\prod _{t=1}^{T^{\prime }}p\left(a_tx_t,\pi \left(a_{1\text{:}t-1}\right)\right).$

FIG. 2B is a conceptual network diagram 210 illustrating an example mapping between an input sequence 212 and a segmented output sequence 214 (again omitting the depiction of the dependence of a segment from the concatenation of preceding segments). Here, element x₁ emits segment {y₁, $}, element x₄ emits segment {y₂, y₃, $}, and elements x₂, x₃, and x₅ each emit an empty segment. To reflect that each input element can emit a non-empty or an empty segment, this network is also referred to as a “Sleep-Wake Network (SWAN).” As with the network for non-sequence input shown in FIG. 2A, elements within an output segment depend on the respective preceding output elements.

FIG. 2C is conceptual network diagram 220 illustrating example dependencies between output segments, in accordance with various embodiments. As shown, input element x_{t_2} emits a segment ending in output element y_j−1, which flows as input into the next, empty segment emitted by input element x_t−1 as well as into the segment beginning with the next output element y_j, which is emitted by input element x_t. In this manner, information is carried over across segments.

With renewed reference to FIG. 1, the probability distribution of the output sequence, constructed from the segment probabilities as described above, is modeled, in accordance with the method 100, by a set of (generally one or more) artificial neural networks (act 104). In various embodiments, the individual segment probabilities are modeled by respective instances of a first recurrent neural network (RNN) (or, put differently, by first RNNs that all share the same network parameters). This RNN may have an associated softmax layer, as is known to those of ordinary skill in the art, to turn the neural network output into a probability distribution over the output vocabulary. To capture the conditional dependence of segment probabilities on preceding segments, a second RNN may be used to model the concatenation of segments. (In some applications, such as in text segmentation where the words and phrases are taken to be interchangeable, the segments are deemed conditionally independent, and the concatenation π(a_1:t-1) is dropped from the above defining equations for the probability of the output sequence, p(y_1:T|x) or p(y_1:T|x_1:T′), obviating the second RNN.) Although embodiments are described herein as implemented using RNNs, this is not intended to limit the scope of the disclosed subject matter. Rather, the probability distribution of the output sequence (as defined above), and the conditional probabilities distributions of the constituent output segments, may also be modeled using other types of neural networks, such as, for example, convolution neural networks or fully connected neural networks.

The set of artificial neural networks (which collectively constitute a larger-scale neural network) is trained, in act 106 of the method 100, using backward propagation of errors (a technique known to those of ordinary skill in the art). In brief, in this supervised-learning algorithm, the input of an input-output pair provided as a training example, is fed into the (larger-scale) neural network to compute, by forward propagation through the network, an associated output or probability distribution over possible outputs. A suitably defined error function (or cost function) is then used to quantify the discrepancy (or error) between the computed output and the actual output as reflected in the training example, or the computed probability of the actual output given the input of the training example (where the error can be taken as the negative logarithm of the probability). Using gradients of the error with respect to the network weights (or other network parameters), the error is then propagated backwards through the network to update the network weights. For RNNs used to compute sequential output (as is generally the case when an output segment in accordance herewith is computed), where the output at each time step is fed back into the computation of the next output, the error propagation additionally proceeds backward in time (e.g., from the error associated with the last output element to the error associated with the first output element in the output sequence), and errors computed for the network weights for the individual time steps are aggregated to determine the overall updates for the network weights. In accordance with various embodiments, training the neural network through backward propagation of errors is rendered computationally tractable by employing dynamic programming techniques to store and reuse solutions to partial computations in a recursive computation of the overall probability of the output sequence (as described in more detail below with respect to FIGS. 4-5B).

Once the (larger-scale) neural network is trained, i.e., the values of the network weights (and any other network parameters) have been determined, the neural network can be used to predict an output sequence for a given input, a process herein also referred to as “decoding” (act 108). For this purpose, a beam search algorithm may be employed. During a beam search, the output sequence is constructed by iteratively expanding partial candidate output sequences, creating a search tree structure in the process. At each level of the search tree, all children (corresponding to extensions of the partial candidate output sequences by another output element) are generated, and only the B most probable partial candidate output sequences are kept; B is referred to as the beam width. In accordance with embodiments hereof, each output segment is generated by a respective beam search, and, since different segmentations can imply the same output sequence, each time the partial candidate output sequences have been extended with a new segment, differently segmented, but otherwise identical partial candidate output sequences are merged. Beneficially, in the context of sequence transduction, modeling the output sequence as described herein allows for incremental (or “online”) prediction of the output sequence during decoding, that is, rather than waiting for the input sequence to be read in and processed in its entirety, output can be generated as the input sequence is read in.

Having provided an overview of segmentation-based output sequence modeling in accordance with various embodiments, the artificial neural networks employed as well as methods for training (act 106) and decoding (act 108) will be described in more detail in the following.

Neural networks generally include an input layer with or more input nodes for receiving input from outside the network, an output layer with one or more output nodes producing the results, and, optionally, one or more “hidden” layers with hidden nodes that modify the data en route from the input layer to the output layer. The hidden nodes and output nodes each generate an output value from one or more input values (corresponding to the output values of nodes to which they are connected, weighted by the network weights assigned to the respective connections) in accordance with a (generally nonlinear) “activation function” or “transfer function.” Nodes may form sub-networks that collectively generate an output value from one or more input values, operating as network “units” from which larger networks can be built. An RNN is a neural network that includes directed cycles between the network nodes or units, allowing information to be fed back from the hidden or output layers to preceding layers. RNNs are therefore useful to model sequential output with dependencies of output elements on preceding output (also often captured with the notion of “memory”).

FIG. 3A schematically illustrates an example RNN 300 for modeling segment probabilities in accordance with various embodiments. The RNN 300 may include one or more hidden layers. In various embodiments, the RNN 300 is built from units such as long short-term memory (LSTM) units, gated recurrent units (GRU), or other units well-known to those of ordinary skill in the art. The output layer of the RNN 300 includes a node for each label in the vocabulary as well as for the end-of-segment symbol $. A softmax layer 302 at the output of the RNN 300 converts the output values into a probability distribution over the vocabulary and end-of-segment symbol (by mapping the output values onto values within the range [0,1] that add up to 1). In accordance with various embodiments, the probability distributions for all segments in the output sequence are modeled separately from each other, but with the same RNN 300 in multiple instances thereof. That is, all instances of the RNN 300 share the same network weights and parameters, while initializations and inputs differ between network instances for different respective segments.

The RNN 300 for segment a_t and its associated softmax layer 302 sequentially produce probability distributions for output elements y_j(t-1)+k that form the segment a_t (where j_t−1 denotes the index of the last output element within the preceding segment a_t−1, and k is an integer, initialized to 0, that is incremented during each pass from the input layer to the output layer of the RNN 300). At each time step k, the “hidden state” h_k of the RNN 300 (where the hidden state may capture the output values of multiple of the hidden nodes) is updated, and the probability distribution for the respective output element y_jt-1+k is computed from the updated hidden state. The initial hidden state h₀ is based, in accordance with various embodiments, on the input element x_t for segment a_t, or the input x in the case of non-sequence input, and on the concatenation of all previous segments as encoded in the hidden state h_jt-1 of a separate neural network (shown in FIG. 3B). The updated hidden state h_k (k≥1) in each time step is computed from the respective preceding hidden state h_k-1 and the preceding element in the output sequence, y_jt-1+k-1; in other words, the previous output element and hidden state constitute the inputs to the RNN 300 for updating the hidden state and computing the new output element. During training of the network, the output elements fed as input into the network 300 are provided by the known output sequence of the input-output pair constituting the training example; note that, in training, only the probability of that known output sequence is of interest, and the remaining information of the computed probability distributions may be disregarded. During decoding, where probabilities for multiple output sequences are computed in a beam search by following multiple paths through the search tree, the current path determines which of the possible values of the output element y_jt-1+k is fed back to the input of the RNN 300 for the next time step k+1. The cycles through the RNN 300 stop when the output element takes the end-of-segment symbol $.

Turning now to FIG. 3B, an example RNN 310 for modeling segment interdependencies in accordance with various embodiments is schematically shown. The RNN 310 may have one or more hidden layers, may be unidirectional or bidirectional, and may be built from LSTM, GRU, or other units. The RNN 310 takes the output sequence y_1:T as input (one element y_j at a time), and updates, in each time step j, its internal state to h_j based on its previous internal state h_j−1 and the output element y_j. The internal state at the end of a segment a_t−1, h_jt-1, is provided to the RNN 300 for segment a_t to initialize the internal state h of RNN 300. (Although the RNN 310 itself outputs predictions for the output element y_j, in various embodiments, these predictions are not themselves used and are, in particular, not fed back as input to the RNN 310, as the RNN 310 may take its input from the output of the RNN 300 for the individual segments.)

FIG. 4 is a flowchart illustrating an example dynamic programming method 400 for training a set of artificial neural networks that model a probability distribution for a segmented output sequence (such as RNNs 300, 310), in accordance with various embodiments. Starting point 402 of the method 400 is a set of training examples, each including a pair of an input sequence (or non-sequence input) and an output sequence. The method 400 involves, for each input-output pair, computing the probability of the output sequence given the respective input, using forward propagation through the network (acts 404, 406). In accordance with various embodiments, dynamic programming is employed to reduce the computational cost of the forward propagation. Specifically, the probability of the output sequence is computed as the sum over products of “forward and backward probabilities,” which, in turn, are computed recursively based on the probabilities of the output segments.

The forward and backward probabilities for the case of sequence input are defined as:

α_t(j)=p(y_1:j|x_1:t)

β_t(j)=p(y_j+1:T|x_t+1:T′,y_1:j),

where the forward probability α_t(j) represents the probability that input x_1:t emits output y_1:j and the backward probability β_t(j) represents the probability that input x_t+1:T′ emits output y_j+1:T. The indices t and j demarcate two-way partitionings of the input sequence and the output sequence, respectively. For any t=0, 1, . . . , T′, the probability of the output sequence can be computed from the forward and backward probabilities for all possible two-way partitionings of the output:

p(y_1:T|x_1:T′)=Σ_j=0^Tα_t(j)β_t(j).

For t=T′ and t=0, this sum reduces to the special case p(y_1:T|x_1:T′)=a_T′(T)=β₀(0) (with α₀(0)=β_T′(T)=1, and α₀(j)=0 for all j≠0 and β_T′(j)=0 for all j≠T).

The forward and backward probabilities can be computed recursively using the following dynamic-programming recursions:

α_t(j)=Σ_j′=0^jα_t−1(j′)p(y_j′+1:j|x_t)

β_t(j)=Σ_j′=j^Tβ_t+1(j′)p(y_j+1:j′|x_t+1).

where p(y_j′+1:j|x_t) is the probability of the segment y_j′+1:j emitted by x_t and p(y_j+1:j′|x_t+1) is similarly defined. For j=j′, the notation y_j′+1:j indicates an empty segment with previous output y_1:j. The segment probabilities are computed, in act 404, using forward propagation through the RNN that models them (e.g., RNN 300), optionally conditioned based on the hidden state of the RNN that models the connections between segments (e.g., RNN 310). The probability of the output sequence as expressed in terms of the forward and backward probabilities can then be efficiently computed, in act 406, using the above recursions for α_t(j) and β_t(j) with initial conditions α₀ (0)=β_T′(T)=1 and α₀(j)=0 for all j≠0 and β_T′(j)=0 for all j≠T.

For non-sequence input, the probability of the output sequence can be analogously constructed from similarly defined forward and backward probabilities:

p(y_1:T|x)=Σ_j=0^Tα(j)β(j),

where α(j)=p(y_1:j|x) and β(j)=p(y_j+1:T|x, y_1:j) can be computed using the dynamic-programming recursions:

α(j)=Σ_j′=0^j-1α(j′)p(y_j′+1:j|x)

β(j)=Σ_j′=j+1^Tβ(j′)p(y_j+1:j′|x)

with α(0)=β(T)=1.

Following computation of the probability of the output sequence, the gradients of the logarithm of the probability with respect to the inputs to the RNN 300 for the segment probabilities (as discussed above with respect to FIGS. 3A and 3B)—that is, the gradients with respect to the input-sequence elements x_t (which depend on network weights of a network used to encode the input sequence), the hidden states h_k of the RNN 300 (which depend on network weights of the RNN 300), and the hidden states h_jt-1 of the RNN 310 (which depend on network weights of the RNN 310)—are computed (act 408). The network weights are then updated by following these gradients in a direction towards a maximum of the probability of the output sequence. With the above recursions for the forward and backward probabilities, the gradient of the logarithm of the probability with respect to the input element x_t, for instance, is:

$\frac{\partial \log p\left(y_{1\text{:}T}x_{1\text{:}T^{\prime }}\right)}{\partial x_t}=\sum _{j^{\prime }=0}^T\sum _{j=0}^{j^{\prime }}w_t\left(j,j^{\prime }\right)\frac{\partial \log p\left(y_{j+1\text{:}j^{\prime }}x_t\right)}{\partial x_t},\mathrm{where}$ $w_t\left(j,j^{\prime }\right)\stackrel{△}{=}{\alpha }_{t-1}\left(j\right){\beta }_t\left(j^{\prime }\right)\frac{p\left(y_{j+1,j^{\prime }}x_t\right)}{p\left(y_{1\text{:}T}x_{1\text{:}T^{\prime }}\right)}.$

Accordingly, the gradient with respect to x_t is a weighted linear combination of contributions from related segments. (The formulas for non-sequence input can be obtained by dropping the subscripts t and t−1, and replacing x_t and x_1:T, with x.) Other gradients can be similarly derived. Based on the gradients, the network weights are updated, using backpropagation through the network (act 410) to maximize the probability of the output sequence given the input (averaged over all training examples). The result of the method 400 is a set of trained neural networks (412) with fixed and optimized network weights and parameters that can thereafter be used to compute an (unknown) output sequence for a given input.

The dynamic programming method 400 involves computing segment probabilities p(y_j+1:j′|x_t) and associated gradients

$\frac{\partial \log p\left(y_{j+1\text{:}j^{\prime }}x_t\right)}{\partial x_t}$

for 0≤j≤j′≤T and 0≤t≤T′, amounting to (T′T²) segments. Considering each recurrent step as a unit of computation, the overall complexity of computing the segment probabilities is (T′T³). Although the computation can be parallelized for different segments, training the network is still computationally expensive. In accordance with various embodiments, therefore, further strategies are employed to reduce the computational cost. One strategy is to limit the segment length to a maximum length L to reduce the computational complexity to (T′T²L); with a sufficiently high value of L, this limit will not be much of a limitation as a practical matter for many applications. For example, when segmenting text into phrases, or words into phonemes, a maximum length set to, for instance, L=10 will cover the vast majority, if not all, of meaningful phrases or word segmentations.

Another strategy for reducing the computational complexity of training the network is based on the observation that, due the structure of segments, computations performed for a longer segment can in part be reused to provide computational results for shorter segments contained within the longer segment. Considering forward propagation for fixed values of j and t, suppose, for example, that p(y_j+1:j′|x_t) is to be computed for any j′=j, . . . , j+L, corresponding to L+1 segments with lengths ranging from 0 to L. The probability of the longest segment, p(y_j+1:j+L|x_t) is determined as the product of the probabilities p(y=y_j+1|x_t,h₀), p(y=y_j+2|y_j+1, x_t, h₁), . . . , p(y=y_j+L|y_j+L-1, x_t, h_L-1), and p(y=$|y_j+L, x_t, h_L), where h_l, l=0, 1, . . . , L are the recurrent states (e.g., of RNN 300). Accordingly, computing the probability of the longest segment also provides the probabilities for the shorter segments, i.e., for j′=j, . . . , j+L−1.

For backward propagation, it can be observed that the contribution of each segment to the gradient of the output-sequence probability has its own respective weight, which is w_t(j, j′) for segment y_j+1:j′. Recognizing that the logarithm of the segment probability p(y_j+1:j+L|x_t) can be determined as the sum of the logarithms of the individual probabilities p(y=y_j+1|x_t, h₀), p(y=y_j+2|y_j+1, x_t, h₁), . . . , p(y=y_j+L|y_j+L-1, x_t, h_L-1), and p(y=$|y_j+L, x_t, h_L), the gradient of the logarithm of the output-sequence probability can be determined by summing over the contributions of all output elements, each weighted by the sum of the weights of all segments in which that element occurs. Accordingly, the contributions of the elements within the longest segment are also used, with merely an adjusted weight, to account for the contributions of the shorter segments.

FIGS. 5A and 5B are conceptual diagrams illustrating the efficient reuse of the segment contributions in the computation of the probability of the output sequence during forward propagation and of the gradient of its logarithm during backward propagation, respectively. In the depicted example, the probabilities and gradients for segments y_j+1:j′ (j′=0, . . . , L) are computed for j=0 and L=3, corresponding to a longest segment {y₁, y₂, y₃, $}, and four shorter segments {y₁, y₂, $}, {y₁, $}, and {$}. The possible values within the output vocabulary of size V are shown along the vertical axis. In FIG. 5A, filled circles indicate the softmax probabilities for computing the probabilities of all four segments. The softmax probabilities for y₂ and y₁ can be reused twice and once, respectively, reducing the overall computational cost. In FIG. 5B, filled circles indicate the gradient contributions of individual elements that flow into the computation of the gradients of all four segments. Each filled circle is annotated with the respective weight of the gradient contribution, w_j′w_t(0, j′); open circles have zero gradients. For example, the gradient for element y₁ has a total weight w₁+w₂+w₃ since y₁ appears in three segments {y₁, y₂, y₃, $*}, {y₁, y₂, $}, and {y₁, $}.

FIG. 6 is a flowchart illustrating a beam search algorithm 600 for computing an output sequence from a given input, using a trained set of artificial neural networks (e.g., as resulting from the method 400 of FIG. 4), in accordance with various embodiments. The algorithm 600 takes a sequence x_1:T′ (602) as input, and iteratively expands a set of partial (candidate) output sequences, retaining only the B most probable candidates, where B is the beam width. After the set of partial output sequences and the set of associated probabilities have been initialized, in act 604, to the null element (corresponding to a length-zero sequence that provides the starting point for extensions) and an associated probability of 1, left-to-right beam searches are performed for all input elements x_t, beginning with t=1 (initialized in act 606), to compute the respective output segments in a loop 608.

Each pass through the loop 608 begins, in act 610, with the initialization of a local beam size b to the global beam size B and empty sets _t and _t for holding any partial output sequences that complete the current segment and the associated probabilities. The partial output sequences in that result from the computation for all previous segments (i.e., segments up to t−1, or, for the first loop at t=1, the null element) are then extended element by element in an inner loop 612 from j=0 (initialized in act 614) to j=L, where L is the maximum permissible segment length. In more detail, each pass through the inner loop 612 involves computing, for each partial output sequence ∈, the probability distribution for the next output element, p(y_j|, x_t) (act 616). As long as j<L (as determined in decision act 618), the algorithm proceeds in act 620 with the selection, among all extended partial output sequences {, y_j} with ∈ and y_j∈{1, . . . , V, $} of b candidate sequences (“candidates”) with the highest probability ()p(y_j|, x_t). For any such candidates that have reached the end of the segment, i.e., for which y_j=$, the partial output sequence is moved from to the set _t of completed-segment sequences, and b is decremented by 1 (in act 622). Further, the probability of the partial output sequence is assigned according to _t()←()p(y_j|,x_t). For any candidates that have not yet reached the end of the segment, the partial output sequence is updated in accordance with the extension: ←{, y_j} (also in act 622), and the probability of the partial output sequence is updated according to ()←()p(y_j|, x_t). The loop index j is then incremented to j+1 (act 624) for the next loop.

Once it is determined, in act 618, that j=L, the b most probable candidates among the extended partial output sequences that have reached the end of the segment (i.e., among {, $}) are selected, the respective partial output sequences (without the end-of-segment symbol) are moved to the set _t of completed-segment sequences, and their respective probabilities are updated according to ()←()p(y_j=$|, x_t) (act 626). The set _t includes, at this stage, B partial output sequences each including segments 1 through t. In act 628, the partial output sequences in _t are moved (back) to the set . It can happen, at this point, that contains duplicate partial output sequences resulting from different segmentations. For example, a partial output sequence {y₁, y₂, y₃, y₄, y₅} may result from the concatenation of first segment {y₁, y₂} and second segment {y₃, y₄, y₅}, or from the concatenation of first segment {y₁, y₂, y₃} and second segment {y₄, y₅}. Following the left-to-right beam search of each loop 608, any such identical partial output sequences obtained for multiple respective segmentations are merged, in act 630, by retaining only one instance thereof in the set and adding the respective probabilities up. Formally stated, for any pair _i, _i′∈ with _i=_i′ the probability is updated to (_i)←(_i)+(_1′), and _i′ is removed from . As long as t<T′ (as determined in decisional act 632), the index t is then incremented to t+1 (act 634), and the algorithm proceeds with the next segment. When the last segment, t=T′, is reached, the algorithm returns the (now complete) output sequence ϵ with the highest associated probability as output 636.

In some application contexts, the output segments are independently modeled (i.e., the network used to model the output-segment probabilities is not conditioned on the concatenation of previous segments). In that case, output segments can be computed in parallel, reducing the time used to compute the output sequence.

The above-described approach to predicting output sequences modeled via segmentations (using, e.g., neural network configurations as illustrated in FIGS. 2A-3B and methods 400, 600 for training and decoding as shown in FIGS. 4 and 6) can be applied to various sequence modeling and sequence transduction tasks, both for sequence and non-sequence input. Example applications include, without limitation, automatic content-based text segmentation (where the input is not a sequence, and the output is a segmented text whose segments may be exchangeable), speech recognition, text-to-speech, DNA sequence segmentation, etc.

FIG. 7 is a flow chart of an example sequence-transduction method 700, in accordance with various embodiments, that incorporates the above-described approach to sequence modeling. Input to the method 700 is raw data 702, which may be, for example, a text in a human language, or audio data representing human speech. The individual items of data 702 are converted, in a first act 704, into respective vector representations. For example, for text input, each word may be represented as a one-hot vector the size of the vocabulary that has a single component equal to 1 uniquely identifying the word, all other vector components being zero. As another example, for audio-data input, the data may be divided into a sequence of short time windows, and the vectors may represent spectral distributions of the audio data within these time windows, e.g., as computed using a filter bank. The sequence of vector representations of the individual data items may then be fed into a neural network for encoding as an input sequence x_1:T, in which element depends on its past and/or future context (act 706). From this input sequence x_1:T′, or, alternatively, directly from the sequence of vector representations output by act 704, a SWAN as described above (and as depicted in FIG. 2B) generates an output sequence, or probability distribution of output sequences (710), modeling the output sequence via segmentations in accordance herewith (act 708). A unique output sequence may be determined from a probability distribution of sequences, e.g., by selecting the sequence with the highest probability.

In various embodiments, the SWAN approach is applied to the problem of machine translation, that is, the automatic translation of human-language input (usually text, although application to speech input is also conceivable) in a first language (e.g., English) into human language output in a second language (e.g., German). In human language, concrete meaning is often attached to phrases rather than individual words. For example, the statement “machine learning is a field of computer science” may be easier to comprehend if represented as “[machine learning] [is] [a field of] [computer science],” where square brackets enclose meaningful segments or phrases (herein used synonymously in the context of machine translation), which have their own meanings and are often reusable in other contexts. Since SWAN models output sequences via phrases, they have the potential to improve the accuracy of machine translation. As an approach to neural phrase-based machine translation, it can capture benefits from both traditional phrase-based statistical machine translation (such as the ability to explicitly model phrases) and conventional attention-based neural machine translation (which often provide better performance).

As described above, SWAN allows for output sequences that are shorter or longer than the input sequence (or equal in length), but assumes monotonic alignment between input and output. In the context of translation, this assumption does not hold for many pairs of source and target languages. To mitigate this problem, the method 700 is modified, in accordance with various embodiments, by introducing a local reordering step in the workflow. FIG. 8 illustrates, in the form of a flowchart, an example layered neural-network architecture 800 for phrase-based machine translation in accordance with various embodiments, which traces the steps of method 700 but for the addition of the reordering layer. The neural network architecture 800 operates on human-language input 802 in a first language (constituting the source language), such as a text document. This raw input is converted into vector representations in a word-embedding layer 804. Word embedding maps the words from the vocabulary of the first language to vectors of real numbers (and thus from a space with one dimension per word to a continuous vector space with generally much lower dimension), and can be accomplished with a neural network, or, alternatively, with any of a number of other language-modeling and feature-learning techniques known in the field of natural language processing (e.g., dimensionality reduction on the word co-occurrence matrix, or using probabilistic models, etc.).

Following word embedding, the “embedded representation” passes through a (soft) local reordering layer 806, explained in more detail below. The reordered representation is fed as input to a bi-directional RNN layer 808 to generate the encoded input sequence x_1:T, that flows into the SWAN layer 810. In the bi-directional RNN layer 808, the input sequence is processed from left to right as well as from right to left to predict each element of the input sequence based on past and future context. Possible alternatives to a bi-directional RNN for input-sequence encoding include, for example, convolutional neural networks (CNNs), LSTM networks, bidirectional LSTM networks, GRU networks, and transformer networks. The SWAN layer 810 may include, e.g., the RNNs 300, 310 described in FIGS. 3A and 3B. Output of the SWAN layer is an output sequence (or probability distribution over output sequences) that can be straightforwardly mapped onto human-language output 812 in a second language (constituting the target language). For example, the elements of the output sequence may be vectors that serve directly as identifiers of words (e.g., one-hot vectors whose components correspond to the words in the vocabulary), or vectors that identify sub-units of words (such as characters or n-grams) that can be merged into words. The network layers 804, 806, 808, 810 of the neural network architecture 800, which may be thought of as a first set of neural networks (corresponding to layer 810) for modeling an output sequence via segmentations in accordance herewith and a second set of neural networks (corresponding to layers 804, 806, 808) for generating the input sequence to layer 810 from a human-language input sequence, may be trained end-to-end based on training data that includes pairs of input and output sequences in the source and target languages, respectively. The training may involve backpropagation of errors as described above with respect to the SWAN layer 810, extended to the other layers within the larger network.

FIG. 9 is a text example illustrating soft reordering in accordance with various embodiments. Here, in the course of translating the German sentence “Sie wollen die Entscheidung wirklich richtig treffen” (900) to the English sentence “You really want to make the decision right” (902), the word order of the German sentence is modified to result in a reordered sequence (904) that is monotonically aligned with the English output.

To formally describe the operation of the local reordering layer 806, let e_1:T, denote the sequence of vector representations resulting from word embedding that constitute the input to the reordering layer 806, and let f_1:T, denote the sequence of output vectors of the reordering layer 806. The reordering is “local” in that the position of each input vector et within the input sequence can change only within a fixed input window of size 2t+1. With these notations, in some embodiments, the output of the reordering layer 806 is computed from the input according to:

f _t=tanh(Σ_i=0^2τσ(w_i^T[e_t−τ; . . . ;e_t; . . . ;e_t+τ])e_t−τ+i),

where σ(⋅) is the sigmoid function (implementing a gate function for the reordering layer), [e_t−τ; . . . ; e_t; . . . ; e_t+τ] is the concatenation of the input vectors within the input window, and w_i^T are vectors (each having a dimensionality of 2τ+1 times the dimensionality of the individual input vectors) that constitute parameters for the gate function at respective positions i of the input window. The sigmoid function applied to the scalar product of the parameter w_i^T and [e_t−τ; . . . ; e_t; . . . ; e_t+τ] determines how much information from element of e_t−τ+i of the input window is accepted through the “gate” σ. The gate may take any value between zero and one; the reordering layer is, in that sense “soft.” The final output f_t is a weighted linear combination of the input vectors e_t−τ, . . . , e_t, . . . , e_t+τ in the window, followed by a non-linear transformation by the tanh(⋅) function (applied separately to each vector element of the combination of input vectors e_t−τ, . . . , e_t, . . . , e_t+τ).

FIG. 10A is a conceptual network diagram 1000 illustrating an example local reordering layer with a window size of 5 (τ=2). Here, σ_t−2+iσ(w_i^T[e_t−2; e_t−1; e_t; e_t+1; e_t+2]), i=0, . . . , 4 are the gates for input vectors e_t−2+i, respectively. Note that the gate for each individual input vector is decided based on information from all input vectors within the window. As a further illustration, FIG. 10B is a conceptual network diagram 1010 showing how local reordering works. In the depicted example, two adjacent windows of length three, corresponding to input vector sequences {e₁, e₂, e₃} and {e₂, e₃, e₄}, respectively, are processed to determine two respective outputs f₂ and f₃. If e₃ gets the largest weight (and is thereby picked) in the first window and e₂ gets the largest weight (and is thereby picked) in the second window, e₂ and e₃ are effectively reordered.

FIG. 11 is an example grid of gate values, further illustrating the behavior of the reordering layer in accordance with various embodiments. In this example, the English source sentence “can you translate it?” is translated to “können man es übersetzen?,” where the mapping between words is: “can→können,” “you→man,” “translate→übersetzen,” “it→es,” and “?→?” This mapping implies a reordering from “translate it” to “es übersetzen.” In the depicted grid of gate values, each row represents a window of size seven (corresponding to τ=3) that is centered at a word of the source sentence. The values in the grid represent the gate values σ(w_i^T[e_t−τ; . . . ; e_t; . . . ; e_t+τ]) for the words within each window, which are to be multiplied with the embedded vectors e_t−τ+i to determine their respective contributions to the vector f_t output by the reordering layer; gate values above a threshold corresponding to selection of the respective input vectors are shaded. Along the vertical axis, the words or phrases emitted by the SWAN layer for the corresponding positions t are shown. As can be seen, the gates mostly focus on the central word, reflected in monotonic alignment between input and output for the first two words of the sentence. Interestingly, for the window centered at the third word, “translate,” the model outputs “$” (corresponding to an empty segment). Then, the model outputs “es” when the model encounters “it.” Finally, in the last window (shown in the top row), the model not only has a large gate value for the center input “?,” but it also has a relatively large gate value for the word “translate,” and outputs, as a result, the translation “übersetzen?” This illustrates the reordering effect achieved using the gating mechanism of the reordering layer.

Beneficially, the neural phrase-based machine translation approach described above, which uses SWAN in conjunction with a local reordering layer, does not use an attention-based decoder with predefined phrases (as commonly used with conventional neural machine translation), but automatically discovers the phrase structures of the target language based on training data. Further, it can decode in linear time, and may outperform conventional machine translation approaches. The reordering layer of the present approach differs from attention mechanisms in multiple ways: it does not rely on a hidden representation to use as a query when decoding a word; the weights w_i^T for the input vectors e_t−τ+i of the layer are not normalized to a sum equal to one, which provides reordering capability that can potentially turn off everything (i.e., allow for all weights to be set to zero); the gate value at any position is determined by all input vectors within the window; and the layer performs locally similar to a convolutional layer and the positional information is encoded by a different parameter w_i^T for each relative position i in the window. Further, since the reordering of elements is local, i.e., confined to a specified input window, the described neural machine-translation approach is amenable, in various embodiments (not employing a bidirectional RNN in layer 808), to incremental output of the translation into the target language as the sequence in the source language comes in.

Although the local reordering layer 808 has been described in the context of a neural network architecture that employs SWAN, the scope of potential applications extends beyond use with SWAN. For instance, local reordering as described herein can also be used to produce input for a CTC layer, RNN layer, CNN layer, or other neural network layer for predicting an output sequence from an input sequence (and substituting for SWAN in the network architecture 800). Furthermore, the disclosed machine-translation approach with local reordering is not limited to a translation from one language to another in the strictest sense, but can also be employed in other sequence-input to sequence-output transduction tasks, including, for example, conversational response modeling, grammar correction, summarization, and the like.

FIG. 12 illustrates an example computing system 1200 for implementing the methods 100, 400, 600, 700, 800 for training artificial neural networks as described herein and using them to predict output sequences. The computing system may generally include any suitable combination of hardware and software, for instance, in accordance with some embodiments, one or more (e.g., general-purpose) computers (e.g., as illustrated in more detail in FIG. 13) that collectively include one or more hardware processors for executing software instructions and one or more machine-readable media storing the instructions as well as the data on which they operate (such as, e.g., the input and output sequences, the weights and parameters of the neural networks, and results of partial computations for reuse). The overall functionality of the computing system 1200 may be organized into multiple software tools, components, or modules. For example, as depicted, the computing system 1200 may include, in addition to the artificial neural network(s) 1202 themselves, a modeling tool 1204, a training component 1206, and a decoder 1208. Each of these components 1202, 1204, 1206, 1208 may be implemented in software, that is, with program code and associated data structures.

The modeling tool 1204 may provide a user interface that allows a human model developer to provide input 1210 to define the structure of the neural network(s) 1202, e.g., by specifying—graphically or otherwise—the number and types of network layers, the dimensionality of the associated inputs and outputs, the types of network units used within the layers, the activation functions associated with the network nodes or units, the connections between layers, and so on. Based on this input 1210, the modeling tool 1204 may build program code and associated data structures implementing the neural network(s) 1202, e.g., using code and data-structure templates. Alternatively, the program code and data structures of the neural network(s) 1202 may be provided directly to the computing system 1200 (e.g., as one or more files downloaded to the computing system 1200). The neural network(s) may include the RNNs 300, 310 used to model a segmented output sequence for sequence or non-sequence input, as well as, depending on the particular application, additional network layers to generate the input to the RNNs 300, 310, e.g., the word-embedding, local-reordering, and sequence-encoding layers 804, 806, 808 of the neural network architecture 800 for machine-translation applications.

The neural network(s) as originally defined generally include a number of network weights or other parameters 1212 that are to be optimized through training. The training component 1206 adjusts these network parameters 1212 based on training data 1214 provided as input to the computing system 1200. The training data includes pairs of an input (which may or may not be a sequence) and an output sequence whose format depends on the overall architecture of the neural network(s) to be trained. For example, to train the neural network architecture 800 for machine translation end-to-end, the input and output sequences may be strings reflecting texts in the source and target languages, respectively. On the other hand, if, for instance, a SWAN layer is trained separately, at least the input is provided in the form of real-valued vectors. The training component 1206 may include algorithms implementing, e.g., the forward and backward propagations through the network that are described above with respect to FIGS. 4-5B.

The decoder 1208 may use the neural network(s), once trained, to generate, from an input (sequence) 1216, a corresponding output sequence 1218: the format of these sequences generally conforms to that of the (application-dependent) format of the training data. The input (sequence 1216) may be provided to the computing system 1200 via a user interface, such as, e.g., an interface that allows the user to type in text, or record speech, to be translated. Alternatively, the input may be provided in the form of a file uploaded to the computing system 1202, or be fed into the decoder directly from another computational component (within or outside the computing system 1200) that produces it as output. Similarly, the output sequence 1218 may be output via a user interface, e.g., in a machine-translation application, in the form of a translated text or audio output; saved to a file; or sent to a downstream computational component for further processing. Other modes of input and/or output may occur to those of ordinary skill in the art. The decoder 1208 may implement algorithms for feeding the input 1216 to the neural network(s) 1202 for generating the output sequence 1218 or elements thereof. For example, a decoder for SWAN may include program code implementing the beam search algorithm described with reference to FIG. 6.

In general, the operations, algorithms, and methods described herein may be implemented in any suitable combination of software, hardware, and/or firmware, and the provided functionality may be grouped into a number of components, modules, or mechanisms. Modules and components can constitute either software components (e.g., code embodied on a non-transitory machine-readable medium) or hardware-implemented components. A hardware-implemented component is a tangible unit capable of performing certain operations and can be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client, or server computer system) or one or more processors can be configured by software (e.g., an application or application portion) as a hardware-implemented component that operates to perform certain operations as described herein.

In various embodiments, a hardware-implemented component can be implemented mechanically or electronically. For example, a hardware-implemented component can comprise dedicated circuitry or logic that is permanently configured (e.g., as a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware-implemented component can also comprise programmable logic or circuitry (e.g., as encompassed within a general-purpose processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a hardware-implemented component mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) can be driven by cost and time considerations.

Accordingly, the term “hardware-implemented component” should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily or transitorily configured (e.g., programmed) to operate in a certain manner and/or to perform certain operations described herein. Considering embodiments in which hardware-implemented components are temporarily configured (e.g., programmed), each of the hardware-implemented components need not be configured or instantiated at any one instance in time. For example, where the hardware-implemented components comprise a general-purpose processor configured using software, the general-purpose processor can be configured as respective different hardware-implemented components at different times. Software can accordingly configure a processor, for example, to constitute a particular hardware-implemented component at one instance of time and to constitute a different hardware-implemented component at a different instance of time.

Hardware-implemented components can provide information to, and receive information from, other hardware-implemented components. Accordingly, the described hardware-implemented components can be regarded as being communicatively coupled. Where multiple such hardware-implemented components exist contemporaneously, communications can be achieved through signal transmission (e.g., over appropriate circuits and buses that connect the hardware-implemented components). In embodiments in which multiple hardware-implemented components are configured or instantiated at different times, communications between such hardware-implemented components can be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware-implemented components have access. For example, one hardware-implemented component can perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware-implemented component can then, at a later time, access the memory device to retrieve and process the stored output. Hardware-implemented components can also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information).

The various operations of example methods described herein can be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions. The components referred to herein can, in some example embodiments, comprise processor-implemented components.

Similarly, the methods described herein can be at least partially processor-implemented. For example, at least some of the operations of a method can be performed by one of processors or processor-implemented components. The performance of certain of the operations can be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processor or processors can be located in a single location (e.g., within an office environment, or a server farm), while in other embodiments the processors can be distributed across a number of locations.

The one or more processors can also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations can be performed by a group of computers (as examples of machines including processors), these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs)).

Example embodiments can be implemented in digital electronic circuitry, in computer hardware, firmware, or software, or in combinations of them. Example embodiments can be implemented using a computer program product, e.g., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable medium for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers.

A computer program can be written in any form of description language, including compiled or interpreted languages, and it can be deployed in any form, including as a standalone program or as a component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

In example embodiments, operations can be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method operations can also be performed by, and apparatus of example embodiments can be implemented as, special purpose logic circuitry, e.g., an FPGA or an ASIC.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In embodiments deploying a programmable computing system, it will be appreciated that both hardware and software architectures merit consideration. Specifically, it will be appreciated that the choice of whether to implement certain functionality in permanently configured hardware (e.g., an ASIC), in temporarily configured hardware (e.g., a combination of software and a programmable processor), or a combination of permanently and temporarily configured hardware can be a design choice. Below are set out hardware (e.g., machine) and software architectures that can be deployed, in various example embodiments.

FIG. 13 is a block diagram of a machine in the example form of a computer system 1300 within which instructions 1324 may be executed to cause the machine to perform any one or more of the methodologies discussed herein. In alternative embodiments, the machine operates as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine can operate in the capacity of a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a web appliance, a network router, switch, or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The example computer system 1300 includes a processor 1302 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 1304, and a static memory 1306, which communicate with each other via a bus 1308. The computer system 1300 can further include a video display 1310 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system 1300 also includes an alpha-numeric input device 1312 (e.g., a keyboard or a touch-sensitive display screen), a user interface (UI) navigation (or cursor control) device 1314 (e.g., a mouse), a disk drive unit 1316, a signal generation device 1318 (e.g., a speaker), and a network interface device 1320.

The disk drive unit 1316 includes a machine-readable medium 1322 on which are stored one or more sets of data structures and instructions 1324 (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. The instructions 1324 can also reside, completely or at least partially, within the main memory 1304 and/or within the processor 1302 during execution thereof by the computer system 1300, with the main memory 1304 and the processor 1302 also constituting machine-readable media.

While the machine-readable medium 1322 is shown in an example embodiment to be a single medium, the term “machine-readable medium” can include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more instructions 1324 or data structures. The term “machine-readable medium” shall also be taken to include any tangible medium that is capable of storing, encoding, or carrying instructions 1324 for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure, or that is capable of storing, encoding, or carrying data structures utilized by or associated with such instructions 1324. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media. Specific examples of machine-readable media 1322 include non-volatile memory, including by way of example semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

The instructions 1324 can be transmitted or received over a communication network 1326 using a transmission medium. The instructions 1324 can be transmitted using the network interface device 1320 and any one of a number of well-known transfer protocols (e.g., HTTP). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, mobile telephone networks, plain old telephone (POTS) networks, and wireless data networks (e.g., Wi-Fi and WiMax networks). The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding, or carrying instructions 1324 for execution by the machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such software.

Although embodiments have been described with reference to specific example embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The accompanying drawings that form a part hereof, show by way of illustration, and not of limitation, specific embodiments in which the subject matter may be practiced. The embodiments illustrated are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. This description, therefore, is not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.

Claims

1. A method comprising: constructing an output-sequence probability as a sum, taken over all valid output-sequence segmentations, of products of output-segment probabilities;modeling a distribution of the output-sequence probability with a set of artificial neural networks, the set of artificial neural networks modeling the distributions of the output-segment probabilities with respective instances of a first recurrent neural network having an associated softmax layer; andusing one or more hardware processors to train the set of artificial neural networks, wherein a dynamic programming algorithm is used to recursively compute the output-sequence probability from the output-segment probabilities.

2. The method of claim 1, wherein the output-segment probabilities depend on respective concatenations of preceding output segments, and wherein the set of artificial neural networks models the concatenations with a second recurrent neural network.

3. The method of claim 1, wherein computing the output-sequence probability comprises recursively computing forward and backward probabilities for two-way output-sequence partitionings.

4. The method of claim 1, wherein, in computing the output-segment probabilities, an output-segment length is limited to a specified maximum value.

5. The method of claim 1, wherein the set of artificial neural networks is trained using backward propagation of errors, and wherein, in computing the output-segment probabilities during a forward propagation phase and in computing gradients of the output-segment probabilities used during a backward propagation phase, contributions computed for longer output segments are reused during computations for shorter output segments contained in the respective longer output segments.

6. The method of claim 1, further comprising: using the one or more hardware processors to perform a beam search algorithm to determine an output sequence for a given input based on the trained set of artificial neural networks.

7. The method of claim 6, wherein the input is an input sequence and the beam search algorithm comprises, for each element of the input sequence, performing a left-to-right beam search and thereafter merging any identical partial candidate output sequences obtained for multiple respective segmentations of the output sequence.

8. The method of claim 1, wherein the output-sequence probability is constructed for non-sequence input and wherein empty segments are not permitted in output sequences.

9. The method of claim 1, wherein the output-sequence probability is constructed for an input sequence, and wherein an output sequence is modeled as monotonically aligned with the input sequence and as having a number of segments equal to a number of elements in the input sequence, empty segments being permitted.

10. The method of claim 9, wherein the input sequence represents a human-language sequence in a first language and the output sequence represents a human-language sequence in a second language that corresponds to a translation from the first language.

11. The method of claim 10, further comprising training a second set of neural networks that generates the input sequence from the human-language sequence in the first language.

12. The method of claim 11, wherein the second set of neural networks comprises a network layer that locally reorders elements of a sequence of embedded representations of elements of the human-language sequence in the first language.

13. The method of claim 12, wherein an output element of the network layer is computed as a non-linear transformation of a weighted liner combination of elements of the sequence of embedded representations within an input window, each element being weighted in the weighted linear combination by gate values that depend on all elements within the input window.

14. The method of claim 12, wherein the second set of neural networks further comprises a word-embedding layer that generates the embedded representations of the elements of the human-language sequence in the first language by word embedding.

15. The method of claim 12, wherein the second set of neural networks further comprises a bidirectional recurrent neural network that generates the input sequence from the reordered sequence of embedded representations.

16. One or more machine-readable media storing: data defining a trained set of artificial neural networks, the set of artificial neural networks modeling an output-sequence probability constructed as a sum of products of output-segment probabilities taken over all valid output-sequence segmentations, the set of artificial neural networks modeling the distributions of the output-segment probabilities with respective instances of a recurrent neural network having an associated softmax layer; andinstructions for execution by one or more hardware processors, the instructions, when executed by the one or more hardware processors, causing the one or more hardware processors to perform operations for creating an output sequence from an input, the operations comprising: using a beam search algorithm to determine the output sequence for the input based on the trained set of artificial neural networks.

17. The one or more machine-readable media of claim 16, wherein the input is an input sequence and the beam search algorithm comprises, for each element of the input sequence, performing a left-to-right beam search and thereafter merging any identical partial candidate output sequences obtained for multiple respective segmentations of the output sequence.

18. The one or more machine-readable media of claim 16, wherein the set of artificial neural networks is trained based on pairs of input and output sequences that represent human-language sequences in first and second languages, the output sequences corresponding to translations from the first language into the second language.

19. The one or more machine-readable media of claim 18, wherein the data further defines a second set of artificial neural networks modeling generation of the input sequences from human-language sequences in the first language, the second set of neural networks comprising a local reordering layer for locally reordering an embedded representation of the human-language sequence in the first language.

20. A system comprising: one or more hardware processors; andone or more machine-readable media storing data defining a set of artificial neural networks modeling an output-sequence probability constructed as a sum of products of output-segment probabilities taken over all valid output-sequence segmentations, the set of artificial neural networks modeling the distributions of the output-segment probabilities with respective instances of a recurrent neural network having an associated softmax layer; andinstructions for execution by the one or more hardware processors, the instructions, when executed by the one or more hardware processors, causing the one or more hardware processors to perform operations to train the set of artificial neural networks, using a dynamic programming algorithm to recursively compute the output-sequence probability from the output-segment probabilities.