Generation and exploitation of an approximate language model

Abstract

A system, method, and computer program for generating and exploiting an approximate language model are provided. The method comprises generating a language model according to an approximate hashing technique. The language model comprises a plurality of event sequences in a target language, and each member of the plurality of the event sequences is associated with at least one count. The language model is queried for a member of the plurality of event sequences. A probability associated with the member of the plurality of event sequences is determined based on results of the query.

Description

BACKGROUND

1. Field of Invention

The present invention relates generally to natural language processing and more specifically to natural language processing applications that use a language model.

2. Description of Related Art

In machine translation, a language model is often used to estimate the correctness of a translated text in a target language. In statistical machine translation, the language model typically assigns a high probability to a grammatically correct sentence and a low probability to a grammatically incorrect sentence. For example, a language model will assign a high probability to the sequence, “Mary has a little lamb” and a low probability to the sequence “Mary little lamb a has.”

Typically, language models include a stochastic generative process that produces structures (e.g., sentences) that are either well-formed or ill-formed. The stochastic generative process typically generates the structures in a sequence of steps where each step is associated with a probability. The probability of the entire structure is calculated by multiplying the probability of all of the steps.

For example, a language model can be generated by choosing a first word in the sentence, choosing other words in the sentence and choosing a last word in the sentence. The first word in the sentence is associated with a probability of p(w₁|BOS) where BOS is the Beginning of the Sentence. The other words in the sentence are associated with a probability p(w_i|w_i-1). The last word in the sentence is associated with a probability p(EOS|w_n) where EOS is the End of the Sentence. P(x|y) is a conditional probability or how likely it is that x follows word y. The probabilities are typically estimated from a large corpus in the target language.

Continuing the “Mary has a little lamb” example, the sentence “mary has a little lamb” receives the probability p(mary|BOS)×p(has|mary)×p(a|has)×p(little|a)×p(lamb|little)×p(EOS|lamb). Because the bigram conditional probabilities reflect that the sequence “Mary has a little lamb” is more likely than those in the sequence, “Mary little lamb a has,” the sequence “Mary has a little lamb” will be selected by a machine translator.

Another example of a language model is a dependency-based language model where a sentence is generated using a tree structure. FIG. 1 is an exemplary depiction of a tree structure 100 used to generate a sentence. From an invisible START symbol 105, the language model generates the head word 110 of the tree “has” with probability p(has|START). The language model then generates left modifiers 115 (i.e., “Mary”) and right modifiers 120 (i.e., “lamb”) with probabilities p_left(mary|has)×p_right(lamb), respectively. Finally, the left modifiers 125 (i.e., “a” and “little”) of “lamb” with probabilities p_left(a|lamb)×p_left(little|lamb) are generated.

To be used in machine translation, the language model is stored in a random access memory (RAM). For a bi-gram language model, such as those described herein, four pieces of information are generally stored in RAM. First, the history of the word, or preceding context on which the decision to use the word was made, is stored. Next, the word (i.e., event) that was generated based on the history is stored. Third, the conditional probability of generating the word given the history is stored. The conditional probability is expressed as a floating point value. Finally, some back-off values that enable one to estimate probabilities of events that have not been seen in the training data are stored.

If a language model becomes too large (e.g., tens of billions or trillions of words), the amount of RAM required to store the language model becomes prohibitive.

SUMMARY OF THE INVENTION

A system, method, and computer program for generating and exploiting an approximate language model are provided. The method comprises generating a language model according to an approximate hashing technique. The language model comprises a plurality of event sequences in a target language, and each member of the plurality of the event sequences is associated with at least one count. The language model is queried for a member of the plurality of event sequences. A probability associated with the member of the plurality of event sequences is determined based on results of the query.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary depiction of a tree structure used to generate a sentence according to the prior art.

FIG. 2 is a flowchart of an exemplary method for generating and exploiting an approximate language model according to some embodiments.

FIG. 3 is a flowchart of a method for generating an approximate language model according to some embodiments.

FIG. 4A is a stratified spectral bloom filter according to various embodiments.

FIG. 4B is a condensed version of the stratified spectral bloom filter according to some embodiments.

FIG. 5 is a flowchart of a method for querying a Stratified Spectral Bloom filter for determining a count according to various embodiments.

FIG. 6 is a diagram of a digital device according to various embodiments.

DETAILED DESCRIPTION

An approximate language model using approximate hashing to load a language model into a RAM is provided. The approximate language model comprises a table which pairs a language model event sequence (e.g., word sequence, word dependency sequence, part of speech sequence, etc.) appearing in training data to a numerical value. The numerical value may indicate a count, a probability, a score, or the like. The approximate language model decreases the amount of data that is loaded into a random access memory (RAM). However, the use of approximate hashing techniques may cause the language model to return inaccurate results.

Approximate hashing may be used to store and query an approximate language model while reducing the amount of data loaded into the RAM of a computer or other machine performing machine translation, speech recognition, and/or other natural language processing applications. Approximate hashing may compactly represent the language model in RAM not storing the history and word/event information. Approximate hashing may be performed by generating one or more hash tables using separate hash functions. For example, a key corresponding to a single input string, when input into a data structure using approximate hashing, is processed using each of the hash functions. The key may return a value for each of the hash functions based on the hash tables associated with the hash function. Each value may be a quantized language model probability. Thus, approximate hashing further involves determining, through selection and/or aggregation of the returned values, a single result. The single result may comprise one or more values.

FIG. 2 is a flowchart of an exemplary method 200 for generating and exploiting an approximate language model according to some embodiments. The method 200 may be performed by a digital device as described in connection with FIG. 6.

In a step 210, data including history and event information is generated. The data may be generated from, for example, a known language model, a parallel corpus, a monolingual corpus, or the like. The data includes a list of unique tuples which indicate history, event, and count information of a word of interest. History information is the context in which the word of interest appears, e.g., the word occurring just before the word of interest. Event information identifies the word of interest itself. The history information and the event information combined may be referred to as an “event sequence.” Count information is the number of times the history and event have occurred. The tuples may be generated according to techniques known to those skilled the art.

In a step 220, the data is stored in a language model using an approximate hashing technique. Approximate hashing techniques include techniques employing data structures such as: spectral Bloom filters, Bloom filter replacements, dynamic Bloom filters, weighted Bloom filters, d-left counting Bloom filters, parallel Bloom filters, hierarchical Bloom filter arrays, stable Bloom filters, dynamic count filters, Bloomier filters, compact approximators, generalized Bloom filters, attenuated Bloom filters, compressed Bloom filters, perfect hashing, or the like.

In some embodiments, the approximate language model comprises a set of data structures organized in a hierarchy. The hierarchy may be ranked based on the number of occurrences of a language model event sequence in the training data. In the hierarchy, the lower levels, corresponding to the event sequences occurring least frequently, may be stored using fewer bits while the higher levels, corresponding to the word sequences occurring most frequently, are stored using more bits. Thus, once multiple values are selected based on a key and more than one hashing function, the hierarchy may determine the result. In exemplary embodiments, the result appearing in the highest ranked data structure containing a value corresponding to the key is returned. An example of using spectral bloom filters organized in a hierarchy is discussed in greater detail in connection with FIGS. 3-4B.

In a step 230, input text is received. The input text is typically an initial translation in a target language of text in a source language. The initial translation may be generated using a statistical machine translation technique.

In a step 240, the language model is queried. The type of query used depends on the approximate hashing technique used to generate the language model as will be apparent to those skilled in the art of approximate hashing techniques. An exemplary query technique based on spectral bloom filters is discussed in connection with FIG. 5. The results of the query include counts of the number of occurrences of the history with the event and the number of occurrences of the history. If either of the counts is zero, the method 200 may back-off to shorter histories (i.e., histories having fewer words) and employ a back-off factor or a range of back-off factors that depend on the length of the history. The back-off factors may be tuned automatically depending on a final evaluation criterion using standard techniques.

In a step 250, at least one query result is selected based on history and event information. The probability of the event is calculated by dividing the count of the number of occurrences of the history with the event by the number of occurrences of the history. Typically, the query result having the highest probability of the event is selected.

FIG. 3 is a flowchart of a method 300 for generating an approximate language model according to some embodiments. The approximate language model generated by the method 300 is generated using a stratified spectral bloom filter. As will be apparent, other approximate hashing techniques may be used. As will be apparent to those skilled in the art, the method 300, which includes four Spectral Bloom filters (SBF), may be modified to accommodate any number of Spectral Bloom filters within the SSBF 400. The method 300 may be performed by a digital device as described in connection with FIG. 6.

In a step 305, a stratified spectral bloom filter (SSBF) is generated. The SSBF includes one or more Spectral Bloom filters. In some embodiments, the SSBF includes six Spectral Bloom filters. The Spectral Bloom filter comprises a space-efficient probabilistic data structure that is used to test whether an element is a member of a set. An empty Spectral Bloom filter comprises a bit vector of m counters, C, representing the multiplicities of items, all set to 0. A Spectral Bloom filter comes with k different hash functions, each of which maps a key value to one of the m array positions. To add an element s to the Spectral Bloom filter, the counters C_h1(s), C_h2(s), . . . , and C_hk(s) are increased by 1. To query for an element s (determine how often it occurred), s is fed to each of the k hash functions to get the values of the counters C_h1(s), C_h2(s), . . . , and C_hk(s). The counter of minimal value is returned. When adding an element s, only the counters equal to m_x, the minimal counter, may be increased. When performing a lookup query, m_x is returned. For inserting c occurrences of an element, the smallest counters are increased by c and every other counter is set to the maximal of their previous value and m_x+c.

An SSBF is a hierarchy of Spectral Bloom filters. The lower levels of the hierarchy store the counts of the elements s that occur very rarely in a multiset S (i.e., the generated data of step 210). The higher levels of the hierarchy store the counts of the elements s that occur very frequently in the multiset S. To save space, the Spectral Bloom filters at the lower level use fewer bits per count than the Spectral Bloom filters at the higher level.

The hierarchy may include any number of Spectral Bloom filters depending on the size of the bit counters. Each SBF_I uses a set of hash functions h_I,1, h_I,2, . . . , h_I,k. Each Spectral Bloom filter uses the same number of hash functions, but the hash functions are distinct to minimize collisions.

FIG. 4A is a stratified spectral bloom filter 400 according to various embodiments. The SSBF 400 includes four Spectral Bloom filters. The first Spectral Bloom filter SBF₁ 405 uses bit counters of size 1. The second Spectral Bloom filter SBF₂ 410 uses bit counters of size 2. The third Spectral Bloom filter SBF₄ 415 uses bit counters of size 4. The fourth Spectral Bloom filter SBF₈ 420 uses bit counters of size 8. As will be apparent to those skilled in the art, the number of bits in each counter may be varied. As will be discussed further herein, the counters depicted in FIG. 4A indicate that an element s occurs six times.

Returning to FIG. 3, in a step 310, at least one hash value is calculated based on an element s (i.e., event) in the set S (i.e., the data generated in step 210). The hash values are calculated using hash functions as will be apparent to those skilled in the art. Because multiple hash functions may be used for each element s, the element s is associated with more than one hash value and, thus, more than one counter C_h1,1(s), C_h1,2(s), . . . , and C_h1,k(s), where k is the number of hash functions used.

In a step 315, the counters are incremented by adding elements s to the SSBF. Referring to FIG. 4A, for any element s from S, the one-bit counters C_h1,1(s), C_h1,2(s), . . . , and C_h1,k(s) corresponding to each hash function are set to 1 in SBF 1 405 if s occurs at least once in S.

In a step 320, a determination is made as to whether the element s occurs more than three times. If the element s does not occur more than three times, the two-bit counter in SBF₂ is incremented to indicate the actual count, in a step 325. If, however, the element s occurs more than three times, the counters C_h2,1(s), C_h2,2(s), . . . , and C_h2,k(s) corresponding to each hash function are set to the maximum value in SBF₂ (i.e., three), in step 330. Thus the counter in SBF₂ is set to the minimum of the count and three, the maximum count that can be stored by the two-bit counter. Referring to FIG. 4A, because the element s occurs more than three times, the counter of SBF 2 410 is set to the maximum amount of three.

In a step 335, a determination is made as to whether the element s occurs more than fifteen times, the maximum count that can be stored in a four-bit counter. If the element s does not occur more than fifteen times, the four-bit counter in SBF₃ is incremented to indicate the actual count, in a step 340. Referring to FIG. 4A, because the element s occurs six times, the counter of SBF 4 410 is set to the binary number 0110, or six. If, however, the element s occurs more than fifteen times, the counters C_h4,1(s), C_h4,2(s), . . . , and C_h4,k(s) corresponding to each hash function are set to the maximum value in SBF₄ (i.e., fifteen), in step 330. Thus the counter in SBF₄ is set to the minimum of the count and fifteen, the maximum count that can be stored by the four-bit counter.

In a step 350, a determination is made as to whether the element s occurs more than 255 times, the maximum count that can be stored in an eight-bit counter. If the element s does not occur more than 255 times, the eight-bit counter in SBF₈ is incremented to indicate the actual count in a step 355. If, however, the element s occurs more than 255 times, the counters C_h8,1(s), C_h8,2(s), . . . , and C_h8,k(s) corresponding to each hash function are set to the maximum value in SBF₈ (i.e., 255), in step 360. Thus the counter in SBF₈ is set to the minimum of the count and fifteen, the maximum count that can be stored by the four-bit counter.

FIG. 4B is a condensed version 450 of the stratified spectral bloom filter 400 according to some embodiments. Once an SSBF-based language model is computed, it can be further compacted by keeping at each SBF level only a simple, 1-bit Bloom filter, indicating whether the count is stored in the Spectral Bloom filter at that level. For example, a count is at the level 3 in the SSBF simply indicates that the count has a value between four and fifteen. To quantize the counts, the average of the values stored at each level may be used.

As depicted in FIG. 4B, Bloom filter (BF) 1 455 is set to one to indicate that the element s occurs at least once. BF 2 460 is also set to one, indicating that element occurs at least two to three times. BF 4 465 is also set to one, indicating that element s occurs at least four to fifteen times. BF 8 470 is set to zero, indicating that element s does not occur at least sixteen times. Thus, the condensed version 450 indicates that the count for element s is between four and fifteen.

To use SSBFs to store a language model, a vector of several SSBFs is generated. To illustrate, a first SSBF, SSBF[1], stores all empty histories and one-word events, for a total key size of 1 (history size of zero+one extra word). A second SSBF, SSBF[2], stores all histories of length 1 and one-word events, for a total key size of 2 (history size of one+one extra word). A final SSBF, SSBF[n], stores all histories of length n−1 and one-word events, for a total key size of n. For example, a five-gram language model has one SSBF that stores all unigram counts; one SSBF that stores all bigram counts; one SSBF that stores all trigram counts; one SSBF that stores all four-gram counts; and one SSBF that stores all five-gram counts.

In one instance, given the counts associated with each event, one-count events can be stored as one bit; two- and three-count events can be stored as two bits; and so forth for four, eight, sixteen, and thirty-two bits. On the basis of this computation and a pre-defined error rate, the number of bits used at each level in an SSBF data structure is calculated. For example, from a corpus of 10M sentences, the counts of all trigrams in the corpus are calculated. In the corpus, there are 50,322,804 trigrams that occur one or more times; 20,983,262 trigrams whose counts fit in two or more bits; 8,624,766 trigrams whose count fit into four or more bits; 1,809187 trigrams whose count fit into eight or more bits; 90,250 trigrams whose count fit into sixteen or more bits; and 36 trigrams whose count fit into thirty-two bits. Assuming that bit arrays eight times larger than the number of keys they store are used and six hash key functions are used per SBF, which corresponds to an error rate of approximately two percent, it follows that all the counts for all trigrams can be stored in an SSBF of 136.096 MB. Further, as will be apparent to those skilled in the art, the counts of all unigrams, bigrams, trigrams, four-grams, and five-grams extracted from the same 10M line corpus can be stored in five SSBFs that require a total of 635.931 MB. When the language model is condensed, as described in connection with FIG. 4B, the approximate counts of all the trigrams in the corpus example can be stored in 80.02 MB.

FIG. 5 is a flowchart of a method 500 for querying an SSBF, such as SSBF 400, to determine a count according to various embodiments. The method 500 traverses the SSBF from the low bit counters to the high-bit counters and returns the counter having the minimum value. While the method 500 describes an SSBF having counters of one, two, four, and eight bits, it will be apparent to those skilled in the art that the method 500 may be modified to accommodate an SSBF having counters of any size. The method 500 is repeated for each hash function used to generate the SSBF.

In a step 505, a determination is made as to whether the one-bit counter stores a value equal to one. If the one-bit counter stores a value of zero, the query results in returning a value of zero in a step 510, indicating that the element s does not exist in the language model.

In a step 515, if the one-bit counter stores a value equal to one, another determination is made as to whether the two-bit counter (or another higher bit counter) stores a value greater than zero. In a step 520, if the two-bit counter stores a value of zero, the method 500 returns the highest count from the lower-bit counter, in this case, one.

In a step 525, if the two-bit counter stores a non-zero value, a determination is made as to whether the two-bit counter stores a value equal to the maximum count that can be stored by the counter, in this case, three. If the two-bit counter does not store a value equal to three, the value stored in the two-bit counter is returned, in a step 530.

In a step 535, if the two-bit counter stores a value equal to three, another determination is made as to whether the next higher bit counter, in this instance, a four-bit counter, stores a value greater than zero. In a step 540, if the four-bit counter stores a value of zero, the method 500 returns the highest count from the lower-bit counter, in this case, three.

In a step 545, if the four-bit counter stores a non-zero value, a determination is made as to whether the four-bit counter stores a value equal to the maximum count that can be stored by the counter, in this case, fifteen. If the four-bit counter does not store a value equal to fifteen, the value stored in the four-bit counter is returned, in a step 550.

In a step 555, if the four-bit counter stores a value equal to fifteen, another determination is made as to whether the next higher bit counter, in this instance, an eight-bit counter, stores a value greater than zero. In a step 560, if the eight-bit counter stores a value of zero, the method 500 returns the highest count from the lower-bit counter, in this case, fifteen.

In a step 565, if the eight-bit counter stores a non-zero value, the value of the eight-bit counter is returned. Because the SSBF in this example contains a highest counter of eight bits, the maximum count that can be stored is 255. It is understood that SSBFs having counters containing more bits may be used to achieve higher counts.

After the counts for each history and event are determined the probabilities associated with each can be calculated to select a query result, as described in connection with steps 240 and 250 of FIG. 2.

FIG. 6 is a diagram of a digital device according to various embodiments. The digital device 600 comprises a processor 602, a random access memory 604, a read-only memory 606, an input/output (“I/O”) interface 608, a communication network interface 610 which are all coupled to a system bus 612. The processor 602 is configured to execute executable instructions. In some embodiments, the processor 602 comprises one or more circuits and/or multiple computation devices capable of processing the executable instructions.

The random access memory 604 is a memory configured to store data such that it can be accessed quickly. Some examples of the random access memory 604 include RAM, or flash memory. The read-only memory 606 is any storage configured to persistently store data (e.g., static memory) but is accessed sequentially. Some examples of the read-only memory 606 include, but are not limited to, flash drives, hard drives, optical drives, and/or magnetic tape. Computer readable medium may comprise all or some of the random access memory 604, the read-only memory 606, or a combination of the random access memory 604 and the read-only memory 606.

The I/O interface 608 is any device that can receive input from or deliver output to a system administrator or a user. The I/O interface 608 can couple to a keyboard, display, mouse, keypad, printer, scanner, or any other input or output device.

The communication network interface 610 is configured to couple to a communication network such as a local area network (LAN), wide area network (WAN), the Internet, or the like via the coupling 614. Moreover, the communication network interface 610 may support communication over many kinds of connections, including, but not limited to, a USB connection, a firewire connection, an Ethernet connection, a serial connection, and/or a parallel connection. The communication network interface 610 may also support wireless communication (e.g., 802.11a/b/g/n or wireless USB). It will be apparent to those skilled in the art that the communication network interface 610 can support many wired and wireless standards.

The above-described modules can be comprised of instructions that are stored on computer-readable storage media. The instructions can be retrieved and executed by the processor 602. Some examples of instructions include software, program code, and firmware. Some examples of storage media comprise memory devices and integrated circuits. The instructions are operational when executed by the processor 602 to direct the processor 602 to operate in accordance with embodiments of the present invention. Those skilled in the art are familiar with instructions, processor(s), and storage media.

While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments.

Claims

1. A method comprising: executing instructions via a processor of a computing system for: generating a language model according to an approximate hashing technique, the language model comprising a plurality of event sequences in a target language, each member of the plurality of the event sequences associated with at least one count, the language model comprising a set of data structures organized in a hierarchy with lower levels corresponding to event sequences occurring less frequently being stored using fewer bits, the hierarchy having three or more levels;querying the language model for a member of the plurality of event sequences; anddetermining a probability associated with the member of the plurality of event sequences based on results of the query.

2. The method of claim 1, wherein the approximate hashing technique uses a plurality of hash functions.

3. The method of claim 1, wherein the approximate hashing technique comprises generating a Stratified Spectral Bloom filter comprising a plurality of levels, each of the levels including a Spectral Bloom filter.

4. The method of claim 3, wherein the at least one count is indicated within each of the levels based on a minimal counter.

5. The method of claim 3, wherein an approximation of the at least one count is indicated within each of the levels using one bit.

6. The method of claim 1, wherein the language model includes a number of Stratified Spectral Bloom filters, the number based on a length of the event sequences.

7. The method of claim 1, wherein the event sequence comprises history information and event information.

8. The method of claim 1, wherein determining the probability is based on a first value associated with event information and a second value associated with history information.

9. The method of claim 1, wherein the approximate hashing technique comprises a filter selected from a group consisting of: Bloom filter replacements, dynamic Bloom filters, weighted Bloom filters, d-left counting Bloom filters, parallel Bloom filters, hierarchical Bloom filter arrays, stable Bloom filters, dynamic count filters, Bloomier filters, compact approximators, generalized Bloom filters, attenuated Bloom filters, perfect hash keys, and compressed Bloom filters.

10. A system comprising: a processor configured to generate a language model based on a training corpus using an approximate hashing technique, the language model comprising a set of data structures organized in a hierarchy with lower levels corresponding to event sequences occurring less frequently being stored using fewer bits, the hierarchy having three or more levels; anda memory configured to store the language model.

11. The system of claim 10, wherein the language model comprises a plurality of Stratified Spectral Bloom filters.

12. The system of claim 11, wherein the approximate hashing technique includes a plurality of hash functions.

13. The system of claim 10, wherein the memory comprises a random access memory.

14. The system of claim 10, wherein the size of the language model is less than six hundred fifty megabytes for a portion of a training corpus comprising ten million sentences.

15. The system of claim 10, wherein the processor is further configured to query the language model based on an initial sequence.

16. The system of claim 10, wherein the approximate hashing technique comprises generating a Stratified Spectral Bloom filter comprising a plurality of levels, each of the levels including a Spectral Bloom filter.

17. The system of claim 16, wherein a count corresponding to an event sequence within the language model is indicated within each of the levels based on a minimal counter.

18. The system of claim 16, wherein the count is indicated within each of the levels using one bit.

19. The system of claim 18, wherein the size of the language model is less than one hundred megabytes for a portion of a training corpus comprising ten million sentences.

20. A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor for performing a method for determining a translation, the method comprising: generating a language model according to an approximate hashing technique, the language model comprising a plurality of event sequences in a target language, each member of the plurality of the event sequences associated with at least one count, the language model comprising a set of data structures organized in a hierarchy with lower levels corresponding to event sequences occurring less frequently being stored using fewer bits, the hierarchy having three or more levels;querying the language model for a member of the plurality of event sequences; anddetermining a probability associated with the member of the plurality of event sequences based on results of the query.