Patent Grant
8260817
The invention relates in general to the field of topic classification, and more particularly to methods of processing free-text queries.
Topic classification systems are a class of machine learning tools designed to classify media based on information that has been extracted from the media. When topic classification systems are applied to the area of natural language processing, natural language inputs are classified and labeled based on the classes or topics that are found within the inputs. Typically, natural language inputs include text intervals. Text intervals are spans of text that need not be well-formed sentences and can come from a variety of sources, such as newspaper articles, books, e-mail, web articles, etc. For example, if the topic within a particular text interval is determined to be “the looting of Iraqi art galleries in 2003”, a number of labels can be assigned, such as Iraqi art galleries, looting in Iraq in 2003, etc.
Although typical topic classification systems classify a large number of text intervals, the labels that are assigned to each text interval generally need to be defined in advance. For example, a database stores searchable text intervals, in which each text interval has been assigned pre-defined labels organized into a topic or keyword listing. When a user performs a database query using several keywords, the system produces a set of candidate text intervals that have labels containing one or more of those keywords.
However, in a situation where the system has no prior knowledge of the labels of the text intervals, the system needs to parse through a text interval to determine its labels. For example, if text intervals are provided at run-time via natural language or free-text formulations, the topic classification system can no longer rely on predefined labels to locate similar text intervals. An example of a free-text formulation of a query is “List facts about the widespread looting of Iraqi museums after the US invasion.” Free-text queries differ from structured queries such as database queries where query terms need to be explicitly provided.
In order to respond to these free-text queries, the topic classification system needs to analyze various text intervals in a natural language document and determine whether the candidate responses are on-topic with the free-text queries. Although the topic classification system can match a free-text query to a candidate response if the query is simple and specific, but the system is limited with respect to matching a free-text query that contains a lengthy or complex description of some topic or event. In addition, human language makes it possible to convey on-topic information without using words that were used in the actual topic formulation. For example, given a free-text query of “List facts about the widespread looting of Iraqi museums after the US invasion,” an on-topic response would be “Many works of art were stolen from Baghdad galleries in 2003.” Furthermore, the presence of topic words in a sentence does not guarantee that the response will be relevant. For example, given the same query as before, an off-topic response would be “There were no known instances of looting of Iraqi museums before the U.S. invasion.” In accordance with the present invention, a topic classification system that provides for matching complex free-text queries to candidate responses is provided.
In one aspect the invention relates to a system that can process a text interval by extracting a proposition from a first text interval, and generating a proposition tree from the proposition, where each proposition tree includes a set of nodes and a set of edges. In one embodiment, in order to match a query to a response, the topic classification system can generate a first proposition tree for the query and a second proposition tree for the response. In order to determine the relevance of the response to the query, the system can calculate a first similarity value based on matching the first proposition tree to the second proposition tree. Finally, the system can select to output the response depending on whether the first similarity value exceeds a threshold.
In another embodiment, in order to determine the redundancy of two text intervals, the topic classification system can generate a first proposition tree for a first text interval and a second proposition tree for a second text interval. After generating proposition trees, the system can determine a second similarity value in addition to the first similarity value by a two-way comparison of the first proposition tree and the second proposition tree. Finally, the system can select not to output the second text interval based on whether the first and second similarity values exceed thresholds.
The matching process between the query and the response can be improved by augmenting the nodes of the first proposition tree to a first augmented proposition tree and augmenting the nodes of the second proposition tree to a second augmented proposition tree. One way to augment is by co-reference. Augmentation by co-reference includes identifying a real-world object, concept, or event in one or more nodes in the first proposition tree. Then the system can search the document that the query belongs to for alternative words that correspond to the same real-world object, concept, or event as the one or more nodes in the first proposition tree. Finally, nodes of the proposition tree can be augmented by the alternative words found using co-reference. Another way to augment nodes of the first proposition tree is by adding synonyms, hypernyms, hyponyms, substitutable labels, etc. to the nodes of the first proposition tree. The same approach can be used to augment the second proposition tree to create a second augmented proposition tree.
The similarity value used to determine the relevance of the response to the query can be calculated by performing a node-to-node match and an edge-to-edge match of the first proposition tree and the second proposition tree. One illustration of the similarity value is the transformation score that transforms the first proposition tree into the second proposition tree. In addition, a pair of nodes can match even if they are not initially identical. This type of matching can be accomplished by matching an augmented node of the first augmented proposition tree to an augmented node of the second augmented proposition tree. Furthermore, a pair of non-identical edges can match by allowing for the match of substitutable semantic relationships. Costs are associated with relaxing the matches of nodes and edges. In a situation where the query and the response generate a first and second set of proposition trees, a similarity value can calculated by aggregating the transformation scores across the two sets of augmented proposition trees, where each augmented proposition tree may be assigned a different weight.
A first set and second set of proposition subtrees can be generated from the first and second text intervals. A third similarity value can be generated by matching the first set of proposition subtrees to the second set of proposition subtrees. A first and second bag of nodes can be generated from the first and second text intervals. A fourth similarity value can be generated by matching the first bag of nodes to the second bag of nodes. The system can select to output the response based on whether the first, third or fourth similarity value exceeds its threshold.
According to another aspect, the invention provides a method of processing a text interval by extracting a proposition from the text interval, and generating a proposition tree from the proposition, where each proposition tree includes a set of nodes and a set of edges. In one embodiment, the method includes generating a first proposition tree for the query and a second proposition tree for the response in order to match a query to a response. The method also includes calculating a first similarity value based on matching the first proposition tree to the second proposition tree in order to determine the relevance of the response to the query. Finally, the method includes selecting to output or not output the response depending on whether the first similarity value exceeds a threshold.
The foregoing and other objects and advantages of the invention will be appreciated more fully from the following further description thereof, with reference to the accompanying drawings. These depicted embodiments are to be understood as illustrative of the invention and not as limiting in any way:
To provide an overall understanding of the invention, certain illustrative embodiments will now be described, including a system and a method for processing free-text queries in a topic classification system. However, it will be understood by one of ordinary skill in the art that the systems and methods described herein may be adapted and modified as is appropriate for the application being addressed and that the systems and methods described herein may be employed in other suitable applications, and that such other additions and modifications will not depart from the scope hereof.
In general, computing device 102 may be any type of computing platform (e.g. one or more general or special purpose computers), and may be connected to network 114. Computing device 102 is exemplary only. Concepts consistent with the present invention can be implemented on any computing device, whether or not connected to a network.
Processor 104 executes program instructions stored in memory 106. Process 104 can be any of a number of well-known computer processors, such as processors from Intel Corporation, of Santa Clara, Calif. Processor 104 can be used to run operating system applications, topic classification applications, and/or any other application. Processor 104 can drive output device 112 and can receive user inputs from input device 110.
Memory 106 includes one or more different types of memory that may be used for performing system functions. For example, memory 106 includes cache, Flash, ROM, RAM, or one or more different types of memory used for temporarily storing data.
Storage device 108 can be, for example, one or more storage mediums. Storage device 108, may store, for example, application data (e.g., documents that can be used to generate candidate responses based on free-text queries).
In the illustrative embodiment, modules are implemented in software for execution by various types of processors, such as processor 104. An identified module of executable code may, for instance, comprise one or more physical or logical blocks of computer instructions which may, for instance, be organized as an object, procedure, or function. Nevertheless, the executables of an identified module need not be physically located together, but may comprise disparate instructions stored in different locations which, when joined logically together, comprise the module and achieve the stated purpose for the module.
Indeed, a module of executable code could be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within modules, and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices, and may exist, at least partially, merely as electronic signals on a system or network.
Topic classification system 200 is preferably implemented as computer readable instructions executable by processor 104 on computing device 102. The computer preferably includes storage device 108 for storing data collected and used by topic classification system 200. Topic classification system 200 derives text intervals from a natural language document. While topic classification system 200 is generally described with respect to text-based inputs, it should be understood that this system may also be used with non-text based inputs, such as verbal or audio inputs, etc.
Topic classification system 200 can take a free-text query and one or more candidate responses and determine the extent to which these candidate responses are relevant for the topic of the query. Topic classification system 200 is useful in application domains including dialog systems, story segmentation systems, and question answering systems, etc. Dialog systems are computer systems that are designed to converse with a human participant. An example of a dialog system is a system frequently employed by airlines that allows travelers to input a number of natural language queries in order to find flight information. Story segmentation systems include systems that segment a story based on various topics found within the story. Each segmentation is organized into sub-stories, where each sub-story has the same topic. Question answering systems include systems that are given a collection of documents and a query posed in a natural language or free-text format and retrieve answers based on the free-text query. In question answering systems, a topic classification system would be useful in situations where queries are focused entirely on a topic or event (e.g., “List facts about events described as follows: [EVENT]”), or the queries inquire about a particular aspect of a topic or the reaction an event solicited (e.g. “How did [COUNTRY] react to [EVENT]?”).
Preprocessing module 204, extracts information from text intervals, such as sentences, phrases, free-text queries, candidate responses, etc., and converts the information into propositions, also called predicate-argument structures. A proposition describes how a particular predicate (e.g., a noun or verb) is related to arguments in a text interval (e.g., part of a text interval that surrounds the noun or verb). The proposition may capture more meaning from a text interval than, for instance, a keyword listing.
Propositions are classified as a particular type (e.g., VERB, NOUN, MODIFIER, SET, etc.) and include a predicate, and one or more arguments, where arguments may also be other propositions. Links define the semantic relationship between the predicate and its arguments. These links are labeled with roles (e.g., <object>, <subject>, <direct-object>, <premodifier>, any preposition, etc.). The standard notation for a proposition is: predicate<predicate-type>(role: argument, . . . , role: argument). For example, given a text interval such as text interval 202 “John throws the baseball,” preprocessing module 204 produces outputs corresponding to propositions 106a-106c. Proposition 212a is of type VERB and has predicate “throws.” Two arguments, m1 and m2, are linked to it. The first argument m1 is labeled as the subject of the predicate, and the second argument m2 is labeled as the object of the predicate. The subject and object labels define the roles of m1 and m2 with respect to the predicate “throws”. Furthermore, two other propositions 212b (a NAME proposition) and 212c (a NOUN proposition) are created to represent m1 and m2. This proposition structure is merely illustrative. It should be understood that other suitable structures can be used.
Preprocessing module 204 includes a parsing module 206, a proposition finding module 208, and a co-reference module 210. Given a text interval, parsing module 206 identifies the parts of speech of words in the text interval and generates a syntactic parse tree of the text interval. The syntactic parse tree generally focuses on the grammatical structure of the text interval and captures the parts of speech for each word in a text interval.
Proposition finding module 208 normalizes the syntactic parse tree to find propositions. The normalization procedure includes: 1) finding the logical subject and object for predicates (e.g., verbs); 2) identifying noun modifications; and 3) identifying arguments of predicates (e.g., verbs) as mentions or propositions. Mentions are references to real-world objects, concepts, or events defined in a text interval. Objects, concepts, or events include any real-world entity, such as a person, place, organization, event, etc., or an abstract entity, such as an idea, number, etc. Mentions include names, nominal expressions, pronouns, etc. Each argument of a proposition has a role, where roles include a closed set of grammatical functions (e.g., subject, object, indirect object, etc.), as well as a set of preposition- and conjunction-based roles (e.g., to, of, that, etc.).
Co-reference module 210 can be used to process all text intervals in a natural language document. For each mention in the natural language document, co-reference module 210 automatically determines alternative words that refer to the same real-word object, concept, or event as the mention by searching through all text intervals in the document. Thus, co-reference module 210 finds other words that occur in the same role as the mention, but belong to other text intervals in the document.
One specific preprocessing module that can be used as preprocessing module 204 is the information extraction module, SERIF, available from BBN Technology Corp. of Cambridge, Mass. SERIF is described further in “Experiments in Multi-Modal Automatic Content Extraction” by L. Ramshaw, et al., published in Proceedings of HLT-01 in 2001, the entirety of which is incorporated herein by reference.
Generation module 214 assembles all of the propositions produced by preprocessing module 204 into proposition trees. For example, generation module 214 assembles the components of proposition 212a into proposition tree 216. In general, a proposition tree is a coherent tree structure representing a text interval's meaning (e.g. semantics), abstracted from the text interval's exact grammatical structure (e.g. syntax). The nodes of the proposition tree include either predicates or arguments (e.g., verbs, nouns, names, adjectives, etc.), and the edges of the proposition tree are labeled with roles that define the semantic relationship between the nodes. It is possible for a predicate nodes to represent a conjunction (e.g., “for,” “and,” “nor,” “but,” etc.) or a disjunction (e.g., “either,” “or”, etc.) of argument nodes.
In step 310, generation module 214 determines whether the expanded proposition tree is already subsumed by another, bigger proposition tree. If it is determined by generation module 214 that the expanded proposition tree has been subsumed by another, bigger proposition tree, the module can ignore the expanded proposition tree. When generation module 214 has determined that the expanded proposition tree has not been subsumed by another, bigger proposition tree, the module determines in step 314 whether the expansion criteria have been satisfied. Expansion criteria are user specified metrics that indicates the desired level of proposition tree expansion. In general, smaller expansions create a greater number of smaller proposition trees, whereas larger expansions create a smaller number of bigger proposition trees. A fully expanded proposition tree is a tree in which all of the argument nodes have been expanded. For example, in
When generation module 214 has determined that the expansion criteria has been satisfied, the module returns in step 316 the resulting proposition tree or set of proposition trees that represents the text interval. If it is determined in step 314 that the expansion criteria has not been satisfied, generation module 214 can go back to step 308 and continue to expand each tree by replacing its argument nodes by propositions.
Therefore, by analyzing all propositions contained within a text interval, a set of proposition trees is created to represent the text interval, where the root node of each proposition tree is a different proposition. In addition, since text intervals can be sentence fragments, method 300 may be used to process free-text queries, which often contains words that do not form a complete sentence.
Referring back to
Given a first text interval such as the first sentence in the paragraph above, co-reference module 210 can produce the following co-references by examining the rest of the paragraph: “he” and “John” refer to “John Smith,” and “ball” refers to “baseball”. As mentioned previously, co-reference module 210 can be provided by SERIF, an information extraction system, available from BBN Technology Corp. of Cambridge, Mass. Augmentation module 218 then uses these co-references to augment the nodes of proposition tree 216 to generate augmented proposition tree 220. By augmenting the nodes of proposition trees, two nodes can match even if they are not initially identical. Augmenting nodes in a proposition tree will be further discussed in relation to
Matching module 222 is for matching two text intervals. Matching module 222 determines if a candidate response is on-topic with a free-text query. This module will be further discussed in
In step 516, external sources can be used to further augment augmented proposition tree 512 to form augmented proposition tree 518. Augmented proposition tree 518 includes nodes 520a-e. In particular, nodes 514a (“arrest”), 514d and 514e have been augmented to yield nodes 520a (“arrest, capture, apprehend, apprehension, detain”), 520d (“his, Abu Abbas, Abbas, the PLF leader, Mohammed Abbas, Abul Abbas”), and 520e (“Baghdad, the Iraqi capital, Baghhdad, Bagadad, Baghdag, Bagdad”), respectively. External sources may include a nominalization table and/or external dictionaries, where synonyms, alternative spellings for names, nationalities, and capitals to country names, etc., can be identified to augment nodes in a proposition tree. Another external source which may be employed is WordNet, which is described in “WordNet, an Electronic Lexical Database” by C. Fellbaum, published in 1998, the entirety of which is incorporated herein by reference. By using WordNet, stemmed versions of a word, as well as the word's synonyms, directly connected hypernyms and hyponyms can be added to a proposition tree node.
Some augmentation sources and augmentation relationships may be more reliable than other sources for generating words to augment a node. As a result, in addition to storing the substitutable word used to augment a node, other information regarding the substitutable word may be stored as well. For example, the source of the substitutable word and the relationship between the substitutable word and the original word (e.g., synonym, hypernym, etc.) may also be stored.
Flowchart 600 provides a method of comparing whether a candidate response is on-topic with a free-text query. In step 602, preprocessing module 204 creates a first parse tree from the free-text query. In step 604, generation module 214 transforms the first parse tree to a first proposition tree that includes a set of nodes. Next, in step 606, augmentation module 218 augments the first proposition tree to a first augmented proposition tree.
A similar process applies when a candidate response is presented to the topic classification system. In step 608, preprocessing module 204 creates a second parse tree from the candidate response. In step 610, generation module 214 transforms the second parse tree to a second proposition tree that includes a set of nodes. Next, in step 612, augmentation module 218 augments the second proposition tree to a second augmented proposition tree. In step 614, matching module 222 matches the first and second augmented proposition trees and calculates similarity values to determine the relevance of the candidate response to the free-text query. Similarity values indicate whether the candidate response is on-topic with the free-text query. The candidate response can then be outputted based on the similarity value.
In step 702, a full tree similarity value θ is calculated between a first augmented proposition tree generated from a free-text query and a second augmented proposition tree generated from a candidate response. In step 704, a subtree similarity value η is calculated between a first set of augmented proposition subtrees generated from a free-text query and a second set of augmented proposition subtrees generated from a candidate response. Proposition subtree formation will be further discussed in
The first process can be applied to exact match formulations, in which two nodes are considered a match if they are labeled with the same word (or phrase), and if they can be reached from their parents over edges with identical roles, whereby the parent nodes have already been found to be a match. In this method, θij is a binary value of either 0 or 1.
The second process can be applied to situations where proposition trees have been augmented, such as in illustrative example 400. For augmented proposition trees, the matching condition is relaxed so that two nodes match if the intersection of the words associated with the node is not the empty set. Furthermore, transformation score θij is computed according to:
θij=#matched*γmatch−
where γmatch is a constant match score awarded to each node-to-node match, # matched is the number of such matches for a proposition tree comparison, and
Other factors can incur cost as well. For example, augmentation step 508 may incur costs. The cost of adding a new word to a node can be computed as the cost of the original word incremented by empirically estimated stemming and/or synonym costs. The estimated stemming and/or synonym costs can be adjusted based on the source of augmentation. For instance, augmentation based on synonyms provided by WordNet may have a different cost than augmentation based on words provided by co-reference module 210.
In addition, situations in which role confusions are allowed can also add costs. As described above, roles are labels on the edges of the proposition tree that define the semantic relationship between the nodes of the proposition tree. For example, proposition trees for “Abu Abbas's arrest” and “the arrest of Abu Abbas” differ only because the edge between arrest and Abu Abbas is labeled <possessive> in the first case and “of” in the second case. Allowing the match of these two proposition trees requires taking into account the cost of substituting one role for another role, which adds to
Finally, when matching trees, given a likely scenario where not all proposition tree nodes of a free-text query (e.g., “the looting of Iraqi museums after the U.S. invasion”) can be transformed to proposition tree nodes of a candidate response (e.g., “the looting of Iraqi museums”), an allowance can be made to relax the match, especially if a significant portion of the rest of the proposition trees in the two sets match with a high match score. However, this incurs a cost that is proportional to the size of the missing portion and the closeness of the missing portion to the root node. That is, the cost increases when the missing portion is closer to the root node of the proposition tree.
In situations where sets of augmented proposition trees have been generated for both the free-text query and the candidate response, transformation scores can be aggregated across the two sets of augmented proposition trees, where each proposition tree may be assigned a different weight. The full tree similarity value θ is calculated by:
where ωi is the weight of each proposition tree generated from the free-text query, θi is the aggregate transformation score, and θimax is the pre-computed highest possible transformation score. For matches that have been performed based on fully expanded proposition trees, ωi can be adjusted based on the number of nodes in a proposition tree.
Subtree similarity value η can be calculated from augmented proposition subtrees in the same way as full tree similarity value θ. The exception is that the weight of each of the proposition subtrees ωi may depend on the size of the full proposition tree that encompasses the proposition subtrees.
Bag of nodes similarity value μ can be calculated by the percentage or weighted sum of augmented nodes in the free-text query that have at least one match in the augmented nodes of the candidate response. The weight of each node ωi can depend on the distance between the node and the root node of the augmented proposition tree. The bag of nodes similarity value μ can incorporate the augmented node cost structure of the augmented proposition trees. Therefore, similarity value μ is more generous than similarity values θ and η.
In another embodiment of the invention, topic classification system 200 can compare proposition trees to eliminate redundant information. As discussed previously, topic classification system 200 can determine whether a text interval is on-topic with another text interval. Moreover, some circumstances may require topic classification system 200 to determine redundancy in text intervals. In order to determine redundancy between a pair of text intervals, matching module 222 can calculate two similarity values from the pair of text intervals. Comparing one text interval proposition tree to another text interval proposition tree generates one similarity value. The other similarity value is generated by the reverse comparison.
For example, topic classification system 200 can be presented with two text intervals. The first text interval is “The shutdown of the Cernadova nuclear plant by the authorities.” The second text interval is “The shutdown of the plant.” The first text interval can be considered an on-topic response to the second text interval, because the first text interval contains additional information about “plant” that is not provided in the second text interval. However, the second text interval is an off-topic response to the first text interval, because “plant” in the second interval can refer to other plants besides the “Cernadova nuclear plant.” Therefore, although the first text interval would be a reasonable candidate response to the query given by the second text interval, the reverse is not true. As a result, only one similarity value should be calculated for this type of matching.
In a situation where the two text intervals are considered for redundancy, generating two-way similarity values may be useful. In the example given above, the first text interval and the second text interval can be regarded as providing redundant information. For instance, if the two text intervals are considered as two candidate responses, matching module 222 may only choose to selectively output the first text interval because it encompasses the information given by the second text interval. Therefore, two similarity values are used to determine redundancy.
The invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The forgoing embodiments are therefore to be considered in all respects illustrative, rather than limiting of the invention.
This application is a continuation of pending U.S. patent application Ser. No. 11/974,022, filed Oct. 10, 2007, which is hereby incorporated by reference herein in its entirety.
The U.S. Government has a paid-up license in this invention and the right in limited circumstances to require the patent owner to license others on reasonable terms as provided for by the terms of Contract No. HR0011-06-C-0022 awarded by DARPA.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4599691 | Sakaki | Jul 1986 | A |
| 4754326 | Kram et al. | Jun 1988 | A |
| 4809351 | Abramovitz et al. | Feb 1989 | A |
| 4914590 | Loatman et al. | Apr 1990 | A |
| 5062143 | Schmitt | Oct 1991 | A |
| 5343537 | Bellegarda et al. | Aug 1994 | A |
| 5396419 | Morimoto | Mar 1995 | A |
| 5406480 | Kanno et al. | Apr 1995 | A |
| 5418717 | Su et al. | May 1995 | A |
| 5438630 | Chen et al. | Aug 1995 | A |
| 5492473 | Shea | Feb 1996 | A |
| 5544257 | Bellegarda et al. | Aug 1996 | A |
| 5694559 | Hobson et al. | Dec 1997 | A |
| 5745113 | Jordan et al. | Apr 1998 | A |
| 5752052 | Richardson et al. | May 1998 | A |
| 5757960 | Murdock et al. | May 1998 | A |
| 5787198 | Agazzi et al. | Jul 1998 | A |
| 5822747 | Graefe | Oct 1998 | A |
| 5839106 | Bellegarda | Nov 1998 | A |
| 5841895 | Huffman | Nov 1998 | A |
| 5862259 | Bokser et al. | Jan 1999 | A |
| 5903858 | Saraki | May 1999 | A |
| 5926180 | Shimamura | Jul 1999 | A |
| 5926784 | Richardson et al. | Jul 1999 | A |
| 5933525 | Makhoul et al. | Aug 1999 | A |
| 5940821 | Wical | Aug 1999 | A |
| 5963940 | Liddy et al. | Oct 1999 | A |
| 5963965 | Vogel | Oct 1999 | A |
| 6006221 | Liddy et al. | Dec 1999 | A |
| 6021403 | Horvitz | Feb 2000 | A |
| 6026388 | Liddy et al. | Feb 2000 | A |
| 6029195 | Herz | Feb 2000 | A |
| 6105022 | Takahashi et al. | Aug 2000 | A |
| 6112168 | Corston et al. | Aug 2000 | A |
| 6167369 | Schultz | Dec 2000 | A |
| 6243669 | Horiguchi et al. | Jun 2001 | B1 |
| 6243670 | Bessho et al. | Jun 2001 | B1 |
| 6260035 | Horvitz et al. | Jul 2001 | B1 |
| 6278967 | Akers et al. | Aug 2001 | B1 |
| 6278968 | Franz et al. | Aug 2001 | B1 |
| 6304870 | Kushmerick et al. | Oct 2001 | B1 |
| 6430552 | Corston-Oliver | Aug 2002 | B1 |
| 6442584 | Kolli et al. | Aug 2002 | B1 |
| 6600501 | Israel et al. | Jul 2003 | B1 |
| 6601026 | Appelt et al. | Jul 2003 | B2 |
| 6606625 | Muslea et al. | Aug 2003 | B1 |
| 6609087 | Miller et al. | Aug 2003 | B1 |
| 6615207 | Lawrence | Sep 2003 | B1 |
| 6681044 | Ma et al. | Jan 2004 | B1 |
| 6691279 | Yoden et al. | Feb 2004 | B2 |
| 6785673 | Fernandez et al. | Aug 2004 | B1 |
| 6795825 | Rishe | Sep 2004 | B2 |
| 6839714 | Wheeler et al. | Jan 2005 | B2 |
| 6853992 | Igata | Feb 2005 | B2 |
| 6892189 | Quass et al. | May 2005 | B2 |
| 6950753 | Rzhetsky et al. | Sep 2005 | B1 |
| 7031970 | Blitzer | Apr 2006 | B2 |
| 7062483 | Ferrari | Jun 2006 | B2 |
| 7081834 | Rukangas et al. | Jul 2006 | B2 |
| 7110939 | Ozawa et al. | Sep 2006 | B2 |
| 7143091 | Charnock et al. | Nov 2006 | B2 |
| 7146030 | Vailaya et al. | Dec 2006 | B2 |
| 7146349 | Benitez-Jimenez et al. | Dec 2006 | B2 |
| 7149687 | Gorin et al. | Dec 2006 | B1 |
| 7191119 | Epstein et al. | Mar 2007 | B2 |
| 7249013 | Al-Onaizan et al. | Jul 2007 | B2 |
| 7287026 | Oommen | Oct 2007 | B2 |
| 7292976 | Hakkani-Tur et al. | Nov 2007 | B1 |
| 7328209 | Das et al. | Feb 2008 | B2 |
| 7337106 | Liu | Feb 2008 | B2 |
| 7346490 | Fass et al. | Mar 2008 | B2 |
| 7496621 | Pan et al. | Feb 2009 | B2 |
| 7530020 | Szabo | May 2009 | B2 |
| 7593967 | Harnsberger et al. | Sep 2009 | B2 |
| 7620538 | Marcu et al. | Nov 2009 | B2 |
| 7630879 | Liu | Dec 2009 | B2 |
| 7650283 | Mizutani et al. | Jan 2010 | B2 |
| 7805302 | Chelba et al. | Sep 2010 | B2 |
| 7809548 | Mihalcea et al. | Oct 2010 | B2 |
| 7853444 | Wang et al. | Dec 2010 | B2 |
| 7890539 | Boschee et al. | Feb 2011 | B2 |
| 7987176 | Latzina et al. | Jul 2011 | B2 |
| 20010029455 | Chin et al. | Oct 2001 | A1 |
| 20020035466 | Kodama | Mar 2002 | A1 |
| 20020107827 | Benitez-Jimenez et al. | Aug 2002 | A1 |
| 20030093613 | Sherman | May 2003 | A1 |
| 20030120640 | Ohta et al. | Jun 2003 | A1 |
| 20030189933 | Ozugur et al. | Oct 2003 | A1 |
| 20030195890 | Oommen | Oct 2003 | A1 |
| 20030208499 | Bigwood et al. | Nov 2003 | A1 |
| 20040039734 | Judd et al. | Feb 2004 | A1 |
| 20040049495 | Lee et al. | Mar 2004 | A1 |
| 20040098670 | Carroll | May 2004 | A1 |
| 20040111253 | Luo | Jun 2004 | A1 |
| 20040243531 | Dean | Dec 2004 | A1 |
| 20050039123 | Kuchinsky et al. | Feb 2005 | A1 |
| 20050278325 | Mihalcea et al. | Dec 2005 | A1 |
| 20060015320 | Och | Jan 2006 | A1 |
| 20060015324 | Pan et al. | Jan 2006 | A1 |
| 20060116866 | Suzuki et al. | Jun 2006 | A1 |
| 20060242101 | Akkiraju et al. | Oct 2006 | A1 |
| 20060245641 | Viola et al. | Nov 2006 | A1 |
| 20060253274 | Miller | Nov 2006 | A1 |
| 20060253476 | Roth et al. | Nov 2006 | A1 |
| 20060288023 | Szabo | Dec 2006 | A1 |
| 20070011150 | Frank | Jan 2007 | A1 |
| 20070136246 | Stenchikova et al. | Jun 2007 | A1 |
| 20070233696 | Ishihara et al. | Oct 2007 | A1 |
| 20080040095 | Sinha et al. | Feb 2008 | A1 |
| 20080215309 | Weischedel | Sep 2008 | A1 |
| 20080319947 | Latzina et al. | Dec 2008 | A1 |
| 20090006447 | Balmin et al. | Jan 2009 | A1 |
| 20090024385 | Hirsch | Jan 2009 | A1 |
| 20090063473 | Van Den Berg et al. | Mar 2009 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20110153673 A1 | Jun 2011 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11974022 | Oct 2007 | US |
| Child | 13012225 | US |
