Patent Grant
10282419
The present invention relates to an architecture for natural language processing.
Natural Language Processing (NLP) and Natural Language Understanding (NLU) involve using computer processing to extract meaningful information from natural language inputs such as human generated speech and text. One recent application of such technology is processing speech and/or text queries in mobile devices such as smartphones.
An ideal architecture for NLP/NLU on a mobile device should permit use of modular components so that it is easy to reuse classifiers, grammars, and entire semantic domains. The arrangement should be compatible with the use of hybrid NLP technologies, for example, combining rule-based and statistical approaches. And it also should be possible to achieve a smooth evolution from an initial prototype to the final system that leverages big data sources.
Embodiments of the present invention embrace the inherent ambiguity in natural language with a multi-domain natural language processing architecture that uses multiple parallel domain pipelines to process natural language inputs such as a speech input query from a smartphone or other mobile device. Ambiguity is preserved throughout the pipelines producing rich N-best lists and the results are ranked and disambiguated late in the process when a global decision can be made.
Embodiments of the present invention are directed to an arrangement and corresponding method for multi-domain natural language processing. Multiple parallel domain pipelines are used for processing a natural language input (also referred to as a “query”). Each domain pipeline represents a different specific subject domain of related concepts. Each domain pipeline includes a mention module that processes the natural language input using natural language understanding (NLU) to determine a corresponding list of mentions, and an interpretation generator that receives the list of mentions and produces a rank-ordered domain output set of sentence-level interpretation candidates. A global evidence ranker receives the domain output sets from the domain pipelines and produces an overall rank-ordered final output set of sentence-level interpretations.
The arrangement may operate in real time to process the natural language input with minimum response latency. The mention module may use both rule-based NLU and statistical-based NLU to determine the corresponding list of mentions. The mention module may also associate additional semantic attachments related to the mentions and include the semantic attachments with the list of mentions to the interpretation generator. The global evidence ranker may also include mentions and semantic attachments with the final output set of sentence-level interpretations. And there may be a confidence value for each mention and/or each interpretation candidate.
The interpretation generator also may determine a query intent representing a semantic meaning of the natural language input to produce the interpretation candidates. The interpretation generator also may extract relationships between mentions to produce the interpretation candidates. And the interpretation generator may perform a semantic re-ranking of the interpretations candidates within the domain output set.
Rule-based mention detection is particularly well suited for handling a core high-frequency use cases. Mention detection using robust parsing grammars can be useful for various reasons including without limitation:
Grammars are naturally somewhat context-free, though some context can be implemented to identify individual mentions. That is, simpler context-free grammars, in effect, are simply big lists of synonyms. More complex contextual grammars can make peripheral “dips” into other grammars. Specific embodiments may actually use many (e.g., hundreds) of individual robust parsing grammars for each mention type, which may including some overlap so that some specific mention types may be parsed by more than one grammar Use of so many distinct grammars may cost significant loading time (though caching helps) and multiple dips into the same grammar for context can be costly and wasteful (and again, caching may help).
Rule-based mention detection with ambiguous terms and free-form text may also be susceptible to over-generation, producing a large number of ambiguous parses that are all equivalent. If training data is available, weighted grammar rules and/or confidence scoring can be used to avoid this issue.
Statistical mention detection may use various specific approaches, for example, based on named entity recognition using a maximum entropy tagger, which can also provide confidence scoring of the detected mentions. Statistical mention detection produces a short easy-to-rank N-best list. Rule-based grammars produce more ambiguity for a longer, richer N-best list, but the ranking gets harder as a result. There may be various useful features built-in—e.g., lexical, syntactic, dictionary, etc.—and external streams of features can be available. In a hybrid system, a full set of grammars can act as feature extractors for statistical mention detection. Statistical mention detection using a probabilistic approach provides a better use of context, which can be helpful for processing free form text. In addition, the generated set of mentions is more likely to be internally consistent, and (unlike grammars) very large dictionaries can be efficiently leveraged.
Mention semantics module 302 discovers additional semantic information (“semantic attachments”) for specific mentions. For example, a semantic attachment for a date could be “2012 Dec. 24” while the surface form of the mention might be “December 24”, “Christmas Eve”, “day before Christmas”, etc. This can be done using rules, for example, Java rules for deriving semantic attachments from mention entity types. In addition or alternatively semantic attachments can be identified using other approaches such as classifiers, information retrieval (e.g., Apache Solr), etc.
In some specific embodiments, the functionality of the mention detection module 301 and the mention semantics module 302 may be combined in and performed by a single mention module, for example, using a Robust Parser Grammar Mention Annotation (RP-GMA) approach. And the specific order in which the mention processing is performed may vary from one embodiment to another. In some embodiments, mention processing may be implemented across multiple different domains where the detected mentions are dumped into a mention pool which is shared by the different domains.
Once a list of mentions has been produced by the mention processing block(s), an interpretation generator 303 is a domain expert interpreter that extracts relationships between mentions and blends the mentions together to form a rank-ordered domain output set of sentence-level interpretation candidates representing the full input utterance. Specific architectural details of the interpretation generator 303 can be chosen to allow independent and individual work in different specific domains so that structural details and system improvements can scale. Adding a new domain need not be risky and the specific arrangement should support performance of parallel computations without introducing accuracy risks. Specific arrangements can profit from mentions that come from multiple different sources such as active application, dialog state, user-specific content, time of day, etc. The interpretation generator 303 may be agnostic of the source of the mentions and it can use default weights or it may be trained using field data.
For example, one specific domain interpretation generator 303 can be defined and invoked by software pseudocode such as:
For each defined child interpretation generator, a root interpretation generator inspects the available mentions and where there exists at least one sufficient mention, the root interpretation generator calls the child interpretation generator with the list of acceptable mentions that are available. At the end of this loop, a global interpretation generator rank can be added based on each query intent score from all domain pipelines.
For each defined parallel domain pipeline, the interpretation generator 303 gathers the admissible mentions from the full list of mentions from the previous stages. A finite state machine (FSM) can be built and trained with bare words and mentions. The constructed FSM can be traversed using a best-first, breadth-first or depth-first search to build semantic annotations which are collected and sent to a query intent classifier that sorts them by language model score.
The 0.693 cost come from the costs applied by a robust parsing grammar as compiled from various experimental data. These weights may not be perfectly suited for a given use-case, but they may serve as a reasonable starting point.
A “longest first” approach may be used: on each node, the arcs are sorted by distance and the “Concept” or “Filler” cost is inserted.
This shows the impact of ambiguity: “from/to Location” can also be understood as just a location or a location and a state. The cumulative cost accumulated for each interpretation as well as the use of semantic templates can help discriminate which choices are better.
The interpretation generator 303 can traverse the FSM using a best-first, breadth-first or depth-first search to build semantic annotations. However, a depth-first algorithm can be wasteful of time, even with pruning enabled. Once all the annotations are collected, the interpretation generator 303 sends these on to the next stage.
The described embodiment of the interpretation generator 303 provides flexibility in blending mentions from various sources to benefit from the stability of grammars and the deduction power of statistical engines. And different algorithms and/or configurations can be used based on the specific subject domain. The described embodiment of the interpretation generator 303 also is robust and works well with mention detections sources that may have good recall but poor precision. Embodiments of the interpretation generator 303 may also be able to filter mentions and skip domains based on quality factors.
Unless a real language model is implemented, the interpretation generator 303 will treat all mentions with equal weight, and it may be difficult to add guessing rules. To distinguish between reasonable and unreasonable mentions it may be useful to rely on a database of templates. The issue of over-generation may also need to addressed, and a lack of data may impair pruning ability. If enough data is available, then it may be viable to use a true language model to direct the search and/or possibly train a probabilistic parser.
The query intent classifier 304 can leverage expert knowledge and training data to determine a query intent representing a semantic meaning of the natural language input to produce the sentence-level interpretation candidates that represent overall intent of the input query. Ideally, any specific design should be able to become operational before a significant data stream of training data is ready, and then seamlessly integrate field data when it becomes available. This suggests use of a fuzzy match approach that does not depend on perfect coverage in the training data.
The query intent classifier 304 can be implemented using a mixture of rule-based, information retrieval and/or classifier algorithms. For example, a Lucene-based k-nn algorithm can be used where the mentions form semantic templates which are used as the Lucene vector instances, where each word is one coordinate of a Lucene term. The query intent classifier 304 performs a search with terms from the query annotation. The distance function may be a sort of tf*idf-based distance, and scores can normalized to 1 using the perfect match score.
For example, for the input query “play boston”, the following annotations are a likely result of the query intent classifier 304:
1[IntentPlay] play [/IntentPlay] [Location] boston [/Location]
Semantic templates are reduced annotations where the mention markers and the text contained in-between are replaced by the mention name. In addition, at some point, relations are found between two mentions. For example, in “add Bob to the meeting”, the mention of the intention to “add” relates to the mention of a person (“bob”). The relationship is different for “add a meeting with Bob”.
The query intent classifier 304 can be bootstrapped with a few hand-written templates, making it very predictable, but it can also be rather sensitive to any annotation errors. It can be difficult to add guessing rules. In specific embodiments, the classification of the query intent classifier 304 may be decoupled from the earlier mention processing passes. The output of the query intent classifier 304 is the same as from the interpretation generator 303 but with the added query's intent. It may be possible to implement the query intent classifier 304 without significant training data. The manual addition of a template can readily fix certain coverage bugs, and the manual templates can seamlessly merge with field data as it becomes available.
There may occasionally be collisions with the same template and two intents. Some embodiments may also lack a notion of frequency, even when data is available. In addition, the template database can tend to become rather unwieldy, especially with complex domains. It may be possible to use reduced templates successfully, but granularity can be lost.
As an interpretation is built in a particular domain pipeline 202, evidence is accumulated that supports or refutes each hypothesis, and a ranking algorithm can be trained based on this evidence. Thus, before exiting the domain pipeline 202, a domain evidence ranker 305 performs a semantic re-ranking of the interpretation candidates within the domain output set. For example, the domain evidence ranker 305 may typically (but not necessarily) be a classifier trained on data representative of the task. The domain evidence ranker 305 uses some knowledge about the list of candidate interpretations to improve discrimination. If designed properly, the features being fed into this classifier may be language and application independent. Rather than trying to figure out absolutely how good a hypothesis is, the domain evidence ranker 305 just needs to know if it's better than some other item in the list of candidate interpretations, which is a relative decision about the hypothesis and easier to do. Here the incoming candidate interpretations come from multiple diverse sources and the domain evidence ranker 305 needs to combine concepts from grammars, statistical methods, DB searches, etc.
Thus the domain evidence ranker 305 could rank the list of interpretation candidates in various different specific ways including concept coverage, likelihood, and/or contextual information about the semantic entities (e.g., a current movie may be more likely than an out-of-print book). Any existing ambiguity will be preserved, the domain evidence ranker 305 just ranks it. A specific embodiment should be trainable, robust and sustainable, and may or may not be application dependent. This can be achieved by a simple implementation without rules that just ranks the candidate interpretations using a trained classifier. Or a more complex implementation may be chosen that includes evidence weighting heuristics, etc.
For example, the domain evidence ranker 305 may use a specific ranking method based on a logistic regression approach or a pairwise ranking approach. In a specific example of a pairwise approach referred to as SVMrank, for each hypothesis hi, the domain evidence ranker 305 extracts a feature vector f(hi). For each pair f(hi), f(hj)
SVMrank then predicts which is better. This approach uses knowledge about the interpretation candidate list, but it is quadratic on the number of candidates, which affects efficiency for both training and testing. The output of the domain evidence ranker 305 is a re-sorted list of sentence-level query interpretations.
Some embodiments may implement the domain evidence ranker 305 using an approach based on conditional random fields. That is, a log-linear model can be characterized:
conditioned on the hypotheses in the N-best:
Z(x,
where
Embodiments of the present invention also may include a confidence value for any or all of the various processing parts including without limitation mentions, semantic attachments, query intent, and/or interpretation candidates. A confidence model can be trained to predict for each independent hypothesis how likely it is to be relevant to determining the semantic interpretation of the natural language input. For example, a confidence model may be based on a logistic regression approach using data features that represent supporting or refuting evidence in order to rank each hypothesis on a relevance probability. A language independent feature may be used to determine a confidence value via a heuristic, a classifier score, or some other means. This may be only a matter of using the output score of the single algorithm that provides the data feature. For example, the confidence of interpretation candidates may be a score from the domain evidence ranker. Or a separate statistical confidence engine can be run using a regression model on the ranked output. The determined confidence values can be included with the outputs of each stage, for example, with the mentions in the N-best list output of the mention processing block.
Overall the individual semantic concept domain pipelines 202 blend mentions from various different sources that combine the stability of grammars with the deduction power of statistical engines. This approach allows for different algorithms and configurations based on the specific subject domain, thereby obtaining desirable robustness working well with mention sources that may have good recall but poor precision. Specific arrangements can filter mentions and/or skip specific concept domains based on quality.
A global evidence ranker 203 receives the domain output sets from the individual domain pipelines 202 and produces an overall rank-ordered final output set of sentence-level interpretations. Thus, the global evidence ranker 203 acts to arbitrate between many different domain experts by ranking their various interpretations. As each particular domain pipeline 202 builds its interpretation candidates, evidence is gathered and accumulated that supports or refutes each hypothesis. A ranking algorithm can be trained based on this evidence. A global rank can be assigned, for example, based on the score from the query intent classifier 304 within each domain pipeline 202. The global evidence ranker 203 may also include mentions and semantic attachments with the final output set of sentence-level interpretations.
Because of the modular nature of the system architecture, whenever more data becomes available or a new statistical model is added, only the mention model needs re-training. Optionally (for optimal performance), the domain re-ranker also might be re-trained. Other system components should be stable. The use of parallel domain pipelines supports many different domain expert interpreters, which helps generate useful competing interpretations. For example, given the natural language input “buy harry potter,” does the user want to buy the book or the dvd? Domains such as “Movie” and “Books” would return good scoring interpretations and make a richer N-Best list of interpretation candidates.
Many individual models and grammars may be language- and application-dependent, but the overall system architecture and some models (such as confidence models, ranking models, etc.) will be independent of the language and application. The system architecture leverages classifiers and information retrieval techniques that use abstract features as much as possible so that, whenever possible, features that are independent of the words themselves are used. That allows the classifiers (and other algorithms) to be used across different languages and different applications. Moreover, the parallel pipeline approach allows easy removal and/or addition of specific domain experts as desired to allow easy plug-and-play of different subject domains. The entire system may operate in real time to process the natural language input with minimum response latency.
Embodiments of the invention may be implemented in whole or in part in any conventional computer programming language such as VHDL, SystemC, Verilog, ASM, etc. Alternative embodiments of the invention may be implemented as pre-programmed hardware elements, other related components, or as a combination of hardware and software components.
Embodiments can be implemented in whole or in part as a computer program product for use with a computer system. Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium. The medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., microwave, infrared or other transmission techniques). The series of computer instructions embodies all or part of the functionality previously described herein with respect to the system. Those skilled in the art should appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies. It is expected that such a computer program product may be distributed as a removable medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web). Of course, some embodiments of the invention may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments of the invention are implemented as entirely hardware, or entirely software (e.g., a computer program product).
Although various exemplary embodiments of the invention have been disclosed, it should be apparent to those skilled in the art that various changes and modifications can be made which will achieve some of the advantages of the invention without departing from the true scope of the invention.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6278967 | Akers et al. | Aug 2001 | B1 |
| 6278968 | Franz et al. | Aug 2001 | B1 |
| 6760695 | Kuno et al. | Jul 2004 | B1 |
| 7085708 | Manson | Aug 2006 | B2 |
| 7363398 | Scott | Apr 2008 | B2 |
| 7464026 | Calcagno et al. | Dec 2008 | B2 |
| 7716056 | Weng et al. | May 2010 | B2 |
| 8175864 | Dubiner | May 2012 | B1 |
| 8185397 | Lee | May 2012 | B2 |
| 8731929 | Kennewick | May 2014 | B2 |
| 20040019588 | Doganata et al. | Jan 2004 | A1 |
| 20040111255 | Huerta | Jun 2004 | A1 |
| 20050278287 | Bao et al. | Dec 2005 | A1 |
| 20060004703 | Spivack | Jan 2006 | A1 |
| 20060074670 | Weng | Apr 2006 | A1 |
| 20060074900 | Nanavati | Apr 2006 | A1 |
| 20060080101 | Chotimongkol | Apr 2006 | A1 |
| 20060229870 | Kobal | Oct 2006 | A1 |
| 20070011434 | Luick | Jan 2007 | A1 |
| 20070100624 | Weng | May 2007 | A1 |
| 20080033938 | Okamoto et al. | Feb 2008 | A1 |
| 20080195601 | Ntoulas | Aug 2008 | A1 |
| 20080306959 | Spivack et al. | Dec 2008 | A1 |
| 20090030697 | Cerra et al. | Jan 2009 | A1 |
| 20090112892 | Cardie | Apr 2009 | A1 |
| 20090150853 | Larvet | Jun 2009 | A1 |
| 20090210244 | Koister | Aug 2009 | A1 |
| 20090222437 | Niu | Sep 2009 | A1 |
| 20090319498 | Zabokritski et al. | Dec 2009 | A1 |
| 20100030552 | Chen | Feb 2010 | A1 |
| 20100076995 | Pan et al. | Mar 2010 | A1 |
| 20100094870 | Narang | Apr 2010 | A1 |
| 20110087673 | Chen et al. | Apr 2011 | A1 |
| 20110258188 | AbdAlmageed | Oct 2011 | A1 |
| 20110301935 | Quirk | Dec 2011 | A1 |
| 20110307252 | Ju | Dec 2011 | A1 |
| 20120016678 | Gruber et al. | Jan 2012 | A1 |
| 20120084086 | Gilbert | Apr 2012 | A1 |
| 20120265528 | Gruber | Oct 2012 | A1 |
| 20130159000 | Ju | Jun 2013 | A1 |
| 20130268522 | Barrett | Oct 2013 | A1 |
| 20130304454 | Kimberly | Nov 2013 | A1 |
| 20130332450 | Castelli et al. | Dec 2013 | A1 |
| 20140006010 | Nor | Jan 2014 | A1 |
| 20140072948 | Boguraev | Mar 2014 | A1 |
| 20140095147 | Hebert | Apr 2014 | A1 |
| 20140101119 | Li | Apr 2014 | A1 |
| 20140108424 | Casella dos Santos | Apr 2014 | A1 |
| 20140108460 | Casella dos Santos | Apr 2014 | A1 |
| 20140165002 | Grove | Jun 2014 | A1 |
| 20140297268 | Govrin | Oct 2014 | A1 |
| 20150161124 | Keshava | Jun 2015 | A1 |
| Entry |
|---|
| Final Office Action received in corresponding U.S. Appl. No. 13/632,380 dated Jun. 9, 2014. |
| “Google Search history Expands, Becomes Web History”; http://searchengineland.com/google-search-history-expands-becomes-web-history-11016. |
| Sep. 2, 2014—U.S. Non-Final Office Action—U.S. Appl. No. 13/632,380. |
| Jun. 24, 2016—U.S. Non-Final Office Action—U.S. Appl. No. 13/632,380. |
| Nov. 18, 2016—U.S. Final Office Action—U.S. Appl. No. 13/632,380. |
| Dec. 5, 2016—U.S. Notice of Allowance and Fee(s) Due—U.S. Appl. No. 13/632,380. |
| Sullivan, Danny, “Google Search History Expands, Becomes Web History,” http://searchengineland.com/google-search-history-expands-becomes-web-history-11016, Apr. 19, 2007. |
| Number | Date | Country | |
|---|---|---|---|
| 20140163959 A1 | Jun 2014 | US |
