Patent Application
20200342178
This invention relates to the field of natural language processing (NLP), information extraction, terminology processing, and human machine interaction (HMI).
Knowledge bases (KBs) are among the most important pillars of modern intelligent systems. KBs are both provided as standalone products or services, and enable downstream applications such as question answering and semantic search: e.g. to answer questions about “direct injection” by automobile professionals, many systems rely on pre-constructed knowledge bases about automobile terminologies. An important first step in such systems is to link the natural language phrase “direct injection” to a node in KB. This is called Entity Linking (EL).
Most existing work on EL focuses on named entities, i.e. person, organization, and location names. But for domain specific knowledge-based applications in the industrial setting, for instance, systems that recommend power tools to DIY users, or systems that support car repairing activities, requires that domain-specific terminologies are linked to KB nodes as well. This invention disclosure presents an EL system with terminology linking capability to support various industrial use cases.
Accurately linking phrases to KB nodes is a difficult task. First, the phrase may not directly correspond to the name of a KB node. For example, the phrase “Hybrid” in documents about repairing cars should probably be linked to “Hybrid_vehicle”, but the two surface forms are not identical. In addition, phrases can be ambiguous: the word “fatigue” can either mean “Fatigue_(medical)” in a medical document or mean “Fatigue_(metal)” in a material science document.
Occurrence and co-occurrence statistics are collected from a large corpus where phrases are manually linked to KB nodes (Wikipedia in practice) to solve these problems. Occurrence statistics to measure how likely a mention is linked to a node, i.e., if “Hybrid” is frequently linked to “Hybrid_vehicle” somewhere in the corpus, the system will be able to propose “Hybrid_vehicle” when it encounters “Hybrid” in unseen text. Co-occurrence statistics measures how likely two entities appear together in the same document, so that the system will be able to determine that “fatigue” probably means “Fatigue_(metal)”, if there is another entity “Cast_iron” in the document based on the association between “fatigue” and “Cast_iron.”
One challenge that makes terminology linking especially hard is that many terminologies in technical documents are emerging or highly-specific concepts that are not yet covered by KBs. As a result, co-occurrence statistics do not exist for these terminologies.
For the purposes of promoting an understanding of the principles of the disclosure, reference will now be made to the embodiments illustrated in the drawings and described in the following written specification. It is understood that no limitation to the scope of the disclosure is thereby intended. It is further understood that the present disclosure includes any alterations and modifications to the illustrated embodiments and includes further applications of the principles of the disclosure as would normally occur to a person of ordinary skill in the art to which this disclosure pertains.
This disclosure is directed to a computing system and methodology for linking terminologies in text media such as academic papers, technical reports, and user-generated contents, to knowledge base (KB) nodes, e.g. Wikipedia pages. In contrast, most systems in prior-art focus on named entities (person, organization, and location names) only.
As used herein, the term “knowledge base” refers to a human annotated text collection having nodes which are associated with entities, such as people, places or things, and terminologies, which refer to the lexicon used in association with a particular field, practice area, or technology. The nodes of the knowledge base may include links to related or referenced nodes. As used herein, the term “annotated” means that entities and terminologies in the text collection are already linked to nodes in the knowledge base. Wikipedia is an example of an annotated knowledge base from which occurrence and co-occurrence statistics may be gathered for the system.
As discussed below, the system first preprocesses a full text KB (i.e. a KB whose nodes have natural language descriptions), and generates occurrence and co-occurrence statistics of KB nodes based on the description text for the nodes. Candidate nodes are generated for new terminologies that are to be linked to the KB. The candidate nodes are identified based on the occurrence statistics. The candidate nodes in the same document are then scored through collective inference, using both occurrence and co-occurrence statistics among these nodes.
For terminologies that have no candidates associated therewith, our system automatically constructs a list of pseudo-candidate nodes which are semantically-related to the terminology. This is a major departure from prior-art. The pseudo-candidate nodes are near-synonyms for the terminology. The co-occurrence statistics of the near-synonyms are used to facilitate joint inference. The terminologies are then linked to the highest scored nodes or pseudo-nodes for the terminology. This system is useful for itself in digital library applications and can be used to facilitate KB construction and semantic search.
The combination of collective inference using occurrence and co-occurrence statistics and the pseudo-link generator, the proposed system achieves state-of-the-art performance on terminology linking. The system can be used to create digital library applications, such as car repair tutorials and technical document explorers, which offer inline links to KBs. It can also be used as pre-processor for other knowledge-based applications.
Referring to
The KB statistics counter 12 calculates two types of statistics, occurrence statistics and co-occurrence statistics based on a human annotated text collection 20. By being annotated, the entities and terminologies in the text collection 20 are already linked as appropriate. Wikipedia is an example of an annotated collection upon for the purposes of this system.
Occurrence statistics measure how likely a mention (m) is linked to a KB node (e). Formally, it is the estimation of the conditional probability p(e|m). Co-occurrence statistics measure how likely two KB entities or terminologies appear in the same document. Formally, it is the estimation of the joint probability p(m1, m2). Both occurrence and co-occurrence statistics can be calculated using the raw count of occurrences of entities and terminology mentions in the annotated text collection. The KB statistics counter 12 may store occurrence statistics and co-occurrence statistics in an occurrence database 22.
The candidate generator 14 accepts a document 24 and a list of terminologies in the document 24 as input (
In one embodiment, the list of terminologies from the document is generated beforehand by an external party or system. Alternatively, the list of terminologies may be generated as part of the functionality of the system. For example, the system may include a terminology identification component which is configured to receive a document or other form of text data and to parse the text data to create the list of terminologies in the document. Any suitable method or algorithm may be used to identify terminologies within the document.
The candidate generator 14 processes the list of terminologies with respect to the KB nodes in the knowledge base to produce a list of candidate nodes 26 for each terminology. Candidate nodes 26 are nodes in the KB to which terminologies in the document can potentially link. Each terminology in the document can only be correctly linked to one KB node, but can have more than one candidate in the candidate generation phase.
Given a terminology (m) and a probability distribution p(e|m) generated by the KB statistics counter 12, the candidate generator 14 returns the KB nodes whose corresponding p(e|m) are not zero. In other words, the candidate generator 14 returns all KB nodes that are linked at least once to the terminology (m) in the annotated text collection 20.
One of the major challenges facing terminology linking is that many terminologies are emerging or highly domain-specific concepts that have no corresponding node in the KB. As a result, the candidate generator 14 is not able to produce candidates for these terminologies and these terminologies have no statistics for the scorer to utilized. These terminologies are referred to herein as NIL terms because the candidate generator returns a NIL value when identifying candidate nodes for the terminology, as well as candidate-less terminologies for obvious reasons.
The proposed system incorporates a pseudo-candidates generator 16 to capture the information behind the NIL terms. The output of the pseudo-candidates generator 16 is a list of pseudo-candidate nodes 28. Pseudo-candidate nodes 28 are KB nodes that are not identical to the NIL term but are semantically related to the NIL term.
These pseudo-candidates can be generated in various ways, depending at least in part on the nature of the application. For example, the pseudo-candidate generator may be configured to identify knowledge base nodes that share words with the NIL terms as pseudo-candidate nodes for the NIL terms. As another example, the pseudo-candidate generator may be configured to identify knowledge base nodes that are distributionally similar to the NIL term as pseudo-candidates for the candidate-less terminology. Distributional similarity between the NIL terms and the nodes of the knowledge base may be determined using vector representations of the knowledge base nodes and the unlinked terminology according to a model, such as a Word2vec representation model.
Once the candidates 26, the pseudo-candidates 28 and their corresponding occurrence and co-occurrence statistics 22 have been identified, the candidate scorer 18 performs collective inference on the document to score the candidates and pseudo-candidates using a scoring algorithm, such as the Loopy Belief Propagation (LBP) algorithm. Other suitable algorithms may be used.
After being scored, each non-NIL terminology is linked to the KB node corresponding to the highest-scored candidate identified for that terminology. Similarly, NIL terms may be linked with the highest scored pseudo-candidate. Alternatively, further processing may be performed to determine whether pseudo-candidate scores are high enough to merit association with the terminology before NIL-terms are linked to pseudo-candidate nodes.
The system may include an output generating system for generating output based on the terminology links to the KB. For example, technical documents may be generated with terms linked to associated nodes in the KB (
The components of the system may be implemented by one or more computing devices having one or more processors, such as a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) device, or a micro-controller. The processor is configured to execute programmed instructions that are stored in the memory. The memory can be any suitable type of memory, including solid state memory, magnetic memory, or optical memory, just to name a few, and can be implemented in a single device or distributed across multiple devices. The programmed instructions stored in memory include instructions for implementing various functionalities in the system, including identifying candidates and candidate nodes for terminologies and using collective inference based on occurrence and co-occurrence statistics to score the candidates. The computing device(s) may include network interface device(s) for accessing and communicating via a network.
The combination of collective inference using occurrence and co-occurrence statistics and the pseudo-link generator, the proposed system achieves state-of-the-art performance on terminology linking. The system can be used to create digital library applications, such as car repair tutorials and technical document explorers, which offer inline links to KBs. It can also be used as pre-processor for other knowledge-based applications.
While the disclosure has been illustrated and described in detail in the drawings and foregoing description, the same should be considered as illustrative and not restrictive in character. It is understood that only the preferred embodiments have been presented and that all changes, modifications and further applications that come within the spirit of the disclosure are desired to be protected.
This application claims priority to U.S. Provisional Application Ser. No. 62/611,575 entitled “SYSTEM AND METHOD FOR DOMAIN-INDEPENDENT TERMINOLOGY LINKING” by Ding et al., filed Dec. 29, 2017, the disclosure of which is hereby incorporated herein by reference in its entirety.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2018/067617 | 12/27/2018 | WO | 00 |
| Number | Date | Country | |
|---|---|---|---|
| 62611575 | Dec 2017 | US |
