Information extraction (IE) problems are becoming increasingly important due to an increasing amount of data to process, such as in sources like the World Wide Web. Information extraction is the process of automatically extracting structured information from semi-structured or unstructured data. An example of unstructured data is natural language text found in a computer-readable document.
Some examples are described with respect to the following figures:
Before particular examples of the present disclosure are disclosed and described, it is to be understood that this disclosure is not limited to the particular examples disclosed herein as such may vary to some degree. It is also to be understood that the terminology used herein is used for the purpose of describing particular examples only and is not intended to be limiting, as the scope of the present disclosure will be defined only by the appended claims and equivalents thereof.
Notwithstanding the foregoing, the following terminology is understood to mean the following when recited by the specification or the claims. The singular forms ‘a,’‘an,’ and ‘the’ are intended to mean ‘one or more.’ For example, ‘a part’ includes reference to one or more of such a ‘part.’ Further, the terms ‘including’ and ‘having’ are intended to have the same meaning as the term ‘comprising’ has in patent law. The term ‘approximately’ when used in reference to a calculation or determination means that the calculation or determination provides an inexact solution, in contrast to a closed analytic solution, for example.
Many high-level information extraction problems include multiple “subtasks”, which are tasks to complete during information extraction. The subtasks may be interdependent on each other. For example, two such subtasks are (1) segmentation, which may involve identifying segments in observed data, and (2) relation discovery, which may involve discovering certain relations between the segments. Each segment may be labeled with a segment type, such as person, location, organization, date, year, time, number, miscellaneous, or the like. Each relation may be labeled with a relation type, such as employee, father, executive, job title, education, or the like. An example problem is to find segments and relations in observed data such as the natural language text “Barack Obama is a member of the Democratic Party and graduated from Harvard University.”
Accordingly, the present disclosure concerns information extraction systems, computer readable storage media, and methods of information extraction from observed data. In solving the example problem, the methods and systems herein may identify segments such as a segment “Barack Obama” whose segment type is “person”, segment “Democratic Party” whose segment type is “organization”, and segment “Harvard University” whose segment type is “school.” Additionally, the methods and systems herein may identify relations such as a relation “executive” between “Barack Obama” and “Democratic Party”, and a relation “education” between “Barack Obama” and “Harvard University.”
The methods and systems herein may effectively optimize and solve the subtasks jointly and simultaneously by using a joint discriminative probability distribution that incorporates first-order logic formulas. As defined herein, a “joint discriminative probabilistic model” or “joint discriminative probability distribution” is a model to predict two unobserved variables a and b from an observed variable c according to a joint conditional probability distribution such as P(α, b|c)=P(α|c)P(b |α, c). Thus, the model may predict the variables a and b jointly such that they can be optimized simultaneously.
The joint discriminative probability distribution may be used in a top-down and bottom-up bidirectional manner to exploit dependencies and interactions between the subtasks, and may provide flexibility to incorporate both uncertainty of probabilistic graph models which may be effective for segmentation, and first-order logic for domain knowledge concisely formulated by first-order logic formulas which may be effective for relation discovery. Thus, employing first-order logic in a joint discriminative probabilistic model may result in high performance for both segmentation and relation discovery, and may reduce cascading error accumulation. “First order-logic formulas” are symbolized formulas that formalize statements that include a subject and a predicate, and in which the predicate modifies or defines the properties of the subject. In first-order logic, a predicate refers to a single subject, not multiple subjects.
The computer system 210 may include a display controller 220 responsive to instructions to generate a textual or graphical display of any of the observed data, likely outputs, intermediate data, or graphical representations of the methods disclosed herein, on a display device 222 such as a computer monitor, camera display, smartphone display, or the like.
The processor 212 may be in communication with a computer-readable medium 216 via a communication bus 214. The computer-readable medium 216 may include a single medium or multiple media. For example, the computer readable medium may include one or both of a memory of the ASIC, and a separate memory in the computer system 210. The computer readable medium 216 may be any electronic, magnetic, optical, or other physical storage device. For example, the computer-readable storage medium 216 may be, for example, random access memory (RAM), static memory, read only memory, an electrically erasable programmable read-only memory (EEPROM), a hard drive, an optical drive, a storage drive, a CD, a DVD, and the like. The computer-readable medium 216 may be non-transitory. The computer-readable medium 216 may store, encode, or carry computer executable instructions 218 that, when executed by the processor 212, may cause the processor 212 to perform any one or more of the methods or operations disclosed herein according to various examples.
At block 302, a sequence of data X={X1,X2, . . . Xn}, designated by reference numeral 402 in
The information extraction method 300 may be able to solve a number of information extraction problems based on the data X. An example problem is to perform two subtasks, segmentation and relation discovery. Y={R,S} is the set of possible segments S of the data X and possible relations R between possible segments S. The problem may be to find the set Y*={R*,S*} of most likely segments S* of the data X and most likely relations R* between potential segments S, where S* is contained in S (S* εS) and R* is contained in R (R*εR). Thus, Y* may have the maximum a posteriori (MAP) probability of the possible assignments Y given the data X, namely Y*=arg maxYP(Y|X).
As defined herein, “segmentation” is the task assigning one or more most likely segments S* to the data X. For example, a segment S1*may be assigned to token X1, and a segment S2*may be assigned to tokens X2 and X3. Thus, a “segment” is a unit assigned to one or more tokens. In some examples, only adjacent tokens may form a segment. In such examples, a segment cannot be assigned to tokens X1 and X3. Segmentation may be used for word segmentation, chunking, and/or entity recognition, for example. Additionally, as defined herein, “relation discovery” is the task of discovering one or more most likely relations R* between pairs of potential segments S. Relation discovery may be used for entity resolution, relation extraction, and/or social relation mining, for example.
At block 304, an information extraction model may be loaded and provided. The model may be a joint discriminative probability distribution P(Y|X) over multiple variables Y, such as segmentation variables representing possible segments S and relation variables representing possible relations R, conditioned on the observed data X. Thus, the joint discriminative probability distribution P(Y|X) may model a first subtask and a second subtask. The joint discriminative probability distribution P(Y|X) may be represented as a factor graph, The joint discriminative probability distribution may take many forms. An example form is as an exponential family, such as Markov random fields or Markov networks. A “Markov random field” or “Markov network” is understood herein a set of random variables that (1) have a “Markov property”, in that they are variables in “Markov chain”, which is a stochastic process that is memoryless, and (2) are represented as an “undirected graph”, which is a graph having edges with no orientation, i.e. no directionality. In some examples, the joint discriminative probability distribution may be defined as:
Z(X) is a normalization function. The exponential of the joint discriminative probability distribution may be factored into a product of factored exponential families φi=exp{Σkμikƒik (Xi, Yi)}, as shown. Each factored exponential family φi may be a real, scalar value over sufficient statistics ƒik(Xi, Yi), each weighted by a parameter μik, of the subset of variables Yi and Xi that are neighbors of φiin the factor graph G. The neighbors may form “cliques”, which are defined herein to be complete subgraphs in which every pair of distinct vertices of the subgraph is connected by a unique edge. This model can represent a large number of random variables as a family of probability distributions that factorize according to an underlying graph, and it can capture complex dependencies between variables.
The factors of the joint discriminative probability distribution P(Y|X) may be partitioned into two or more factors each representing a particular subtask. For example, the joint discriminative probability distribution P(Y|X) may be factored into a product of: (1) a probability distribution P(S|X) over possible segmentations S, designated by reference numeral 404 in
The “Hammersley-Clifford theorem” states that a probability distribution with a positive density can be factorized over its cliques, if and only if it satisfies a Markov property with respect to an undirected graph. Thus, because as discussed earlier P(Y|X) may satisfy a Markov property, the segmentation and relation factors may be factored over their cliques.
The feature functions gi may be weighted by a first subset λic of the parameter weights λic and θjd, and the first-order logic formulas ƒj may be weighted by a second subset θjd of the parameter weights λic and θjd. “Parameter weights” are weights given to functions in the joint discriminative probability distribution.
Each exponential family exp{93i=1wicgi}corresponds to one candidate segment Sc of all possible segments S of the data X, where Ws is the number of feature functions gi, which may model the first subtask and the first variables, e.g. segmentation variables representing segments S. Each “feature function” gi defines a particular rule that results in segmentation of the data X into the candidate segment Sc. Additionally, to effectively capture properties of segmentation, the feature functions gi may be semi-Markovian to form “semi-Markov chains”, in that each feature function gi may depend on the current segment Sc, the previous segment Sc-1, and the data X, such that gi=gi(Sc,Sc-1, Sc-1, X). The likelihood that the data X are correctly segmented into candidate segment Scbased on a particular feature function gi is represented by a real-valued parameter weight λic. Thus, the total likelihood of segmenting into candidate segment Sc is provided by the set of all parameter weights of a given Sc, namely λc={λi=1,c, λi=2,c, . . . λi=W
The following demonstrates how this model may be applied to the example problem mentioned earlier to find segments in the natural language text “Barack Obama is a member of the Democratic Party and graduated from Harvard University,” In this example, each labeled token may be represented with the letter I along with a segment type, and each non-labeled token may be represented with an O. Thus, the 15 tokens, including 14 words and 1 period, may be sequentially labeled as {I-PERSON,/PERSON,O,O,O,O,O-ORGANIZATION,I-ORGANIZATION,O,O,O,-I-SCHOOL,I-SCHOOL,O}. The correct corresponding sequence of segments may be {<1,2,1-PER>,<3,3,O>,<4,4,O>,<5,5,O>,<6,6,>,<7,7,O>,<8,9,I-ORG>,<10,10,>,<11,11,O>,<12,12,O>,<13,14,I-SCHOOL>,<15,15,O>}, where each segment is represented as <starting position, end position, label>. Two possible feature functions gi for the segment <8,9,I-ORG>may be g(I-ORG,O,X,8,9) and g(I-ORG, I-ORG,X,8,9). In the former, the current 8th token is labeled with I-ORG and the previous 7th token is labeled with O, and in the latter, both the current 9th token and previous 8th tokens are labeled with I-ORG.
Each exponential family exp{Σj=1W
The following demonstrates how this model may be applied to the example problem mentioned earlier to find relations in the natural language text “Barack Obama is a member of the Democratic Party and graduated from Harvard University.” In this example, correct relations may be the relation “executive” between segment “Barack Obama” i.e. <1,2,I-PER> and segment “Democratic Party” i.e. <8,9,I-ORG>, and the relation “education” between segment “Barack Obama” and segment “Harvard University” i.e. <13,14,I-SCHOOL>. One possible first-order logic formulas ƒj may represent the claim that “people attend school”. Thus, this formula may be equal to (1) a high probability value if the segment comprising tokens 8 and 9 is labeled as a person and the segment comprising tokens 13 and 14 is labeled as a school, in which case the relation may be labeled as “education”, or (2) a low probability value if the segment comprising tokens 8 and 9 is not labeled as a person or the segment comprising tokens 13 and 14 is not labeled as a school. If the first-order logic formula ƒj correctly represents a relation between these segments, its parameter weight θjd may be likely to be high. Otherwise, its parameter weight θjd may be likely to be low.
In
At blocks 306 to 312, the parameters weights λic, of each of the first variables and the parameter weights θjd of each of the second variables may be determined. For example, the parameter weights may be estimated approximately by a “variational expectation maximization (VEM) algorithm”, which is an iterative method for finding maximum likelihood or maximum a posteriori (MAP) estimates of variational parameter weights, using V, E, and M steps such as those discussed at blocks 306 to 312. The VEM algorithm may, in some examples, operate in a top-down and bottom-up manner to optimize subtasks, e.g. segmentation and relation discovery, iteratively and collaboratively using hypotheses from each other, such that information may flow bi-directionally between the subtasks to obtain mutual benefits for each of the subtasks. The VEM algorithm may, for example, provide a fast, deterministic approximation, whose convergence time may be independent of dimensionality of the exponential family of P(Y|X). In some examples, the VEM algorithm may operate as follows.
At block 306, in the V step, a variational distribution Q indexed by a set of variational parameters weights, such as variational segmentation parameter weights and variational relation parameter weights, may be generated and provided. “Variational parameters weights” are parameter weights that are varied toward particular values. The variational distribution Q may be an approximation of the target distribution P(Y|X). The variational distribution Q may be selected from a family of variational distributions, such that it may be most feasible and most mathematically tractable to perform inference at block 314 on the selected variational distribution Q relative to other possible variational distributions.
The variational distribution Q may be a naive (i.e. non-structured) variational distribution. A structured variation distribution involves performing exact probability calculations on tractable substructures, combined with variational methods to capture the interactions between substructures, However, in cases where the probability distribution to be calculated is fully factorized, such that the interacting variables are independent and the joint distribution is a product of single variable marginal probabilities, a nave non-structured variational distribution may be used.
At blocks 308 to 312, an expectation maximization (EM) based optimization algorithm may be applied to iteratively update the variational parameter weights such that the values of the variational parameter weights may converge toward the values of the parameter weights λic and θjd.
At block 308, in the E-step, the variational segmentation parameter weights of the variational distribution Q may be held fixed while bottom-up learning may be performed, using the hypotheses from segmentations, to converge the variational relation parameter weights of the variational distribution Q toward the values of the relation parameter weights θjd.
At block 310, in the M-step, the variational relation parameter weights may be held fixed while top-down learning may be performed, using the hypotheses from relation discovery, to converge the variational segmentation parameter weights toward the values of the segmentation parameter weights λic.
The variational parameters may converge to an equilibrium, such that the Kullback-Leibler (KL) divergence between the variational distribution Q and the target distribution P(Y|X) may reach a stable minimum, which may be an optimal solution according to nave mean-field variational theory. Such iterative optimization allows information to flow bi-directionally to boost both the segmentation and relation discovery performance. Thus, at equilibrium, the values of the parameter weights λic and θjd may be estimated to be equal to the values of the equilibrium variational parameter weights.
At decision block 312, whether equilibrium values of the variational parameter weights have been reached may be determined. If equilibrium is reached, the method may proceed to block 314. If equilibrium is not reached, the method 300 may proceed back to block 308. In some examples, the decision may be made based on whether a threshold number of iterations of blocks 308 and 310 has been reached, rather than based on whether equilibrium is reached.
At block 314, a first likely output of the first variables and a second likely output of the second variables may be determined based on the parameter weights λic, and θjd, where the first likely output associated with the first subtask and the second likely output associated with the second subtask. For example, inference may be performed to find Y′=arg maxY P(Y|X), which may be the maximum a posteriori (MAP) probability of the possible assignments Y given the data X. Inference may be performed approximately, given the large data set of possible segments S and possible relations R. For example, inference may be performed by a bidirectional Markov chain Monte Carlo (MCMC) algorithm to find the maximum a posteriori (MAP) assignment Y*, which represents likely segments S* and likely relations R*, as discussed earlier. An “MCMC algorithm” is understood herein to sample the probability distribution P(Y|X) by generating a Markov chain having the probability distribution P(Y|X) as its equilibrium distribution after a number of steps in the Markov chain. In some examples, the MCMC algorithm may be guaranteed to converge to the equilibrium distribution. In some examples, a Metropolis-Hastings (MH) algorithm, which is a type of MCMC algorithm, may be used. An “MH algorithm”, in addition to the general properties of MCMC algorithms, is understood herein to sample the probability distribution P(Y|X) indirectly, for example by generating a histogram or integral that approximates the probability distribution P(Y|X). The MCMC algorithms above may sample from both semi-Markov chains of the segmentation factor ΠcεC
By modeling the segments S and relations R simultaneously, the methods herein may provide natural ways to perform joint information extraction, and may reduce error propagation. As the segments from the semi-Markov chains of the segmentation factor may be dynamically changed, the relation factor may correspondingly change based on the changed segmentations. Likewise, changed relation factor may influence segmentation. Thus, the model captures bidirectional top-down and bottom-up dependencies between multiple subtasks for joint information extraction problems.
In one example, experiments were performed on segmentation and relation discovery from 1,127 paragraphs from 441 pages of English encyclopedic articles in Wikipedia. For reference, initially information was extracted manually from the data. This yielded 7,740 entities labeled into 8 categories, including 1,243 person, 1,085 location, 875 organization, 641 date, 1,495 year, 38 time, 59 number, and 2,304 miscellaneous names. This data also contained 4,701 relation instances and 53 labeled relation types. To compare the manual performance to the performance of the tested example method, a standard measure was used of Precision (P), Recall (R), and F-measure, which is the harmonic mean of P and R, namely (2PR)/(P+R), for both segmentation and relation discovery. The token-wise labeling accuracy was also determined. The test example method achieved high performance. For segmentation, the test example method achieved an accuracy of 97.55, a precision of 94.03, a recall of 93.89, and an F-measure of 93.96. For relation discovery, the test example method achieved an accuracy of 96.92, a precision of 72.89, a recall of 64.20, and an F-measure of 68.27. It should be noted that these results are applicable to only one example of the methods herein.
Thus, there have been described examples of information extraction systems, computer readable storage media, and methods of information extraction. In the foregoing description, numerous details are set forth to provide an understanding of the subject disclosed herein. However, examples may be practiced without some or all of these details. Other examples may include modifications and variations from the details discussed above. It is intended that the appended claims cover such modifications and variations.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/CN2013/083415 | 9/12/2013 | WO | 00 |