Patent Grant
8346534
The present invention relates generally to the field of data processing and, more particularly, to a method, computer program, apparatus, and system for automatic keyword extraction.
Books represent one of the oldest forms of written communication and have been used since thousands of years ago as a means to store and transmit information. Despite this fact and given that a large fraction of the electronic documents available online and elsewhere consist of short texts such as Web pages, news articles, scientific reports, and others, the focus of natural language processing techniques to date has been on the automation of methods targeting short documents. A change however can be seen: more and more books are becoming available in electronic format, in projects such as the Million Books project, the Gutenberg project, or Google Book Search. Similarly, a large number of the books published in recent years are often available—for purchase or through libraries—in electronic format. This means that the need for language processing techniques able to handle very large documents such as books is becoming increasingly important.
A back-of-the-book index typically consists of the most important keywords addressed in a book, with pointers to the relevant pages inside the book. The construction of such indexes is one of the few tasks related to publishing that still requires extensive human labor. Although there is a certain degree of computer assistance, consisting of tools that help the professional indexer to organize and edit the index, there are no methods that would allow for complete or nearly-complete automation of the task.
In addition to helping professional indexers in their task, an automatically generated back-of-the-book index can also be useful for the automatic storage and retrieval of a document; as a quick reference to the content of a book for potential readers, researchers, or students; or as a starting point for generating ontologies tailored to the content of the book.
Keywords are not only used as entries in back-of-the-book indexes, but can be used to give a concise, high-level description of a document's contents that can help to determine a document's relevance, or as a low-cost measure of similarity between documents. They are also used in a topic search, in which a keyword is entered into a search engine and all documents with this particular keyword attached are returned to a user. It can be seen that improved keyword extraction methods have a wide range of applications for short documents as well as in back-of-the-book index generation for large documents.
Unfortunately, only a small fraction of documents have keywords assigned to them, and manually attaching keywords to existing documents is a very laborious task. Therefore, automation of this process using artificial intelligence, for example, machine learning techniques, is of interest. In implementing keyword extraction, any phrase in a new document can be identified—extracted—as a keyword. Then, machine learning or another computational technique is used to determine properties that distinguish candidate words that are keywords from those that are not.
The state-of-the-art in keyword extraction is currently represented by supervised learning methods, where a system is trained to recognize keywords in a text based on lexical and syntactic features. This approach was first suggested in Turney, 1999; and U.S. Pat. No. 6,470,307, where parameterized heuristic rules are combined with a special-purpose genetic algorithm into a system for keyword extraction (GenEx) that automatically identifies keywords in a document. Training GenEx on a new collection is computationally very expensive. A different learning algorithm was used in Kea [Frank et al., 1999]. Very briefly, Kea is a supervised system that uses a Naïve Bayes learning algorithm and several features, including information theoretic features such as tf.idf and positional features reflecting the position of the words with respect to the beginning of the text. Training Kea is much quicker than for training GenEx. Finally, in recent work, [Hulth, 2003] a system for keyword extraction from abstracts has been proposed that uses supervised learning with lexical and syntactic features, which were shown to improve keyword extraction significantly over previously published results.
A related task that requires keyword extraction is that of annotating a document with links to sources of additional information. An example of a collection of such documents is found in Wikipedia, an online encyclopedia, which is provided with manually-assigned keywords in the form of annotations consisting of hyperlinks to pages within or outside Wikipedia that are embedded within the text of each article. These annotations are currently performed by human contributors of articles to Wikipedia by hand following a Wikipedia “manual of style,” which gives guidelines concerning the selection of important concepts in a text, as well as the assignment of links to appropriate related articles. A system that could automatically perform the annotation task would aid contributors to Wikipedia, but there are also many other applications that could benefit from such a system.
Thus, there are many benefits to be gleaned from the capability of automatic extraction of keywords and automatic annotation of electronic text using this capability, and a large number of potential applications for these technologies. However, even state-of-the art systems for automatic keyword extraction and annotation still perform at relatively low levels using the standard information retrieval metrics: precision, recall, and F-measure; and they often fail to produce keywords or annotations approaching the quality of those that are manually constructed by human authors or professional indexers. There is, therefore, a need for improved keyword extraction methods and systems to enhance the quality of automatically-generated indexes and for use in linking other relevant information to electronic documents.
The present invention provides a method, a computer program, an apparatus, and a system for automatically generating one or more keywords from an electronic document. Examples of collections of keywords used for finding useful information include a back-of-the-book index for a book-length document, and keywords used as annotation links within electronic encyclopedias such as Wikipedia. The present invention provides a method of automatically processing electronic documents in order to extract useful collections of keywords, which achieves a goal of more closely approaching a quality of output like that generated by human authors and/or professional indexers. Both unsupervised and supervised methods of automatic keyword extraction algorithms are provided, each with advantages in speed and performance. Novel features for use by machine learning algorithms in keyword extraction are introduced that also have further applications in other areas of natural language processing using computer data processing systems. By combining keyword extraction with word sense disambiguation, a system for automatically annotating electronic documents with links to related information in an electronic encyclopedia is also described that can be used to enrich text for educational and other purposes.
More specifically, the present invention provides a method for automatically generating one or more keywords from an electronic document. Candidate entries for the keywords are identified. A feature vector is constructed for each candidate entry, wherein the feature vector includes at least one feature among one or more discourse comprehension features, one or more part-of-speech pattern features, or one or more encyclopedic annotation features. A numeric score is then assigned to each candidate entry based on the feature vector for that candidate entry. A specified number of entries are selected as the keywords. In addition, the present invention can be implemented as a computer program embodied on a computer readable medium wherein one or more code segments perform the steps described above.
The present invention also provides an apparatus for automatically generating one or more keywords from an electronic document that includes an interface to a document storage repository, a memory, and a processor communicably connected to the interface and the memory. The processor identifies candidate entries for the keywords, constructs a feature vector for each candidate entry, where the feature vector includes at least one feature from among one or more discourse comprehension features, one or more part-of-speech pattern features, and one or more encyclopedic annotation features, assigns a numeric score to each candidate entry based on the feature vector for that candidate entry, and selects a specified number of entries to be the keywords.
In addition, the present invention provides a system for automatically generating one or more keywords from an electronic document that includes a network, one or more client computers communicably connected to the network, one or more server computers communicably connected to the network, one or more document storage repositories communicably connected to the network, to one or more of the client computers, or to one or more of the server computers, and a processor within at least one of the client computers or server computers. The processor identifies candidate entries for the keywords, constructs a feature vector for each candidate entry, where the feature vector includes at least one feature from among one or more discourse comprehension features, one or more part-of-speech pattern features, and one or more encyclopedic annotation features, assigns a numeric score to each candidate entry based on the feature vector for that candidate entry, and selects a specified number of entries to be retained as the keywords.
Other features and advantages of the present invention will be apparent to those of ordinary skill in the art upon reference to the following detailed description taken in conjunction with the accompanying drawings.
The above and further advantages of the invention may be better understood by referring to the following description in conjunction with the accompanying drawings, in which:
While the making and using of various embodiments of the present invention are discussed in detail below, it should be appreciated that the present invention provides many applicable inventive concepts that can be embodied in a wide variety of specific contexts. The specific embodiments discussed herein are merely illustrative of specific ways to make and use the invention and do not delimit the scope of the invention.
Keyword extraction is the identification, within a supplied document, of one or more keywords which can be used to locate information of interest within the document. An example of a list or collection of keywords that can be used for finding useful information is a back-of-the-book index for a book-length document. Such back-of-the-book indexes are generated manually by human indexing professionals. Automation of this task would save a great deal of labor and facilitate possible information lookup on documents that would otherwise never have had indexes. There are also many uses for keyword extraction on shorter documents, such as online web pages, abstracts, or articles. However, most documents are not supplied with a collection of related keywords generated either by the human author, another human, or automatically. Therefore, it would be desirable to have a means for processing any document automatically using a computer to generate the keywords. A related task that requires keyword extraction is that of annotating a document with links to sources of additional information. It is very important that keywords that are generated be useful and relevant to human readers as well as to automatic systems that might process the document or collections of documents that are so indexed. The standard for quality of collections of keywords remains indexes and keyword collections that are generated by humans. The present invention provides a method of automatically processing electronic documents containing text, in order to extract useful collections of keywords, which achieves a goal of more closely approaching a quality of output like that generated by human authors and/or professional indexers.
The present invention applies recent advances in artificial intelligence, specifically machine learning techniques, together with novel features that provide machine learning or ranking algorithms with high-quality and numeric quantitative information as to the relevance of extracted candidate keywords. The inventive methods of deriving and applying these new features to generate collections of keywords is shown to result in improved performance over state-of-the-art keyword extraction algorithms, and to enable further novel applications in annotating text with links to relevant reference information.
Two types of machine learning algorithms are referred to as supervised learning and unsupervised learning. In supervised learning, an algorithm generates a function that maps inputs to desired outputs, often formulated as a “classification” problem, in which the training data is provided with several input-output examples demonstrating a desired mapping. In unsupervised learning, an algorithm functions as an agent which models a set of inputs, but labeled examples are not available to train the algorithm in making classification decisions. Both types of algorithms use the concept of a “feature.” Features are individual measurable heuristic properties of the phenomena being observed that can be used to create a numerical representation of the phenomena, which are in this case word patterns. Removing irrelevant and redundant features from the input data improves processing speed and efficiency by reducing the dimensionality of the problem. Feature selection, that is, choosing discriminating and independent features, is key to any pattern recognition algorithm, and also helps researchers to better understand the data, which features are important, and how they are related to one another.
In one embodiment of the present invention, a supervised learning approach (supervised method) uses a set of (mostly numerical) features (an n-dimensional “feature vector”) that are chosen for their effectiveness in separating desired and undesired entries, and examples of documents together with collections of keywords that have been generated by humans (manually-constructed) are provided as training data to a machine learning algorithm. In another embodiment, an unsupervised method can use similar features selected for their sensitivity to parameters of relevance in ranking keywords, but in the absence of training data, it might use numeric values derived from the feature vectors to perform scoring and ranking of candidate entries. Subsequently, a number of candidate entries to be retained in a keyword collection can be selected using predetermined criteria for quality or for a desired number of entries. Thus the present invention provides both unsupervised and supervised embodiments of an automatic keyword extraction method.
Examples of algorithms and corresponding classifiers used in supervised and unsupervised methods include Naïve Bayes, Support Vector Machine (SVM), Relevance Vector Machine (RVM), decision tree, genetic algorithm, rule induction, k-Nearest Neighbors, Gaussian, Gaussian Mixture Model, artificial neural networks, multilayer perceptron, and radial basis function (RBF) networks.
Referring now to
The unsupervised method 100 shown in
Referring now to
In order to use some collections of documents as a gold standard collection for automatic index construction, it may be necessary to eliminate the inversions, which are typical in human-built indexes. Inversion is a method used by professional indexers by which they break the ordering of the words in each index entry, and list the head first, thereby making it easier to find entries in an alphabetically ordered index. As an example, consider the entry indexing of illustrations, which, following inversion, becomes illustrations, indexing of. To eliminate inversion, an approach is used that generates each permutation of the composing words for each index entry, looks up the frequency of that permutation in the book, and then chooses the one with the highest frequency as the correct reconstruction of the entry. In this way, the form of the index entries as appearing in the book are identified, which is the form required for the evaluation of extraction methods. Entries that cannot be found in the book, which were most likely generated by the human indexers, are preserved in their original ordering.
Once the training data set is provided, it is processed in block 204 to identify candidate entries. This process is similar to that of the unsupervised method in
A test data set is provided in block 212 from documents in need of automatic keyword extraction. In block 214, candidate entries are identified in the test data set, which may be the same type of process that has already been mentioned for block 204 (and 102) and that will be elaborated in
An example data set from which to derive both training and test data to use in evaluating the algorithms and features of the present invention is a collection of books and monographs from the eScholarship Editions collection of the University of California Press (UC Press), consisting of 289 books, each with a manually constructed back-of-the-book index. The average length of the books in this collection is 86053 words, and the average length of the indexes is 820 entries. The UC Press collection is provided in a standardized XML format, following the Text Encoding Initiative (TEI) recommendations, and thus it is relatively easy to process the collection and separate the index from the body of the text. For training and evaluation purposes, a random split of the collection can be made into 90% training and 10% test. This yields a training corpus of 259 documents and a test data set of 30 documents. The following describes a process for identifying candidate entries using this training and test data set, giving exemplary values for numbers of entries and data set sizes.
Every sequence of words in a document such as a book represents a potential candidate for an entry in the keyword collection such as a back-of-the-book index. Thus, a first step is to extract from the training and the test data sets all the n-grams (in this example, up to the length of n=4), not crossing sentence boundaries. An “n-gram” is a sequence of n consecutive words out of the text. These represent the candidate index entries that will be used in the classification algorithm. Candidate entries from a training data set are then labeled as positive or negative, depending on whether the given n-gram was found in the back-of-the-book index associated with the book.
Using an n-gram-based method to extract candidate entries has the advantage of providing high coverage, but the unwanted effect of producing an extremely large number of entries. In fact, the resulting set is unmanageably large for any machine learning algorithm. Moreover, the set is extremely unbalanced, with a ratio of positive and negative examples of 1:675, which makes it unsuitable for most machine learning algorithms. In order to address this problem, it is desirable to find ways to reduce the size of the data set, possibly eliminating the training instances that will have the least negative effect on the usability of the data set.
The first step to reduce the size of the data set is to use candidate filtering techniques for unsupervised back-of-the-book index. Namely, stopword and comma filters are applied to both the training and the test collections. These filters work by eliminating all the n-grams that begin or end with a stopword (for example, a list of 300 most frequent English words), as well as those n-grams that cross a comma. This results in a significant reduction in the number of negative examples in the example training data set, from 48 to 11 million instances, with a loss in terms of positive examples of only 7.6%.
The second step is to use a technique for balancing the distribution of the positive and the negative examples in the data sets. There are several methods proposed in the existing literature, focusing on two main solutions: undersampling and oversampling. Undersampling means the elimination of instances from the majority class (in our case negative examples), while oversampling focuses on increasing the number of instances of the minority class. Aside from the fact that oversampling has hard-to-predict effects on classifier performance, it also has the additional drawback of increasing the size of the data set, which in our case is undesirable. For this example, an undersampling solution was adopted, where 10% of the negative examples are randomly selected for retention in the training data. This undersampling is applied only to the training set.
The following table shows the number of positive and negative entries in the UC Press data set, for the different preprocessing and balancing phases:
These steps for identifying candidate entries in either test or training data are summarized in the method 300 shown in
We next describe novel features, as defined earlier, that can be used in the evaluation or classification of candidate phrases for inclusion in a collection of automatically-extracted keywords. As previously explained, an important step in the development of a supervised system is the choice of features used in the learning process. Ideally, any property of a word or a phrase indicating that it could be a good keyword should be represented as a feature and included in the training and test examples. A number of features, including information-theoretic features used in unsupervised keyword extraction, as well as a novel set of features based on syntactic and discourse properties of the text, or on information extracted from external knowledge repositories, are aspects of the present invention. These novel features are also of great utility in increasing the effectiveness of unsupervised methods.
First, a set of information-theoretic features are described, which are referred to here as the phraseness and informativeness features. A keyphrase is a sequence of one or more words that together can act as a keyword. Phraseness refers to the degree to which a sequence of words can be considered a phrase. It can be used as a measure of lexical cohesion of the component terms and can be treated as a parameter to be used in a collocation discovery problem. Informativeness represents the degree to which a keyword or keyphrase is representative for the document at hand, and it correlates to the amount of information conveyed to the user.
To measure the informativeness of a keyword or keyphrase, various methods can be used:
The independence score is calculated based on the observed (O) and expected (E) counts:
where i, j are the row and column indices of the contingency table. The O counts are the cells of the table. The E counts are calculated from the marginal probabilities (the sum of the values of a column or a row) converted into proportions by dividing them with the total number of observed events (N):
N=O1,1+O1,2+O2,1+O2,2
Then the expected count for seeing the phrase in the document is:
To measure the phraseness of a candidate phrase, a technique based on the χ2 independence test can be used. Such a test measures the independence of the events of seeing the components of the phrase in the text. This method has been found to be one of the best performing models in collocation discovery. For n-grams where n>2, the χ2 independence test may be applied by splitting the phrase in two (e.g. for a 4-gram, by measuring the independence of the composing bigrams).
Next, novel features for keyword extraction, and methods for deriving them, are described that were inspired by work on discourse comprehension. A construction integration framework is used, which is the backbone used by many discourse comprehension theories.
Very few existing keyword extraction methods look beyond word frequency. Excepting the use of pointwise mutual information to improve the coherence of the keyword set, there does not appear to be any other work that attempts to use the semantics of the text to extract keywords. The fact that most systems rely heavily on term frequency properties poses serious difficulties, since many index entries appear only once in the document, and thus cannot be identified by features based solely on word counts. For instance, as many as 52% of the index entries in the previously-described example training data set appeared only once in the books they belong to. Moreover, another aspect not typically covered by current keyword extraction methods is the coherence of the keyword set, which can also be addressed by discourse-based properties.
Discourse comprehension is a field in cognitive science focusing on the modeling of mental processes associated with reading and understanding text. The most widely accepted theory for discourse comprehension is the construction integration theory. According to this theory, the elementary units of comprehension are propositions, which are defined as instances of a predicate-argument schema. As an example, consider the sentence The hemoglobin carries oxygen, which generates the predicate CARRY[HEMOGLOBIN,OXYGEN]. A processing cycle of the construction integration model processes one proposition at a time, and builds a local representation of the text in a working memory, called the propositional network.
Referring to
During a construction phase, propositions 502 are extracted from a segment of the input text (typically a single sentence) using linguistic features. The propositional network is represented conceptually within the working memory as a graph, with nodes consisting of propositions, and weighted edges representing the semantic relations between them. All the propositions generated from the input text are inserted into the graph, as well as all the propositions stored in a short-term memory 510. The short-term memory 510 contains the propositions that compose the representation of the previous few sentences. The second phase of the construction step is the addition of past experiences (or background knowledge), which are stored in a semantic (long-term) memory 504. This is accomplished by adding new elements to the graph in block 506, usually consisting of the set of closely related propositions from the semantic memory.
After processing a sentence, an integration step performed within working memory 514 establishes the role of each proposition in the meaning representation of the current sentence, through a spreading activation applied on the propositions derived from the original sentence. Once the weights are stabilized, the set of propositions with the highest activation values give the mental representation of the processed sentence. Propositions with the highest activation values are added to the short-term memory 510, the working memory 514 is cleared and the process moves to the next sentence.
The main purpose of the short-term memory 510 is to ensure the coherence of the meaning representation across sentences. By keeping the most important propositions 508 in the short-term memory, the spreading activation process transfers additional weight to semantically related propositions in the sentences that follow. This also represents a way of alleviating one of the main problems of statistical keyword extraction, namely the sole dependence on term frequency. Even if a phrase appears only once, the construction integration process ensures the presence of the phrase in the short-term memory as long as it is relevant to the current topic, thus being a good indicator of the phrase importance.
Note that the embodiment of a construction integration process for the present invention requires a number of adaptations of previous work used for different applications. A traditional construction integration model is not directly applicable to keyword extraction without modification due to a number of practical difficulties. The first implementation problem is the lack of a propositional parser. This problem can be solved in one embodiment of the present invention by using a shallow parser to extract noun phrase chunks from the original text. Second, since spreading activation is a process difficult to control, with several parameters that require fine tuning, a different graph centrality measure may be used, namely PageRank [Brin and Page, 1998].
Finally, to represent the relations inside the long term semantic memory, a variant of latent semantic analysis (LSA) is used, for example as implemented in the InfoMap package, trained on a corpus consisting of the British National Corpus, the English Wikipedia, and the books in another collection. To alleviate a data sparsity problem, pointwise mutual information (PMI) can also be used to calculate the relatedness of the phrases based on the book being processed.
An embodiment of a method 400 for deriving discourse comprehension features for automatic keyword extraction using a construction integration approach works by performing the steps enumerated in
At least three different exemplary useful features can be derived based on the construction integration (CI) model:
Next, a method for deriving useful numeric features based on syntactic patterns is described. Previous work has pointed out the importance of syntactic features for supervised keyword extraction, but such features have had discrete and not numeric values, as would better suit them to ranking and classification algorithms. The construction integration model just described is already making use of syntactic patterns to some extent, through the use of a shallow parser to identify noun phrases. However, that approach does not cover patterns other than noun phrases. To address this limitation, a new numeric feature is introduced that captures the part-of-speech of the words composing a candidate phrase.
There are multiple ways to represent a part-of-speech feature. The simplest is to create a string feature consisting of the concatenation of the part-of-speech tags, as in the prior work. However, this representation imposes limitations on the machine learning algorithms that can be used, since many learning systems cannot handle string features. A second solution is to introduce a binary feature for each part-of-speech tag pattern found in the training and the test data sets. In many applications this is again unacceptable, given the size of documents such as books and the large number of syntactic patterns that can be extracted. Instead, a novel solution is presented which, rather than using the part-of-speech pattern directly, determines the probability of a phrase with a certain tag pattern to be selected as a keyphrase. Formally:
where C(pattern, positive) is the number of distinct phrases having the tag pattern pattern and being selected as a keyphrase, and C(pattern) represents the number of distinct phrases having the tag pattern pattern. This probability is estimated based on the training collection, and is used as a numeric feature. The name of this feature is part-of-speech pattern.
A method 600 to implement the calculation of a syntactic feature such as the numeric part-of-speech pattern feature is detailed in
Next another novel type of feature is introduced, called an encyclopedic annotation feature. Recent work has suggested the use of domain knowledge to improve the accuracy of keyword extraction. This is typically done by consulting a vocabulary of plausible keywords, usually in the form of a list of subject headings or a domain-specific thesaurus. The use of a vocabulary has the additional benefit of eliminating the extraction of incomplete phrases (e.g. “States of America”).
Since large documents like books can cover several domains, the construction and use of domain-specific thesauruses is not plausible, as the advantage of such resources is offset by the time it usually takes to build them. Instead, encyclopedic information can be used as a way to ensure high coverage in terms of domains and concepts. Such encyclopedic information may be found online on the Internet in the form of articles that are linked by annotations to each other. For example, Wikipedia is an online encyclopedia that has grown to become one of the largest online repositories of encyclopedic knowledge, with millions of articles available for a large number of languages. In fact, Wikipedia editions are available for more than 200 languages, with a number of entries varying from a few pages to more than one million articles per language. Besides being the largest and the fastest growing encyclopedia available today, the structure of Wikipedia has the additional benefit of being particularly useful for the task of automatic keyword extraction. Wikipedia includes a rich set of links that connect important phrases in an article to their corresponding articles. These links are added manually by the Wikipedia contributors, and follow the general guidelines of annotation provided by Wikipedia. The guidelines coincide with the goals of keyword extraction, and thus the Wikipedia articles and their link annotations can be treated as a vast keyword annotated corpus.
Such encyclopedic annotations can be used in two ways. First, if a phrase is used as the title of a Wikipedia article, or as the anchor text in a link, this is a good indicator that the given phrase is well-formed. Second, the probability of a term W to be selected as a keyword in a new document can be estimated by counting the number of documents count(Dkey) where the term was already selected as a keyword, divided by the total number of documents count(DW) in which the term appeared:
These counts are collected from the entire set of Wikipedia articles. This probability can be interpreted as “the more often a term was selected as a keyword among its total number of occurrences, the more likely it is that it will be selected again.” In the following, this feature is referred to as Wikipedia keyphraseness.
If there is a specific goal to create keywords that point to Wikipedia (or other electronic encyclopedia) articles, such as in the annotation application to be discussed later, then a controlled keyword vocabulary can be constructed that is intended to contain only the Wikipedia article titles (1,406,039 such titles are included in the March 2006 version of Wikipedia), and use this controlled vocabulary to extract keywords. One could simply construct such a vocabulary out of article titles. However, this would greatly restrict the potential to find all the keywords, since the actual use of a phrase within an article (surface form) may differ from the corresponding article title. For instance, different morphological forms such as e.g. “dissecting” or “dissections” can be linked to the same article title “dissection.” By ignoring these morphological variations, it is likely that a good fraction of the keywords that appear in a form different than the Wikipedia titles would be missed. To address this problem, the controlled vocabulary can be extended with all the surface forms collected from all the Wikipedia articles. The Wikipedia keyphraseness probability feature could become unreliable for marginal cases where counts are very low. To fix this problem, one can simply subsequently discount all the occurrences that were used less than a predetermined small number (in this example, five) of times. After this process, the resulting controlled vocabulary consists of 1,918,830 terms.
Now refer to
Other features may also be included among those used in constructing feature vectors operated upon during the automatic keyword extraction process. In addition to the features described before, several other features can be added that are frequently used in keyword extraction: the frequency of the phrase inside the book (term frequency orf); the number of documents that include the phrase (document frequency or df); a combination of the two (tf.idf); the within-document frequency, which divides a book into for example ten equally-sized segments, and counts the number of segments that include the phrase (within document frequency); the length of the phrase (length of phrase); and finally a binary feature indicating whether the given phrase is a named entity, according to a simple heuristic based on word capitalization.
The features described above were integrated into a machine learning framework and was evaluated on the UC Press data set, consisting of 289 books, randomly split into 90% training (259 books) and 10% test (30 books). Three learning algorithms, selected for the diversity of their learning strategy, were evaluated: multilayer perceptron, SVM, and decision trees. For all three algorithms, their implementation available in the Weka package was used.
For evaluation, the standard information retrieval metrics were used: precision, recall, and F-measure. Two different mechanisms were used for selecting the number of entries in the index. In the first evaluation (ratio-based), a fixed ratio of 0.45% from the number of words in the text was used; for instance, if a book has 100,000 words, the index will consist of 450 entries. This number was estimated based on previous observations regarding the typical size of a back-of-the-book index (Csomai and Mihalcea, 2006). In order to match the required number of entries, all the candidates were sorted in reversed order of the confidence score assigned by the machine learning algorithm, and consequently the top entries in this ranking were selected. In the second evaluation (decision-based), the machine learning algorithm was allowed to decide on the number of keywords to extract. Thus, in this evaluation, all the candidates labeled as keywords by the learning algorithm will be added to the index. Note that all the evaluations are run using a training data set with 10% undersampling of the negative examples, as described before.
The results of the evaluation were:
where P is Precision, R is Recall and F is F-measure. As can be seen, the multilayer perceptron and the decision tree provide the best results, for an overall average F-measure of 27%. Interestingly, the results obtained when the number of keywords is automatically selected by the learning method (decision-based) are comparable to those when the number of keywords is selected a-priori (ratio-based), indicating the ability of the machine learning algorithm to correctly identify the correct keywords.
Additionally, an experiment was run to determine the amount of training data required by the system. While the learning curve continues to grow with additional amounts of data, the steepest part of the curve is observed for up to 10% of the training data, which indicates that a relatively small amount of data (about 25 books) is enough to sustain the system.
It is worth noting that the task of creating back-of-the-book indexes is highly subjective. In order to put the performance figures in perspective, one should also look at the inter-annotator agreement between human indexers as an upper bound of performance. For example, the consistency studies that have been carried out on the MEDLINE corpus (Funk and Reid, 1983) found an inter-annotator agreement of 48% on an indexing task using a domain-specific controlled vocabulary of subject headings.
The performance of the system was compared with two other methods for keyword extraction. One is the tf.idf method, traditionally used in information retrieval as a mechanism to assign words in a text with a weight reflecting their importance. This tf.idf baseline system uses the same candidate extraction and filtering techniques as the supervised systems. The other baseline is the KEA keyword extraction system (Frank et al., 1999), a state-of-the-art algorithm for supervised keyword extraction. Very briefly, KEA is a supervised system that uses a Naïve Bayes learning algorithm and several features, including information theoretic features such as tf.idf and positional features reflecting the position of the words with respect to the beginning of the text. The KEA system was trained on the same training data set as used in the experiments.
The following table shows the performance obtained by these methods on the test data set. Since none of these methods have the ability to automatically determine the number of keywords to be extracted, the evaluation of these methods is done under the ratio-based setting, and thus for each method the top 0.45% ranked keywords are extracted.
Using an information gain weight as assigned by running a learning algorithm, an indication of the role played by each feature, and how well they discriminate between wanted and unwanted keywords, was determined using the UC Press data set described earlier. The following table lists features that are described above together with the weight associated with each feature. Higher weights indicate higher information gain, that is, they make a bigger difference in the sorting of the input data. Features having higher information gain weights are more effective features for classification, speeding the algorithm, and are thus preferred over other features.
The features with the highest weight are the linguistically motivated features of the present invention, including syntactic (part-of-speech) patterns, the discourse comprehension (CI) features, and the Wikipedia keyphraseness encyclopedic annotation feature. The smallest weights belong to the information theoretic features, including term frequency and document frequency, that are most commonly used in prior art keyword extraction algorithms.
A number of features have been listed that are useful in either supervised or unsupervised methods for automatic keyword extraction. Either type of algorithm uses these features to perform either a decision process for inclusion with the final keywords (decision-based) or to perform a ranking of candidate entries based on numeric scores (score-based). Now it will be shown how a decision can be made regarding how many entries to retain in the final output.
Referring now to
According to an information gain measure of feature importance, the new features performed significantly better than the traditional frequency-based techniques, leading to a system with an F-measure of 27%. This represents an improvement of 140% with respect to a state-of-the-art supervised method for keyword extraction. The present invention proved to be successful both in ranking the phrases in terms of their suitability as index entries, as well as in determining the optimal number of entries to be included in the index. The present invention can be used for computer-assisted back-of-the-book indexing, as well as on the use of the automatically extracted indexes to improve the browsing of digital libraries.
Given a capability for high-quality automatic keyword extraction, and methods for deriving useful features and properties from electronic encyclopedia annotations, an advanced system is now described using these methods, together with the addition of word sense disambiguation, to provide the function of automatically annotating any text with links to electronic encyclopedia articles, following e.g. the Wikipedia guidelines.
Referring now to
More detail is now provided about what is meant by word sense disambiguation, and about the processes occurring within the word sense disambiguation module 920.
Ambiguity is inherent to human language. In particular, word sense ambiguity is prevalent in all natural languages, with a large number of the words in any given language carrying more than one meaning. For instance, the English noun plant can either mean green plant or factory; similarly the French word feuille can either mean leaf or paper. The correct sense of an ambiguous word can be selected based on the context where it occurs, and correspondingly the problem of word sense disambiguation is defined as the task of automatically assigning the most appropriate meaning to a polysemous word within a given context. Word sense ambiguity is also present within Wikipedia, with a large number of the concepts mentioned in the Wikipedia pages having more than one possible explanation (or “sense”). In the Wikipedia annotations, this ambiguity is solved through the use of links or piped links, which connect a concept to the corresponding correct Wikipedia article.
For instance, ambiguous words such as e.g. plant, bar, or chair are linked to different Wikipedia articles depending on the meaning they have in the context where they occur. Note that the links are manually created by the Wikipedia contributors, which means that they are most of the time accurate and referencing the correct article. The following represent five example sentences for the ambiguous word bar, with their corresponding Wikipedia annotations (links):
Interestingly, these links can be regarded as sense annotations for the corresponding concepts, which is a property particularly valuable for the entities that are ambiguous. The ambiguity is related to the surface form of the concepts defined in Wikipedia, e.g. the word bar that can be linked to five different Wikipedia pages depending on its meaning Note that although Wikipedia defines the so-called disambiguation pages, meant as a record of word meanings, the disambiguation pages do not always account for all the possible surface form interpretations. For instance, there are several Wikipedia pages where the ambiguous word bar is sometimes linked to the pages corresponding to nightclub or public house, but these meanings are not listed on the disambiguation page for bar.
Regarded as a sense inventory, Wikipedia has a much larger coverage than a typical English dictionary, in particular when it comes to entities (nouns). This is mainly due to the large number of named entities covered by Wikipedia (e.g. Tony Snow, Washington National Cathedral), as well as an increasing number of multi-word expressions (e.g. mother church, effects pedal). For instance, in the March 2006 version, there were counted a total of 1.4 million entities defined in Wikipedia, referred by a total of 4.5 million unique surface forms (anchor texts), accounting for 5.8 million unique Wikipedia word “senses” (where a “sense” is defined as the unique combination of a surface form and a link to a Wikipedia entity definition). Other sense inventories are available, but Wikipedia seems to provide the largest. The sense inventory and access to its entries is indicated in
Two different disambiguation algorithms have been implemented and evaluated, inspired by two main trends in word sense disambiguation research. These are a knowledge-based method and a data-based method. Note that the two approaches can be used individually by choosing one, or by using both in combination in order to achieve higher accuracy, as described in the following.
The first disambiguation method uses a knowledge-based approach, which relies exclusively on information drawn from the definitions provided by the sense inventory. This method is inspired by the Lesk algorithm [Lesk, 1986], and attempts to identify the most likely meaning for a word in a given context based on a measure of contextual overlap between the dictionary definitions of the ambiguous word—here approximated with the corresponding Wikipedia pages, and the context where the ambiguous word occurs (for example, the current paragraph may be used as a representation of the context). Function words and punctuation are removed prior to the matching. For instance, given the context “it is danced in ¾ time, with the couple turning 180 degrees every bar”, and assuming that “bar” could have the meanings of bar music or bar counter, the Wikipedia pages are processed for both the music and counter meanings, and consequently determine the sense that maximizes the overlap with the given context. Block 914 in
The second approach is a data-driven method that integrates both local and topical features into a machine learning classifier. For each ambiguous word, a training feature vector is extracted for each of its occurrences inside a Wikipedia link, with the set of possible word senses being given by the set of possible links in Wikipedia. To model feature vectors, the current word and its part-of-speech are used, a local context of three words to the left and right of the ambiguous word, the parts-of-speech of the surrounding words, and a global context implemented through sense-specific keywords determined as a list of at most five words occurring at least three times in the contexts defining a certain word sense. The numbers of words in these groups are given here in the context of an example, and thus can be changed without departing from the spirit and scope of the present invention. The parameters for sense-specific keyword selection were determined through cross-fold validation on the training set. The features can be integrated for example in a Naïve Bayes classifier, selected mainly for its performance in previous work showing that it can lead to a state-of-the-art disambiguation system given the features that have been listed, or any of the similar types of classifiers and machine learning algorithms that have been previously listed. Block 916 indicates processing of keywords using a data-driven algorithm.
Given the orthogonality of the knowledge-based and the data-driven approaches, a voting scheme as mentioned earlier can optionally be implemented, meant to filter out the incorrect predictions by running both and seeking agreement between the two methods. Since it has been noticed that the two methods disagree in their prediction in about 17% of the cases, this disagreement can be used as an indication of potential errors, and consequently annotations that lack agreement can be ignored or removed from consideration. In the system shown in
Again, when all the annotations are ready, the structure of the original hypertext document is reconstructed in block 922, and the newly added reference links are included in the text in block 924, and the method of 900 is completed, resulting in at least one fully-processed electronic document containing links to encyclopedia articles as indicated by e.g. highlighted forms of the keywords.
Thus, a method for automatically annotating electronic documents with links to related information in an electronic encyclopedia has been demonstrated. Specifically, given an input document, the system 900 has the ability to identify the important concepts in a text (keyword extraction), and then link these concepts to corresponding encyclopedia articles such as Wikipedia pages (word sense disambiguation).
To evaluate the accuracy of the disambiguation algorithms, a gold-standard data set consisting of a collection of pages from Wikipedia was used, containing manual “sense” annotations made by the Wikipedia contributors. As mentioned before, the “sense” annotations correspond to the links in a Wikipedia page, which uniquely identify the meaning of the corresponding words. The same set of pages used during the keyword extraction evaluation was used, namely 85 Wikipedia pages containing 7,286 linked concepts.
Since the focus of this particular evaluation is on the quality of the disambiguation system, the keyword extraction and the word sense disambiguation evaluations were detached, and it was assumed that the keyword extraction stage produces 100% precision and recall. This assumption helps avoid the error propagation effect, and consequently isolate the errors that are specific to the word sense disambiguation module. An evaluation of the entire system is reported in the following section.
The set of keywords were manually selected by the Wikipedia contributors within the dataset of 85 pages, and for each such keyword the word sense disambiguation method was used to automatically predict the correct “sense,” i.e. the correct link to a Wikipedia definition page.
For instance, given the context “Jenga is a popular beer in the [[bar (establishment)|bar]]s of Thailand.”, we will attempt to disambiguate the word “bar,” since it has been marked as a candidate Wikipedia concept. We therefore try to automatically predict the title of the Wikipedia page where this concept should be linked, and evaluate the quality of this prediction with respect to the gold standard annotation bar (establishment).
Evaluations of word sense disambiguation systems typically report on precision and recall, where precision is defined as the number of correctly annotated words divided by the total number of words covered by the system, and recall is defined as the number of correct annotations divided the total number attempted by the system.
The gold standard data set includes all the words and phrases that were marked as Wikipedia links in the 85 test articles, which amount to a total of 7,286 candidate concepts. Out of these, about 10% were marked as “unknown”—indicating that the corresponding surface form was not found in other annotations in Wikipedia, and therefore the system did not have any knowledge about the possible meanings of the given surface form. For instance, the surface form “Conference Championship” is a candidate concept in one of our test pages; however, this surface form was not encountered anywhere else in Wikipedia, and therefore since we do not have any sense definitions for this phrase, we mark it as “unknown.” These cases could not be covered by the system, and they account for the difference between the total number of 7,286 concepts in the data set, and the “attempted” counts listed in Table 2 below.
Precision, recall and F-measure figures for the three disambiguation algorithms are shown below:
The table also shows the performance of an unsupervised baseline algorithm that for each candidate concept randomly selects one of its possible senses, and the performance of the most frequent sense baseline using counts derived from Wikipedia.
Perhaps not surprising, the data-driven method outperforms the knowledge-based method both in terms of precision and recall. This is in agreement with previously published word sense disambiguation results on other sense annotated data sets. Nonetheless, the knowledge-based method proves useful due to its orthogonality with respect to the data-driven algorithm. The voting scheme combining the two disambiguation methods has the lowest recall, but the highest precision. This is not surprising since this third system tagged only those instances where both systems agreed in their assigned label. We believe that this high precision figure is particularly useful for the Wikify! system, as it is important to have highly precise annotations even if the trade-off is lower coverage.
Note that these evaluations are rather strict, as we give credit only to those predictions that perfectly match the gold standard labels. We thus discount a fairly large number of cases where the prediction and the label have similar meaning For instance, although the system predicted Gross domestic product, as a label for the concept “GDP”, it was discounted for not matching the gold-standard label GDP, despite the two labels being identical in meaning There were also cases where the prediction made by the system was better than the manual label, as in e.g. the label for the concept football in the (British) context playing football, wrongly linked to Association football by the Wikipedia annotator, and correctly labeled by the automatic system as football (soccer).
The final disambiguation results are competitive with figures recently reported in the word sense disambiguation literature. For instance, the best system participating in the recent Senseval/Semeval fine-grained English all-words word sense disambiguation evaluation reported a precision and recall of 59.10%, when evaluated against WordNet senses. In the coarse-grained word sense disambiguation evaluation, which relied on a mapping from WordNet to the Oxford Dictionary, the best word sense disambiguation system achieved a precision and recall of 83.21%.
The present invention also provides a Wikify! System that integrates the keyword extraction algorithm, automatically identifies the important keywords in the input document, and the word sense disambiguation algorithm that assigns each keyword with the correct link to a Wikipedia article. The Wikify! system brings together the capabilities of the keyword extraction and the word sense disambiguation systems under a common system that has the ability to automatically “wikify” any input document. Given a document provided by the user or the URL of a webpage, the system processes the document provided by the user, automatically identifies the important keywords in the document, disambiguates the words and links them to the correct Wikipedia page, and finally returns and displays the “wikified” document. The interface allows the user to either (1) upload a local text or html file, or (2) indicate the URL of a webpage. The user also has the option to indicate the desired density of keywords on the page, ranging from 2%-10% of the words in the document (default value: 6%), as well as the color to be used for the automatically generated links (default color: red). The Wikify! system is then launched, which will process the document provided by the user, automatically identify the important keywords in the document, disambiguate the words and link them to the correct Wikipedia page, and finally return and display the “wikified” document. Note that when an URL is provided, the structure of the original webpage is preserved (including images, menu bars, forms, etc.), consequently minimizing the effect of the Wikify! system on the overall look-and-feel of the webpage being processed.
In addition to the evaluations concerning the individual performance of the keyword extraction and word sense disambiguation methods, the overall quality of the Wikify! System was evaluated. A Turing-like test concerned with the quality of the annotations of the Wikify! system as compared to the manual annotations produced by Wikipedia contributors was designed. In this test, human subjects were asked to distinguish between manual and automatic annotations. Given a Wikipedia page, we provided the users with two versions: (a) a version containing the original concept annotations as originally found in Wikipedia, which were created by the Wikipedia contributors; and (b) a version where the annotations were automatically produced using the Wikify! system. Very briefly, the second version was produced by first stripping all the annotations from a Wikipedia webpage, and then running the document through the Wikify! system, which automatically identified the important concepts in the page and the corresponding links to Wikipedia pages.
The dataset for the survey consisted of ten randomly selected pages from Wikipedia, which were given to 20 users with mixed professional background (graduate and undergraduate students, engineers, economists, designers). For each page, the users were asked to check out the two different versions that were provided, and indicate which version they believed was created by a human annotator. Note that the order of the two versions (human, computer) was randomly swapped across the ten documents, in order to avoid any bias.
Over the entire testbed of 200 data points (20 users, each evaluating 10 documents), the “human” version was correctly identified only in 114 cases, leading to an overall low accuracy figure of 57% (standard deviation of 0.15 across the 20 subjects).
An “ideal” Turing test is represented by the case when the computer and human versions are indistinguishable, thus leading to a random choice of 50% accuracy. The small difference between the accuracy of 57% achieved by the subjects taking the test and the ideal Turing test value of 50% suggests that the computer-generated and human-generated Wikipedia annotations are hardly distinguishable, which is an indication of the high quality of the annotations produced by the Wikify! system.
Now referring to
The client computers 1014 can each be a mainframe, desktop, minicomputer, computer workstation, laptop computer, terminal, mobile computing device, mobile communications device, personal data assistant (“PDA”), or any other local or remote device in which it is desirable to access, use or process text, hypertext, or speech-to-text documents. Client computers 1014 typically include a browser, terminal program, or other software that allows a user to interface with the other client computers 1014, server computers 1016, document storage repositories 1018, or other resources communicably connected to the network 1012. The various inventions described above in reference to
There are many applications that could benefit from such a system of automatic annotation and linking. First, the vision of the Semantic Web is to have semantic annotations readily available inside the webpages, which will allow for a new semantically-oriented way of accessing information on the Web. The annotations produced by the system can be used to automatically enrich online documents with references to semantically related information, which is likely to improve the Web users' overall experience. Second, in educational applications, it is important for students to have fast access to additional information relevant to the study material. The system could serve as a convenient gateway to encyclopedic information related to assignments, lecture notes, and other teaching materials, by linking important terms to the relevant pages in Wikipedia or elsewhere. In addition, the system can also be used by the Wikipedia users, where the system can provide support for the manual annotation process by suggesting keywords and links. Finally, a number of text processing problems are likely to find new solutions in the rich text annotations produced by the system. Wikipedia has already been successfully used in several natural language processing applications, and automatic Wikipedia-style annotation of documents will likely prove useful in a number of text processing tasks such as e.g., summarization, entailment, text categorization, and others.
The present invention provides improved methods for automatic keyword extraction and word sense disambiguation. Novel features for use by machine learning algorithms in keyword extraction are introduced, that have further applications in other areas of natural language processing using computer data processing systems. For example, language processing can ultimately be applied to processing speech as well as text. In addition, these inventive methods can be combined to produce automatically annotated text with links to electronic encyclopedias. Other combinations of the inventive elements can lead to other automated applications in natural language processing. While the algorithms and derivations presented here were all described in relation to English language examples, the methods can be equally well applied to other languages. For example, Wikipedia editions are available in more than 200 languages, and thus the encyclopedic features and applications herein described can be readily translated.
The following references are hereby incorporated by reference in their entirety:
Although preferred embodiments of the present invention have been described in detail, it will be understood by those skilled in the art that various modifications can be made therein without departing from the spirit and scope of the invention as set forth in the appended claims.
This patent application is a non-provisional patent application of U.S. patent application 61/112,182 filed on Nov. 6, 2008 and entitled “Method, System and Apparatus for Automatic Keyword Extraction,” which is hereby incorporated by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5960383 | Fleischer | Sep 1999 | A |
| 6470307 | Turney | Oct 2002 | B1 |
| 7266545 | Bergman et al. | Sep 2007 | B2 |
| 7581227 | Lawrence et al. | Aug 2009 | B1 |
| 7707204 | Li et al. | Apr 2010 | B2 |
| 7809548 | Mihalcea et al. | Oct 2010 | B2 |
| 8090723 | Cao et al. | Jan 2012 | B2 |
| 20040181389 | Bourigault et al. | Sep 2004 | A1 |
| 20050278325 | Mihalcea et al. | Dec 2005 | A1 |
| 20070233656 | Bunescu et al. | Oct 2007 | A1 |
| 20080126311 | Rowley | May 2008 | A1 |
| 20100114863 | Steelberg et al. | May 2010 | A1 |
| 20110072023 | Lu | Mar 2011 | A1 |
| 20120197864 | Bourdoncle et al. | Aug 2012 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20100145678 A1 | Jun 2010 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61112182 | Nov 2008 | US |
