Patent Application
20070106493
This invention relates to methods of defining ontologies, word disambiguation methods, computer systems, and articles of manufacture.
Collection, integration and analysis of large quantities of data are routinely performed by intelligence analysts and other entities. Vast numbers of different types of communications (e.g., documents, electronic mail, etc.) may be analyzed and perhaps associated with one another in an attempt to gain information or insight which is not readily comprehensible from the communications taken individually. Various analyst tools process communications in attempts to generate, identify, and investigate hypotheses. However, many words present in the documents may have a plurality of senses or different meanings which may lead to unacceptable processing of the documents. At least some aspects of the present disclosure are directed towards apparatus and methods of disambiguating the meaning of words occurring in text to facilitate analysis of the documents.
Preferred embodiments of the invention are described below with reference to the following accompanying drawings.
At least some aspects of the disclosure relate to apparatus and/or methods of defining a lexical database ontology. The lexical database ontology may be used in textual analysis to disambiguate textual content being analyzed.
Additional aspects of the disclosure provide apparatus and/or methods of disambiguating textual content. One disambiguation embodiment utilizes a lexical database ontology. Another disambiguation embodiment discloses selection of certain classification models and/or ontologies corresponding to subject matter of the textual content being analyzed. In one embodiment, the textual content may be annotated with disambiguated meanings of the textual content resulting from analysis. The disambiguation may facilitate additional analysis of the textual content, for example, in analysis including artificial intelligence. Additional embodiments are described below.
Computing system 10 may be implemented in exemplary embodiments using any appropriate processing device, such as a personal computer, workstation, server, etc. The exemplary depicted computing system 10 includes a communications interface 12, processing circuitry 14, storage circuitry 16 and a user interface 18 according to one embodiment. Other embodiments are possible including more, less and/or alternative structure. Further, additional computing systems 10 may be utilized in other embodiments. For example, one computing system 10 may be configured to define a lexical database ontology while one or more other systems 10 may be configured to disambiguate textual content (e.g., which may include using a lexical database ontology determined by another device) in one exemplary embodiment.
Communications interface 12 is arranged to implement communications of computing system 10 with respect to external devices (not shown). For example, communications interface 12 may be arranged to communicate information bi-directionally with respect to computing system 10. Communications interface 12 may be implemented as a network interface card (NIC), serial or parallel connection, USB port, Firewire interface, flash memory interface, floppy disk drive, or any other suitable arrangement for communicating with respect to computing system 10.
In one embodiment, processing circuitry 14 is arranged to process data, control data access and storage, issue commands, and control other desired operations. Processing circuitry 14 may comprise circuitry configured to implement desired programming provided by appropriate media in at least one embodiment. For example, the processing circuitry 14 may be implemented as one or more of a processor and/or other structure configured to execute executable instructions including, for example, software and/or firmware instructions, and/or hardware circuitry. Exemplary embodiments of processing circuitry 14 include hardware logic, PGA, FPGA, ASIC, state machines, and/or other structures alone or in combination with a processor. These examples of processing circuitry 14 are for illustration and other configurations are possible.
The storage circuitry 16 is configured to store programming such as executable code or instructions (e.g., software and/or firmware), electronic data, databases, and/or other digital information and may include processor-usable media. Processor-usable media may be embodied in any computer program product or article of manufacture which can contain, store, or maintain programming, data and/or digital information for use by or in connection with an instruction execution system including processing circuitry 14 in the exemplary embodiment. For example, exemplary processor-usable media may include any one of physical media such as electronic, magnetic, optical, electromagnetic, infrared or semiconductor media. Some more specific examples of processor-usable media include, but are not limited to, a portable magnetic computer diskette, such as a floppy diskette, zip disk, hard drive, random access memory, read only memory, flash memory, cache memory, and/or other configurations capable of storing programming, data, or other digital information.
At least some embodiments or aspects described herein may be implemented using programming stored within appropriate storage circuitry 16 described above and/or communicated via a network or other transmission media and configured to control appropriate processing circuitry. For example, programming may be provided via appropriate media including for example embodied within articles of manufacture, embodied within a data signal (e.g., modulated carrier wave, data packets, digital representations, etc.) communicated via an appropriate transmission medium, such as a communication network (e.g., the Internet and/or a private network), wired electrical connection, optical connection and/or electromagnetic energy, for example, via a communications interface, or provided using other appropriate communication structure or medium. Exemplary programming including processor-usable code may be communicated as a data signal embodied in a carrier wave in but one example.
User interface 18 is configured to interact with a user including conveying data to a user (e.g., displaying data for observation by the user, audibly communicating data to a user, etc.) as well as receiving inputs from the user (e.g., tactile input, voice instruction, etc.). Accordingly, in one exemplary embodiment, the user interface 18 may include a display (e.g., cathode ray tube, LCD, etc.) configured to depict visual information and an audio system as well as a keyboard, mouse and/or other input device. Any other suitable apparatus for interacting with a user may also be utilized.
Referring to
At a step S10, a lexical database is accessed. In one embodiment, the lexical database which is accessed may be useful to facilitate generation of an ontology. For example, in exemplary embodiments, the lexical database may include a lexical hierarchy wherein words are arranged by synonyms, hypernyms, hyponyms, and/or troponyms in illustrative arrangement examples. Words may refer to entire words or lemmas of the words. An example of a lexical database which may be used is WordNet 2.0 available at http://wordnet.princeton.edu, the teachings of which are incorporated herein by reference. Other lexical databases may be used in other embodiments.
A lexical database, such as WordNet 2.0, may include a plurality of different concepts. The concepts may be constructed as a plurality of synonym sets associated with respective ones of the concepts. For example, WordNet 2.0 includes approximately 13,500 concepts or synonym sets for verbs. Concepts or synonym sets are also present in WordNet 2.0 for nouns, adjectives, adverbs, etc. The concepts or synonym sets may individually include a plurality of words. For example, in WordNet 2.0, a plurality of lemmas of synonyms and the senses thereof are associated with individual ones of the concepts and form synonym sets.
At a step S12, a document corpus is accessed to assist with defining a lexical database ontology. An exemplary corpus which may be used is the British National Corpus. Other textual content may be used in other embodiments.
At a step S14, words present in the documents of the corpus are identified. In one embodiment, occurrences of lemmas present in the documents of the corpus are identified to provide the identification of the words.
At a step S16, a plurality of frequency counts indicative of the number of occurrences of the concepts in the documents of the corpus are counted or determined. As mentioned above, in one embodiment, the British National Corpus may be used. Words of the documents of the British National Corpus are not annotated with WordNet word senses. Accordingly, concept counts may be distributed across all possible senses of a word or lemma in one embodiment. In one embodiment, the number of times a lemma occurs in the corpus is determined and the frequency counts of the concepts which include a sense or hypernym of the respective lemma are incremented according to the number of occurrences of the lemma in the corpus. In one embodiment, the frequency count of each concept including a sense of the word lemma may be increased by a measure. The measure may be calculated to be the same for all of the concepts which include a sense of the word lemma and wherein the sum of the measures for the concepts being incremented equal one (e.g., increase each concept by ½ if two concepts include a sense of the word lemma, increase each concept by ⅓ if three concepts include a sense of the word lemma, etc.). For example, if there are two senses of a lemma, then each of the frequency counts of the concepts including one of the senses is updated by 0.5 when the lemma is observed to occur within the corpus. In addition, the frequency count of a concept which is a hypernym of the concept(s) which are increased as a result of including a sense of the word lemma may also be increased by the same measure (e.g., ½, ⅓, etc.). If the words of the corpus are annotated with senses, the frequency counts of the respective concepts which include the words and appropriate senses are incremented when the words and respective senses are observed in the corpus. Other methods of determining frequency counts may be used in other embodiments.
At a step S18, the frequency counts are analyzed. For example, the frequency counts may be compared with a threshold to identify the most frequently occurring concepts present in the documents of the corpus. The most frequently occurring concepts may be chosen as event classes in one embodiment. In an example wherein the British National Corpus is used, frequency counts for verb concepts or synonym sets of WordNet 2.0 ranged from 0 to 2,060,415. In one embodiment, an exemplary frequency cut-off value may be selected as a threshold for selecting the verb concepts as verb event classes. A threshold value of 10,000 was selected in one embodiment and resulted in 1077 verb event classes out of a total of 24,632 verb concepts or synonym sets using the British National Corpus and WordNet 2.0.
At a step S20, the concepts whose frequency counts exceed the threshold are selected as event classes of a lexical database ontology.
Additional details regarding analysis of a corpus of documents to extract WordNet concept frequencies are discussed in Resnik, P. (1995) Using Information Content To Evaluate Semantic Similarity in a Taxonomy, Proceedings of the 14th International Joint Conference on Artificial Intelligence, Vol. 1, 448-453, Montreal, August 1995; Pedersen, T., S. Banerjee and S. Patwardhan Maximizing Semantic Relatedness to Perform Word Sense Disambiguation, University of Minnesota Supercomputing Institute Research Report UMSI Mar. 25, 2005 (available at http://www.msi.umn.edu/general/Reports/rptfiles/2005-25.pdf); and http://search.cpan.org/dist/WordNetSimilarity/utils/BNCFreg.pl, the teachings of which are incorporated herein by reference.
In one embodiment, the above exemplary procedure may select concepts as event classes that are less specific in meaning (e.g., selection of a general concept or verb synonym set {communicate#2, intercommunicate#2} versus selection of a more specific concept {gesticulate#1, gesture#1, motion#1}). As used herein, the numbers following a word or lemma indicate the sense of the word or lemma.
Other and/or additional criteria may be used in one embodiment to identify concepts or synonym sets as event classes. For example, in one embodiment, an exemplary criterion in addition to the above-recited method for identifying classes may be imposed that concepts which are selected as event classes include hyponyms in the lexical database being utilized. Other criteria are possible for selecting concepts.
In one example, 386 top-level verb synonym sets had no hyponyms or were below the frequency cut-off value. Some of the verb synonym sets which did not meet the criteria to become event classes may be mapped to other verb synonym sets using a “similar sense” function of WordNet 2.0 while the remaining synonym sets may pertain to relatively rare or more specific concepts, have very few or no hyponyms and may not be well suited as ontology classes.
Once a given concept is selected as an event class of the ontology, one embodiment of the disclosure provides a method to select the name of the event class. In one example, the lemma of the concept or synonym set which was found to be most frequently occurring from the analysis of the corpus compared to other lemmas of the concept may be selected as the name of the concept or synonym set. For example, for the concept or verb synonym set {communicate#2, intercommunicate#2}, the more frequently occurring lemma and sense “communicate#2” may be selected to name the concept responsive to it being more frequently occurring in the British National Corpus compared with “intercommunicate #2.”
Following the identification of an event class of the ontology, instances of the identified event class may be determined. In one example, the lemmas (and senses) of the concept or synonym set may be selected as instances (e.g., communicate#2 and intercommunicate#2) of the concept “communicate#2”. In addition, the troponyms (e.g., gesticulate#1, gesture#1, motion#1 of communicate#2; and grimace#1, make_a_face#1, pull_a_face#1 of intercommunicate#2) and hyponyms of the lemmas may also be selected as instances of the event class. Other methods may be used to select the instances in other embodiments.
In one example, the event classes and instances of the ontology may be developed using an OWL (Web Ontology Language) ontology described at http://www.w3.org/TR/owl-ref using Protégé described at http://protege.stanford.edu as an ontology editor environment and Jena described at http://jena.sourceforge.net as a semantic web framework in which to implement the ontology, handle reification, issue queries, and perform logical inference. The teachings of the references cited in this paragraph are incorporated by reference.
As mentioned above, lexical database ontologies may be generated for nouns, adjectives, adverbs, etc. using the method of
Referring to
Referring to
Initially, at a step S30, a lexical database ontology is accessed. An exemplary method of determining an appropriate ontology is described above with respect to
At a step S32, textual content to be analyzed may be accessed. In one example, the textual content may include one or more text items (e.g., documents, articles, correspondence, speech transcriptions, etc.) or other arrangements of textual content. Analysis of the textual content may be performed to disambiguate the subject matter of the textual content. At least some of the words of the textual context may include a plurality of senses which cause ambiguation as to the words' meanings. The disambiguation may facilitate searching of the textual content, increase the accuracy of search results compared with no disambiguation, facilitate analysis to associate a plurality of documents with one another (e.g., cluster documents using In-Spire ™) and/or improve the accuracy of association of documents.
At a step S34, word senses of the words of the textual content accessed in step S32 are determined. In one embodiment, classification models (also referred to as classifiers) may be used to identify the senses of the words. The following discussion with respect to classification models pertains to analysis of verbs but may also be applied to nouns, adverbs, or adjectives.
Classification models may be created for verbs of the English language in one embodiment. In a more specific exemplary embodiment, for each verb lemma, a classification model may be defined which predicts the sense of a verb of the textual content being analyzed according to the context of the content in which the verb lemma occurs.
In one embodiment, classification models may be defined using a training corpus of documents. In one example for defining classification models for use in analyzing documents of general subject matter, a SemCor training corpus discussed at http://www.cs.unt.edu/˜rada/downloads.html may be used. An OpenNLP MaxEnt implementation (http://maxent.sourceforge.net) of a maximum entropy classification algorithm described at Berger, A., S. Della Pietra and V. Della Pietra (1996) A Maximum Entropy Approach to Natural Language Processing. Computational Linguistics, volume 22, number 1, pages 39-71, may be used to define the classification models in one embodiment. The teachings of the references cited in this paragraph are incorporated by reference.
In one embodiment, contextual, syntactic and semantic information may be used to train and define classification models in one embodiment. Contextual information may be obtained in one example by including three tokens on each side of a verb lemma of a training corpus under analysis wherein the tokens do not cross sentence boundaries. Tokens may include word tokens as well as punctuation. Syntactic information includes grammatical dependencies (e.g., subject, object) and morphosyntactic features such as part of speech, case, number and tense. In one embodiment, a parser may be used to extract syntactic information. Details of one exemplary parser which may be used are described at Tapanainen, P. and Timo Järvinen (1997) A Nonprojective Dependency Parser, In Proceedings of the 5th Conference on Applied Natural Language Processing, pages 64-71, Washington D.C. Association for Computational Linguistics and www.connexor.com, the teachings of which are incorporated herein by reference. A sample output of a Connexor parser for a sentence “The engine throbbed into life.” is shown in Table 1.
As mentioned above, semantic information may also be used to define the classification models. Semantic information includes named entity types (e.g., person, location, organization) and hypernyms. In one embodiment, LCC's Cicero Lite described at http:www.languagecomputer.com/solutions/information_extraction/cicero_lite, the teachings of which are incorporated herein by reference, may be used to extract named entity types and replace strings identified as named entities (e.g., Joe Smith) with corresponding entity types (e.g., PERSON). Hypernyms may be retrieved from WordNet 2.0 in the described example. In one embodiment, only the hypernyms of sence 1 of lemmas are expanded but an entire hypernym chain (e.g., motor, machine, device, instrumentality, artifact, object, whole, entity) may be included.
Additional details of usage of contextual, syntactic and/or semantic information are described in Dang, H. T. and M. Palmer The Role of Semantic Roles in Disambiguating Verb Senses, Proceedings of the 43 Annual Meeting of the Association for Computational Linguistics, Ann Arbor, Mich., Jun. 26-28, 2005, and Kohomban, U. and W. Lee (2005) Learning Semantic Classes For Word Sense Disambiguation, Proceedings of the 43rd Annual meeting of the Association for Computational Linguistics, Ann Arbor Mich., the teachings of which are incorporated by reference. An example of a sample of resulting feature vectors which were both for training and recognition is provided in Table 2.
As the example of Table 2 indicates, combination of contextual, syntactic and semantic information types may result in a relatively large number of features. A feature selection procedure may be used to reduce a final feature set to a feature subset that is most relevant to verb class assignment for each verb lemma (e.g., some of the features may be noise for any particular lemma). One procedure is to score each potential feature according to a particular feature selection metric and to use the best k features. In one embodiment, an Information Gain selection metric, measuring the decrease in entropy when the feature is given versus absent may be used. Details regarding this procedure are discussed in Yang, Y. and J. O. Pedersen (1997) A Comparative Study on Feature Selection in Text Categorization, In Proceedings of the 14th International Conference on Machine Learning (ICML), pages 412-420; and Forman, G. (2003) an Extensive Empirical Study of Feature Selection Metrics for Text Classification, Journal of Machine Learning Research 3, pages 1289-1305, the teachings of which are incorporated herein by reference. It is believed that feature selection may be improved by developing a best subset selection procedure based on Information Gain by scoring subsets of features simultaneously rather than individual features thereby identifying high value feature combinations.
For noun disambiguation, different sets of features may be selected. For a noun under analysis, features may be selected as the noun and three tokens found on each side of the noun within sentence boundaries and all verbs within sentence boundaries (e.g., tokens may include words and punctuation), morphological information about all tokens chosen (e.g., part of speech, case, number), syntactic dependency of the noun, an indication of what dependents the noun has (e.g., “hasDet”), a specification of what verb is related to the noun, the syntactic dependency of the words dependent on the noun (e.g., ‘det:the’), and the hypernym chain of all nouns selected. Other methods may be used to select features of nouns in other embodiments.
Once the classification models are created for the respective words or lemmas, the words of the textual content are analyzed with respect to the classification models in an attempt to determine which of the senses of the words are appropriate corresponding to the context in which the words or lemmas are used in the textual content. The words of the textual content may be parsed, for example using a Connexor or other suitable parser. The parsed output may be processed as described above with respect to the creation of the classification model which identifies feature vectors for the words or lemmas. Thereafter, the procedure may match groups of features (e.g., feature vectors) of the word or lemma of the textual content with a closest one of the groups of features of the classification model for the word or lemma. The sense of the word or lemma associated with the selected or matched group of features may be selected as the sense for the word or lemma.
Referring again to
At a step S38, an annotation may be created using the identified event classes and the annotation associated with the textual content. The annotation may include the event classes which were identified for the respective words or lemmas of the textual content and the event classes provide disambiguated meanings of the words or lemmas of the textual content in one exemplary embodiment. As discussed above, an event class may include a word or word lemma and the respective sense thereof corresponding to the respective event class and which provides a more clear and/or generic meaning of the word or lemma being disambiguated in the textual content.
The method of
Referring to
In at least some of the above-described embodiments, accuracy of word sense disambiguation depends on the word sense annotations found in the training corpus and the lexical database ontology. Accordingly, the nature and quality of the corpus used to create classification models or different ontologies may impact the results of the disambiguation. The method of
The different word sense disambiguation models may include a plurality of different classifiers or classification models which may be generated using the teachings described above. However, the training corpus of documents may be changed during the training of the classification models corresponding to the word sense disambiguation models being generated. More specifically, in one embodiment, different types of classification models may be generated by different types of training corpus and corresponding to different subject matter domains of textual content to be analyzed. For example, if textual content to be analyzed concerns medicine, biology or sports, different classification models may be trained using medicine, biology or sports related training corpora, respectively, to increase the accuracy of the resultant analysis. According to additional aspects, corpora of different subject matter domains may be used to define a plurality of lexical database ontologies. For example, at step S12 of
At a step S40, textual content to be analyzed may be accessed. The textual content may be the same as the textual content described above with respect to step S30 of
At a step S42, the topical content or subject matter domain of the textual content being analyzed is determined. In one embodiment, text categorization techniques may be utilized to sort a set of documents into categories from a predefined set. During text categorization preparation, a training set for each domain of interest (e.g., general English, medical, sports, etc.) may be generated. Further, a text categorization model may be developed that, for each input text, predicts to which domain the input text belongs to. Using the text categorization model, the domain which is most appropriate for the topical content of the textual content being analyzed is identified. Finally, word sense disambiguation models (e.g., classification models) may be generated for each of the domains. Additional details of text categorization modeling are described in Fabrizio Sebastiani (2002) Machine Learning In Automated Text Categorzation, ACM Computing Surveys, volume 34, number 1, pages 1-47, available at http://www.math.unipd.it/˜fabseb60/Publications/ACMS02.pdf, the teachings of which are incorporated herein by reference.
At a step S44, the results of the domain prediction of step S42 may be used to predict which of the lexical database ontologies and/or word sense disambiguation models should be utilized for disambiguating the textual content being analyzed.
At a step S46, the textual content may be disambiguated and annotated using the selected lexical database ontology and/or classification models of the selected word sense disambiguation model.
In compliance with the statute, the invention has been described in language more or less specific as to structural and methodical features. It is to be understood, however, that the invention is not limited to the specific features shown and described, since the means herein disclosed comprise preferred forms of putting the invention into effect. The invention is, therefore, claimed in any of its forms or modifications within the proper scope of the appended claims appropriately interpreted in accordance with the doctrine of equivalents.
This invention was made with Government support under Contract DE-AC0676RLO1830 awarded by the U.S. Department of Energy. The Government has certain rights in the invention.
