Patent Grant
12135944
The present invention generally relates to context-sensitive keyword disambiguation services. More particularly, the invention relates to a novel technique for determining the semantic context for one or more homographs (i.e., words with identical written forms but varying definitions) in a corpus of data.
This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present techniques, which are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
In computational linguistics, homographs, which are words that can have multiple meanings, may present many challenges, as it may be difficult to identify the sense of the word as it used in a sentence. For example, the term “bat” could refer to a flying mammal or an object used in baseball. Systems designed to perform context-sensitive services may analyze and evaluate a corpus of text to determine the semantic context of one or more keywords within the corpus of text. For the context-sensitive services to be performed effectively, the system may need to know the semantic context of the words within the corpus of text. When encountering a homograph, the system may use a variety of tools to determine the semantic context of the homograph, e.g., to determine what definition or part of speech is intended. This issue is known as “word-sense disambiguation.”
One method for accomplishing word-sense disambiguation may be part of speech tagging. A model or algorithm may be programmed to determine the part of speech of a homograph based on the words surrounding the homograph. For example, the model or algorithm may analyze the sentences “Can you help me?” and “He kicked a metal can,” determine that the word “can” has different parts of speech in each sentence, and tag each with its proper part of speech. After performing part of speech tagging, the model or algorithm may analyze the sentences as “Can|VERB you help me?” and “He kicked a metal can|NOUN.” This may assist the model or algorithm in determining the semantic context of the homograph “can” the next time it is encountered. It should be noted that, in some embodiments, part of speech tagging may operate on every word in a sentence. For example, the sentence “He kicked a metal can” may be tagged as “He|PRON kicked|VERB a|ART metal|ADJ can|NOUN.”
However, an additional level of difficulty is encountered when a homograph has multiple semantic contexts with the same part of speech. In this case, part of speech tagging is ineffective. For example, the word “bat” has multiple semantic contexts with the same part of speech; i.e., “bat” may describe multiple different common nouns. A word-sense disambiguation algorithm that encounters the word “bat” may not know whether the word refers to the animal, a bat used in baseball or some other sport, or some other meaning.
A summary of certain embodiments disclosed herein is set forth below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of these certain embodiments and that these aspects are not intended to limit the scope of this disclosure. Indeed, this disclosure may encompass a variety of aspects that may not be set forth below.
In an embodiment, a data store, such as an electronic data corpus, may include an ambiguous word (e.g., words with multiple possible definitions). A word-sense disambiguation service may be performed on the electronic data corpus to disambiguate the ambiguous word by selecting one of the plurality of possible contexts for the ambiguous word and providing a context indication of the selected context for the ambiguous word. A context-sensitive service may receive the context indication for the ambiguous word from the word-sense disambiguation service, and perform a service that is dependent upon the context indication to provide a context-sensitive result.
In another embodiment, a word-sense disambiguation service may separate an electronic corpus of text into individual portions of text. The word-sense disambiguation service may then search the individual portions of text for occurrences of ambiguous target words by iteratively selecting a main word and creating a context window around the main word. The context window may have a main word (e.g., a target word), a number of possible context words occurring prior to the main word, and a number of possible context words occurring after the main word in the individual portions of text. The word-sense disambiguation service may then determine, for the context window created for each occurrence of the main word, which of the possible context words co-occur with the main word, and may filter, from the possible context words that co-occur with the main word, resulting in defining context words (e.g., context words that may be helpful in defining the target word). The word-sense disambiguation service may then identify one or more word clusters, where each word cluster may be associated with a particular context of the main word and each word cluster may contain the main word and the defining context words that co-occur with the main word in the context window, wherein the defining context words indicate that the ambiguous target word has the particular context associated with a corresponding cluster. In the final step of the iterative process, the word-sense disambiguation service may identify the main word as an occurrence of an ambiguous word having a plurality of possible contexts when two or more word clusters are identified as corresponding to the main word. Then, for each occurrence of the identified ambiguous target word, the word-sense disambiguation service may identify a corresponding one of the one or more word clusters, based upon the defining context words that co-occur with the occurrence and associate a disambiguation tag identifying the corresponding one of the one or more word clusters with occurrence.
In yet another embodiment, a context-sensitive service may be performed by receiving input from a user of the context-sensitive service, wherein the input is a keyword. The context-sensitive service may then determine whether the keyword is associated with a plurality of possible semantic contexts and, in response to determining that the keyword is in fact associated with a plurality of possible semantic contexts, render a graphical user interface (GUI) prompt requesting the user to select one of the plurality of possible semantic contexts. The context-sensitive service may then receive a selection indicating the semantic contexts that was selected, may search a data corpus for the keyword associated with the selected context, and finally may provide results of the search via the GUI. The search may refer to searching the data corpus for vectors similar to the vector generated and assigned to the selected semantic context.
These and other features, aspects, and advantages of the present disclosure will become better understood when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings, wherein:
One or more specific embodiments of the present disclosure will be described below. In an effort to provide a concise description of these embodiments, all features of an actual implementation may not be described in the specification. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it should be appreciated that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.
When introducing elements of various embodiment of the present disclosure, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of these elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements.
Context-sensitive services may be utilized in a variety of fields, such as in advertising, marketing, business, research and development, or the like. A context-sensitive service may involve parsing a corpus of text or other data to locate and identify a keyword or a set of keywords. An issue may arise when an algorithm encounters a homograph, or a word with multiple definitions and/or multiple parts of speech. If the intended semantic context of a word cannot be determined, the ability to provide context-sensitive service may be significantly impaired. The process of determining the semantic context of a word is known as word-sense disambiguation.
As previously discussed, part of speech tagging may be used to determine the context of a word by identifying the part of speech of a particular instance of the word. A word-embedding model may be programmed to determine the part of speech of a homograph based on the words surrounding the homograph. For example, the word-embedding model may analyze the sentences “Can you help me?” and “He kicked a metal can,” determine that the word “can” has different parts of speech in each sentence, and tag each with its proper part of speech. The word-embedding model may use the relationship between the words of a sentence with different parts of speech to identify the context of a word. For instance, the model may use the relationship between verbs, nouns, and adjectives to determine the part of speech of a word and tag the relevant part of speech accordingly. After performing part of speech tagging, the word-embedding model may analyze the sentences as “Can|VERB you help me?” and “He kicked a metal can|NOUN.” This may assist the word-embedding model in determining the semantic context of the homograph “can” the next time it is encountered. Alternatively, each word in a sentence or sentence clause may be tagged with its part of speech. For example, the sentence “He kicked a metal can” may be tagged as “He|PRON kicked|VERB a|ART metal|ADJ can|NOUN.” In some cases, tagging each word may provide more accurate results (e.g., upon a search by the context-sensitive service).
However, when a word has multiple contexts with the same part of speech, part of speech tagging is ineffective as a word-sense disambiguation tool. For example, the word “bat” may describe a flying mammal or an implement used in sports such as baseball and cricket. “Foot” may describe a unit of measurement or a body part. “Bass” may describe a type of fish or a low pitch in music. Part of speech tagging would not assist a word-sense disambiguation algorithm in determining the context of a word. One or more additional algorithmic processes may be used to determine the semantic context of a homograph that has multiple definitions with the same part of speech.
With the foregoing in mind,
To disambiguate words in the data corpus 102, the word-sense disambiguation service may be trained via training data 101. The training data 101 may be the data found in the data corpus 102 or may be independent data, used to identify different contexts of particular words. When training data 101 that is independent from the data corpus 102 is used, the training data 101 to process may be filtered based upon terms relevant to the data corpus 102. That is, training data 101 used by the word-sense disambiguation service 104 may be specifically tailored to the data corpus 102, by filtering out data not found in the data corpus 102. For example, the training data 101 that is available may include data related to elephants, but when the data corpus 102 does not include data related to elephants, the elephant data in the available training data 101 may be filtered out, resulting in more efficient processing of training data by the word-sense disambiguation service 104.
Once the word-sense disambiguation service 104 is performed and returns ambiguous keywords (e.g., keywords with multiple semantic contexts), the user of the context-sensitive service may select the desired context it wishes to target. Using this selected context, the context-sensitive service may analyze the data corpus 102 and identify instances of the ambiguous keywords having the selected context, and identify the segments in which those instances of the keywords occur as relevant to the user of the context-sensitive service.
As previously discussed, an issue may arise if the data corpus 102 contains homographs. For example, one of the target words chosen for the baseball equipment advertiser may be “bat.” However, the word “bat” may have several definitions and/or parts of speech. To address this issue, the present methods and systems may locate and analyze a set of target words and the words surrounding the target words (e.g., “context words”) to determine the context in which a particular target word is used. This process of determining the context in which a particular word is used may be referred to as word-sense disambiguation.
The process 200 may identify the context words by creating context windows around each instance of a target word, and analyzing the other words that occur within each context window. For example, if the target word is “bat,” each instance of “bat” is found in the data corpus. For each found instance of the word, the algorithm may identify a context window around the target word as a particular defined number of context words surrounding the target word. Each word that occurs within any context window may be identified by the process 200 as a potentially defining context word.
In process block 302 the corpus of text may be separated into sentences or sentence clauses. Each sentence or clause may be analyzed by the algorithm as a separate document.
Returning to
In process block 306, the first word in the sentence that remains after the initial filtering process is complete is located, and may set that word as the initial target word. In process block 308, a context window may be created around the first target word (e.g., 404A or “This” in
The context window and surrounding context words may be evaluated for the purpose of calculating co-occurrence statistics for the target word and the context words. In process block 310, after evaluating a window for the first target word, processing may move to the next remaining target word, and repeat the evaluation of process block 308 on that target word.
In the illustration 400 of the rolling context window operation in
Returning to
IDF may be utilized to filter words that occur too frequently in a corpus of text (such as the data corpus 102). Words that occur with a high frequency may not be useful in defining the context of a target word. Thus, IDF may be used to filter out words that occur too frequently.
For the purposes of IDF, each sentence or sentence clause may be treated as a separate document. IDF may be calculated by dividing the total number of documents (i.e., sentences or sentence clauses) by the number of documents in which a certain context word occurs. For example, the algorithm may analyze a corpus of text containing 1,000 documents (i.e., 1,000 sentences or sentence clauses). If the word “brought” occurs in 200 documents, the IDF will be 1000/200, or 5. However, if in the same corpus of text the word “swung” occurs in only 20 documents, the IDF will be 1200/20, or 50. Therefore, a lower IDF correlates to greater frequency throughout the corpus of text. A sufficiently low IDF may indicate that the context word occurs too frequently in the corpus of text, and thus may not be useful in identifying the context of a target word. The algorithm may be programmed with a threshold IDF (e.g., 10, 20, 50 etc.), at or below which a context word may be filtered out. Further, the algorithm may calculate an average IDF and filter out words with an IDF some degree lower than the average; e.g., may filter out words with an IDF that is 50% of the average IDF or less.
The second filtering technique that may be used in process block 202 is context word co-occurrence.
In process block 506, the context word co-occurrence is counted. Context word co-occurrence is a measure of how many times a context word appears in the context window of the specified target word. For example, if the word “time” occurs within one or more context windows of the word “bat,” it may be identified as a context word. However “time” may occur throughout the data corpus in instances outside of the context windows of “bat.” These instances may not be counted, as they do not co-occur with the target word.
In process block 508, the context word co-occurrence percentage may be calculated. This may be calculated by dividing context word co-occurrence (i.e., the sum of the operation in process block 506) by total context word count (i.e., the sum of the operation in process block 504). In process block 510, it may be determined whether the calculated context word co-occurrence percentage satisfies a threshold percentage (e.g., 10%, 51%, or 75%). If the context word co-occurrence percentage of a certain context word fails to meet the threshold percentage, the context word may occur too frequently outside of the context window of the given target word to be useful in defining the semantic context of the target word.
Table 1 below provides examples of context word co-occurrence calculations. In Table 1, the context word “baseball” may occur 100 times total throughout a corpus of text (e.g., the data corpus 102), and may co-occur (i.e., in a context window) with the target word “bat” 80 times. Thus, the context word co-occurrence percentage is 80% (80/100). In contrast, the word “time” occurs much more frequently than the word “baseball” throughout the corpus of text, occurring a total of 20,000 times. Additionally, “time” has a higher co-occurrence count than “baseball,” co-occurring with the target word “bat” 90 times. However, due to its prevalence throughout the corpus of text, the co-occurrence percentage for “time” is a mere 0.45% (90/20,000). If the algorithm in this example were set with a context word co-occurrence percentage threshold of anything above 0.45% (e.g., 1%), “time” would be filtered out.
Returning to
Additionally or alternatively, the algorithm may pinpoint word lemmas instead of the actual words themselves. For instance, the word “swung” would turn into “swing.” This may be especially useful in the situation where the corpus of data is not large enough, since different tenses of the same word will be assigned the same token.
In process block 208 of
In certain embodiments, whether or not the target word is ambiguous (e.g., has multiple meanings) may be determined by whether or not the target word has more than one associated cluster. For instance, if word-sense disambiguation is being performed on a data corpus (e.g., the electronic scripts of several seasons of a television program), the word “saxophone” may only have one associated context word cluster, as “saxophone” may only have one common definition. However, the word “bass” may have multiple associated context word clusters. The word-sense disambiguation service may identify, and thus create context word clusters for, instances of “bass” meaning “the lowest adult male singing voice;” “a four-stringed guitar;” and “a common freshwater perch.” Therefore, in some embodiments, it may be determined that a target word is ambiguous by the presence of multiple context word clusters associated with a single target word.
In process block 210, sets of defining context words for particular contexts may be identified by the generated clusters, where each cluster represents a particular context. For example, for the target word “bat,” the algorithm may identify the words that separately co-occur with the varying contexts of the word “bat.” One cluster may contain words relating to a baseball bat (e.g., “swung,” “baseball,” “glove”), while another cluster may contain words relating to the animal (e.g., “cave,” “wings,” “nocturnal”). In general, the algorithm may accomplish the identification by determining that the clusters do not share a sufficient number of defining context words; e.g., the context words “cave” and “wings” never co-occur with the context words “baseball” and “glove,” thus the algorithm may determine that the generated clusters represent more than one semantic context of the target word “bat.”
Cluster 650 may represent instances of the target word with a different context than the instances represented by cluster 610. For example, cluster 650 may represent the target word “bat” referring to a baseball bat. Similarly to sets associated with cluster 610, the sets 652 and 654 may represent individual context windows containing the defining context words identified by the algorithm through the filtering processes. For example, set 652 may represent a context window with the defining context words “baseball,” “swung,” and “slugger;” while set 654 may represent a context window with the defining context words “baseball,” “swung,” and “glove.” Again, the edges 656 and 658 indicate the same context word occurring in multiple context windows.
The identification of sets of defining context words described in process block 210 may be accomplished in several ways. For example, the algorithm may determine that the identifying context words “nocturnal” and “mammal” never appear in cluster 650 in
Alternatively, the target word may be tagged not with a key (e.g., “Bat1” or “Bat2”), but with the context words with which the target word is associated. For example, instead of being tagged as “Bat1,” an instance of “bat” referring to the animal may be tagged as “Bat|NOUN (cave|NOUN wings|NOUN nocturnal|ADJ).”
In process block 702, a particular cluster associated with a target word may be identified from a set of clusters associated with the target word, the identified cluster indicating a particular context of the target word. Returning to the example used for
In process block 704, a disambiguation tag indicating the identified cluster and the associated context of the target word may be generated. For example, the algorithm may recognize, using the process 200 described in
In process block 706, the generated disambiguation tag may be associated with the target word. For example, if the algorithm receives the word “bat” as the target word, it may then associate the disambiguation tags indicating identifier Bat1 or Bat2 with the received target word based on the corresponding identifying context words. An illustration 1000 of this process may be seen in
A word-embedding model may analyze a word and represent the word as a vector of a real number. The mapping of a word to a vector may be accomplished using a number of methods such as neural networks and probabilistic models, among others. In the case of homographs, the word-embedding model may represent the same word (e.g., the target word “bat”) as multiple different vectors based on the context. Continuing with the above example, an instance of the target word “bat” correlating to the disambiguation tag Bat1 may be represented as one vector, while an instance of the word “bat” correlating to the disambiguation tag Bat2 may be represented as a different vector. Additionally, the contexts words correlating to their associated target word may have a vector similar to the vector of the target word. For example, “baseball” may have a vector that is similar to the instances of “bat” meaning a baseball bat. Further, the vectors for “baseball” and “bat” may be similar to the vectors for “glove,” “base,” and “swung.” Each of these vectors may be dissimilar from the vectors for “bat” indicating the animal, as well as vectors for “cave,” “wings,” etc.
The training of the word-embedding model in process block 802 may consist of teaching the model which words are associated with which disambiguation tags. For example, the word-embedding model may learn (e.g., through machine learning) that the context words “baseball” and “swung” correlate to disambiguation tag Bat2. Thus, when those words appear in a context window with “bat,” that instance of “bat” may be associated with Bat2.
In process block 804, the algorithm, using the trained word-embedding model, may merge separate context word clusters if corresponding vectors are sufficiently similar. However, the clusters may not always be properly separated. Indeed, multiple clusters may be generated for the same semantic context of a target word. For example, a cluster may be formed for “bat” corresponding to Bat2 (e.g., a bat used in sports such as baseball) with context words like “baseball,” “base,” and “swung.” Another cluster may be formed containing context words such as “wooden,” “implement,” and “swung.” Upon initially associating the target word with a disambiguation tag as described in
Once the word-embedding model is trained, the algorithm may determine that the cosine similarity (i.e., the measure of similarity between two non-zero vectors) of the two clusters is sufficiently high. If the cosine similarity between the two context word clusters is indeed sufficiently high, the algorithm may merge the two clusters. The cosine similarity may be based on a similarity in context words; i.e., a larger number of context words shared by both clusters may correlate to a higher cosine similarity between the clusters. Continuing with the above example, the word-embedding model may merge the clusters based on the fact that “swung” appears in both clusters.
If the algorithm does not merge two or more context word clusters with the same semantic context, the clusters may be merged manually. This may occur if there are not enough examples of the given target word in the data. Gathering more data and analyzing different parts of speech of the target word may resolve this issue.
In process block 806, the word-embedding model is retrained on the merged context word clusters. Once multiple clusters with sufficiently high cosine similarity are merged, the context words may be associated with a relevant disambiguation tag so as to be identified with the semantic context of the target word. Continuing with the example set forth above, the second cluster would be assigned the same disambiguation tag Bat2 as the first cluster. Thus both the first and second clusters would be associated with Bat2. The word-embedding model would then be retrained to identify the context words “wooden,” “implement,” “swung,” “baseball,” and “base” as corresponding to Bat2.
In process block 1102, an input indicative of a defined target word having two or more contexts may be received. For example, an advertiser may wish to have an advertisement play during a television program involving a baseball game. As previously stated in the discussion of the word-sense disambiguation service 104 in
In process block 1104, an identification of context word clusters and their defining context words may be retrieved for the words having multiple semantic contexts. For example, clusters 610 and 650 may be retrieved, where the cluster 610 may refer to the context of “bat” meaning the animal, as well as the defining context words corresponding to cluster 610; e.g., “nocturnal,” “wings,” etc. Similarly, the cluster 650 may be retrieved, where “bat” refers to a baseball bat, and defining context words such as “baseball” and “swung.” In an aspect, the client may provide a grouping of keywords “baseball,” “bat,” “glove,” “homerun,” such that one keyword within the group may provide context for one or more other keywords within a group. For example, “baseball,” “glove,” and “homerun” may provide the context for “bat” so that the system may determine that, between the tags “Bat1” and “Bat2,” “Bat2” is the more likely match given the context. This matching may be associated with a confidence level that can be a function of the number of other target words (e.g., baseball, glove, homerun) that also show up in the context cluster of a first target word (e.g., “bat”).
In process block 1106, an affordance for selecting one or more of the context clusters to apply to the received input may be provided. In an example, if the confidence level is above a threshold (e.g., 90%), then the affordance may simply present the highest ranked context cluster as the recommended context cluster to the advertiser (or the user) rather than providing all of the context clusters while giving the user an option to see additional context clusters if the recommended cluster is incorrect.
For example, if a client is a caving adventure service provider, the client may wish to have an advertisement play during a television program featuring the word “bat” in the context of the animal, and thus the client may select affordance 1204A associated with the disambiguation tag “Bat1.” Based on the selection of the affordance 1204A, the context-sensitive service 106 may perform a search on the data corpus 102, identifying portions of the data corpus 102 that may include the target word with the desired context indicated by the selection of the affordance 1204A. As previously stated, a word-embedding model may assign a vector to each instance of an ambiguous target word and associated context words to indicate the semantic context. Upon receiving a selection of the affordance 1204A (e.g., the affordance associated with the disambiguation tag “Bat1”), the context-sensitive service may search through the data corpus 102 to identify words with a vector similar to the vector representing “Bat1” (e.g., indicating greater cosine similarity). For example, if affordance 1204A is selected, the context-sensitive service 106 may identify vectors representing the context words “wings,” “cave,” “fruit” and “nocturnal” based on their similarity to the vector representing “Bat1.” The context-sensitive service 106 may then return results based on the vectors representing the context words. For example, the context-sensitive service 106 may provide, in a search result GUI 1210, an affordance 1212A that may correspond to a script for a vampire movie, or may provide an affordance 1212B that may correspond to a script for a cave exploration movie. If the client is a caving adventure service provider, the client may be interested in advertising their products and/or services during the cave exploration movie, and thus may select search result affordance 1212B.
Similarly to
By employing the techniques described in the present disclosure, the systems and the methods described herein may allow for the efficient and accurate performance of the word-sense disambiguation service 104 and context-sensitive services 106. The word-sense disambiguation algorithm may break the data corpus 102 down into sentences and evaluate each sentence as its own document. The algorithm may then filter the words in each sentence based on part of speech to remove unnecessary tokens (e.g., articles, punctuation, etc.). The algorithm may then create rolling context windows 406 for each sentence and compute context word co-occurrence for each target word. The algorithm may then use co-occurrence statistics to filter out context words that are not useful in defining the context of the target word. The algorithm may then generate context word clusters (e.g., 610 and 650) using the defining context words that remain after the filtering processes, associating each instance of the context word and the target word with a disambiguation tag corresponding to the associated context word cluster. The algorithm may then train a word-embedding model on the defining context words based on their associated disambiguation tags. The algorithm may then merge two or more context word clusters if their cosine similarity is sufficiently high, and train the word-embedding model on the merged context word clusters. The context-sensitive service 106 may then generate affordances (e.g., 1204, 1404) in a GUI (e.g., 1202, 1400) based on the context word clusters 610 and 650. The context-sensitive service 106 may, upon receiving selection of the affordance 1204 or 1404, return portions of the data corpus 102 as search results based on the selection.
While only certain features of the present disclosure have been illustrated and described herein, many modifications and changes will occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the embodiments described herein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7869989 | Harvey | Jan 2011 | B1 |
| 10460229 | Yuan | Oct 2019 | B1 |
| 20060136223 | Brun | Jun 2006 | A1 |
| 20080065621 | Ellis | Mar 2008 | A1 |
| 20100004925 | Ah-Pine | Jan 2010 | A1 |
| 20110251839 | Achtermann | Oct 2011 | A1 |
| 20130013400 | Kim | Jan 2013 | A1 |
| 20180157644 | Mandt | Jun 2018 | A1 |
| 20180285781 | Mizobuchi | Oct 2018 | A1 |
| 20210027771 | Hall | Jan 2021 | A1 |
| 20210150631 | Resheff | May 2021 | A1 |
| 20220215047 | Banipal | Jul 2022 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20220405482 A1 | Dec 2022 | US |
