Patent Grant
8676795
Due to the number of sources in which data and/or other information is provided, disseminated, and otherwise available, it has become increasingly difficult to monitor these sources in an attempt to identify information of interest. Moreover, information sources, such as email providers, blogs, news sites, and social media networks (e.g., Twitter®, Facebook®, Flickr®, etc.), etc., may provide a wealth of constantly changing information. As a result of the growing number of these sources, and the vast amount of information generated by these sources, existing systems are unable to efficiently identify specific information on an ongoing basis.
For instance, existing methods may collect data relating to the performance, latency, and/or customer behavior/feedback associated with a system and then utilize this data to develop new features for the system and/or to correct identified problems of the system. More particularly, the collection of data and actions taken based on this data may be automated, such as by writing software code. In the above example, the software code may be written so that the software performs some type of function in response to one or more instructions. However, since the author of the software code is generally aware of the type of information being sought, the foregoing methods may be limited to identifying and collecting data that is expected to be available. Therefore, because it is very difficult, if not impossible, to know what type of information or documents will be provided via the information sources noted above, existing systems may be unable to detect what topics and content are currently being mentioned and/or discussed on these information sources.
The detailed description is set forth with reference to the accompanying figures, in which the left-most digit of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in the same or different figures indicates similar or identical items or features.
This disclosure describes processing natural language documents from one or more sources and visually representing the phrases to a user based at least in part on the frequency in which the phrases appear within documents associated with the one or more sources and/or the number of sources in which those documents appear. In particular, phrases within documents associated with one or more document sources may be extracted and filtered such that the phrases are likely to represent topics and/or the content of their respective documents. The filtered phrases may be processed to determine a frequency at which the phrases appear in the documents and the processed documents are then are weighted (e.g., scored) accordingly. Further, the filtered phrase may also be processed to determine the frequency at which the filtered phrases appear in different document sources. Once the phrases are processed and weighted, a visual representation of the subset of phrases may be output to a computing device whereby a user interface may visually present the phrases to a user. Optionally, the table depicting information associated with the represented phrases may also be displayed either independently or in conjunction with the visual representation of the phrases.
In various embodiments, a system may utilize one or more algorithms to process documents from a variety of sources in order to extract meaningful phrases (e.g., nouns, verbs, adjectives, etc.) from these documents. The phrases may then be weighted based at least in part on their respective importance. In some embodiments, the importance of a particular phrase may be determined by the frequency in which that phrase appears in a single document, multiple documents from a particular source, each of the documents within that source, and/or each of the documents associated with multiple sources. The importance of a phrase may also depend upon the number of document sources in which that phrase appears. Moreover, the system may render the phrases into a document (e.g., an HTML document) for display to a user via a user interface (e.g., web browser). In certain embodiments, each phrase within the visual representation may be emphasized (e.g., variations in size, font, color, etc., of each phrase) to indicate an overall frequency of the phrase among the document sources and/or the number of documents sources in which the phrase appears. In addition, a table indicating this information, such as identification of the phrases and their corresponding weights, may be displayed to the user in a phrase table.
Furthermore, any type and/or source of information that can be organized into one or more documents may be processed in the manner described above. Examples of document sources include e-mail messages, blog posts, news articles, social media information (Twitter® tweets, Facebook® posts and/or comments, etc.), and any other source that may create or provide access to natural language documents. In various embodiments, each document source may be periodically polled or continuously monitored to determine the existence of new documents and, as additional phrase extracted and processed, the weight of each phrase may change.
For instance, since the weight of phrases may depend on their frequency within the documents, due to the presence of new phrases or additional instances of previously identified phrases, the weights associated with the new phrases and the existing phrases may be updated. As weights for new phrases are calculated and as the scores for the existing phrases are periodically updated, the representation of the phrases (e.g., text, images, symbols, etc.) within the visual representation may also change (e.g., increase/decrease in size, change colors, font variation etc.). As a result, the visual representation of the phrases identified in the documents may constantly evolve and change based on the frequency in which the phrases appear in those documents and the documents sources associated therewith. That is, the systems described herein may display a snapshot of what phrases are currently trending (e.g., being mentioned, described, and/or discussed) amongst a multitude of different information sources at any particular point in time.
The discussion begins with a section, entitled “Example Environment,” describing a system for providing a visual representation of phrases extracted from various document sources. The discussion then moves to a “Phrase Analysis Service” section that describes components and functions associated with the phrase analysis service. Subsequently, a “Providing Phrase Data” section describes the generation and production of phrase data. Next, the discussion includes a section, entitled “Phrase Extraction,” that describes a system for processing phrases extracted from documents associated with various document sources. The discussion then moves on to a “Visual Representation of Phrase Data” section that describes a system for visually representing the phrases based at least in part on weights assigned thereto. The discussion then includes a section, entitled “Example Processes,” that illustrates and describes example processes for implementing the described techniques. Lastly, the discussion includes a brief “Conclusion”.
This brief introduction, including section titles and corresponding summaries, is provided for the reader's convenience and is not intended to limit the scope of the claims, nor the proceeding sections. Furthermore, the techniques described above and below may be implemented in a number of ways and in a number of contexts. Several example implementations and contexts are provided with reference to the following figures, as described below in more detail. However, the following implementations and contexts are but a few of many.
Example Environment
In various embodiments, the phrase analysis service 110 may monitor and/or may poll one or more document sources 106 to identify documents 118 associated with those document sources 106. From the identified documents 118, the phrase analysis service 110 may extract phrases and filter those phrases to determine meaningful words that substantively represent topics and/or the content of those documents 118. The phrase analysis service may then process the filtered phrases to determine which phrases are trending (e.g., being commonly referred to) at a particular point in time. To determine which phrases are currently trending, the phrase analysis service 110 may determine how frequently the phrases occur in the documents 118. The phrase analysis service 110 may also determine the number of document sources 106 in which each phrase appears. For instance, for a particular phrase, the system 100 may determine the number of times that the phrase has been mentioned amongst the documents 118 and document sources 106 and the number of different document sources 106 in which that phrase appears.
In further embodiments, for the phrases that appear most frequently in the documents 118, those phrases may be visually represented (e.g., via a web browser) such that a user may determine which phrases are trending at a particular point time. To determine which phrases have been most commonly referred to, and/or to determine which phrases have appeared in the greatest number document sources 106, the visual representation of each phrase may vary in appearance. For instance, each phrase may vary in size, font, color, or in any other manner, to indicate the importance of that particular phrase. By viewing this visual representation, the user 102 may determine which phrases are currently being mentioned and/or discussed more than other phrases in the documents 118 and the various document sources 106 (e.g., blog posts, news articles, e-mail messages, information associated with social media applications, etc.).
In some embodiments, the user 102 may operate the computing device 104 to perform various functions associated with the computing device 104, which may include the one or more processor(s) 112, the memory 114, and the display 116. Furthermore, the computing device 104 may receive the visual representation of the processed phrases and display those to the user 102. The computing device 104 may be a cellular phone, a personal digital assistant (PDA), a laptop computer, a tablet device, an electronic book (e-Book), or any other computing device 104. Moreover, the user 102 may use the computing device 104 to access the network 108. For instance, the user 102 may utilize the computing device 104 to access and navigate between one or more web pages or web documents that may be of interest to the user 102, such as the visual representation of the phrases and the corresponding phrase table, as described above. The computing device 104 shown in
The processor(s) 112 of the computing device 104 may execute one or more modules and/or processes to cause the computing device 104 to perform a variety of functions, as set forth above and explained in further detail in the following disclosure. In some embodiments, the processor(s) 112 may include a central processing unit (CPU), a graphics processing unit (GPU), both CPU and GPU, or other processing units or components known in the art. For instance, the processor(s) 112 may allow the visual representation of phrases output by the phrase analysis service 110 to be displayed to the user 102. Additionally, each of the processor(s) 112 may possess its own local memory, which also may store program modules, program data, and/or one or more operating systems.
In at least one configuration, the memory 114 of the computing device 104 may include any components that may be used to display a visual representation of phrases processed by the phrase analysis service 110. Depending on the exact configuration and type of the computing device 104, the memory 114 may also include volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, miniature hard drive, memory card, or the like) or some combination thereof.
In various embodiments, the computing device 104 may also have input device(s) such as a keyboard, a mouse, a pen, a voice input device, a touch input device, etc. The computing device 104 may also include the display 116 and other output device(s), such as speakers, a printer, etc. The user 102 may utilize the foregoing features to interact with the computing device 102 and/or the phrase analysis service 110 via the network 108. More particularly, the display 116 of the computing device 104 may include any type of display known in the art that is configured to present (e.g., display) information to the user 102. For instance, the display 116 may be a monitor communicatively coupled to the computing device 104 such that when the computing device 104 receives data from the phrase analysis service 110 via the network(s) 108, the computing device 104 may render this data for presentation to the user 102 via the display 116.
In further embodiments, the document sources 106 may include any type of information source that contains, distributes, and/or provides access to natural language documents (e.g., the documents 118). That is, the document sources 106 may include any database, website, and/or any other source that is associated with electronic information. For instance, the document sources 106 may include any source that generates, transmits, provides access to, or is otherwise associated with the documents 118. Moreover, the document sources 106 may store or archive the documents 118 and may also generate, transmit, and/or provide access to new documents 118 when the new documents 118 are received or generated. In various embodiments, the documents 118 may include blog posts, news articles, news alerts, email messages, instant messages, word-processing documents, posts or comments associated with any social media site (e.g., Facebook®, Twitter®, etc.), and any other type of natural language document. Moreover, the phrase analysis service 110 may, at any time, access the documents 118 via the network(s) 108 and extract phrases from within the documents 118.
In some embodiments, the network(s) 108 may be any type of network known in the art, such as the Internet. Moreover, the computing device 104, the document sources 106, and the phrase analysis server 110 may communicatively couple to the network 108 in any manner, such as by a wired or wireless connection. The network(s) 108 may also facilitate communication between the computing device 104, the document sources 106, and/or the phrase analysis server 110 and also allow for the transfer of data therebetween. For instance, the phrase analysis service 110 may access the document sources 108 and the documents 118 via the network(s) 106 and also extract phrases from the documents 118 utilizing the network(s) 108. The phrase analysis server 110 may also output data, such as the scored phrase data, that may be rendered by the computing device 104 and/or presented to the user 102 using the display 116 of the computing device 104.
In addition, and as mentioned previously, the phrase analysis service 110 may include one or more processor(s) 120 and a memory 122, which may include the extraction module 124, the extracted phrases module 126, the filtering module 128, the SIP module 130, the scoring module 132, and the output module 134. The phrase analysis service 110 may also include additional components not listed above that perform any function associated with the phrase analysis service 110. The phrase analysis service 110 may be a server or may be stored on a server, such as a cloud server, for example. Moreover, in some embodiments, the phrase analysis service 110 may access the document sources 106 for the purpose of extracting one or more phrases from the documents 118 associated with the document sources 106. These documents 118 may include documents 118 that were existing at the document sources 106 or documents 118 that have recently become available. Furthermore, the phrase analysis service 110 may periodically monitor, poll, and/or access each document source 106 to determine whether new or additional documents 118 are available. If so, the phrase analysis service 110 may access those new documents 118.
Upon accessing the documents 118 from the multiple document sources 106, the extraction module 124 of the phrase analysis service 110 may extract phrases from the documents 118. In various embodiments, each phrase may include one word or multiple words with each word having any number of characters. Moreover, each word may be a noun, a verb, an adjective, and/or any other type of speech identified in the documents 118 (e.g., pronouns, adverbs, etc.). In some embodiments, the phrases may include one or more words that relate to content and/or describe topics corresponding to the documents 118 from which those phrases are extracted. Furthermore, any number of phrases may be extracted from the documents 118 and the number of phrases that are extracted may depend upon the number of document sources 106 accessed by the phrase analysis service 110 and the number of accessible documents 118 associated with each document source 106. The phrases may be extracted from any documents 118 associated with a particular document source 106, such as archived documents, documents that are currently available at the document source 106, and/or newer and recently released documents 118.
Once the phrases are extracted from the documents 118 associated with the document sources 106, the phrase analysis service 110 may store the extracted documents 118 in the extracted phrases module 126. As the extraction module 124 continues to extract additional phrases from the documents 118, more extracted phrases may be stored in the extracted phrases module 126. The extracted phrases may remain in the extracted phrases module 126 until processed or various phrases within the extracted phrase module 126 may be periodically deleted or removed.
In other embodiments, the filtering module 128 may filter the extracted phrases stored in the extracted phrases module 126. Moreover, since the phrases extracted by the extraction module 124 may include any type and/or number of words, the extracted phrases may include phrases that do not convey meaningful information. For instance, phrases that do not describe the content or topics relating to a particular document 118, phrases in a foreign language (e.g., not in the English language), and/or other phrases that do not otherwise convey meaningful content may be filtered out such that the number of extracted phrases is reduced. The filtering module 128 may utilize one or more rules to remove such phrases prior to the extracted phrases being processed by the SIP module 130, which will be described in detail with respect to at least
Therefore, the filtering module 128 may filter the extracted phrases to eliminate highly unique or other phrases that are unlikely to accurately describe topics and/or content of the documents 118 in which those phrases were extracted. Furthermore, the filtering module 128 may also remove phrases that include more common words (e.g., and, about, regarding, etc.) and grammatical terms that are also unlikely to convey the subject matter being mentioned and/or described in those documents 118. As a result, once the extracted phrases are filtered by the filtering module 128, there may be a higher likelihood that the remaining extracted phrases include phrases that are sufficiently unique to substantively describe the document 118 from which those phrases were previously extracted.
The phrases that have been filtered by the filtering module 130 may then be processed by the SIP module 130. In some embodiments, the SIP module 130 may process the filtered phrases to determine SIPs (statistically improbable phrases) for the documents 118 associated with the filtered phrases. For the purpose of this discussion, SIPs may refer to the most distinctive phrases in the text of each of the documents 118 that appear in a higher frequency than in other documents 118. To identify whether each extracted phrase is a SIP, the SIP module 130 may determine the number of times the phrase appears in the text of the document 118 in which it was extracted, versus the frequency in which that phrase appears in other documents 118. If the phrase appears more frequently in a particular document 118 relative to the other documents 118, that phrase may be referred to as a SIP. Moreover, the phrases that are determined to be SIPs are not necessarily improbable within a particular document 118, but they may be improbable relative to the other documents 118. For instance, the phrase “laptop computer” may appear in a particular document 118 a great number of times but may rarely be present in other documents 118. As a result, this phrase would likely be a SIP with respect to this particular document 118.
In various embodiments, the SIP module 130 may employ one more algorithms that are configured to identify phrases that appear in a particular document 118 at a higher frequency or concentration than in other documents 118. Theses phrases may also appear in other documents 118 but at a lower frequency or concentration than the particular document 118 mentioned above. That is, the SIP module 130 may determine how frequently different phrases appear in each document 118. The SIP module 130 may determine these frequencies or concentrations by generating statistics relating to phrases that appear in a first document 118 much more than those phrases appear in a second document 118 and/or all other documents 118.
As a result, for each filtered phrase, the SIP module 130 may measure the relative frequency by which the phrase appears in a first document 118 compared to other documents 118. Since this phrase appears more frequently in the first document 118, the phrase analysis service 110 may determine that the first document 118 is more likely to contain content and/or topics relating to this phrase than the other documents 118 in which that phrase appears less frequently. For instance, assume that the phrase “laptop computer” appears in a first document 118 (e.g., an email message, a news article, a blog post, etc.) seventeen times whereas this phrase appears in the other documents 118 (e.g., an email message, a news article, a blog post, etc.) no more than five times. Since the concentration of the phrase “laptop computer” is much higher in the first document 118, there may be a much higher likelihood that the first document 118 relates to laptop computers. Although the other documents may mention the phrase “laptop computers” one or more times, because this phrase is mentioned at a much lower frequency, there may be a lesser likelihood that the topic and/or content of the other documents 118 is primarily related to laptop computers.
Therefore, the SIP module 130 may determine a frequency of phrases in a particular document 118 as compared to a subset of other documents 118, or all other documents 118 associated with the document sources 106. The SIP module 130 may also measure the relative frequency of phrases from one document source 106 to another document source 106. That is, for each filtered phrase, the SIP module 130 may determine whether a particular phrase appears within documents 118 from one document source 106 (e.g., email messages) or from multiple document sources 106 (e.g., email messages, blogs, news sites, social media sites, etc.). For instance, the SIP module 130 may consider each of the documents 118 within the multiple document sources 106 and determine that a particular phrase appears in documents 118 across multiple document sources 118 or appears in documents 118 within only one document source 106. Therefore, the SIP module 130 may apply its corresponding algorithm(s) to multiple document sources 106 to determine whether a particular phrase is being mentioned frequently but within only one document source 106 or whether this phrase is being discussed and described more generally across multiple sources of information.
Accordingly, in various embodiments, the SIP module 130 may process the filtered phrases to determine (1) a frequency in which phrases occur within a particular document 118, multiple documents 118 within a particular document source 106, and/or documents 118 from multiple document sources 118; and (2) the number of document sources 106 in which each phrase appears. In other words, the SIP module 130 may process each phrase to determine the frequency in which the phrases appear in the documents 118 in which they were extracted. The SIP module 130 may then process each phrase to determine the frequency in which the phrases appear within documents 118 from other document sources 106.
As stated above, SIPs may refer to unusual but frequently occurring phrases that appear in a document 118 relative to other documents 118. The SIP module 130 may attempt to identify and pare down the SIP phrases so that the number of SIP phrases within the documents 118 may be determined. To accomplish this, the SIP module 130 may determine the SIP phrases for a particular document 118 and then determine the frequency that these SIP phrases occur in other places, such as other documents 118 within a particular document source 106 and/or documents 118 within multiple different document sources 106. The SIP module 130 may also filter out SIP phrases that are unlikely to convey meaningful content or that are unlikely to relate to a topic and/or content of a document 118. As the SIP module 130 continues to process the newly extracted and filtered phrases, the phrase analysis service 110 may identify trends and/or velocity changes of the SIP phrases. That is, the phrase analysis service 110 may be able to determine whether phrases are appearing more or less frequently across a variety of different document sources 106, which may indicate the importance and/or popularity of those phrases at a particular point in time.
In some embodiments, the SIP module 130 may utilize a SIP algorithm. However, the SIP module 130 may also utilize any other algorithm, method, and or technique to determine a frequency or concentration in which a phrase appears in different documents 118 and document sources 106.
Once the extracted and filtered phrases are processed by the SIP module 130, the scoring module 132 may assign a weight (e.g., score) or multiple weights to some or all of the SIP phrases. For the purposes of this discussion, the terms “score” and “weight”, and any variations thereof, may be used interchangeably. The weight of each SIP phrase may be based at least in part on a frequency in which the phrases appear in the documents 118 extracted by the extraction module 124 and/or the number of documents sources 106 in which each phrase appears. In various embodiments, phrases that appear more frequently in the extracted documents 118 and/or that appear in a greater number of document sources 106 may be assigned a greater weight. Moreover, SIP phrases having a higher weight may indicate a relative importance of the phrase. For instance, a phrase having a higher weight may indicate that the phrase is being mentioned and/or discussed at a higher frequency than other phrases, possibly meaning that a topic or content relating to this phrase is becoming more relevant. Furthermore, if the weight of the phrase represents that the phrase is also being mentioned and/or discussed in multiple information sources, this may be further evidence that topics relating to this phrase are being more commonly mentioned and/or discussed by various individuals and/or entities. In addition, as additional phrases are extracted, filtered, and then processed by the SIP module 130, the new phrases may be weighted against the phrases that have already been processed and weighted. Since the additional phrases may increase the frequency of existing phrases, the weights associated with the existing phrases may periodically change. For example, if the phrase “laptop computer” has already been assigned a weight, this phrase appearing in more recently available documents 118 may cause the weight to increase.
In further embodiments, after the phrases are weighted or scored by the scoring module 132, the output module 134 may output the scored phrase data to the computing device 104 via the network(s) 108. Alternatively, the output module may output a visual representation of the scored phrase data and optionally, a table including the scored phrase data, to the computing device 104 for presentation via the display 116. The visual representation of the scored phrases may take any form, and will be discussed in additional detail with respect to
The system 100 shown in
Phrase Analysis Service
As mentioned previously with respect to
In various embodiments, the extraction module 124 may extract phrases from one or more documents 118 (e.g., e-mail messages, blog posts, news articles, etc.) from a variety of document sources 106 (e-mail server, blog sites, news sites, etc.) and may store those phrases in the extracted phrases module 126. The phrases may include one or any combination of words, letters, characters, and/or other symbols. Moreover the phrases may be one or any combination of nouns 202, verbs 204, and/or adjectives 206. In various embodiments, the phrase analysis service 110 may be configured such that only nouns 202, verbs 204, or adjectives 206 may be extracted as phrases. Moreover, the phrase analysis service 110 may also select any combination of nouns 202, verbs 204, and adjectives 206 as phrases.
As described with respect to
For instance, the dictionary module 208 may be applied to help make sure that each phrase includes at least one word that appears in an English language dictionary. In various embodiments, the dictionary module 208 may specify that the dictionary being utilized should include at least a predetermined number of words (e.g., 75,000), which may be any number of words. This may ensure that at least one word of a particular phrase is an established word in the English language and, therefore, is more likely to appear in other documents 118. If a particular phrase does not include at least one word in a dictionary having a predetermined number of words, that particular phrase may be removed and not processed by the SIP module 130. Otherwise, that phrase may remain and be addressed by the remaining modules of the filtering module 128.
In further embodiments, the phrase count module 210 of the filtering module 128 may determine whether each phrase is at least a predetermined number of words in length (e.g., two words) and/or no more than a predetermined number of words length (e.g., four words). The phrase count module 210 may help ensure that the phrase does not contain too few words, thereby allowing the phrase to appear in too great a number of documents 118 and/or document sources 106. Furthermore, the phrase count module 210 may help ensure that the phrase does not contain too many words, which may avoid the phrase only appearing in a limited number of documents 118 and/or document sources 106. In various embodiments, any number of words may be specified and if the amount of words in a particular phrase does not fall within the specified length parameters, that phrase may be removed.
The word count module 212 of the filtering module 128 may determine whether each phrase includes any words that are greater than predetermined number of characters in length (e.g., fourteen). In example embodiments, the predetermined number may be any number. Moreover, if at least one word in a particular phrase exceeds this predetermined number, the phrase may be removed and not subsequently processed by the SIP module 130.
In addition, the stop phrase module 214 may determine whether the phrase includes any words that may be classified as “stop phrases” or “stop words.” For the purpose or this disclose, stop phrases or stop words may refer to words and/or characters that indicate a stopping point in a document 118. For instance, a stop phrase may indicate the end of a sentence, the end of a paragraph, the end of a page, or the end of the document 118. The stop phrases may also refer to phrases that begin or end with a stop word, such as “and”, “it”, “or”, etc., or numbers. As a result, these stop phrases may be less likely to convey meaningful content and, therefore, may be less likely to represent topics and/or content associated with the document 118 in which the phrase was extracted. Furthermore, if the stop phrase module 214 determines that a particular phrase includes one or more stop phrases, the filtering module 128 may remove that phrase. In further embodiments, the filtering module 128 may include any number of additional modules that further filter the extracted phrases.
Once the extracted phrases have been filtered by the filtering module 128, the filtered phrases may then be processed by the SIP module 130. As mentioned previously with respect to
Following processing at the SIP module 130, the SIP phrases may be weighted or scored by the scoring module 132. In some embodiments, the scoring module 132 may assign a weight to each phrase based on their respective frequency determined by the SIP module 130. For instance, phrases that appear more frequently across the document sources 106 and/or that appear in a greater number of different document sources 106 may be weighted higher than other phrases. Furthermore, the weights associated with the phrases may constantly, or periodically, be updated. For example, as additional instances of existing phrases are extracted and processed, the weights for those phrases may be increased. On the contrary, the weight for existing phrases that have not recently appeared in the various document sources 106 may remain the same. Further, newly identified phrases may be weighted and when those phrases become more frequent, the weights associated with the newly identified phrases may surpass those of existing phrases. As a result, weighting of the phrases allows the phrase analysis service 110 to determine which phrases are most commonly being referred to or discussed within the monitored document sources 106 at any particular point in time.
As illustrated in
Once the processed phrases are weighted by the scoring module 132, the collection component 216 may compile the weighted phrases into a single file or a location stored in the memory 122. Subsequently, the ranking module 218 may rank the weighted phrases based at least in part on the weights assigned to each phrase. For instance, the ranking module 218 may rank the phrases in a hierarchical manner, meaning that the phrases with the highest weights are ranked first and the phrases with the lowest weights are ranked last. Alternatively, the ranking module 218 may rank the phrases in any manner and based on any characteristic or attribute associated with the phrases. Moreover, upon the weighted phrases being ranked, the display module 220 may create a visual representation of the ranked phrases. In various embodiments, the visual representation may include a representation of each phrase (e.g., by text, images, symbols, etc.) that varies based on the weight assigned to each phrase. For instance, various characteristics of each phrase, such as font, color, size, etc., may be emphasized or deemphasized to indicate the relative frequency associated with each phrase. In these embodiments, the visual representation of the phrases for a particular point in time may be output to the computing device 104. Moreover, the visual representation may be constantly or periodically updated either at the phrase analysis server 110 and/or at the computing device 104 as additional phrases are extracted, filtered, processed, and weighted.
Providing Phrase Data
Initially, the document sources 106 (e.g., blogs, news sites, social media sites/networks, etc.) may generate, store, and/or provide access to a plurality of the documents 118 (e.g., blog posts, news articles, e-mail messages, etc.). Moreover, the extraction module 124 of the phrase analysis service 110 may extract one or more phrases from the documents 118. In various embodiments, the phrases may include any number of words, characters, symbols, etc., and may relate to or describe the documents 118 in which they are extracted. In various embodiments, the extracted phrases may be stored in the extracted phrases module 126 to prior to processing by the phrase analysis service 110. In particular, the extracted phrases may then be filtered by the filtering module 128, which may utilize one or more rules to reduce the number of extracted phrases. That is, the filtering module 126 may filter out certain phrases that have a lower likelihood of representing the actual content of and/or topics associated with the document 118 in which the phrase was extracted. In various embodiments, the filtering process may be based on one or more dictionaries, the number of words in the phrase, the number of characters in at least one word in the phrase, and/or the presence of stop phrases or stop words in the phrase, as described above with respect to
Once the phrases have been filtered (e.g., reduced in number), the SIP module 130 may determine whether each of the filtered phrases are SIPs. Moreover, the SIP module 130 may determine the frequency in which those phrases appear in the documents 118 and may also determine the number of different document sources 106 in which each phrase appears. In various embodiments, the scoring module 132 may then assign a weight to each SIP phrase based at least in part on the frequencies determined by the SIP module 130. For instance, a phrase that appears more frequently in the documents 118 and/or appears in a greater number of different document sources 106 may be assigned a higher weight than other phrases that appear less frequently.
At this point, the output module 134 may collect the weighted phrases, rank the weighted phrases based at least in part on their respective weights, and assemble the weighted data into phrase data 302. In some embodiments the phrase data 302 may include a visual representation 304 of the phrases and/or a phrase table 306 that includes various information associated with each phrase. For instance, for each phrase, the phrase table 306 may include an identity of the phrase, a weight assigned to that phrase, and the corresponding frequencies determined by the SIP module 130. The visual representation 304 and/or the phrase table 306 may be formulated and output by the output module 134. In other embodiments, the phrase data 302 may include the ranked phrase data but the computing device 104 may create the visual representation 304 and/or the phrase table 306 for presentation to the user 102. In any event, the phrase data 302 is output to the computing device 104, which presents the visual representation 304 and/or the phrase table 306 to the user 102 via the display 116. The visual representation 304 and the phrase table 306 will be discussed in additional detail in
Phrase Extraction
As shown, the extracted phrases 402 may then be filtered by the filtering module 128, resulting in a subset of filtered phrases 404. In some embodiments, the filtering module 128 may utilize one or more rules in order to eliminate the extracted phrases 402 that are less likely to represent topics and/or content associated with the documents 118. For instance, the filtering module 128 may reduce the number of extracted phrases 402 based at least in part on whether at least one word within the phrase is included in a dictionary, the number of words within the phrase, the number of characters of at least one word within the phrase, the presence of stop phrases or stop words within the phrase, and/or any other factor that may help ensure that the phrases relate to meaningful content.
In further embodiments, the resulting filtered phrases 404 may then be processed by the SIP module 130. More particularly, for each filtered phrase 404, the SIP module 130 may determine the frequency that each filtered phrase 404 appears in the document 118 in which it was extracted, in comparison to the other documents 118. That is, the SIP module 130 may determine whether each filtered phrase 404 occurs in a higher concentration with respect to its corresponding document 118 as opposed to the remaining documents 118. For those filtered phrases 404 that are determined to be SIPs (e.g., local SIP phrases 406), those SIP phrases may be weighted or scored by the scoring module 132. These SIP phrases may be referred to as local SIP phrases 406 since they may be considered SIPs with respect to the document 118 in which they were extracted. Furthermore, the scoring module 132 may assign a weight to the local SIP phrases 406 in any manner, such as by assigning a higher weight to the local SIP phrases 406 based on their respective frequency within the document 118 in which they were extracted. For instance, more frequently occurring local SIP phrases 406 may be assigned a weight or scored higher than less frequently occurring local SIP phrases 406. Once the local SIP phrases 406 are weighted or scored by the scoring module, they may be referred to as scored local SIP phrases 408.
In some embodiments, the scored local SIP phrases 408 may be processed by the SIP module 130 to generate global SIP phrases 410. More particularly, for each of the scored local SIP phrases 408, the SIP module 130 may determine the number of document sources 106 (e.g., news sites, blogs, e-mail providers, etc.) in which the scored local SIP phrases 408 occur. Therefore, not only may the SIP module 130 determine the frequency of the phrases relative to other documents 118, but the SIP module 130 may also determine whether, and to what extent, the phrases appear in different document sources 106. Alternatively, as opposed to the SIP module 130 processing the scored local SIP phrases 408, the SIP module 130 may also process the filtered phrases 404 in a similar manner. That is, for each filtered phrase 404, the SIP module 130 may determine the number of document sources 106 in which those filtered phrases 404 occur. The resulting global SIP phrases 410 are considered global since the calculated frequency relates to a number of different document sources 106 in which those phrases are present.
Subsequently, the global SIP phrases 410 may be assigned a weight or scored by the scoring module 132, resulting in scored global SIP phrases 412. As mentioned above, the weight associated to each global SIP phrase 410 may be based in part on the relative frequencies determined by the SIP module 130. Furthermore, the scored global SIP phrases 412 may be collected, ranked, and output by the output module 134. In particular, the scored global SIP phrases 412 may be ranked based on their respective frequencies and the ranked phrases may be depicted as a visual representation 304 of the phrases and/or a phrase table 306 that includes information about each phrase. The visual representation 304 and/or the phrase table 306 may be output by the output module 134 as the phrase data 302. Moreover, the phrase data 302 may then be presented to the user 102 via the display 116 of the computing device 104.
Visual Representation of Phrase Data
In various embodiments, the phrases may be represented by text (e.g., words, characters, etc.), symbols, images, or any other means. Furthermore, the representation of the phrases may be emphasized or deemphasized in any manner, such as by increasing/decreasing the size, changing the colors, and/or adjusting the font of the phrase representations. The visual representation 304 and/or the phrase table 306 may also be presented to the user in multiple ways, such as by presenting the information in a web browser on the display 116.
In the example embodiments illustrated in
In the above embodiments, the visual representation 304 includes multiple phrases including data 502, attacker 504, insider 506, and invocation 508, among others. As shown the phrase data 502 appears to have the largest text, followed by attacker 504, insider 506, and invocation 508. In these embodiments, the larger text size associated with the phrase data 502 may indicate that this term appeared more frequently in the documents 118 relative to the phrases attacker 504, insider 506, and invocation 508. Further, due to the sizes of each phrase, the phrase attacker 504 may have appeared more frequently in the documents 118 than the phrases insider 506 and invocation 504, but less frequently than the phrase data 502. As a result, by viewing the text size of these phrases in the visual representation 304, the user 102 may determine which phrases are appearing more frequently in the documents 118 at a particular point in time. Although text size is used to highlight the frequency of the phrases, any method of emphasizing the phrases may be utilized (e.g., changing fonts, colors, etc.).
In addition, the visual representation 304 may indicate to the user 102 that certain phrases appear in documents 118 from multiple document sources 106 whereas other phrases appear only within a single document source 106. Although not shown in the visual representation 304, the color assigned to the phrases may indicate the number of different document sources 106 in which those phrases appear. For instance, the brightness of colors associated with the text of the phrases may correspond to the number of document sources 106 in which those phrases are found. That is, a first phrase that appear in multiple document sources 106 may have a bright color, such as bright red, yellow, or orange, whereas as second phrase that appears in a single document source 106 may be represented in a darker color, such as black or brown. Moreover, a third phrase that appears in a number of document sources 106 that is between the first and the second phrases may be assigned an intermediate color, such as blue or green. As stated above, although colors may be used to highlight the number of document sources 106 in which the phrases appear, this may also be represented in any manner, such as by varying the size or font of the phrases, or utilizing upper case letters, lower case letters, and/or a combination thereof. Moreover, the orientation of the representations of the phrases within the visual representation 304 may also indicate frequencies of the phrases within the documents 118 and/or the document sources 106. For instance, the phrases may appear vertically, horizontally, diagonally, and/or at any other angle within the visual representation 304.
Provided that the phrases depicted in the visual representation 304 are emphasized in the manner set forth above, a phrase that appeared many times in the documents 118 and also appeared in a great number of document sources 106 may have a larger text size that is displayed in a bright color. On the contrary, a phrase that appeared a limited number of times in the documents 118 and only appeared in a single document source 106 may be represented by small text and a darker color. Furthermore, a phrase that appeared very frequently in the documents 118 but appeared in a relatively low number of document sources 106 would likely have a larger text size, but would be displayed in a darker color. Lastly, a phrase that appeared only several times in the documents 118 but appeared in multiple different document sources 106 would likely be represented by small text in a brighter color.
In various embodiments, the visual representation 304 may indicate to the user the frequency in which phrases are being mentioned and/or discussed in various documents and/or the number of different information sources in which those phrases are appearing at a particular point in time. That is, at any particular time, the visual representation 304 provides a snapshot of what phrases are trending and, based on the representation of these phrases, the user 102 may be able to determine topics and/or content that are currently being mentioned on a more frequent basis across multiple information sources. However, the visual representation 304 may also be periodically or dynamically updated based on the availability of more recently accessible documents 118. For instance, new phrases and additional instances of existing phrases may be extracted, processed, and weighted by the phrase analysis service 110. As this occurs, the new phrases may be added to the visual representation and the existing phrases may be adjusted based on their updated frequency within the documents 118 and among the different document sources 106.
Therefore, if a representation of a particular phrase within the visual representation 304 is modified, the user 102 may determine that the appearance of this phrase is becoming more or less frequent. Since the phrase analysis service 110 may be able to immediately and dynamically highlight changes in the representation of the phrases, the user 102 may choose to examine documents 118 associated with the changing phrases to examine why such changes are occurring. Furthermore, as additional documents 118 become available, the visual representation 304 of the phrases may continue to change to illustrate what phrases are trending at that particular time.
In additional embodiments, the phrase table 306 may also be presented to the user 102, either independently or in conjunction with the visual representation 304. The phrase table 306 may include data associated with the phrases illustrated in the visual representation 304. Although the phrase table 306 may include any type of information associated with these phrases, the phrase table 306 shown in
In various embodiments, the phrases illustrated in the visual representation 304 and the phrase table 306 may remain without being deleted. Although phrases that have been existing in the visual representation 304 for long periods of time may tend to have higher frequencies, as newer phrases appear more frequently in documents 118 and/or different document sources 106, the frequency of those newer phrases may eventually surpass the older, existing phrases. As a result, the weight assigned to the newer phrases may also exceed the weight associated with the older, existing phrases. An advantage of maintaining the phrases in the visual representation 304 is that, over time, the visual representation 304 and the corresponding phrase table 306 may constitute an archive that describes what phrases were trending at different times. For instance, a user 102 may be interested in determining which phrases were trending and, therefore, which topics and/or content was being mentioned or discussed at a particular time in the past. Since this data remains, the user 102 may be able to search for a particular time period and access the visual representation 304 associated with that time. In other embodiments, both a current visual representation 304 and an archived visual representation 304 may be maintained for this purpose. As a result, the user 102 may have access to currently trending phrases and/or phrases that were trending at a specific time period in the past but no longer frequently appear in the documents 118.
Moreover, the visual representation 304 and the phrases associated therewith may also be manipulated based on time. For instance, the weights assigned to the phrases may be degraded or decreased over time in order to prevent recent frequently appearing phrases to be overshadowed by phrases that used to appear frequently but do not presently occur as frequently. Otherwise, since older phrases are likely to have appeared a relatively large amount over time, their respective scores may be quite large. Therefore, it may take a significant amount of time for a newer phrase to attain a weight that actually surpasses the older phrases. By degrading the weights of the phrases over time, newer phrases that are currently appearing very frequently may be displayed as such in the visual representation 304.
In other embodiments, newer and/or more frequently appearing phrases may be weighted more heavily in order to emphasize that these phrases are currently becoming important and/or more relevant. In different embodiments, phrases that have become more frequent in a short period of time but that do not surpass the weights assigned to phrases that have been trending for a significant amount of time may be highlighted and/or emphasized in the visual representation 304. That way, the user 102 may be able to determine that these newer phrases are actually more relevant than the older phrases at this particular point in time.
Furthermore, the user 102 may be able to configure the visual representation 304 based on his or her preferences. For instance, the visual representation 304 may be customized by the user 102 such that the visual representation 304 either includes or excludes certain document sources 106. That is, the user 102 may desire to view phrases appearing in news articles and blogs, but not social media sites or networks. Moreover, the user 102 may customize the visual representation such that the phrases displayed therein relate to a specific geographic location, demographic, or any other factor.
Example Processes
Block 602 illustrates extracting a plurality of phrases. More particularly, the extraction module 124 of the phrase analysis service 110 may extract phrases from multiple documents 118 that correspond to multiple different sources of information. For instance, the documents 118 may include e-mail messages, blog posts, news articles, any posts or comments associated with social media sites or networks (e.g., Facebook®, Twitter®, etc.), and/or any other type of natural language document 118. Moreover, document sources 106 may be the source(s) of such documents 118, such as an origin of the documents 118 or a location of where the documents 118 are accessible. Moreover, the phrases may include any number of words and/or characters and may include nouns, verbs, adjectives, or any other type of language, that may reference a topic or content associated with the document 118 in which the phrase is extracted.
Block 604 illustrates filtering the plurality of phrases. In particular, the extracted plurality of phrases may be filtered (e.g., by the filtering module 128) to eliminate phrases that are not likely to reflect topics or content associated with the documents 118. In some embodiments, the plurality of phrases may be filtered based on any characteristic associated with the phrases. For instance, the plurality of phrases may be filtered based at least in part on whether any words in the phrases are included in a dictionary having a predetermined number of words, the amount of words in the phrases, the amount of characters in any word included within the phrases, and the existence of any stop phrases or stop words included in the phrases. As a result, based on whether a particular phrase falls into one of the foregoing categories, the phrase may be eliminated or maintained for further processing.
Block 606 illustrates processing the plurality of phrases. In some embodiments the plurality of phrases that have not been filtered out may be processed (e.g., by the SIP module 130) by determining whether each phrase is a SIP. Furthermore, each may phrase may be processed to determine a frequency in which the phrases appear in the documents 118 and/or a number of the document sources 106 in which each phrase is present. That is, the plurality of phrases may be processed to determine the extent to which these phrases are being mentioned and/or discussed amongst various information sources.
Block 608 illustrates assigning a weight to each of the plurality of phrases. In various embodiments a weight or score may be assigned (e.g., by the scoring module 132) based at least in part on the frequency in which the phrases appear in the documents 118 and/or a number of the document sources 106 in which each phrase is present. For instance, phrases that appear more frequently in the documents 118 and/or phrases that appear in a greater number of document sources 106 may be assigned higher weights. Once the plurality of phrases are weighted, they may be collected, ranked, and output (e.g., by the output module 134). Moreover, the plurality of phrases may also be assembled into a visual representation 304 that may be provided to the computing device 104 for presentation to the user 102 via the display 116.
Block 610 illustrates providing a visual representation of the plurality of phrases based at least in part on the assigned weights. In particular, the visual representation 304 may include representations of the plurality of phrases, such as the text of the phrases or symbols, images, or characters associated with the phrases. Moreover the representation of each phrase may vary, such as by varying any characteristic associated with the representation (e.g., size, color, shape, font, etc.). As stated above with respect to
Block 612 illustrates dynamically modifying the visual representation. In various embodiments, the representations of the phrases within the visual representation 304 may be dynamically modified as additional phrases are extracted, processed, and weighted. For instance, as additional instances of existing phrases and new phrases are extracted, the representations of the phrases may be modified to indicate changes in frequency of the phrases in the documents 118 and/or the document sources 106. As a result, after the visual representation 304 is modified, the user 102 may then view which phrases are currently trending (e.g., being mentioned and/or discussed more frequently across the multiple information sources). In some embodiments, if a user 102 determines that phrase of interest to the user 102 is increasing in frequency, the user 102 may access the documents 118 in which that phrase appears to determine the context of why the phrase is being utilized.
Block 702 illustrates presenting a visual representation of a plurality of phrases. In some embodiments, upon extracting phrases from the documents 118 associated with one or more document sources 106 (e.g., by the extraction module 124), filtering the phrases (e.g., by the filtering module 128), processing the phrases (e.g., by the SIP module 130), and assigning a weight to the phrases (e.g., by the scoring module 132), the phrases may be transmitted to the computing device 104 and presented to the user 102 as a visual representation 304. As stated above, in the visual representation 304, each phrase included therein may be represented in a manner (e.g., changes in size, color, shape, font, etc.) that indicates a current frequency in which those phrases are occurring in the documents 118 and/or the multiple document sources 106.
Block 704 illustrates extracting additional phrases. More particularly, the phrase analysis service 110, and the extraction module 124 in particular, may monitor (either continuously or periodically) the document sources 106 for newly provided documents 118. From these documents 118, additional phrases may be extracted. These phrases may include additional instances of existing phrases already included in the visually represented phrases and/or new phrases that have yet to be processed. Once these phrases are identified, they then may be filtered, processed, and weighted by the various components of the phrase analysis service 110.
Block 706 illustrates adjusting a weight assigned to the plurality of phrases. Based on the frequency in which the phrases now appear in the documents 118, the weight assigned to the phrases may be adjusted. For instance, phrases that currently appear more frequently in the documents 118 may have their corresponding weights be increased. Moreover, the weights of existing phrases that have the same frequency as previously may remain the same. Therefore, the weights of the phrases may be adjusted based on their overall frequency in the documents 118 and among the different document sources 106.
Block 708 illustrates dynamically modifying the visual representation. More particularly, based on the weight adjustments, the visual representation 304 of various phrases may be modified. For instance, a representation for a phrase that now appears more frequently in the documents 118 and/or the number of document sources 106 may be modified to reflect this, such as by increasing a size, changing a color, changing a shape, and/or changing a font of that particular phrase. By viewing the visual representation 304, the user 102 may readily determine which phrases are appearing more frequently in the documents 118 and various information sources and take any action in response thereto.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as illustrative forms of implementing the claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20070282826 | Hoeber et al. | Dec 2007 | A1 |
| 20080231644 | Lempel et al. | Sep 2008 | A1 |
| 20100235353 | Warnock et al. | Sep 2010 | A1 |
| 20110131207 | Jonsson | Jun 2011 | A1 |
| 20120053927 | Kulkarni et al. | Mar 2012 | A1 |
| 20120278341 | ogilvy et al. | Nov 2012 | A1 |
