Patent Grant
8838633
The present disclosure relates to methods, techniques, and systems for providing sentiment analysis and, in particular, to methods, techniques, and systems for providing sentiment analysis using natural language processing to determine sentiment of objects in a corpus.
Every minute of every day people express their sentiments and write them down in news articles, blog posts, other web content, and the like. Some people may regard themselves as too famous to write down their sentiments, but journalists, bloggers and other content creators are more than willing to document their feelings. Often times a famous radio commentator will bash a politician, or a politician will thrash a Hollywood actress. On occasion, a true act of heroism will be recognized, and all sorts of famous folk will follow with praise. Whether depressing or uplifting, disturbing or unnerving, tapping in to the sentiments of key actors on the world stage can be highly informative and engaging.
Determining the underlying sentiment of an article using a computing system may be difficult because of the variety of styles people employ in expressing sentiment—a comment may be an offhanded compliment in amongst an otherwise negative article, for example. Current techniques often involve traditional keyword searching for particular negative or positive words (verbs) such as “hate,” “like,” “distaste,” etc. to guesstimate the underlying sentiment of an article.
The headings employed herein are used to assist in the presentation and organization of the material and are not to be used to limit the scope of the described techniques.
A. Overview
Embodiments described herein provide enhanced computer- and network-based methods, techniques, and systems for providing sentiment analysis and for presenting the results of such analysis. Example embodiments provide a Sentiment Analysis System (“SAS”), which provides tools to enable authors, programmers, users, developers, and the like to incorporate sentiment analysis into their content, such as into their web pages, and other web blogs or textual content. In some embodiments such tools are provided in the form of an Application Programming Interface (“API”). In other embodiments, such tools are provided in the form of an “ready-made” Sentiment Widget, which is programmed to analyze sentiment for a particular topic, entity, or facet (e.g., characteristic of an entity). Other embodiments provide other mechanisms and examples of user interfaces which incorporate the techniques of the SAS and deliver information via NLP-based sentiment analysis to a consumer of such results.
The SAS works to understand the sentiments, or positive and negative expressions by and about entities. Many types of applications can be built using the sentiment API in areas including, but not limited to: market intelligence, market research, sports and entertainment, brand management, product reviews and more. For example, using the sentiment API, one can:
Find the percentage of positive and negative expressions of sentiment made by an entity, or about an entity. For example, one can find out what percentage of things being written about the iPhone are positive and which percent are negative.
Discover who is criticizing and who is praising a particular person, place or thing. For example, see who is criticizing and praising IBM right now.
Read what praisers and critics are saying about an entity. For example, see what the GOP are saying about the Democrats.
Discover who or what your favorite entity is bashing and why. For example, see who Lance Armstrong is complaining about.
Discover who or what your favorite entity is praising and why. For example, see who the World Health Organization is commending and why.
The SAS uses natural language based processing techniques, such as parts of speech tagging and relationship searching, to identify sentence components such as subjects, verbs, and objects, and to disambiguate and identify entities so that the SAS can recognize whether the underlying relationships (e.g., between subjects, verbs, and objects) in the content are expressed in a negative or positive sentiment. Example relationship searching technology, which uses natural language processing to determine relationships between subjects and objects in ingested content, is described in detail in U.S. Pat. No. 7,526,425, issued on Apr. 28, 2009, and entitled “METHOD AND SYSTEM FOR EXTENDING KEYWORD SEARCHING FOR SYNTACTICALLY AND SEMANTICALLY ANNOTATED DATA,” and entity recognition and disambiguation technology is described in detail in U.S. patent application Ser. No. 12/288,158, filed Oct. 15, 2008, and entitled “NLP-BASED ENTITY RECOGNITION AND DISAMBIGUATION,” both of which are incorporated herein by reference in their entireties. As explained therein, relationship searching uses queries which attempt to understand the underlying content through the use of natural language processing and to recognize and understand the various relationships between entities (e.g., persons, locations, things, events, and the like) using syntactic and semantic analysis of the underlying content. The use of relationship searching, enables the SAS to establish second order (or greater order) relationships between entities and to store such information.
Although the example embodiments described below utilize the EVRI™ relationship searching described in the above listed references to achieve more robust and precise sentiment analysis, other natural language systems and Boolean keyword matching systems may be used to identify content of a particular sentiment, which can then be integrated into the user interface and presentation tools of an SAS as described further herein.
In addition, as used herein, entities are generally identifiable people, places or things, such as people, locations, organizations, products, events, and the like. Facets are generally more finely granular characteristics of entities such as categories, types, and/or characteristics of certain entities such as actor, politician, nation, drug, automobile, and the like. Topics are subjects of interest that may involve a group of entities and/or facets. Any hierarchy or non-hierarchical division of the subjects (e.g., nouns and modifiers) and objects (e.g., nouns and modifiers) searched for in content may be used.
In one example embodiment, the Sentiment Analysis System comprises one or more functional components/modules that work together to provide sentiment analysis of a set of content stored in, for example, a corpus of documents. For example, a SAS may comprise an analysis engine, an API, and example user interface tools such as real time updated widgets that are embeddable in other content (for example, a third party website). The sentiment analysis engine is responsible for determining and categorizing the various relations (e.g., S-A-O triplets, or other forms) in the underlying content according to their sentiment. Different embodiments of the sentiment analysis engine may use different techniques for discovery of sentiment, for example, relationship searching using particular verbs, phrases, and heuristics, and/or modifications of same incorporating machine learning techniques. Example implementations of the sentiment analysis engine are discussed further below.
The sentiment analysis API (application programming interface) provides a programmatic interface to the capabilities of the sentiment analysis engine to uncover sentiment from underlying content. The API may provide different forms of the results of analyzing and categorizing the content such as in summary form or with specific details. For example, EVRI™ currently supports an API adhering to a REST interface (a REST API) that is found on www.evri.com/developer in the Reference API Specification and the descriptions of the various available functions. With the EVRI API, a developer or other would be consumer of relationship query data can automatically, cost effectively and in a fully scalable manner: analyze text, get recommendations, discover relationships, mine facts and get popularity data. A particular REST API, the GetSentiment API, is provided to query and organize content according to positive and negative sentiment. A full description of this “GET sentiment” API is provided below in Section D, “Example Sentiment API Specification.” The GET sentiment API offers summary data of content sentiment as well as particular details of sentiment data according to a specified source of the sentiment, type of sentiment, and/or subject (target) of the sentiment. Other API can similarly be incorporated into the SAS to provide the sentiment data, which can then be displayed with the user interface tools of the SAS.
Example user interface tools (or widgets) are shown in
B. Example User Interface Tools
When a user clicks on the “positive” link (which defaults to positive sentiment about any entity) or the “[anything]” link shown in the left column of
Here, the input parameter sentimentSource refers to Barack Obama, indicating interest in vibes or sentiment expressed by Obama, as opposed to about him. Next, the input parameter sentimentType is set to positive, indicating interest in positive sentiment expressions. Finally, the input parameter sort is set to date, indicating interest in obtaining the latest results first. A discussion of how this API may be implemented to achieve such sentiment analysis using an example embodiment of the SAS is discussed with reference to
The API call results also provide a specific snippet from the article, as well as a time stamp, the article title, and a link off to the source article for each result having sentiment that corresponds to the input specification. For example, in one snippet there appears a sentence stating that “the president commended . . . ” The SAS, through the use of its advanced relationship searching mechanisms, recognizes “the president” to be the source of the vibes, or sentiment, and commendation to be the prime justification for his positive sentiment expression. As illustrated, this kind of analysis is beyond mere keyword (straight pattern matching) recognition and shows the ability of the SAS to use NLP to “understand” the underlying relationships expressed in the article.
Next, as the result of the user selecting (clicking on) the “France” link in
The input parameter entityURI references Barack Obama, indicating that the returned sentiment is about Barack Obama. In addition, the input parameter sentimentType is set to negative, indicating that returned sentiment expressions will be negative in nature. Also, the input parameter sentimentSource references Rush Limbaugh. The URI that refers to Rush Limbaugh can be obtained from the sentiment summary results of the request shown above in reference to
Another interface/presentation of sentiment can be seen in the AttackMachine example embodiment described below in Section F, “Attack Machine Example —Specification.” The AttackMachine is an application (here, a web site) targeting an in depth assessment of all things “attack” oriented that have been written about over the indexed set of web pages, recently. It is built on top of the EVRI subject/verb/object style data extraction (using EVRI relationship searching), and can easily be extended to other verbs, or actions, such as: LoveMachine, HateMachine, KillMachine, etc. In each case, the equivalent of “attackers” and “victims” are present. For example, for LoveMachine, the source/targets of the verb love are “The lover” and “The loved”.
Other applications, interfaces, tools, and widgets can be developed using the SAS API, and equivalents that interface to the Sentiment Analysis Engine capabilities.
As mentioned above, the SAS provides a sentiment analysis engine to classify and discover sentiment in content, such as from a corpus of articles. In one embodiment, the sentiment analysis engine uses a determined list of verbs or sentiment phrases in sentiment queries against the content to derive sentiment. These lists may be pre-formulated or provided by means of an external storage so that they can be defined somewhat automatically or dynamically at runtime, or through some combination of both. Certain post filtering activities may also be incorporated, such as to compare the title of a corresponding article where a relationship having sentiment is found with a relationship found therein, to make sure the title the sentiment doesn't indicate that the relationship is a backhanded compliment or criticism or sarcastic. An example of using this technique for uncovering sentiment is discussed with reference to
In other embodiments, machine learning techniques can be incorporated to derive the sentiment verbs and phrases that are searched for in the relationship queries. A description of one example machine learning technique is described below with respect to Section E, “Sentiment Analysis—Machine Learning Example Embodiment.” It is intended to show a general approach to using machine learning to reveal the sentiment of content. Other approaches as they are developed can similarly be incorporated.
Example embodiments described herein provide applications, tools, data structures and other support to implement a Sentiment Analysis System to be used for presenting sentiment of certain content. Other embodiments of the described techniques may be used for other purposes, including for marketing or intelligence information, branding, advertising, and the like. Also, although described primarily with respect to textual content, the techniques described herein can be extrapolated to address visual content, or combined text and visual content, for example, when meta data labeling the visual content can be similarly mined for sentiment information. For example, in some cases the title of a picture may indicate that the picture contains positive or negative sentiment. Also, although certain terms are used primarily herein, other terms could be used interchangeably to yield equivalent embodiments and examples. In addition, terms may have alternate spellings which may or may not be explicitly mentioned, and all such variations of terms are intended to be included.
In the following description, numerous specific details are set forth, such as data formats and code sequences, etc., in order to provide a thorough understanding of the described techniques. The embodiments described also can be practiced without some of the specific details described herein, or with other specific details, such as changes with respect to the ordering of the code flow, different code flows, etc. Thus, the scope of the techniques and/or functions described are not limited by the particular order, selection, or decomposition of steps described with reference to any particular routine.
C. Example Computing System and Processes
The computing system 300 may comprise one or more server and/or client computing systems and may span distributed locations. In addition, each block shown may represent one or more such blocks as appropriate to a specific embodiment or may be combined with other blocks. Moreover, the various blocks of the Sentiment Analysis System 310 may physically reside on one or more machines, which use standard (e.g., TCP/IP) or proprietary interprocess communication mechanisms to communicate with each other.
In the embodiment shown, computer system 300 comprises a computer memory (“memory”) 301, a display 302, one or more Central Processing Units (“CPU”) 303, Input/Output devices 304 (e.g., keyboard, mouse, CRT or LCD display, etc.), other computer-readable media 305, and one or more network connections 306. The SAS 310 is shown residing in memory 301. In other embodiments, some portion of the contents, some of, or all of the components of the SAS 310 may be stored on and/or transmitted over the other computer-readable media 305. The components of the Sentiment Analysis System 310 preferably execute on one or more CPUs 303 and manage the discovery and mining of sentiment data, as described herein. Other code or programs Y30 and potentially other data repositories, such as data repository 306, also reside in the memory 301, and preferably execute on one or more CPUs 303. Of note, one or more of the components in
In a typical embodiment, the SAS 310 includes one or more Sentiment Analysis Engines 311, one or more Entity and Relationship Identifiers 312 as described in patent application Ser. No. 12/288,158, one or more SAS APIs 313, and other (optional) support, such as machine learning support 314, rules for finding sentiment 315, for example if expressed externally to the Sentiment Analysis Engines 311. In at least some embodiments, the heuristics and rules 315 are provided external to the SAS and is available, potentially, over one or more networks 350. Other and/or different modules may be implemented.
In addition, the SAS may interact via a network 350 with application or client computing device 360 that calls the API 313 to incorporate sentiment data for other purposes, e.g., uses results computed by engine 311, one or more content sources 355, and/or one or more third-party systems 365, such as machine learning tools that can be integrated with engine 311. Also, of note, the Entity Data Store 316 and the Indexed Article Data 317 may be provided external to the system and accessible over one or more networks 350. The network 350 may be any combination of media (e.g., twisted pair, coaxial, fiber optic, radio frequency), hardware (e.g., routers, switches, repeaters, transceivers), and protocols (e.g., TCP/IP, UDP, Ethernet, Wi-Fi, WiMAX) that facilitate communication between remotely situated humans and/or devices. The mobile devices 360 include notebook computers, mobile phones, smart phones, personal digital assistants, tablet computers, desktop systems, kiosk systems, and the like.
In an example embodiment, components/modules of the SAS 310 are implemented using standard programming techniques. However, a range of programming languages known in the art may be employed for implementing such example embodiments, including representative implementations of various programming language paradigms, including but not limited to, object-oriented, functional, procedural, scripting, declarative, etc.
The embodiments described above may also use well-known or proprietary synchronous or asynchronous client-server computing techniques. However, the various components may be implemented using more monolithic programming techniques as well, for example, as an executable running on a single CPU computer system, or alternately decomposed using a variety of structuring techniques known in the art, including but not limited to, multiprogramming, multithreading, client-server, or peer-to-peer, running on one or more computer systems each having one or more CPUs. Some embodiments are illustrated as executing concurrently and asynchronously and communicating using message passing techniques. Equivalent synchronous embodiments are also supported by an SAS implementation.
In addition, programming interfaces to the data stored as part of the SAS 310 (e.g., in the data repositories 316 and 317) can be available by standard means such as through C, C++, C#, and Java APIs; libraries for accessing files, databases, or other data repositories; through scripting languages such as XML; or through Web servers, FTP servers, or other types of servers providing access to stored data. The Entity Data Store 316 and the Indexed Article Data 317 may be implemented as one or more database systems, file systems, or any other method known in the art for storing such information, or any combination of the above, including implementation using distributed computing techniques.
Also the example SAS 310 may be implemented in a distributed environment comprising multiple, even heterogeneous, computer systems and networks. For example, in one embodiment, the engine 311, the API functions 313, and the article data repository 317 are all located in physically different computer systems. In another embodiment, various modules of the SAS 310 are hosted each on a separate server machine and may be remotely located from the tables which are stored in the repositories 316 and 317. Also, one or more of the modules may themselves be distributed, pooled or otherwise grouped, such as for load balancing, reliability or security reasons. Different configurations and locations of programs and data are contemplated for use with techniques of described herein. A variety of distributed computing techniques are appropriate for implementing the components of the illustrated embodiments in a distributed manner including but not limited to TCP/IP sockets, RPC, RMI, HTTP, Web Services (XML-RPC, JAX-RPC, SOAP, REST, etc.) etc. Other variations are possible. Also, other functionality could be provided by each component/module, or existing functionality could be distributed amongst the components/modules in different ways, yet still achieve the functions of an SAS.
Furthermore, in some embodiments, some or all of the components of the SAS may be implemented or provided in other manners, such as at least partially in firmware and/or hardware, including, but not limited to one or more application-specific integrated circuits (“ASICs”), standard integrated circuits, controllers executing appropriate instructions, and including microcontrollers and/or embedded controllers, field-programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), and the like. Some or all of the system components and/or data structures may also be stored as contents (e.g., as executable or other machine-readable software instructions or structured data) on a computer-readable medium (e.g., as a hard disk; a memory; a computer network or cellular wireless network or other data transmission medium; or a portable media article to be read by an appropriate drive or via an appropriate connection, such as a DVD or flash memory device) so as to enable or configure the computer-readable medium and/or one or more associated computing systems or devices to execute or otherwise use or provide the contents to perform at least some of the described techniques. Some or all of the system components and/or data structures may be stored as non-transitory content on one or more tangible computer-readable mediums. Some or all of the system components and data structures may also be stored as data signals (e.g., by being encoded as part of a carrier wave or included as part of an analog or digital propagated signal) on a variety of computer-readable transmission mediums, which are then transmitted, including across wireless-based and wired/cable-based mediums, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). Such computer program products may also take other forms in other embodiments. Accordingly, embodiments of this disclosure may be practiced with other computer system configurations.
As described in
In one example embodiment, in the Find Sentiment Data Routine 400, implements the SAS API discussed earlier. Different portions of the routine 400 may be executed, commensurate with the input parameters in the API call. In particular, in block 401, the SAS runs a relationship query using the NLP mechanisms described in detail in U.S. Pat. No. 7,526,425 to find positive sentiment. An example such query is:
where * (any matching) or a particular Source is specified as an entity type, particular entity ID, facet, topic, or string; CommaSeparatedQuotedListOfPositiveVerbs is exactly that —a comma separated list of verbs to be treated as positive verbs, e.g., “like,” “hug,” “praise,” or the like; * (any matching) or a particular Subject is specified as an entity type, particular entity ID, facet, topic, or string. Different relationship query engines may require different syntax (e.g., unquoted verbs, different separators, or the like). Note that the context operator “˜” is used to eliminate relationships that include negative phrases closely situated (within “n” sentences, configurable) with the found relationship. This attempts to eliminate relationships where the sentiment is backhanded or a parody, etc.
In block 402, the SAS runs a similar relationship query using the NLP mechanisms to find negative sentiment. An example such query is:
where * (any matching) or a particular Source is specified as an entity type, particular entity ID, facet, topic, or string; CommaSeparatedQuotedListOfNegativeVerbs is exactly that—a comma separated list of verbs to be treated as negative verbs, e.g., “kill,” “attack,” “hate,” or the like; * (any matching) or a particular Subject is specified as an entity type, particular entity ID, facet, topic, or string. Different relationship query engines may require different syntax (e.g., unquoted verbs, different separators, or the like). Note that the context operator “˜” is used to eliminate relationships that include positive phrases closely situated (within “n” sentences, configurable) with the found relationship. This attempts to eliminate relationships where the sentiment is a quote or a parody, etc.
In block 403, the SAS performs available post relationship filtering. For example, in some embodiments, each returned relationship is compared with the corresponding title of the article in which the relationship is found. Relationships may be eliminated if the corresponding title reflects an opposite sentiment from the relationship. Many other filtering rules containing phrases, rules, identifying specific entities to avoid, etc. may be specified at this block of execution.
In block 404, the SAS determines (computing or otherwise) the percentage of positive sentiments (e.g., number of positive relationships returned in block 401) after filtering in block 403 as a percentage of the total relationships after filtering. Similarly, in block 405, the routine determines the percentage of negative sentiments (e.g., number of negative relationships returned in block 402) after filtering in block 403 as a percentage of the total relationships after filtering. These are stored as part of the output, to support the summary statistics of the sentiment analysis API.
In block 405, the SAS ranks and aggregates the determined remaining relationships and issues resultant output, for example as specified in the API described below with respect to Section D, “Example Sentiment API Specification,” and ends the routine 400. The ranking, for example, may result in only the top most “n” recent relationships being returning for a given source or subject. The aggregation allows the ranking to be performed on like relationships.
Although many different ways are available to rank and aggregate the determined remaining relationships, one such method is provided in
In block 502, the source and target (subject of sentiment) is determined for each relationship found in
In block 503, the relationships are grouped according to their sources and targets determined in block 502. If one or more facets or categories are specified in the sentiment API, then the SAS constrains (filters) the relationships to only those whose sources and/or subjects belong to the facet/category specified, for example, by the input parameters.
In block 504, the results of block 503 are then ranked—for sources and for subjects/targets. In one embodiment, the following criteria may be used for such rankings:
Routine 500 then returns an indication of the ranked and aggregated results.
As stated, routines 400 and 500 can be used to generate the summary sentiment data, shown for example in
“Barack Obama”>CommaSeparatedListOfNegativeVerbs>*
which lists the top “n” relationships in which Barack Obama has expressed negative sentiment towards any subject. Other sentiment queries can be similarly handled.
D. Example Sentiment API Specification
1.1 Input Specifications
1.1.2 HTTP Method: GET sentiment
1.1.3 Resource: Get sentiment of something about something
Description: Returns sentiment by someone or something about someone or something. A subject may or may not be specified. If a subject is specified, it may include types, facets, entities, keywords, or a Boolean combination of all. The sentimentSource may or may not be specified. If it is not, then the source of the specified sentiment is anything. If it is, it may be a specific entity (e.g., Barack Obama), one or more of a facet (e.g., actor or musician), or one or more of a type (e.g., person or organization).
Usage:
sentiment/[summary]/[about]?[inputParameters]
where applicable inputParameters include: type, facet, entityURI, phrase, search, includeDomains, excludeDomains, includeDates, includeMatchedLocations, startId, resultsPerPage, sort, and callback.
Table 1, below, describes inputParameters that are supported.
1.2 Examples: Table 2, below lists various examples of the API in use.
1.3 Result Format
Below is an output independent listing of result elements:
1.3.1 XML Example: Table 3 below shows an XML example:
1.3.2 JSON
JSON is generated using the badgerfish convention:
[/v1/sentiment/by/person/barack-obama-0x16f69.json]
1.4 Response Codes
An embodiment of an NLP SVO (subject-verb-object) style triplet (i.e., a relationship) extraction based sentiment extraction system (SES) includes:
Training:
1. Ground Truth Construction
An explicit feedback mechanism is constructed into a sentiment user interface (SUI). The example SUI consists of the following use cases:
1.1 Summary Determination
For summary determination, the source or subject of sentiment is specified. If the source is specified, a list of subjects for sentimentType positive, and a list of subjects for sentimentType negative will be returned. For example, if the source is Barack Obama, and the returned sentimentType is negative, a list of subjects of Obama's negative sentiment will be returned such as: GOP, Rush Limbaugh, North Korea, and AIG.
1.2 Sentiment Determination
For sentiment determination, the source, sentimentType, and subject are specified. The source and subject may be an entity or facet. Either the source or subject may not be specified, indicating “anything” is an acceptable value. For example, source=Barack Obama, sentimentType=“negative”, and subject=“anything” will return a list of sentiments made up of a snippet, an article title, link to the article, and a date for the article where Barack Obama is mentioning a negative sentiment about anything. If the subject is AIG, then sentiments with Obama mentioning something negative about AIG will be returned.
The explicit feedback mechanism enables a user to rank the quality of each returned result by selecting one of the following options: “very poor”, “poor”, “average”, “good”, “great.” Users will be provided these options for both use cases shown above. The results will be stored into a ground truth style training set. For case 1 (the summary sentiment) shown above, the training consists of the corresponding source or subject entities, along with a rating and the entities rank position. For case 2 shown above, the training set consists of the article URI (or other indicator), title, snippet, rank position, and rating.
Algorithm Training:
Training of an indexing time sentiment classifier will begin against the ground truth data set. One strategy is to take a combined rule based and statistical approach. On the statistical side, compute:
where the topic is determined ad hoc, such as sports, health, everything else. The idea is the word lists are likely different for each topic. Now, the word lists will have an actual score indicating the degree of positivity or negativity for the topic, likely on a 0 to 1 scale. It might help to manually eliminate undesired words.
Indexing Time Application:
For a given document, first determine its topic (simple sum of facet occurrences where each facet is mapped to 1 of the N topic sets; the topic with the greatest number of corresponding facet occurrence wins). Next, use the appropriate word lists to compute a relationship score for every relationship in the document. A given raw relationship score for a given sentimentType can be computed as follows:
Next, an average sentiment score D_st for each sentimentType is computed across all relationships in the document. The final relationship score stored is:
where T is the minimal score threshold required to mark the relationship as belonging to sentimentType st.
Search Time Application for Sentiment Determination:
At search time, one can search for the source/subject/sentimentType combination as follows:
Now, for each of the returned results, a final ranking score needs to be determined that fuses the score for the relationship search itself, along with R_st.
One method is to start with a simple weighted combination of the form: w1*Rr+w2*R_st.
Search Time Application for Sentiment Summary Determination:
At search time, search for the source/subject/sentimentType combination as follows:
Now, for each relationship, extract entities/facets, and compute the entity score as:
Finally, the efficacy is computed against the ground truth, some weights are tweaked, and the process repeated. In some embodiments, the weight determination is automated.
Conclusion:
The above algorithms are examples to show how machine learning techniques may be applied to improve sentiment analysis that uses NLP based (SVO) relationship information. Nuances of the above specifics may yield more precise results and may be similarly integrated.
F. Attack Machine Example
Description
The AttackMachine is an example site targeting an in depth assessment of all things “attack” oriented. AttackMachine is intended to highlight differentiating technology available by EVRI from the EVRI API. AttackMachine is intended to showcase what can be built on top of the EVRI “triplet” or subject/verb/object style data extraction (using EVRI relationship searching). The AttackMachine site template can easily be extended to other verbs, or actions, such as: LoveMachine, HateMachine, KillMachine, etc. In each case, the equivalent of “attackers” and “victims” are present. For example, for LoveMachine, the source/targets of the verb love are “The lover” and “The loved”.
Visual Specification
The following functional specification and other instructions shows how to build such a site using the Sentiment Analysis API and other EVRI API. A detailed explanation of the current EVRI API can be found on the EVRI website, currently located at www.evri.com/develor/REST. It is to be understood that a variety of variations of the presentation of visuals and a variety of content can be similarly shown on a website or in other forms of content presentable through displaying or otherwise (e.g., audio, streaming, etc.) to a user or other consumer of such information.
Example Embodiment of Visual Specification
The following bullet points correspond to reference numbers indicated by numbered green dots shown in
Reference 1 in
Reference 2 in
Determining the top entities:
Every “n” minutes (e.g., 15), execute the following queries:
For any returning cell containing an entityId, convert the ID to hex and preceed with /organism/NAME-HEXID, where NAME is the cell's Desc value with case dropped and whitespace substituted with “-”. Example:
For any returning cell containing an entityId, convert the id to hex and preceed with /person/NAME-HEXID (see above example)
Reference 3 in
For remaining steps, process find top entities similarly to steps 1-6 above for most popular attacker list, making changes in the queries as appropriate.
Reference 4 in
Reference 5 in
Reference 6 in
Reference 7 in
Reference 8 in
Reference 9 in
Now, for each of the targets returned, check the type portion of the URI. Then break out the display according to the grouping described in Green Dot step 7.A. above, i.e., people, places, organization and things with things being either type condition, concept or organism.
yields no results, execute:
Reference 9.1 in
where shark is obtained by dropping the hex portion of the entity URI if it is present, and the word attack is appended.
Reference 10 in
Reference 11 in
Reference 12 in
Reference 13 in
Reference 14 in
Reference 15 in
then send the user to that entities attack page.
Note: /concept/shark with no hex-id is used to represent the keyword shark.
References 16-21 in
Same as the attack case, with differences addressed.
Reference 22 in
All of the above U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications and non-patent publications referred to in this specification including but not limited to U.S. Pat. No. 7,526,425, issued on Apr. 28, 2009, and entitled “METHOD AND SYSTEM FOR EXTENDING KEYWORD SEARCHING FOR SYNTACTICALLY AND SEMANTICALLY ANNOTATED DATA;” U.S. patent application Ser. No. 12/288,158, filed Oct. 15, 2008, and entitled “NLP-BASED ENTITY RECOGNITION AND DISAMBIGUATION;” and U.S. Provisional Patent Application No. 61/372,684, filed Aug. 11, 2010, and entitled “NLP-BASED SENTIMENT ANALYSIS” are incorporated herein by reference in their entireties
From the foregoing it will be appreciated that, although specific embodiments have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the invention. For example, the methods, systems, and techniques for performing sentiment analysis discussed herein are applicable to other architectures other than an NLP architecture. Also, the methods, systems, and techniques discussed herein are applicable to differing protocols, communication media (optical, wireless, cable, etc.) and devices (such as wireless handsets, electronic organizers, personal digital assistants, tablet computers, smart phones, portable email machines, game machines, pagers, navigation devices such as GPS receivers, etc.).
| Number | Name | Date | Kind |
|---|---|---|---|
| 4839853 | Deerwester et al. | Jun 1989 | A |
| 5301109 | Landauer et al. | Apr 1994 | A |
| 5317507 | Gallant | May 1994 | A |
| 5325298 | Gallant | Jun 1994 | A |
| 5331556 | Black, Jr. et al. | Jul 1994 | A |
| 5377103 | Lamberti et al. | Dec 1994 | A |
| 5619709 | Caid et al. | Apr 1997 | A |
| 5634051 | Thomson | May 1997 | A |
| 5778362 | Deerwester | Jul 1998 | A |
| 5794050 | Dahlgren et al. | Aug 1998 | A |
| 5794178 | Caid et al. | Aug 1998 | A |
| 5799268 | Boguraev | Aug 1998 | A |
| 5848417 | Shoji et al. | Dec 1998 | A |
| 5857179 | Vaithyanathan et al. | Jan 1999 | A |
| 5884302 | Ho | Mar 1999 | A |
| 5933822 | Braden-Harder et al. | Aug 1999 | A |
| 5950189 | Cohen et al. | Sep 1999 | A |
| 5982370 | Kamper | Nov 1999 | A |
| 6006221 | Liddy et al. | Dec 1999 | A |
| 6006225 | Bowman et al. | Dec 1999 | A |
| 6026388 | Liddy et al. | Feb 2000 | A |
| 6061675 | Wical | May 2000 | A |
| 6064951 | Park et al. | May 2000 | A |
| 6122647 | Horowitz et al. | Sep 2000 | A |
| 6192360 | Dumais et al. | Feb 2001 | B1 |
| 6202064 | Julliard | Mar 2001 | B1 |
| 6246977 | Messerly et al. | Jun 2001 | B1 |
| 6363373 | Steinkraus | Mar 2002 | B1 |
| 6405190 | Conklin | Jun 2002 | B1 |
| 6411962 | Kupiec | Jun 2002 | B1 |
| 6460029 | Fries et al. | Oct 2002 | B1 |
| 6510406 | Marchisio | Jan 2003 | B1 |
| 6584464 | Warthen | Jun 2003 | B1 |
| 6601026 | Appelt et al. | Jul 2003 | B2 |
| 6728707 | Wakefield et al. | Apr 2004 | B1 |
| 6732097 | Wakefield et al. | May 2004 | B1 |
| 6732098 | Wakefield et al. | May 2004 | B1 |
| 6738765 | Wakefield et al. | May 2004 | B1 |
| 6741988 | Wakefield et al. | May 2004 | B1 |
| 6745161 | Arnold et al. | Jun 2004 | B1 |
| 6757646 | Marchisio | Jun 2004 | B2 |
| 6859800 | Roche et al. | Feb 2005 | B1 |
| 6862710 | Marchisio | Mar 2005 | B1 |
| 6910003 | Arnold et al. | Jun 2005 | B1 |
| 6996575 | Cox et al. | Feb 2006 | B2 |
| 7051017 | Marchisio | May 2006 | B2 |
| 7054854 | Hattori et al. | May 2006 | B1 |
| 7171349 | Wakefield et al. | Jan 2007 | B1 |
| 7283951 | Marchisio et al. | Oct 2007 | B2 |
| 7398201 | Marchisio et al. | Jul 2008 | B2 |
| 7403938 | Harrison et al. | Jul 2008 | B2 |
| 7451135 | Goldman et al. | Nov 2008 | B2 |
| 7526425 | Marchisio et al. | Apr 2009 | B2 |
| 7788084 | Brun et al. | Aug 2010 | B2 |
| 8132103 | Chowdhury et al. | Mar 2012 | B1 |
| 8412557 | Lloyd et al. | Apr 2013 | B1 |
| 20020007267 | Batchilo et al. | Jan 2002 | A1 |
| 20020010574 | Tsourikov et al. | Jan 2002 | A1 |
| 20020022988 | Columbus et al. | Feb 2002 | A1 |
| 20020059161 | Li | May 2002 | A1 |
| 20020078041 | Wu | Jun 2002 | A1 |
| 20020091671 | Prokoph | Jul 2002 | A1 |
| 20020103789 | Turnbull et al. | Aug 2002 | A1 |
| 20020156763 | Marchisio | Oct 2002 | A1 |
| 20030004716 | Haigh et al. | Jan 2003 | A1 |
| 20030101182 | Govrin et al. | May 2003 | A1 |
| 20030115065 | Kakivaya et al. | Jun 2003 | A1 |
| 20030115191 | Copperman et al. | Jun 2003 | A1 |
| 20030233224 | Marchisio et al. | Dec 2003 | A1 |
| 20040010508 | Fest et al. | Jan 2004 | A1 |
| 20040044669 | Brown et al. | Mar 2004 | A1 |
| 20040064447 | Simske et al. | Apr 2004 | A1 |
| 20040103090 | Dogl et al. | May 2004 | A1 |
| 20040125877 | Chang et al. | Jul 2004 | A1 |
| 20040167870 | Wakefield et al. | Aug 2004 | A1 |
| 20040167883 | Wakefield et al. | Aug 2004 | A1 |
| 20040167884 | Wakefield et al. | Aug 2004 | A1 |
| 20040167885 | Wakefield et al. | Aug 2004 | A1 |
| 20040167886 | Wakefield et al. | Aug 2004 | A1 |
| 20040167887 | Wakefield et al. | Aug 2004 | A1 |
| 20040167907 | Wakefield et al. | Aug 2004 | A1 |
| 20040167908 | Wakefield et al. | Aug 2004 | A1 |
| 20040167909 | Wakefield et al. | Aug 2004 | A1 |
| 20040167910 | Wakefield et al. | Aug 2004 | A1 |
| 20040167911 | Wakefield et al. | Aug 2004 | A1 |
| 20040221235 | Marchisio et al. | Nov 2004 | A1 |
| 20040243388 | Corman et al. | Dec 2004 | A1 |
| 20050108262 | Fawcett, Jr. et al. | May 2005 | A1 |
| 20050138018 | Sakai et al. | Jun 2005 | A1 |
| 20050144064 | Calabria et al. | Jun 2005 | A1 |
| 20050149494 | Lindh et al. | Jul 2005 | A1 |
| 20050177805 | Lynch et al. | Aug 2005 | A1 |
| 20050197828 | McConnell et al. | Sep 2005 | A1 |
| 20050210000 | Michard | Sep 2005 | A1 |
| 20050216443 | Morton et al. | Sep 2005 | A1 |
| 20050267871 | Marchisio et al. | Dec 2005 | A1 |
| 20060149734 | Egnor et al. | Jul 2006 | A1 |
| 20060279799 | Goldman | Dec 2006 | A1 |
| 20070156669 | Marchisio et al. | Jul 2007 | A1 |
| 20080005651 | Grefenstette et al. | Jan 2008 | A1 |
| 20080059456 | Chowdhury et al. | Mar 2008 | A1 |
| 20080097975 | Guay et al. | Apr 2008 | A1 |
| 20080097985 | Olstad et al. | Apr 2008 | A1 |
| 20080288456 | Omoigui | Nov 2008 | A1 |
| 20080306899 | Gregory et al. | Dec 2008 | A1 |
| 20090144609 | Liang et al. | Jun 2009 | A1 |
| 20090228439 | Manolescu et al. | Sep 2009 | A1 |
| 20090319342 | Shilman et al. | Dec 2009 | A1 |
| 20100010994 | Wittig et al. | Jan 2010 | A1 |
| 20100023311 | Subrahmanian et al. | Jan 2010 | A1 |
| 20100145940 | Chen et al. | Jun 2010 | A1 |
| 20100299301 | Busch et al. | Nov 2010 | A1 |
| 20100306251 | Snell | Dec 2010 | A1 |
| 20110112995 | Chang et al. | May 2011 | A1 |
| Number | Date | Country |
|---|---|---|
| 0 280 866 | Sep 1988 | EP |
| 0 597 630 | Jul 2002 | EP |
| WO 0014651 | Mar 2000 | WO |
| WO 0057302 | Sep 2000 | WO |
| WO 0122280 | Mar 2001 | WO |
| WO 0180177 | Oct 2001 | WO |
| WO 0227536 | Apr 2002 | WO |
| WO 0233583 | Apr 2002 | WO |
| WO 03017143 | Feb 2003 | WO |
| WO 2004053645 | Jun 2004 | WO |
| WO 2004114163 | Dec 2004 | WO |
| WO 2006068872 | Jun 2006 | WO |
| Entry |
|---|
| Abraham, “FoXQ—Xquery by Forms,” Human Centric Computing Languages and Environments, Proceedings 2003 IEEE Symposium, Oct. 28-31, 2003, Piscataway, New Jersey, pp. 289-290. |
| Cass, “A Fountain of Knowledge,” IEEE Spectrum Online, URL: http://www.spectrum.ieee.org/WEBONLY/publicfeature/jan04/0104comp1.html, download date Feb. 4, 2004, 8 pages. |
| Feldman et al., “Text Mining at the Term Level,” Proc. of the 2nd European Symposium on Principles of Data Mining and Knowledge Discover, Nantes, France, 1998. |
| Ilyas et al., “A Conceptual Architecture for Semantic Search Engine,” IEEE, INMIC, 2004, pp. 605-610. |
| Jayapandian et al., “Automating the Design and Construction of Query Forms,” Data Engineering, Proceedings of the 22ndInternational Conference IEEE, Atlanta, Georgia, Apr. 3, 2006, pp. 125-127. |
| Kaiser, “Ginseng—A Natural Language User Interface for Semantic Web Search,” University of Zurich, Sep. 16, 2004, URL=http://www.ifi.unizh.ch/archive/mastertheses/DA—Arbeiten—2004/Kaiser—Christian.pdf, pp. 1-84. |
| Liang et al., “Extracting Statistical Data Frames from Text,” SIGKDD Explorations, Jun. 2005. vol. 7, No. 1, pp. 67-75. |
| Littman et al., “Automatic Cross-Language Information Retrieval using Latent Semantic Indexing,” In Grefenstette, G., editor, Cross Language Information Retrieval. Kluwer, 1998. |
| Nagao et al., “Semantic Annotation and Transcoding: Making Web Content More Accessible,” IEEE Multimedia, IEEE Computer Society, US. 8(2):69-81, Apr. 2001. |
| Nguyen et al., “Accessing Relational Databases from the World Wide Web,” SIGMOD Record, ACM USA, Jun. 1996, vol. 25, No. 2, pp. 529-540. |
| Pohlmann et al., “The Effect of Syntactic Phrase Indexing on Retrieval Performance for Dutch Texts,” Proceedings of RIAO, pp. 176-187, Jun. 1997. |
| Number | Date | Country | |
|---|---|---|---|
| 20120041937 A1 | Feb 2012 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61372684 | Aug 2010 | US |
