Users often provide feedback, in the form of reviews, regarding offerings (products or services) of different enterprises. As examples, users can be external customers of an enterprise, or users can be internal users within the enterprise. An enterprise may wish to use feedback to improve their offerings. However, there can be potentially a very large number of received reviews, which can make meaningful analysis of such reviews difficult and time-consuming.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
Some embodiments are described with respect to the following figures:
An enterprise (e.g. a company, educational organization, government agency, an internal department within any of the foregoing entities, etc.) may collect feedback from users (which can either be external users or internal users) to better understand user sentiment regarding an offering of the enterprise. Feedback can be received in the form of reviews. An offering can include a product or a service provided by the enterprise (either to an external user or to an internal user). A “sentiment” refers to an attitude, opinion, or judgment of a human with respect to the offering.
An enterprise can provide an online website to collect feedback from users. Alternatively or additionally, the enterprise can also collect feedback through telephone calls or through paper survey forms. Furthermore, feedback can be collected at third party sites, such as travel review websites, product review websites, and so forth. Some third party websites provide professional reviews of offerings from enterprises, as well as provide mechanisms for users to submit their individual reviews.
Additionally, if the users are internal users of enterprise, various mechanisms can also be provided within the enterprise for internal users to submit feedback. If there are a relatively large number of users, then there can be relatively large amounts of user feedback.
Generally, sentiment analysis involves identifying each term appearing in the reviews (which can be in the form of unstructured data) and assigning some score to the term, which can be a negative score, neutral score, or positive score to express whether the term is associated with negative sentiment, neutral sentiment, or positive sentiment. Determining the score can be based on opinion words appearing in portions (e.g. sentences, paragraphs, other sections) that are near a corresponding term. “Unstructured data” refers to data that does not have a predefined format or schema (such as a schema of a relational database management system).
A “term” refers to a word or a combination of words for which a sentiment can be expressed. As examples, a term can be a noun or compound noun (a noun formed of multiple words, such as “customer service”) that exists in the feedback information. As other examples, a term can be any other word or combination of words that an analyst wishes to consider, where the word(s) can be an attribute (noun or compound noun), an adjective, a verb, and so forth. Sentiment words (or opinion words) in the feedback information can also be identified, where sentiment words include individual words or phrases (made up of multiple words) that express an attitude, opinion, or judgment of a human. Examples of sentiment words include “bad,” “poor,” “great performance,” “fast service,” and so forth.
Sentiment scores can be assigned to respective terms based on use of any of various different sentiment analysis techniques, which involve identifying words or phrases in the data records that relate to sentiment expressed by users with respect to each attribute. A sentiment score can be generated based on the identified words or phrases. The sentiment score provides an indication of whether the expressed sentiment is positive, negative, or neutral. The sentiment score can be a numeric score, or alternatively, the sentiment score can have one of several discrete values (e.g. Positive, Negative, Neutral).
Although assigning sentiment scores to terms that may appear in reviews may be useful for various purposes, it is noted that identifying individual terms by themselves may not adequately allow for identification of patterns of terms that may be present in the reviews. Patterns of terms may be based on co-occurrence of the terms within the reviews, which can be co-occurrence of the terms in sentences within the reviews, paragraphs within the reviews, other sections of the reviews, or the entirety of the reviews. For example, in the context of reviews of a given hotel, the hotel owner may wish to find which term is most closely related to the term “hotel room.” Example terms that can be related to “hotel room” can include “bathroom,” “carpet,” and so forth.
In accordance with some implementations, an association data structure (which can be in the form of an association matrix or other type of data structure) can be provided to visualize association among co-occurring terms in input data (which can include reviews in the form of documents or other objects). An association between or among two or more terms refers to co-occurrence of the two or more terms in a review or some portion of the review (e.g. sentence, paragraph, or other section). The visualized association data structure shows association patterns of the co-occurring terms that may be of interest to users. In some implementations, the visualized association data structure allows for visualization of the association patterns in a single display even if there are a large number of co-occurring terms. In accordance with some implementations, terms are visualized only as part of the association data structure. In this association data structure, visual elements representing the terms are assigned respective colors (or other visual indicators) to indicate corresponding sentiments as expressed in sentences (or other portions of a review) with respect to the terms.
Binary association measures can be computed using any one of various different techniques. As examples, such techniques include a hypothesis testing technique (in which a tester starts with a null hypothesis and an alternative hypothesis performs an experiment, and then decides whether to reject the null hypothesis in favor of the alternative hypothesis—the hypothesis testing is basically a binary classification of the hypothesis under study); a likelihood statistics technique, such as a likelihood ratio test technique (which is a statistical test used to compare the fit of two models, one of which (the null model) is a special case of the other, the alternative model), where the test is based on a likelihood ratio that expresses how many times more likely the data is under one model than the other); a phi correlation technique (which is a technique for correlating the association between two variables); an information theory technique, such as a mutual information technique (which is a technique to determine a quantity, referred to as the mutual information, that measures the mutual dependence of two variables), or some other association or correlation technique for correlating pairs of variables (which in some implementations include terms found in feedback reviews).
The process of
Extended associations are derived based on binary associations. Stated differently, binary associations can be extended beyond binary relations to depict relations among more than two terms. In some examples, binary associations can be merged to form extended associations. In the following example, the following binary associations can be merged: (a, b), (a, c), (b, c), where a, b, c represent terms that can be found in reviews, and each of (a, b), (a, c), (b, c) represents a corresponding binary association between the respective pair of terms in parentheticals. The foregoing binary associations are a subset of a collection (A) of binary associations, which can be a collection of hypothesis test associations, a collection of likelihood ratio associations, a collection of phi associations, or a collection of mutual information associations, as examples.
In some examples, the binary associations (a, b), (a, c), and (b, c) can be merged if the following condition is satisfied:
(a,b)εA(a,c)εA
(b,c)εA, (the “
” symbol represents logical AND)
I(a,b,c)>max(I(a,b),I(a,c),I(b,c)),
count(a,b,c)>lowerbound.
In the foregoing, I( ) represents a function for computing an association measure. For example, I( ) can represent a function for computing a pointwise mutual information, according to the following formula (in the binary case):
I(a,b)=p(a,b)/(p(a)*p(b)),
where p( ) represents a probability of the corresponding item—e.g. p(a) represents the probability of the term a occurring in received feedback, and p(a,b) represents the probability of both terms a and b occurring in received feedback.
Thus, I(a,b) represents an example score (pointwise mutual information) indicating the binary association between terms a and b. In the more general sense, when correlating more than two terms, the following extended association measure can be used:
I(a,b, . . . ,n)=p(a,b, . . . ,n)/(p(a)*p(b)* . . . *p(n)),
where I(a, b, . . . , n) represents an example measure of an extended association among terms a, b, . . . , n. In other words, the extended association measure for the extended association of terms a, b, c is represented by I(a, b, c) in the foregoing example.
Also, count(a) represents the count of the number of sentences that contain term a, and lowerbound represents a predefined threshold. In the condition above, count(a, b, c) represents the count of the number of sentences (or reviews or other sections of reviews) that contain all of the terms a, b, c.
The specific condition set forth above for merging the foregoing binary associations is true if each of the binary associations is a member of A, the extended association measure I(a, b, c) is greater than the maximum of the following binary association measures I(a, b), I(a, c), and I(b, c), and the count(a, b, c) is greater than the lower bound predefined threshold, lowerbound. Although a specific condition for merging binary associations is provided above, it is noted that in alternative examples, other conditions can be specified for merging binary associations to form extended associations, where such condition for merging is based on binary association measures.
The process then constructs (at 116) an association data structure according to the binary and extended associations, similar to task 104 in
Each visual element is associated with a respective color (or alternatively, another type of visual indicator), which can be used to indicate the corresponding sentiment expressed with respect to the term, where the sentiment can be a positive sentiment, a neutral sentiment, or a negative sentiment. In some examples, a green color (light green or darker green) can indicate a positive sentiment, where the darker shade of green represents a more positive sentiment than a lighter shade of green. A gray color assigned to a visual element indicates a neutral sentiment associated with the corresponding term, while a red color (lighter shade of red or darker shade or red) represents a negative sentiment expressed with respect to the respective term. A darker shade of red represents a more negative sentiment than a lighter shade of red.
Entries 202B and 202P each contains only one visual element (206 in entry 202B and 208 in entry 202P)—this indicates that no co-occurring terms are associated with entries 202B and 202P.
In
Each entry 202 of the association matrix shown in
In some examples, each association (binary association or extended association) is represented by a high-dimensional numerical vector (“association vector”) that contains one dimension for each review in the corpus. This association vector can have a relatively large number of bit positions, where each bit position corresponds to a respective review. If a review contains the respective association (binary association or extended association), then the association vector corresponding to the association has an entry “1” at the respective bit position, and “0” otherwise. Although “1” and “0” are used, it is noted that in alternative implementations, different values can be used to indicate whether the corresponding review contains the respective association.
Each entry 202 in
In other implementations, instead of using lines to interconnect the entries 202 of the association data structure, other interconnecting elements can be used, with each interconnecting element connecting at least two entries of the association data structure, and with each interconnecting element having an indicator to indicate a degree of association between or among the entries.
In some examples, various visual analytic techniques can be applied to the visualized association data structure. For example, a user can move a cursor (with a mouse or other input device) over a portion of the visualized association data structure (e.g. over a visual element corresponding to a term), and view further details regarding the term and its association(s) with other terms. Moreover, a user can select a portion of the visualized association data structure (such as by drawing a box around the selected portion using a rubber-banding operation, for example) to zoom (drill down) into the selected portion. As further examples, a user can click on the visual element of a term of interest to quickly find association(s) of this term.
The storage media 406 can be implemented as one or multiple computer-readable or machine-readable storage media. The storage media can include different forms of memory including semiconductor memory devices such as dynamic or static random access memories (DRAMs or SRAMs), erasable and programmable read-only memories (EPROMs), electrically erasable and programmable read-only memories (EEPROMs) and flash memories; magnetic disks such as fixed, floppy and removable disks; other magnetic media including tape; optical media such as compact disks (CDs) or digital video disks (DVDs); or other types of storage devices. Note that the instructions discussed above can be provided on one computer-readable or machine-readable storage medium, or alternatively, can be provided on multiple computer-readable or machine-readable storage media distributed in a large system having possibly plural nodes. Such computer-readable or machine-readable storage medium or media is (are) considered to be part of an article (or article of manufacture). An article or article of manufacture can refer to any manufactured single component or multiple components. The storage medium or media can be located either in the machine running the machine-readable instructions, or located at a remote site from which machine-readable instructions can be downloaded over a network for execution.
In the foregoing description, numerous details are set forth to provide an understanding of the subject disclosed herein. However, implementations may be practiced without some or all of these details. Other implementations may include modifications and variations from the details discussed above. It is intended that the appended claims cover such modifications and variations.