Patent Application
20170200205
Field
This disclosure is generally related to user review analysis. More specifically, this disclosure is related to a method and system for identifying and analyzing surprises in user reviews.
Related Art
With the advancement of the computer and network technologies, various operations performed by users from different applications lead to extensive use of web services. This proliferation of the Internet and Internet-based user activity continues to create a vast amount of digital content. For example, multiple users may concurrently provide reviews (e.g., fill out surveys) about a business entity via different applications, such as mobile applications running on different platforms, as well as web-interfaces running on different browsers in different operating systems. Furthermore, users may also use different social media outlets to express their reviews about the business entity.
An application server for the business entity may store the reviews in a local storage device. A large number of users providing reviews can lead to a large quantity of data for the application server, which may not be possible for humans to identify and process. As a result, different data mining technique can be applied to obtain overall insight into the user reviews. However, these data mining techniques typically focus on mainstream features. As a result, these data mining techniques may fail to capture discrepancies in user reviews (e.g., positive opinion about that mainstream feature but a negative overall opinion).
Although a number of methods are available for review analysis, some problems still remain in analysis of discrepancy in user reviews.
One embodiment provides a system that detects and analyzes surprises in user reviews. During operation, the system stores, in a storage device, a plurality of user reviews. A user review includes a recommend score indicating a likelihood of recommending, and one or more feature values indicating opinions about individual features in the user review. The system determines a first user review from the plurality of user reviews to be a first surprise in response to detecting a discrepancy between a recommend score and feature values of the first user review. The system then performs a text analysis on the first surprise to discover impactful features in the surprise.
In a variation on this embodiment, the system identifies the impactful features based on the respective importance of features of a respective user review in the plurality of user reviews. The system trains a prediction model to predict a recommend score based on feature values of the identified impactful features.
In a further variation, the system determines the first surprise by determining whether a predicted recommend score deviates from the recommend score of the first user review.
In a further variation, prior to identifying the impactful features, the system fills in missing values of features of a respective user review in the plurality of user reviews.
In a variation on this embodiment, the system identifies a plurality of surprises from the plurality of user reviews. The system clusters synonymous words in the identified surprises into a word cluster, and associates the word cluster and reviews comprising the synonymous words with a corresponding meaningful feature.
In a further variation, the system determines a sentiment category for the feature. The sentiment category is one of: positive, negative, no opinion, and mixed opinion.
In a further variation, the system displays in a presentation interface one or more surprises associated with the feature in response to a user selecting the feature in the presentation interface.
In a variation on this embodiment, the system determines one or more clusters of user reviews from the plurality of user reviews by grouping the user reviews with similar feature values. The system then identifies the outlier user reviews, which deviate significantly from the determined clusters, as the surprises.
In the figures, like reference numerals refer to the same figure elements.
The following description is presented to enable any person skilled in the art to make and use the embodiments, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Thus, the present invention is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.
Embodiments of the present invention provide a system, which analyzes surprises in user reviews. Due to ease of access via the Internet, a large number of users provide review about a business entity. Such reviews can include surveys (e.g., regarding customer experience) comprising numerical data (e.g., on the scale of 1-10, how would you rate the cleanliness of the guestroom), and textual comments (e.g., a social media post). However, a review can include a discrepancy. In this disclosure, a review with such a discrepancy can be referred to as a surprise. For example, in the context of a customer experience survey about a service, individual numerical data fields of the survey can indicate a good experience but the survey can have an negative recommend score (e.g., a low likelihood of recommending the service). These surprises usually indicate specific problems, which a business entity can address.
Surprises can offer key insights, such as isolated problems associated with a business entity. Isolated problems are often more informative than multiple coexisting problems, as the former gives a clearer attribution than the latter. For instance, an unsatisfied customer can report a single problem. This is an isolated problem, and a solution to this problem may satisfy this customer and improve his/her experience. On the other hand, if that problem coexists with several other problems, identifying the key factors of customer dissatisfaction becomes harder.
However, with existing technologies, the data mining techniques provide analysis of specific mainstream features (e.g., how a particular feature of the business entity is resonating with the users). As a result, these techniques may fail to recognize the surprises. To solve this problem, embodiments of the present invention provide a system that facilitates detection and analysis of surprises from a large set of user reviews. The system screens a large number of reviews and detects the reviews with surprises (e.g., with significant data discrepancies) based on feature extraction, prediction, and outlier detection. The system then processes the detected surprises using text analytics techniques, such as feature discovery and sentiment analysis, to find insights (e.g., common features and sentiment) into the detected surprises. The system can also provide representative examples based on information retrieval techniques via a presentation interface.
Typically, a review includes an overall indication whether a user has expressed a positive or negative sentiment in the review. This overall indication can be referred to as a “recommend score” (e.g., how likely the user is going to commend the service of the business entity). If a user expresses a positive “recommend score” in a review (e.g., a 9 or 10 out of 10), the user can be referred to as a “promoter.” On the other hand, if the user expresses a negative “recommend score” in a review (e.g., a 6 or lower), the user can be referred to as a “detractor.” Otherwise, the user can be referred to as a “neutral.” A review can also include opinions about specific features (e.g., for a hotel, the opinion can be about the cleanliness of a guestroom and friendliness of the staff). These opinions can be represented by different data fields in the review.
Suppose that review 152 is an instance of an “expected” review, which indicates that user 122 is a promoter and review 152 has positive opinions about individual features, or user 122 is a detractor and review 152 has negative opinions about individual features. In this example, user 124 is a promoter and review 154 has negative opinions about individual features. Here, based on the negative opinions, review 154 should have indicated user 124 to be a detractor. However, the observed recommend score of review 154 indicates user 124 to be a promoter. Since the opinions of individual features shows significant deviation from the observed recommend score, review 154 can be considered as a surprise. In the same way, review 156 can also be a surprise, where user 126 is a detractor and review 156 has positive opinions about individual features. These surprises can indicate specific problems, which the business entity can address.
However, with existing technologies, the data mining techniques may not be able to recognize surprises 154 or 156 from expected review 152. For example, such a technique may reveal that users 122 and 124 have negative opinions about a specific feature, without detecting that user 124 might be a promoter. To solve this problem, embodiments of the present invention provide a surprise analysis system 160 that facilitates detection and analysis of surprises from a large set of reviews 152, 154, and 156. System 160 can operate on an analysis server 146, which can be a separate computing device, a virtual machine on a host machine, or an appliance. It should be noted that, since a data mining technique running on a generic computing system may not be able to identify the surprises, system 160 improves the functioning of server 146.
During operation, server 146 obtains reviews 152, 154, and 156 from application server 142 and stores these reviews in storage device 148. System 160 includes a surprise detection module 162, which screens a large number of reviews 152, 154, and 156 and detects surprises 154 and 156 based on feature extraction, prediction, and outlier detection. The system also includes a text analysis module 164, which processes detected surprises 154 and 156 using text analytics techniques, such as feature discovery and sentiment analysis, to find insights into surprises 154 and 156. In some embodiments, the system also includes a presentation interface 166, which provides visual representations of the insights and representative examples based on information retrieval techniques.
In some embodiments, system 160 derives whether a user is a promoter based on textual analysis of a review. For example, in a social media post, a user may not numerically express a recommend score. However, based on a textual analysis of the words or word combinations (e.g., “stay again” or “won't go back”), system 160 can determine whether the user is a promoter or a detractor. Similarly, system 160 can derive whether the user's opinion about a particular feature is positive or not based on the textual analysis (e.g., “clean” or “smelly”) and can assign a corresponding feature value.
Text analysis module 164 obtains the detected surprises and analyzes them for insights. Text analysis module 164 includes a feature discovery mechanism 173, which uses text analytics techniques to determine the features that caused the surprise. Text analysis module 164 also includes a sentiment analysis mechanism 174, which determines the sentiment associated with those features. In this way, text analysis module 164 provides insights (e.g., common features and sentiment) into the detected surprise. Text analysis module 164 can also include an information retrieval mechanism 175, which facilitates interaction with text analysis module 164 by allowing a user to retrieve examples on demand. Information retrieval mechanism 175, in conjunction with presentation interface 166, allows users to retrieve the examples based on a feature (e.g., sentences/surveys associated with a feature) or an example (e.g., sentences/surveys similar to the current example).
In some embodiments, presentation interface 166 obtains the insights and examples from text analysis module 164. Presentation interface 166 can be an interface for a computing device (e.g., a monitor of a desktop or laptop), or an adjusted interface for a cellular (e.g., a cell phone or a tablet) device. Presentation interface 166 includes a visual representation mechanism 176, which presents the insights and sentiments in a graphical or textual representation. Presentation interface 166 can also include an interactive interface 177, which allows the user to use information retrieval mechanism 175 to extract features and examples for a specific feature. In some embodiments, interactive interface 177 also provides recommendations (e.g., from a user's suggestions) associated with a particular feature or example. Examples of a presentation interface include, but are not limited to, a graphical user interface (GUI), a text-based interface, and a web interface.
In this way, surprise analysis system 160 can filter out a few surprises from a large set of reviews 150. For example, surprise detection module 162 filters out surprises from a large number of reviews so that the user workload of reading the surprises stays manageable. Surprise analysis system 160 can further analyze the surprises to provide a handful of insights, which the business entity can address. In addition, based on the detected surprises, the business entity can determine whether important data aspects are captured in a survey.
On the other hand, a larger range of scores can indicate a detractor (e.g., [0, 6)). Since set of reviews 150 is large, such an uneven range of scores can create a bias for the detractors in the surprise detection process. Preprocessing mechanism 302 thus uses the piece-wise linear scaling mapping to reduce the bias. In some embodiments, the piece-wise linear scaling mapping for the recommend scores is from [0, 10] to [4, 10]. Compressing the overall value range, and in particular, the detractor value range enables a more accurate prediction (e.g., as performed by prediction mechanism 172 of
Feature extraction mechanism 171 includes a preprocessing mechanism 304 for the data fields representing the opinions about the features. Preprocessing mechanism 304 identifies the missing values for a particular feature (e.g., a question missing an answer in a survey) and can fill in these values. Preprocessing mechanism 304 calculates correlation with other similar users' opinions about the feature (e.g., how other similar users have answered the corresponding survey question). In some embodiments, preprocessing mechanism 304 can derive whether the user's opinion about the feature is positive or not based on the textual analysis. For example, if the review is a social media post for a hotel, preprocessing mechanism 304 can look for specific words associated with a hotel stay (e.g., “cleanliness” and “lobby”).
Feature extraction mechanism 171 also includes a feature selection mechanism 306 for selecting impactful features of a review. In this way, feature selection mechanism 306 facilitates “noise reduction” for the surprise detection. For example, feature selection mechanism 306 removes the features that are empty or insignificant (e.g., can have only one meaningful answer). Feature selection mechanism 306 can also discard the sparsely populated features, which do not have enough data samples (e.g., less than 30% populated). Feature selection mechanism 306 then orders the features based on a correlation coefficient or mutual information associated with the features. This ordering represents the features that are most significant in indicating whether a user is a promoter or a detractor.
Prediction mechanism 172 obtains the ordered impactful features from feature selection mechanism 306 and applies a prediction model, as described in conjunction with
If a recommend score deviates significantly from a predicted score of a review (e.g., more than a threshold value), outlier detection mechanism 310 marks that review as a surprise. In some embodiments, system 160 maintains the surprises in a database in storage device 148. System 160 can also have a flag indicating a surprise in the database storing the reviews. In the example in
A prediction model can be supervised, where an observed value of a recommend score and respective values of impactful features in a respective review are used to train the prediction model. In some embodiments, system 160 uses unsupervised clustering to compute clusters of the respective values of the impactful features. These values can represent the expected reviews. If system 160 identifies data points away from the clusters, system 160 identifies the review associated with the identified data points as a surprise. Examples of clustering include, but are not limited to, K-means, density-based clustering, spectral clustering, Density-based spatial clustering of applications with noise (DBSCAN), and mixture models.
The system then predicts a recommend score for a review by applying a prediction model to the respective values of the impactful features (operation 360). The system compares the predicted recommend score with the recommend score in the review (operation 362) and checks whether they have significant deviation (operation 364). If the predicted recommend score significantly deviates from the recommend score in the review, the system determines the review to be a surprise (operation 366). Otherwise, the system determines the review to be consistent (operation 368). It should be noted that if an unsupervised clustering mechanism is used instead of a prediction mechanism, a user review is compared against the identified clusters. If the review is an outlier significantly away from any cluster, the review is detected as a surprise.
The system also groups synonymous words into word clusters and generates a seed to identify cluster heads for the word clusters (operation 438). For example, similar words, such as “taxi,” “cab,” “bus,” and “shuttle” can be grouped into a cluster. In the context of the reviews, if the word “taxi” most frequently represents a feature, “taxi” can be selected as the seed and the head for the cluster. Other words, such as “cab,” “bus,” and “shuttle,” can be clustered to the seed. The system then associates features with corresponding word clusters and textual sentences comprising the synonymous words for feature labeling (operation 440). This allows the system to present examples of a feature to a user.
Surprise analysis system 160 uses text analytics methods, such as feature discovery, sentiment analysis, and information retrieval, to obtain insights, such as common features and sentiments from the identified surprises. Surprise analysis system 160 can further ease a user's effort at understanding the surprises by representing them in a presentation interface 166.
Presentation interface 166 provides a visual representation 512 of the impactful features. Visual representation 512 can be generated by visual representation mechanism 176 and can represent the insights (e.g., emotions) obtained from text analysis module 164, as described in conjunction with
Presentation interface 166, in conjunction with text analysis module 164 in the example in
Surprise analysis system 618 can include instructions, which when executed by computer and communication system 602, can cause computer and communication system 602 to perform the methods and/or processes described in this disclosure. Surprise analysis system 618 further includes instructions for detecting surprises from user reviews (surprise detection mechanism 620). Surprise analysis system 618 can also include instructions for analyzing text in the detected surprises (text analysis mechanism 622). Surprise analysis system 618 can include instructions for presenting the analyzed surprises in a presentation interface (presentation mechanism 624). Surprise analysis system 618 can also include instructions for exchanging information with other devices (communication mechanism 628).
Data 632 can include any data that is required as input or that is generated as output by the methods and/or processes described in this disclosure. Specifically, data 632 can store one or more of: a first database comprising the user reviews, and a second database comprising the surprises. In some embodiments, the first database can include a flag indicating a review to be a surprise.
The data structures and code described in this detailed description are typically stored on a computer-readable storage medium, which may be any device or medium that can store code and/or data for use by a computer system. The computer-readable storage medium includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs), DVDs (digital versatile discs or digital video discs), or other media capable of storing computer-readable media now known or later developed.
The methods and processes described in the detailed description section can be embodied as code and/or data, which can be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and/or data stored on the computer-readable storage medium, the computer system performs the methods and processes embodied as data structures and code and stored within the computer-readable storage medium.
Furthermore, the methods and processes described above can be included in hardware modules or apparatus. The hardware modules or apparatus can include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors that execute a particular software module or a piece of code at a particular time, and other programmable-logic devices now known or later developed. When the hardware modules or apparatus are activated, they perform the methods and processes included within them.
The foregoing descriptions of embodiments of the present invention have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the present invention to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the present invention. The scope of the present invention is defined by the appended claims.
