Patent Application
20160253684
Currently, many online sites allow users to write reviews for objects available on or referred to by the site, such as applications (i.e., an “app”), movies, books, hotels, restaurants, or the like. The reviews can be associated with the object and can be relied upon by other users in making viewing or purchasing decisions. A user interface (“UI”) for providing a review often includes a text box for the user to enter the review. There can also be a place for the user to enter an overall score for the reviewed object.
It can be difficult for a user to write a review on a smartphone or other mobile device because such devices often have smaller screen sizes and lack full-sized keyboards. As a result, users are often reluctant to provide reviews, or else provide abbreviated text that may be less useful to others. Another problem relates to the large number of reviews that may be posted for an object. For example, a popular movie may have hundreds of screen pages of reviews submitted by users. Since few users have the time to read all or even most reviews for some objects, it can be difficult for the user to gain an accurate sense of other users' experiences of the object.
Embodiments of the disclosed subject matter provide systems and methods to allow users to efficiently provide high-quality and structured reviews with minimal input to a user interface. The quality of reviews and the reduced input required to enter review may attract more users to provide reviews and/or read reviews.
According to an embodiment of the disclosed subject matter, a method includes providing, with a computing device having an input device and a display device, a user interface to receive a review for an object from a reviewer, selecting a set of tags from an object tag collection stored in a database communicatively coupled to the computing device according to the object and the reviewer, displaying, by the display device of the computing device, the selected set of tags on a display, receiving an input, by the input device, to remove one or more of the displayed tags, and storing, by a storage device, the remaining tags of the set of tags that are submitted according to the received input for the object.
According to an embodiment of the disclosed subject matter, a system is provided that includes a computing device to provide a user interface to receive a review for an object from a reviewer, a database communicatively coupled to the computing device to store an object tag collection, a display device, coupled to the computing device, to display a set of tags selected by the computing device from the database, an input device, coupled to the computing device, to receive an input to remove one or more of the displayed tags on the display device, and a storage device, coupled to the computing device, to store the remaining tags of the set of tags that are submitted according to the received input for the object.
An embodiment of the disclosed subject matter may provide a non-transitory computer readable medium including instructions, that when executed by a computer, perform a method including selecting, with a server, a set of tags from an object tag collection stored in a database communicatively coupled to the server according to an object to be reviewed and a reviewer, providing the selected set of tags from the server to a computing device of the reviewer, receiving, with the server, input data from the computing device to remove one or more of the tags from the selected set of tags, and storing, by a storage device, the remaining tags of the set of tags so as to be associated with the reviewer and the object.
According to an embodiment of the disclosed subject matter, means for structured reviews through auto-generated tags are provided so as to include providing, with a computing device having an input device and a display device, a user interface to receive a review for a object from a reviewer, selecting a set of tags from an object tag collection stored in a database communicatively coupled to the computing device according to the object and the reviewer, displaying, by the display device of the computing device, the selected set of tags on a display, receiving an input, by the input device, to remove one or more of the displayed tags, and storing, by a storage device, the remaining tags of the set of tags that are submitted according to the received input for the object.
Additional features, advantages, and embodiments of the disclosed subject matter may be set forth or apparent from consideration of the following detailed description, drawings, and claims. Moreover, it is to be understood that both the foregoing summary and the following detailed description are illustrative and are intended to provide further explanation without limiting the scope of the claims.
The accompanying drawings, which are included to provide a further understanding of the disclosed subject matter, are incorporated in and constitute a part of this specification. The drawings also illustrate embodiments of the disclosed subject matter and together with the detailed description serve to explain the principles of embodiments of the disclosed subject matter. No attempt is made to show structural details in more detail than may be necessary for a fundamental understanding of the disclosed subject matter and various ways in which it may be practiced.
Embodiments of the disclosed subject matter can efficiently collect high-quality user reviews of objects while requiring less textual input from the reviewers. This can make it easier for users to submit more accurate reviews. Further, the submitted reviews may be more accurately summarized. Even a large number of reviews can thus be made more useful to other users.
In embodiments of the disclosed subject matter, structured reviews can be read by computers and/or machines so that useful information may be mined and/or extracted from the reviews. For each object (e.g., an app, a movie, a book, a hotel, a restaurant, a product, or the like), a collection of machine-readable tags can be generated that may be related to the object. For each reviewer, a set of tags from the collection can be selected based on their possible relevance to the object and presented to the reviewer. The reviewer may remove the tags that are not related to the object and/or are inconsistent with the reviewer's opinion about the object. The tags left by the reviewer may be stored by the system in connection with the object as a structured review.
In an embodiment of the disclosed subject matter, a method 100 shown in
A set of tags can be selected from an object tag collection stored in a database communicatively coupled to the computing device according to the object and the reviewer at operation 120. In some embodiments, as shown in
Topic tags may describe the topics and/or genres about the objects. For example, a social networking application to be reviewed (i.e., an object to be reviewed) can be tagged with “Topic: Social Network”, “Topic: Instant Messaging”, or the like. Quality tags may describe the quality of the objects. For example, when the object to be reviewed is an application, the quality tags may be “Quality: Slow”, “Quality: Fast”, “Quality: Stable”, “Quality: Poor”, or the like. That is, the quality tags may relate to the speed, stability, and/or overall usefulness of the application. In another example, where the object to be reviewed is a hotel, the quality tags may include “Quality: Clean”, “Quality: Dirty”, “Quality: Friendly”, “Quality: Quiet”, or the like. That is, the quality tags may relate to the cleanliness, the demeanor of the staff, the ambiance of the hotel, or the like. Price tags may describe the price of the object with respect to value. For example, the price tags may include “Price: Expensive”, “Price: Free”, “Price: $50-$100” or the like. In some embodiments, types of tags may be defined depending on the characteristics of the object or objects to which they pertain. Such tags may be referred to as definable tags.
The display device (e.g., display 22 shown in
Embodiments of the disclosed subject matter may include storing, by the database, a date and time of a review activity by the reviewer. That is, a date and time of any user selections, for example, in UI 300 shown in
The method of the disclosed subject matter may include predicting, with the computing device (e.g., device 20 shown in
In embodiments of the disclosed subject matter, the set of objects reviewed and the review tags associated with each object submitted according to the received user input from the reviewer may be stored in the database (e.g., database 15 shown in
The selecting the set of tags from an object tag collection in the method 100 described above may include, for example, selecting the set of tags so as to minimize the reviewer inputs for each review and maximize information obtained from the review. That is, the system may use the data stored in the database (e.g., amount of review time or the like) to select tags that may reduce the amount of time and/or the number of inputs provided by a reviewer for each review. By increasing the accuracy of selection of tags, the number of reviewer inputs may be reduced and the review information obtained from a user about an object may be increased.
Object tagging module 210 of the structured review system 200 may annotate each object with a collection of machine-readable tags. The tags may include topic tags, quality tags, price tags, and definable tags. For example, a social transit application to be reviewed can be tagged with “Topic: Transit”, “Topic: Subway”, or the like. Quality tags may describe the quality of the objects. For example, when the object to be reviewed is an application, the quality tags may be “Quality: Accurate”, “Quality: Unreliable”, or the like. In another example, where the object to be reviewed is a restaurant, the quality tags may include “Quality: High-End”, “Quality: Fast Food”, “Quality: Fancy”, “Quality: Casual”, or the like. Price tags may describe the price of the object with respect to value. For example, the price tags may include “Price: Expensive”, “Price: Moderate”, “Price: Cheap,” “Price: $20-$50,” “Price: $50-100,” or the like. In some embodiments, types of tags may be defined depending on the characteristics of the objects. Such tags may be referred to as definable tags.
The review user interface (UI) 220 of the structured review system 200 provides a user interface for a reviewer. Given the object (e.g., an app, a movie, a book, a hotel, a restaurant, a product, or the like) and the reviewer, the Tag Selection module 270 (as described in detail below) selects a set of tags from an object tag collection and present the tags within the UI for the reviewer. The UI may be displayed for example, on display 22 of device 20 shown in
UI 300 shown in
As shown in
The reviewer model 240 of the structured review system 200 may predict the probability that the reviewer keeps a tag, given the reviewer's behavior history and object attributes. That is, the reviewer model 240 may predict the probability (P) of keeping a tag, given the reviewer, the object (“doc”), and the tag, as represented by P (keep|reviewer, doc, tag).
The reviewer model 240 may be built and/or developed in a plurality of ways. The following example provides a machine-learning based solution to build the reviewer module 240. Each triple (reviewer, doc, tag) can be represented by a feature vector. For example, the reviewer can have features like demographics, language, interests, or the like. The object (“doc”) can have features like the topic, number of likes, number of dislikes, quality, or the like.
The reviewer model 240 may process the review logs (e.g., the review logs of the review log module 230). Each previous user review may be treated by the reviewer module 240 as a machine learning training sample. If the reviewer elects to keep the tag presented to him/her (e.g., by the UI 300 shown in
The reviewer model 240 may train a classification model from the training samples. Machine learning models that can be used as a classification model include a SVM (Support Vector Machine) model, logistic regression, or the like.
The tag corpus module 250 of the structured review system 200 shown in
The tag model 260 of the structured review system 200 shown in
That is, the above probability considers the number of times that a tag presented (e.g., in a user interface 300) to a reviewer is retained (e.g., tags that are not removed by a reviewer), and the number of times a tag is presented to a reviewer. It can indicate the relative popularity or unpopularity of a tag among reviewers. This information may be stored in the database, and may be associated with a particular reviewer and/or a particular object.
The probability that the object does not have a tag can be determined by:
P
The entropy (E) of each tag in a object (i.e., “doc”) may be computed from the above probabilities by the following equation:
In the equation above, t may be an individual tag that is part of the set of tags T.
The more confidence that the tag is associated with the object, the smaller the entropy. The information gained from the tags can be measured by:
H(doc)=−ΣtagE(tag|doc, s(tag))
The tag selection module 270 of the structured review system 200 shown in
K(T)=|T|−ΣtεTP (keep|reviewer, doc, t)
In the equation above, T may be set of tags selected and presented to the reviewer, and K is the approximate number of reviewer inputs (e.g., with K(T) being the approximate number of reviewer inputs for the set of tags selected and presented to the user). It may be assumed that more information can be gained if the reviewer can submit tags that fewer previous reviewers have evaluated. For example, if most of previous reviewers have submitted the tag “Topic: Social Network” for a social networking application, then the information gain can be very small if this tag is presented again to a present reviewer. The information gain can be approximated by information change ΔH(doc, T) after the tags are kept or removed by the reviewer:
ΔH(doc, T)=H(doc, T)−H(doc)
where H(doc, T) and H(doc) are the information after and before the reviewer submits the tags, respectively.
In the systems and methods disclosed herein, there can be different ways to select tags. In some embodiments, the systems and methods may find the set of tags T that maximize:
In the above equation, α may be a predefined parameter that controls an importance of K or ΔH, where ΔH is the information change (e.g., the amount of information to be gained from the reviewer input regarding an object). The objective for each individual tag t may be computed by the following:
K(t)=1−P(keep|reviewer, doc, t)
ΔH(doc, t)=E(t|doc, s(t))−E(t|doc, s′(t))
F(t)=−K(t)+αΔH(doc, t)))
In the above equations, s(t) and s′(t) are review stats before and after a current reviewer adds tag t to the object, and F(t) relates to the number of user inputs and information gained.
The above objective function can then be factorized as
That is, the optimal set of tags T is the tags that have largest F(t).
In situations in which implementations collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features collect user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current location), or to control whether and/or how to receive content from the content server that may be more relevant to the user. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. Thus, the user may have control over how information is collected about the user and used by a system as disclosed herein.
Embodiments of the presently disclosed subject matter of structured review systems and methods may be implemented in and used with a variety of component and network architectures.
The bus 21 allows data communication between the central processor 24 and one or more memory components, which may include RAM, ROM, and other memory, as previously noted. Typically RAM is the main memory into which an operating system and application programs are loaded. A ROM or flash memory component can contain, among other code, the Basic Input-Output system (BIOS) that controls basic hardware operation such as the interaction with peripheral components. Applications resident with the computer 20 are generally stored on and accessed via a computer readable medium, such as a hard disk drive (e.g., fixed storage 23), an optical drive, floppy disk, or other storage medium.
The fixed storage 23 may be integral with the computer 20 or may be separate and accessed through other interfaces. The network interface 29 may provide a direct connection to a remote server via a wired or wireless connection. The network interface 29 may provide such connection using any suitable technique and protocol as will be readily understood by one of skill in the art, including digital cellular telephone, WiFi, Bluetooth(R), near-field, and the like. For example, the network interface 29 may allow the computer to communicate with other computers via one or more local, wide-area, or other communication networks, as described in further detail below.
Many other devices or components (not shown) may be connected in a similar manner (e.g., object scanners, digital cameras and so on). Conversely, all of the components shown in
The user interface 13, database 15, and/or processing units 14 may be part of an integral system, or may include multiple computer systems communicating via a private network, the Internet, or any other suitable network. One or more processing units 14 may be, for example, part of a distributed system such as a cloud-based computing system, search engine, content delivery system, or the like, which may also include or communicate with a database 15 and/or user interface 13. In some arrangements, an analysis system 5 may provide back-end processing, such as where stored or acquired data is pre-processed by the analysis system 5 before delivery to the processing unit 14, database 15, and/or user interface 13. For example, a machine learning system 5 may provide various prediction models, data analysis, or the like to one or more other systems 13, 14, 15.
More generally, various embodiments of the presently disclosed subject matter may include or be embodied in the form of computer-implemented processes and apparatuses for practicing those processes. Embodiments also may be embodied in the form of a computer program product having computer program code containing instructions embodied in non-transitory and/or tangible media, such as floppy diskettes, CD-ROMs, hard drives, USB (universal serial bus) drives, or any other machine readable storage medium, such that when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing embodiments of the disclosed subject matter. Embodiments also may be embodied in the form of computer program code, for example, whether stored in a storage medium, loaded into and/or executed by a computer, or transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, such that when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing embodiments of the disclosed subject matter. When implemented on a general-purpose microprocessor, the computer program code segments configure the microprocessor to create specific logic circuits.
In some configurations, a set of computer-readable instructions stored on a computer-readable storage medium may be implemented by a general-purpose processor, which may transform the general-purpose processor or a device containing the general-purpose processor into a special-purpose device configured to implement or carry out the instructions. Embodiments may be implemented using hardware that may include a processor, such as a general purpose microprocessor and/or an Application Specific Integrated Circuit (ASIC) that embodies all or part of the techniques according to embodiments of the disclosed subject matter in hardware and/or firmware. The processor may be coupled to memory, such as RAM, ROM, flash memory, a hard disk or any other device capable of storing electronic information. The memory may store instructions adapted to be executed by the processor to perform the techniques according to embodiments of the disclosed subject matter.
The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit embodiments of the disclosed subject matter to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to explain the principles of embodiments of the disclosed subject matter and their practical applications, to thereby enable others skilled in the art to utilize those embodiments as well as various embodiments with various modifications as may be suited to the particular use contemplated.
