Patent Application
20150324065
The present invention relates to providing a user terminal with a list of entities and/or objects extracted from a messaging conversation. The invention further relates to an enhanced messaging method which selectively displays a list of entities or objects and a device and computer readable medium for same.
User terminals, such as smart phones and tablet computers, permit the exchange of text messages and imagery through messaging applications.
As can be seen from the example of
Illustrative embodiments of the present invention address the above problems by providing a real-time semantic search of messaging conversations, and display of supplemental information related to any identified concepts. By using a semantic search, synonyms and words similar to a given named entity are found, providing an advantage over prior art keyword searching. Moreover, using the algorithms explained in detail below, named entity recognition, topic identification, and sentiment analysis may be performed to provide supplemental information in the context of the conversation graphical environment. To provide better scalability and response time, and to enhance the privacy of users, these algorithms have been designed to execute under the limited resources of a user terminal, so that only their sanitized results are sent to a central server for further processing. Various embodiments may perform these functions without a separate search interface, providing an enjoyable user experience.
Therefore, there is provided in a first embodiment a computerized method of transmitting contextual information, from a server system toward a user terminal, the user terminal executing a messaging application having a graphical user interface. The method requires three computer processes executing on the server system. The first process is determining, from data pertaining to a conversation of the messaging application, a named entity, a topic, and a sentiment that each pertain to at least one message of the conversation. The second process is identifying supplemental information that pertains to a combination of one or more of the determined named entity, topic, and sentiment. The third process is transmitting the named entity and indicia pertaining to the identified supplemental information, from the server system toward the user terminal for display of the indicia on the user terminal by the messaging application using the graphical interface.
There is also provided an embodiment from the perspective of the user terminal for searching after a conversation has been concluded. This embodiment is a computerized method of displaying contextual information on a user terminal executing a messaging application having a graphical user interface. This method includes five computer processes executing on the user terminal. The first process is determining, from data pertaining to a conversation of the messaging application, a named entity, a topic, and a sentiment that each pertain to at least one message of the conversation. The second process is transmitting the determined named entity, topic, and sentiment to a server system. The third process is receiving from the server system indicia identifying supplemental information that pertains to the determined named entity, the server having determined the supplemental information as a function of the transmitted named entity, the topic, and the sentiment. The fourth process is displaying the indicia on a display of the user terminal, by the messaging application using the graphical interface. The fifth process is, upon receiving a selection of an indicium associated with the named entity using the graphical interface, displaying the received supplemental information on the display of the user terminal, by the messaging application using the graphical interface.
Variations on these methods are also contemplated. For example, the user terminal may be a smart phone, a personal digital assistant, a tablet computer, or a laptop computer. The messaging application may be a bulletin board application, an email application, an instant messaging application, or an SMS application. Determining the named entity may include using one or more of a dictionary search, a regular expression match, and a first probabilistic model derived from a first machine learning algorithm. The first probabilistic model may be stored on the user terminal for use by the first computer process. The topic and sentiment may be determined simultaneously using a second probabilistic model derived from a second machine learning algorithm. The second probabilistic model may be stored on the user terminal for use by the first computer process. The second machine learning algorithm may be trained using a corpus of publicly-available messages that each identify a topic and a sentiment, respectively, using textual markings. For example, the corpus may be TWITTER messages, in which case the topic textual markings are hashtags, and the sentiment textual markings are emoticons. The supplemental information may be, for instance, a web document, a wiki article, a search suggestion, a movie listing, a sports score, information pertaining to a local business, weather information, product information, an audiovisual presentation, or advertising. The indicia may include at least one indicium comprising a text string that is associated with a URL. If so, in one embodiment the methods described may be extended by first receiving a selection of the text string using the graphical interface, then displaying, on the user terminal, data retrieved from the URL
Using these methods, messages from and to user terminals are automatically reviewed, key facts are summarized, concepts and sentiment associated with the entities are identified inside a messaging conversation, and all parties are presented with the key facts, entities, and related supplemental information.
The foregoing features of the invention will be more readily understood by reference to the following detailed description, when read with reference to the accompanying drawings, in which:
Illustrative embodiments of the present invention are directed to a method for providing a user with easy access to content during a messaging conversation (also referred to herein as a conversation) as well as display advertisements based on sentiments displayed inside a conversation. There is no known prior art displaying easy to browse entities such as names, places, events and other identified entities related to the content of a messaging conversation while inside a messaging conversation. Furthermore, known methods are not capable of providing advertising that is supplemental to a sentiment of the entity in this connection, while illustrative embodiments of the present invention are capable of doing so. For example, illustrative methods of the present invention may provide a user terminal with a listing of good restaurants in a particular area based on a sentiment such as “love pizza”, and not show an advertisement for a local pizza if a user types “hates pizza” or “allergic to pizza”.
Definitions. As used in this description and the accompanying claims, the following terms shall have the meanings indicated:
A “messaging application” is a computer program or collection of programs that is able to transmit, by and between a plurality of individuals, textual messages that are meaningful to the individuals. Messaging applications include, without limitation: bulletin board applications, email applications, instant messaging applications, and short messaging service (“SMS”) applications.
A “user terminal” or “terminal” is an electronic device that is configured to execute a messaging application. A user terminal may be, for example, a cell phone, a smart phone, a personal digital assistant, a tablet computer, a laptop computer, or a desktop computer.
A “messaging conversation” or “conversation” is a series of one or more topical messages transmitted between individuals using a messaging application.
An “entity” is a concept of potential interest to an individual, that is meaningful within the context of a topic of a messaging conversation. An entity may be one or more recognized words associated with an ontology, for example (and not by way of limitation): a person, people, place, event, product, domain url, e-mail, phone, address location, company, date, time, money, percent, airline name, airline flight number, holiday greetings, relations and other categories that are relevant to the subject matter and topical area or areas of a messaging conversation.
A “named entity” is an entity that has a well-defined name. For example, “the automotive company created by Henry Ford in 1903” is an entity, but it also has the rigid designators “Ford” and “Ford Motor Company” so it is a named entity. Many entities that appear in conversations are well-defined, but some, such as “this past June”, are not because resolution of their meanings requires additional context. If it is possible to provide additional context, for example that the year in which the messaging conversation took place is 2012, then the entity may be further defined as “June of 2012”, which is a named entity.
“Named entity recognition” (also known as “entity identification” and “entity extraction”) is a process that locates and classifies entities into predefined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, and other categories depending on the context.
A “sentiment” is a favorability, or polarity of an entity or a topic in a messaging conversation, and could be positive, negative or neutral. “Sentiment analysis”, also known as “opinion mining”, refers to the application of natural language processing, computational linguistics, and text analytics to identify and extract favorability information in one or more messaging conversations.
“Semantic search” is a process that seeks to determine searcher intent and the contextual meaning of terms as they appear in a searchable data space to determine related terms that may be found in a conversation. Semantic search generally produces more relevant results than a simple keyword or Boolean search.
As discussed in the background,
The indicia and advertisements that appear in the GUI 30 may be updated on a real-time basis by the named entity recognizer in cooperation with a topic extractor and a sentiment analyzer. That is, the named entities that are displayed in the GUI 30 are chosen by an embodiment as a function of the most recent messages in the conversation, because the execution of the named entity recognizer, topic extractor and sentiment analyzer is able to keep pace with an ongoing conversation as it occurs.
In accordance with another embodiment of the invention, different users may receive different advertisements.
As another example of an improvement that is possible,
It will be understood that various embodiments of the invention may include indicia that represent the identified named entities using text, still images, moving images, or any other useful visual representations. It is also contemplated that the indicia may be vocalized or otherwise communicated to the user by audible means, for example by way of accessibility add-on software or hardware. As will be understood from the diverse nature of the above examples, any messaging application that has a GUI may be improved in accordance with embodiments of the present invention. Other arrangements that include additional elements (such as information regarding the conversation or search parameters) or omit various elements (such as the advertisement) are contemplated.
Various embodiments of the invention perform the above-identified functions in a number of different circumstances. In one embodiment, the functions are executed as a conversation is occurring. In this way, supplemental information pertaining to the ongoing conversation may be conveniently accessed in real-time. In another embodiment, the processes are executed after the conversation has concluded. Thus, a search may be performed to locate supplemental information about a named entity that the user remembers discussing only vaguely (so that the user does not have enough specific information to formulate an effective query in another search engine). For example, the user may remember that she is supposed to meet a friend at a restaurant, but doesn't remember which restaurant. This embodiment advantageously allows her to search for the restaurant, even if the word “restaurant” does not appear in any conversation.
The functional components of the server system 71 and the user terminal 73 depicted in
Next, in process 92, the user terminal 73 transmits the named entities, topic, and sentiment to the server system 71, along with geographic data. The user terminal 73 then receives from the server system 71 indicia identifying supplemental information that pertains to the named entities. The server system 71 determines the supplemental information for each named entity as a function of the transmitted named entities, the topic, and the sentiment using the semantic processing module 715, but does not need to use the named entity recognizer 711, topic extractor 712, or sentiment analyzer 713. Then, in process 93, the user terminal 73 displays the indicia, in a selectable manner, on the display 735, under the direction of the messaging application 736. Finally, in process 94, upon receiving a selection of an indicium associated with the named entity using the graphical interface, the user terminal 73 displays the received supplemental information on the display 735.
The processes discussed in connection with
In either case, a named entity recognizer may use any number of techniques to recognize named entities 102 in messages 101. These techniques include, in order of increasing complexity, a dictionary lookup 103, regular expression matching 104, grammar processing 105, and a probabilistic model 106. A dictionary lookup 103 simply matches words in a message 101 against named entities listed in a dictionary. The dictionary, which may be stored in the database 714 for example, may also contain references to supplemental information in case there is a match.
The named entity recognizer 711 or 731 also may employ regular expression matching 104. Unlike a dictionary lookup 103 that matches a message 101 against dictionary text, regular expression matching 104 matches a message 101 against a pattern. For example, if the symbol * indicates any character, then the pattern “*ing” matches all words that end in the letters I-N-G (in that order). As is known in the art, regular expression matching 104 may be used to extract currency amounts, dates, and other types of text that are not easily matched using a dictionary lookup 103. Some named entities 102 may be identified by the fact that they are proper nouns and therefore capitalized; a regular expression may be easily used to identify such named entities.
The named entity recognizer 711 or 731 also may employ grammar processing 105. In this process, the text of a message 101 is analyzed to determine parts of speech, such as verbs, adjectives, and (most importantly) nouns. Any noun (or phrase including a noun) that is detected may be an entity. If the noun or phrase uniquely identifies a concept, then it is a named entity 102. In accordance with some embodiments, non-named entities detected by the named entity recognizer may be further processed by a semantic processing module to resolve any ambiguities, as described below.
The named entity recognizer 711 or 731 also may employ a first probabilistic model 106, either as a standalone entity or in conjunction with grammar processing 105. The probabilistic model 106 is the result of training a machine learning algorithm to recognize words and phrases that are named entities. A lexical analyzer breaks each text message 101 into a sequence of tokens (for example, using dictionary lookups 103, regular expression matching 104, or grammar processing 105). The model 106 provides probabilities, as a function of the sequence of tokens, that any given token is an entity (or a named entity). Typically, the machine learning algorithm is trained on a large corpus of pre-classified text. That is, a large number of messages are tokenized, and the tokens are manually classified either as being named entities or not named entities. Statistical measures are derived from this analysis. As a simplistic example, the probability that a named entity in the corpus follows a non-named entity (versus another named entity) is determined. Later, when automatically applied to a message 101, this manually-derived measure provides a probability that an unknown token following a non-named entity is actually a named entity. Other, more complex measures known in the art that classify tokens, for example, by parts of speech, length, position in a sentence, date and time, geographic location, and other factors may be used in an embodiment of the invention. Advantageously, the machine learning algorithm itself does not need to be executed on a user terminal 73. Instead, in some embodiments, only the probabilities themselves are stored on the user terminal 73, resulting in a tremendous savings in computing cost and permitting the user terminal to perform named entity recognition directly.
The topic extractor 712 or 732 also may employ a second probabilistic model 113. This probabilistic model 113 may be trained on messages in a manner similar to the first probabilistic model 106 is trained on tokens. Thus, the model 113 may store probabilities that if a previous message is known to pertain to topic X, that the current message 101 will pertain to topics X, Y, or Z. Given the underlying assumption that consecutive messages often related to the same topic, it should be clear that, given a first message 101 taken from a conversation, a second consecutive message 101 taken from the same conversation has a high probability of pertaining to the same topic 111. Thus, sequencing information (for example as determined by date and time data) relating to the messages 101 of a conversation often provides a strong clue as to the topic of each message 101 of the conversation. Once the model 113 has been established, it may be applied to unknown messages 101 to determine probabilistically whether they belong to any given topic. A particularly useful embodiment may store a model 113 that is limited to topics for which detailed supplemental information may be obtained, as described more fully below in connection with
The second probabilistic model 113 may be trained automatically in a particularly advantageous way. As noted above, TWITTER messages often contain hashtags that provide a strong clue about the topic of the message. Moreover, there exists a large corpus of public TWITTER messages. Therefore, in a machine learning process that need not occur on the user terminal 73, hashtags are extracted from large numbers of TWITTER messages and used to classify them automatically. These automatic classifications are then used to form the second probability model 113. This technique may be applied only once, and the model 113 stored on the user terminal 73, or it may be applied on an ongoing basis by the server system 71, with the stored model 113 receiving periodic updates.
The sentiment analyzer 713, 733 may employ token analysis 123 to determine sentiment. Certain tokens are strongly indicative of sentiment. Emoticons or “smilies” are an example: if a message 101 contains a smiling emoticon, then the sentiment is likely favorable, while if the message 101 contains a frowning or crying emoticon, then the sentiment is likely negative. Once identified, the tokens of the message 101 therefore may be used to identify a sentiment.
In some situations, for example in complex messages that discuss multiple named entities, token analysis 123 is insufficient to determine the sentiments for each named entity. The sentiment analyzer 713, 733 therefore may employ sentence analysis 124 to determine sentiment. In particular, once a named entity has been identified, nearby tokens are more likely to provide information about the favorability of that named entity than distant tokens.
In other situations, a sentiment 102 expressed in a message 101 may pertain to the topic of the conversation. For this purpose, the sentiment analyzer 713, 733 may employ a topic analysis 125 to determine sentiment. A single sentiment 102 may be present with respect to a number of named entities, across multiple messages 101, that are related by a topic or classification. Topic analysis 125 correlates multiple messages 101 from the conversation to determine whether the topic itself is favorable or unfavorable.
Other techniques from natural language processing, computational linguistics, and textual analysis that are known in the art may be used to extract a sentiment 121 from a message 101, and the description of the above techniques should not be viewed as limiting the scope of the invention. Moreover, the techniques of named entity recognition, topic extraction, and sentiment analysis share many similarities, and may be efficiently combined in some embodiments into a single semantic engine.
In process 132 the module 715 determines whether there are more entities to be processed. If so, the method continues to process 133, in which one of the entities is selected for further processing. Next, in process 134, the module 715 resolves any ambiguity in the entity to provide it with a unique name. For example, “Frank” may be a given name, a surname, a group of people, a type of currency, a community in Alberta, the name of a girl band, the name of an album, a character in a comic strip, or a type of meat (among other things). If any entities have not been uniquely identified, they may be combined with information about the user, about the geographic location of the user terminal 73, about the date and time, about the topic of the conversation, or other information to resolve the ambiguity. For instance, if the topic of conversation is food, then “Frank” has a high probability of referring to a hot dog or similar food, rather than a comic strip.
In process 135, the module 715 locates supplemental information about the named entity. Continuing the previous example, the module 715 may engage the database 714 or the web browser 718 to locate information about hot dogs, such as nearby restaurants that offer this food on their menus, a WIKIPEDIA page that describes hot dogs, or advertising by hot dog makers. The server system 71 may be provided with various application programming interfaces (APIs) that can retrieve this information from knowledgeable sources. APIs may provide, for instance, access to a web document, the WIKIPEDIA web site or other repositories of information such as DBPEDIA, a search suggestion for a web search engine such as GOOGLE, movie listings such as those found at NETFLIX or movie information found at IMDB, sports scores at ESPN, listings for local businesses through the YELLOW PAGES, weather information, product information, audiovisual presentation such as POWERPOINT presentations, advertising, or other sources of supplemental information.
In process 136, the module 715 determines a favorability for the named entry. This favorability may be obtained through the sentiment analyzer 713 or 733. It should be noted that different users may have different sentiments with respect to the same named entities, so this process 136 may produce different results depending on the user to which it is applied. Thus, in some embodiments the module 715 determines a favorability for the given named entity with respect to each user that is participating in the conversation. Once this process 136 is complete, the method returns to process 132 in which it is determined whether there are more entities to process.
Once all entities in a message has been analyzed and supplemental information located, the module 715 proceeds to process 137 in which it ranks the named entities in order of decreasing favorability. When the named entities have been ranked, the server system 71 transmits the ranked list, and indicia pertaining to the named entities, toward the user terminal 73 for eventual display. The indicia themselves may be, for example, text strings that have associated URLs. For example, the named entity “Frank” (once properly disambiguated to mean a hot dog) may be associated with a URL for a WIKIPEDIA page that describes hot dogs. These data are sent to the user terminal 73, which displays the word “Frank” in an appropriate part of its GUI. Upon selection of the indicium, the URL is accessed and the supplemental information displayed. Optionally, the server system 71 also transmits advertising pertaining to a named entity in process 139 for display in the GUI. Preferably, this named entity has the highest favorability of all detected named entities, so the advertising is as relevant as possible to the user's interests. As noted above, different users may have different sentiments with respect to the same named entities, so the advertisement that each user sees may be different, as illustrated in
Various embodiments of this invention have been described. However, this disclosure should not be deemed to be a limitation on the scope of the invention. Accordingly, various modifications, adaptations, and alternatives may occur to one skilled in the art without departing from the spirit and scope of the claimed invention.
It should be noted that the logic flow diagrams are used herein to demonstrate various aspects of the invention, and should not be construed to limit the present invention to any particular logic flow or logic implementation. The described logic may be partitioned into different logic blocks (e.g., programs, modules, functions, or subroutines) without changing the overall results or otherwise departing from the true scope of the invention. Often times, logic elements may be added, modified, omitted, performed in a different order, or implemented using different logic constructs (e.g., logic gates, looping primitives, conditional logic, and other logic constructs) without changing the overall results or otherwise departing from the true scope of the invention.
The present invention may be embodied in many different forms, including, but in no way limited to, computer program logic for use with a processor (e.g., a microprocessor, microcontroller, digital signal processor, or general purpose computer), programmable logic for use with a programmable logic device (e.g., a Field Programmable Gate Array (FPGA) or other PLD), discrete components, integrated circuitry (e.g., an Application Specific Integrated Circuit (ASIC)), or any other means including any combination thereof.
Computer program logic implementing all or part of the functionality previously described herein may be embodied in various forms, including, but in no way limited to, a source code form, a computer executable form, and various intermediate forms (e.g., forms generated by an assembler, compiler, linker, or locator). Source code may include a series of computer program instructions implemented in any of various programming languages (e.g., an object code, an assembly language, or a high-level language such as Fortran, C, C++, JAVA, or HTML) for use with various operating systems or operating environments. The source code may define and use various data structures and communication messages. The source code may be in a computer executable form (e.g., via an interpreter), or the source code may be converted (e.g., via a translator, assembler, or compiler) into a computer executable form.
The computer program may be fixed in any form (e.g., source code form, computer executable form, or an intermediate form) either permanently or transitorily in a tangible storage medium, such as a semiconductor memory device (e.g., a RAM, ROM, PROM, EEPROM, or Flash-Programmable RAM), a magnetic memory device (e.g., a diskette or fixed disk), an optical memory device (e.g., a CD-ROM), a PC card (e.g., PCMCIA card), or other memory device. The computer program may be fixed in any form in a signal that is transmittable to a computer using any of various communication technologies, including, but in no way limited to, analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), networking technologies, and internetworking technologies. The computer program may be distributed in any form as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the communication system (e.g., the Internet or World Wide Web).
Hardware logic (including programmable logic for use with a programmable logic device) implementing all or part of the functionality previously described herein may be designed using traditional manual methods, or may be designed, captured, simulated, or documented electronically using various tools, such as Computer Aided Design (CAD), a hardware description language (e.g., VHDL or AHDL), or a PLD programming language (e.g., PALASM, ABEL, or CUPL).
Programmable logic may be fixed either permanently or transitorily in a tangible storage medium, such as a semiconductor memory device (e.g., a RAM, ROM, PROM, EEPROM, or Flash-Programmable RAM), a magnetic memory device (e.g., a diskette or fixed disk), an optical memory device (e.g., a CD-ROM), or other memory device. The programmable logic may be fixed in a signal that is transmittable to a computer using any of various communication technologies, including, but in no way limited to, analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), networking technologies, and internetworking technologies. The programmable logic may be distributed as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the communication system (e.g., the Internet or World Wide Web).
This application is a continuation of U.S. application Ser. No. 13/734,051, filed Jan. 4, 2013, which claims the benefit of U.S. Provisional Application No. 61/583,091 filed Jan. 4, 2012. The entire contents of both prior applications are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61583091 | Jan 2012 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 13734051 | Jan 2013 | US |
| Child | 14803635 | US |
