Patent Application
20200389420
The following relates generally to messaging systems, and more specifically to a messaging system for automatically generating semantic contextual messages.
Electronic mail (email or e-mail) is a method of exchanging messages (“mail”) between people using electronic devices. Email operates across computer networks such as the Internet. An email client is a computer program used to access and manage a user's email. A user can set the email client to automatically respond to an email with an out of office message during a period the user is unavailable.
A short message service (SMS) is a text messaging service component of most telephone Internet, and mobile-device systems, it uses standardized communication protocols to enable mobile devices to exchange short text messages. For example, a user can use an SMS client on their Smartphone to send a text message to a phone number associated with one of their contacts. Further, some. SMS clients enable users to automatically respond to a text message with a default text message.
A voicemail system is a computer-based system that allows users and subscribers to exchange voice messages. The voicemail system enables a user associated with a telephone number to record a voice message that is played for a caller that has called the telephone number if the user does not answer the call within a predetermined amount of time.
However, since the out of office message, the default text message, and the voice message are all static messages, the caller (i,e,, the one sending the email, text message, or placing a telephone call) has no way to know what the user is currently doing. For example, if the caller is a close colleague, friend, or family member, the user is unable to use the static message to convey a private message to only that caller since all other callers would have access to the private message.
According to an exemplary embodiment of the inventive concept, a messaging system for responding with an automatically generated semantic contextual messages is described. The system includes: a network interface configured to interface with a network; a presentation unit configured to present a graphical user interface (GUI) on a display device of a mobile communication device connected to the network; a monitoring unit, a message generation unit, and an output unit. The GUI prompts the user to grant access to a plurality of data sources of the user and to a plurality of contacts of the user, and enables the user to set a preference for each contact listing which of the data sources are shareable with the corresponding contact. The monitoring unit monitors the network for an incoming communication from a caller to the user and determines whether the user is available to receive the incoming communication. The message generation unit identifies one of the contacts associated with the incoming communication, collects data from the data sources listed by the preference of the identified one contact, generates text based on the collected data, when it is determined that the user is not available. The output unit outputs a semantic contextual message based on the text across the network to the caller.
According to an exemplary embodiment of the inventive concept, a non-transitory computer readable medium storing code for responding with an automatically generated semantic contextual message is described. In some examples, the code comprises instructions executable by a processor to: prompt a user to grant access to a plurality of data sources of the user and to a plurality of contacts of the user; prompt the user to set a preference for each contact listing which of the data sources are shareable with the corresponding contact, monitor a network for an incoming communication from a caller to the user; determine whether the user is available to receive the incoming communication; identify one of the contacts associated with the incoming communication, collect data from the data sources listed by the preference of the identified one contact, generate text based on the collected data, when it is determined that the user is not available; and output a semantic contextual message based on the text across the network to the caller.
A method for responding with an automatically generated semantic contextual message is described. The method may include prompting a user to grant access to a plurality of data sources of the user and to a plurality of contacts of the user; prompting the user to set a preference for each contact listing which of the data sources are shareable with the corresponding contact; monitoring a network for an incoming communication from a caller to the user; determining whether the user is available to receive the incoming communication; identifying one of the contacts associated with the incoming communication, collecting data from the data sources listed by the preference of the identified one contact, generating text based on the collected data, when it is determined that the user is not available; and outputting a semantic contextual message based on the text across the network to the caller.
FIGS, 2 through 4 show examples of a process for automatically generating a semantic contextual message in accordance with aspects of the present disclosure.
Systems and methods for responding with an automatically generated semantic contextual message are described. The system includes a messaging system that is capable of interfacing with a network to receive incoming communications (e.g., emails, text messages, or telephone calls). The messaging system is capable of responding to a caller with an automatically generated semantic contextual message tailored for the caller when a user (e.g., a callee), the caller is attempting to reach, is unable to answer the call.
The semantic contextual message is automatically generated by sensing different data. A user (e.g., the callee) can select which callers and what sensing data will he used to compose the semantic contextual message when the user misses their incoming communication (e,g., a telephone call, a text message, or an email).
The system automatically generates the semantic contextual message from a user's sensed environment and actions, The user can set from their list of contacts who can receive a semantic contextual message and what data can be used to form the semantic contextual message. If one of these contacts calls and the user misses the incoming communication, the system will use the available data and contextual information to compose the semantic contextual message.
The sensed data may be sensed from one or more wearable devices. The sensed data may include health related data such as temperature, pulse, a heart rate, an oxygen saturation level, and body posture (e.g., seated, standing, dancing, jumping, surfing, etc.). For example, the wearable device could include a gyro sensor to indicate the body posture. For example, a wearable device such as a medical bracelet may include one or more physical sensors (e.g., a medical sensor) capable of sensing body temperature, pulse, heart rate, or an oxygen saturation level. For example, the medical sensor could analyze the data sensed to determine whether they are normal or abnormal so that the sensed data indicates whether the sensed body temperature, pulse, heart rate or oxygen saturation level is normal or abnormal, For example, the semantic contextual message resulting from the sensed data could be “my heart rate is normal”.
The sensed actions may include activities such as spending money, cooking, eating, running, watching television sleeping, etc. As an example, an internet of things (IOT) device, a smart home device, or a calendar program of the user could he accessed to determine whether these activities have occurred and their duration. For example, the semantic contextual message resulting from the sensed actions could be “I have been watching television for 2 hours”, “I had dinner”, etc.
The sensed environment may include the location of the user and the names of others that the user is accompanied by (e.g., I'm with John). For example, once the user grants access to their calendar program, the system can determine from events listed in the calendar program, the identities of one or more individuals the user is meeting with during the time of the call. For example, once the user grants access to their GPS, the system is able to determine the present location of the user.
Semantic contextual messages can be used by busy people. These messages allow users to focus on doing/enjoying what they are doing and not on picking up the phone or accessing a computer to reply or send a message to let the caller know what they are doing or how they are doing.
Semantic contextual messages can be used to monitor the elderly, teenagers, loved ones, and family members.
Semantic contextual messages can reduce cost by eliminating the need to text/call the caller every time to let them know about what the receiver is doing.
As shown in
For example, the VoIP device 166 could send a packet to the packet network 18202, the packet network can forward the packet to the wireless network 182-1, and the wireless network 182-1 could then forward the packet to the messaging system 100. For example, the VoiP device 166 could send a packet to the packet network 182-2, the packet network 182-2 can forward the packet to the public switched telephone network 82-3, and the public switched telephone network 182-3 could forward the packet to the messaging system 100.
The messaging system 100 may include a presentation unit 105, a monitoring unit 110, a message generation unit 115, an output unit 120, and a network interface 125.
The monitoring unit 110 may monitor the network 182 for an incoming communication (e.g., a telephone call, an email, or a text message) from a caller 165 to the callee 175, and determine whether the callee 175 is available to receive the incoming communication, The caller 165 generates the incoming communication using the calling device 160. For example, if the callee 175 does not answer an incoming communication within a certain amount of time, the monitoring unit 110 may determine that the callee 175 is not available, and accordingly that a greeting needs to be output to the caller 165. The content of the greeting will depend on whether the messaging system 100 can identify the caller 165 and the preferences that the callee 175 has set for the caller 165.
The message generation unit 115 may identify a contact associated with the incoming communication from among a plurality of contacts of the callee 175, collect data from the data sources 130 listed by a preference associated with the identified contact, generate text based on the collected data, and generate a sematic contextual message, when it is determined that the callee 175 is not available (e.g., the cal lee 175 does answer the call in time).
For example, the message generation unit 115 may determine a phone number of the caller 165 that placed the incoming communication front data received from the calling device 160 during the incoming communication, and compare the determined phone number with phone numbers in contact information of a user associated with the callee 175, to identify a known contact associated with the caller 165. The contact information may include a plurality of contacts of the user and phone numbers of each contact. The contact information may be received from the data sources 130. For example, the data sources 130 may include a contact program of the user that includes the contact information.
For example, if the user associated with callee 175 has 10 contacts in their contact information, and the determined phone number matches the telephone number of the 4th contact, then the message generation unit 115 would conclude that the 4th contact of the user placed the call, For example, the message generation unit 115 may determine a name of the caller 165 placing the incoming communication from data received from the caller device 160 during the incoming communication, and compare the determined name with names in the contact information to identify the contact of the caller 165 placing the incoming communication, For example, if the determined name matches the name of the 7th contact, then the message generation unit 115 would conclude that the 7th contact of the user 170 placed the incoming communication.
The message generation unit 115 may also generate the text from a message pre-recorded by the user associated with the callee 175 when the preference indicates that none of the data sources 130 are shareable with the corresponding contact or if the message generation 115 was unable to identify a contact to associate with the callee 175. The message generation unit 115 may convert the text to speech when the system 100 is being implemented as a voice mail system (VMS). The speech may be stored in an audio file that is later output to the caller 165.
In an exemplary embodiment of the inventive concept, the message generation unit 115 identifies contextual keywords from the collected data, in an exemplary embodiment of the inventive concept, the message generation unit 115 orders the contextual keywords in a chronological order. In an exemplary embodiment, the message generation unit 115 passes the ordered contextual keywords to an artificial neural network to generate the text. For example, the message generation unit 115 can take keywords from the data and consider chronological order of the keywords, to then compose a semantic contextual message (short sentences grammatically correct to form a story of events).
In an exemplary embodiment, the artificial neural network (ANN) includes two recurrent neural networks chained together. Each recurrent neural network (RNN) is a time sequence model that takes any arbitrary input, converts it into a vector of numbers, and combines it with its previous state (a vector of numbers) to produce a new state. In an exemplary embodiment of the inventive concept, the first RNN of the chain is referred to as an encoder and the second RNN of the chain is referred to as a decoder.
When the system 100 is implemented in a voice mail system, the text generated by the ANN is converted to speech, and then the output unit 120 outputs the speech across the network 182 to the calling device 160. When the system 100 is used to send text messages, the output unit 120 outputs the text in a text message to a telephone number associated with the calling device 160 or a user of the calling device 160. When the system is used to send emails, the output unit 120 formats an email to include the text and outputs the email to an email address associated with the calling device 160 or a user of the calling device 160.
In some examples, the data sources 130 include a UPS receiver 180 of the called device 170 and the data includes a location of the callee 175 or the called device 170 determined from the GPS receiver 180. In some examples, the data sources 130 include a smart home device 150 or an Internet of things (IOT) device 145, and the data includes an activity of the user associated with the callee 175 determined by the smart home device 150 or the IOT device 145. In some examples, the data sources 130 include social media data 135 of the user 175, and the data indicates an activity of the user 175 determined from the social media data 135. In some examples, the data sources 130 includes a calendar program 155 of the user 175, and the data indicates an activity listed by the calendar program 155 that is scheduled to be performed by the user 175 at a time of the incoming communication. In some examples, the data sources 130 include a wearable device 140 of the user 175, the wearable device 140 includes a medical sensor and the data is medical data sensed by the medical sensor.
Data sources 130 may include social media data 135, wearable device 140, IOT device 145, smart home device 150, and calendar program 155.
Caller device 160 may be used by a caller 165.
User device 170 may include a GPS receiver 180 and be used by user 175.
At step 200, system 100 may prompt a user 175 to grant access to a plurality of data sources of the user 175 and to a plurality of contacts of the user 175. In some cases, the operations of this step may refer to, or be performed by, a presentation unit 105 as described with reference to
At step 205, system 100 may prompt the user 175 to set a preference for each contact listing which of the data sources are shareable with the corresponding contact. In some cases, the operations of this step may refer to, or he performed by, the presentation unit 105 as described with reference to
At step 210, the system 100 may monitor the network 182 for an incoming communication from a caller 165 to the user 175. In some cases, the operations of this step may refer to, or be performed by, the monitoring unit 110 as described with reference to
At step 215, the system 100 may determine whether the user 175 is available to receive the incoming communication. In some cases, the operations of this step may refer to, or be pertbrmed by, the monitoring unit 110 as described with reference to
At step 220, a system may connect the caller to the user when it is determined that the user is not available. in some cases, the operations of this step may refer to, or be performed by, a connection unit of the messaging system 100.
At step 225, a system may generate a semantic contextual message from the data sources listed by the preference as being shareable with one of the contacts associated with the caller. In some cases, the operations of this step may refer to, or be performed by, a message generation unit as described with reference to
At step 235, a system may output the semantic contextual message across the network to the calier. In some cases, the operations of this step may refer to, or be performed by, an output unit as described with reference to
At step 300, a system may identify one of the contacts associated with the incoming communication. In some cases, the operations of this step may refer to, or he performed by, a message generation unit as described with reference to
At step 305, a system may collect data from the data sources listed by the preference of the identified one contact, in some cases, the operations of this step may refer to, or he performed by, a message generation unit as described with reference to
At step 310, a system may generate text based on the collected data. In some cases, the operations of this step may refer to, or be performed by, a message generation unit as described with reference to
At step 400, a system may identify contextual keywords from the collected data, In some cases, the operations of this step may refer to, or be performed by, a message generation unit as described with reference to
At step 405, a system may order the contextual keywords in a chronological order, In some cases, the operations of this step may refer to, or be performed by, a message generation unit as described with reference to
At step 410, a system may pass the ordered contextual keywords to a recurrent neural network to generate the text, In some cases, the operations of this step may refer to, or be performed by, a message generation unit as described with reference to
An embodiment of the messaging system 100 of
The callee 175 (i.e., a user of the system 100) can use an interface (e.g., 105) of the system 100 to set preferences to indicate which of these data sources to share with some of their contacts. For example, the user could set these preferences to indicate he wants to share data source B with contact 1 and share data sources A and C with contact 2. Thus, contextual messages generated for contact 1 will only be derived from data source B and not be derived from data sources A and C, and contextual messages generated for contact 2 will only be derived from data sources A and C and not be derived from data source B.
In an exemplary embodiment of the invention, the system 100 enables to user create text statements including pre-defined text (e.g., a string) and placeholders that reference the previously captured records and records that reference the previously captured time stamps. The system 100 stores the created text statements and associates them with a given one of the data sources so they can be later loaded per data source for creating part or all of a semantic contextual message, based the data sources currently shareable with a given caller. An example of the text statement for the TV watching record captured from data source B includes “1 watched the TV show [TV watching record] from [start timestamp] to [end timestamp]”. Thus, when the system 100 determines from its access to data source B that the user watched Antiques Roadshow from 10:00 am to 11:00 am, it replaces activity placeholder [TV watching record] with “Antiques Roadshow”, replaces time placeholder [start timestamp] with 10:00 am, and replaces placeholder [end timestamp] with 11:00 am to generate first finalized text such as “I watched the TV show Antiques Roadshow from 10:00 am to 11:00 am, An example of the text statement for the activity record captured from data source A includes “I have done [activity record] from [start timestamp] to [end timestamp]. Thus, when the system 100 determines from its access to data source A that the user walked 1000 steps between 09:00 am to 10:00 am, it replaces placeholder [activity record] with “1000 steps”, replaces placeholder [start timestamp] with 09:00 am, and replaces placeholder [end timestamp] with 10:00 am to generate second finalized text such as “I have done 1000 steps from 09:00 am to 10:00 am”. An example of the text statement for the food record captured from data source C includes “I have eaten [food record] from [start timestamp] to [end timestamp], Thus, when the system 100 determines from its access to data source C that the user ate a tuna fish sandwich between 12:00 pm to 12:30 pm, it replaces placeholder [food record] with “tuna fish sandwich”, replaces placeholder [start timestamp] with 12:00 pm, and replaces placeholder [end timestamp] with 12:30 pm to generate third finalized third text such as “I have eaten a tuna fish sandwich from 12:00 pm to 12:30 pm”. The system 100 may enable the user to change text of the text statements, and use other placeholders, so they can personalize the messages. The finalized text can then he converted to speech so that it can be played to the caller 175 as the semantic contextual message, For example, if the caller 175 is determined to he contact 1, then the semantic contextual message is the first finalized text. For example, if the caller 175 is determined to he contact 2, then the semantic contextual message is the second finalized text and the third finalized text. The system 100 may insert a textual conjunction (e.g., “AND”) between a pair of finalized text (e,g., I have done 1000 steps from 09:00 am to 10:00 am and I have eaten a tuna fish sandwich from 12:00 pm to 12:30 pm).
The description and drawings described herein represent example configurations and do not represent all the implementations within the scope of the claims. For example, the operations and steps may be rearranged, combined or otherwise modified. Also, structures and devices may be represented in the form of block diagrams to represent the relationship between components and avoid obscuring the described concepts. Similar components or features may have the same name but may have different reference numbers corresponding to different figures.
Some modifications to the disclosure may be readily apparent to those skilled in the art, and the principles defined herein may be applied to other variations without departing from the scope of the disclosure. Thus, the disclosure is not limited to the examples and designs described herein, but is to he accorded the broadest scope consistent with the principles and novel features disclosed herein.
The described methods may he implemented or performed by devices that include a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, a conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration). Thus, the functions described herein may be implemented in hardware or software and may be executed by a processor, firmware, or any combination thereof if implemented in software executed by a processor, the functions may he stored in the form of instructions or code on a computer-readable medium.
Computer-readable media includes both non-transitory computer storage media and communication media including any medium that facilitates transfer of code or data. A non-transitory storage medium may be any available medium that can be accessed by a computer. For example, non-transitory computer-readable media can comprise random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk (CD) or other optical disk storage, magnetic disk storage, or any other non-transitory medium fur carrying or storing data or code.
Also, connecting components may be properly termed computer-readable media. For example, if code or data is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, or microwave signals, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology are included in the definition of medium. Combinations of media are also included within the scope of computer-readable media.
