Patent Grant
10761910
This disclosure relates generally to apparatuses, methods, and computer readable media for interacting with people, services, and devices across multiple communications formats and protocols.
A growing number of service providers allow users to request information or services from those service providers via a third party software applications. Additionally, a growing number of smart devices allow users to obtain information from and control those smart devices via a third party software application. Meanwhile, individuals communicate with each other using a variety of protocols such as email, text, social messaging, etc. In an increasingly chaotic digital world, it's becoming increasingly difficult for users to manage their digital interactions with service providers, smart devices, and individuals. A user may have separate software applications for requesting services from a number of service providers, for controlling a number of smart devices, and for communicating with individuals. Each of these separate software applications may have different user interfaces and barriers to entry.
The subject matter of the present disclosure is directed to overcoming, or at least reducing the effects of, one or more of the problems set forth above. To address these and other issues, techniques that enable seamless, multi-format, multi-protocol communications are described herein.
Disclosed are apparatuses, methods, and computer readable media for integrating communications for computing devices across multiple formats and multiple protocols. More particularly, but not by way of limitation, this disclosure relates to apparatuses, methods, and computer readable media to permit computing devices, e.g., smartphones, tablets, laptops, wearables, and the like, to present users with a multi-protocol, multi-format inbox for communicating with people and service providers and for controlling devices via multiple formats and communication protocols.
Referring now to
The computer networks 110 may include any communications network that allows computers to exchange data, such as the internet 111, local area networks 112, corporate networks 113, cellular communications networks 114, etc. Each of the computer networks 110 may operate using any number of network protocols (e.g., TCP/IP). The computer networks 110 may be connected to each other and to the various computing devices described herein (e.g., the interaction platform devices 120, the client devices 130, the third party communications devices 140, the third party service provider devices 150, and the smart devices 160) via hardware elements such as gateways and routers.
The interaction platform devices 120 may include one or more servers 121 and one or more storage devices 122. The one or more servers 121 may include any suitable computer hardware and software configured to provide the features disclosed herein. The storage devices 122 may include any tangible computer-readable storage media including, for example, read-only memory (ROM), random-access memory (RAM), magnetic disc storage media, optical storage media, solid state (e.g., flash) memory, etc.
The client devices 130 may include any number of computing devices that enable an end user to access the features disclosed herein. For example, the client devices 130 may include, for example, desktop computers 131, tablet computers 132, mobile phone 133, notebook computers 134, etc.
The third party communications devices 140 may include email servers such as a GOOGLE® or YAHOO! ® email server (GOOGLE is a registered service mark of Google Inc. YAHOO! is a registered service mark of Yahoo! Inc.), third party instant message servers such as a YAHOO! ° Messenger or AOL® Instant Messaging server (AOL is a registered service mark of AOL Inc.), third party social network servers such as a FACEBOOK® or TWITTER® server (FACEBOOK is a registered trademark of Facebook, Inc. TWITTER is a registered service mark of Twitter, Inc.), cellular service provider servers that enable the sending and receiving of messages such as email messages, short message service (SMS) text messages, multimedia message service (MMS) messages, or any other device that enables individuals to communicate using any protocol and/or format.
The third party service devices 150 may include any number of computing devices that enable an end user to request one or more services via network communication.
The smart devices 160 may include any number of hardware devices that communicate via any of the computer networks 110 and are capable of being controlled via network communication.
As described in more detail below, the universal interaction platform allows users to interact with individuals, service providers, and smart devices 160 by sending a message (in the form of a message object) from a client device 130. The message object is output by the client device 130 for transmittal to the server 121. When the user is interacting with a service provider, the universal interaction platform formats an instruction for the third party service device 150 associated with the service provider and outputs the instruction from the server 121 for transmittal to the third party service device 150. Similarly, when the user is interacting with a smart device 160, the universal interaction platform formats an instruction for the smart device 160 and outputs the instruction from the server 121 for transmittal to the smart device 160. The server 121 may also receive a response from the third party service device 150 or smart device 160, format a response message (in the form of a response message object) for the user, and output the response message object for transmittal to the client device 130.
Referring now to
System unit 205 may be programmed to perform methods in accordance with this disclosure. System unit 205 comprises one or more processing units, input-output (I/O) bus 225 and memory 215. Access to memory 215 can be accomplished using the communication bus 225. Processing unit 210 may include any programmable controller device including, for example, a mainframe processor, a mobile phone processor, or, as examples, one or more members of the INTEL® ATOM™, INTEL® XEON™, and INTEL® CORE™ processor families from Intel Corporation and the Cortex and ARM processor families from ARM. (INTEL, INTEL ATOM, XEON, and CORE are trademarks of the Intel Corporation. CORTEX is a registered trademark of the ARM Limited Corporation. ARM is a registered trademark of the ARM Limited Company). Memory 215 may include one or more memory modules and comprise random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), programmable read-write memory, and solid-state memory. As also shown in
Referring now to
The processing unit core 210 is shown including execution logic 280 having a set of execution units 285-1 through 285-N. Some embodiments may include a number of execution units dedicated to specific functions or sets of functions. Other embodiments may include only one execution unit or one execution unit that can perform a particular function. The execution logic 280 performs the operations specified by code instructions.
After completion of execution of the operations specified by the code instructions, back end logic 290 retires the instructions of the code 250. In one embodiment, the processing unit core 210 allows out of order execution but requires in order retirement of instructions. Retirement logic 295 may take a variety of forms as known to those of skill in the art (e.g., re-order buffers or the like). In this manner, the processing unit core 210 is transformed during execution of the code 250, at least in terms of the output generated by the decoder, the hardware registers and tables utilized by the register renaming logic 262, and any registers (not shown) modified by the execution logic 280.
Although not illustrated in
Multi-protocol, Multi-format Inbox
The inbox 300 enables user to interact with people across multiple formats and multiple protocols (e.g., email, text, voice, video, social messaging, etc.) as described, for example, in U.S. patent application Ser. No. 14/141,551, filed Dec. 27, 2013; U.S. patent application Ser. No. 14/168,815, filed Jan. 30, 2014; U.S. patent application Ser. No. 14/461,967, filed Aug. 18, 2014; and U.S. patent application Ser. No. 14/579,712, filed Dec. 22, 2014. The disclosure of each of the foregoing patent applications (collectively referred to herein as “The Entefy personal communications patent applications”) are hereby incorporated by reference. Each person is represented on a row. In the example of
As is shown across the top row 302 of the inbox 300 also enables users to interact with service providers and devices via multiple formats and communication protocols. All messages may be combined together into a single, unified inbox, as is shown in
In the example of
In the example of
Referring now to
The top portion 321 of the interface 320 includes a pictorial icon 322 and the name of the particular smart device 160 (in this example, a smart thermostat) as well as buttons to view and/or change settings related to the smart device 160. The interface 320 also includes a device interface 325 to control the smart device 160. In this instance, up arrow 326 increases the target temperature 327 of the smart thermostat and the down arrow 328 decreases the target temperature 327. As described in more detail below, the universal interaction platform allows a user to interact with the smart device 160 by sending messages to the smart device 160 using natural language. In the example shown in
Referring now to
As described in more detail below, the interface 340 allows users to interact with the smart device 160 by sending messages using natural language. The universal interaction platform also receives responses from the smart device 160, formats the responses in the form of messages for the user from the smart device 160, and outputs the messages to the user via the interface 340. The interface 340 includes message history 342 showing messages from the user to the smart device 160 and responses from the smart device 160 in the form of messages to the user from the smart device 160. Messages to the smart device 160 may include text (input, for example, by a keyboard or virtual ‘on screen’ keyboard output by a touchscreen) or sound (input, for example, by a microphone). The interface 340 may include a text entry box 344 and a microphone icon 346 to record sound or convert speech to text. A back button 348 may be used to return to the interface 320 and a send button 349 may be used to send the message to the smart device 160.
Referring now to
The interface 350 includes a message area 352 where messages from the provider of the smart device 160 are displayed to the user. The interface 350 may also include a call button 354 to place a telephone call to the provider of the smart device 160. Similar to the interface 340, the interface 350 may also include a text entry box 344 to enter text-based response to the provider of the smart device 160, a send button 349 to send the text-based response, a microphone icon 346 to record sound or convert speech to text, and a back button 348 to return to the interface 320.
Referring now to
The top portion 361 of the interface 360 includes a pictorial icon 362 and the name of the particular service provider (in this example, a ride service) as well as buttons to view and/or change settings related to the service provider. The interface 360 also includes a service provider interface 362 to interact with the service provider. As described in more detail below, the universal interaction platform allows a user to request services from a service provider by sending messages using natural language. In the example shown in
Referring now to
As described in more detail below, the interface 370 allows users to interact with the service provider by sending messages using natural language. The universal interaction platform also receives responses from the service provider, formats the responses in the form of messages for the user from the service provider, and outputs the messages to the user via the interface 370. The interface 370 includes message history 372 showing messages from the user to the service provider and responses from the service provider in the form of messages to the user from the smart device 160. Messages to the service provider may include text (input, for example, by a keyboard or virtual ‘on screen’ keyboard output by a touchscreen) or sound (input, for example, by a microphone). Similar to the interface 340, the interface 370 may include a text entry box 344 to enter text-based messages to the service provider, a microphone icon 346 to record sound or convert speech to text, a back button 348 to return to the interface 360 and a send button 349 to send the message to service provider. Each service provider may have more than one protocol for requesting services from that service provider and/or for outputting responses to service requests. For example, sending a message requesting a car may require outputting instructions to an API while the ride service may send responses by SMS and email. SMS messages may be received from more than one phone number. Email messages may be received from more than one email address (e.g., receipts@rideservice.xyz, support@rideservice.xyz, etc.). Regardless of protocol or format, the universal interaction platform described herein combines all messages to and from each service provider regarding service requests into a single interface 370.
Referring now to
Similar to the interface 350, the interface 380 includes a message area 352 where messages from the service provider are displayed to the user, a call button 354 to place a telephone call to the service provider, a text entry box 344 to enter text-based response to the service provider, a send button 349 to send the text-based response, a microphone icon 346 to record sound or convert speech to text, and a back button 348 to return to the interface 360.
Referring now to
The messaging service 420 sends and receives messages to and from the client devices 130 (via the inbox 300 and the interfaces 360, 380, etc. as described above). The messaging service 420 also communicates with the third party communications devices 140 as described, for example, in the Entefy personal communications patent applications, which are referenced above and incorporated by reference.
The IoT service 430 includes an IoT handler 432 for each service provider or smart device 160 supported by the universal interaction platform 400. Each IoT handler 432 may interact with a particular service provider or type (e.g., brand, model, etc.) of smart device 160. For example, IoT handler 432a may interact with the ride service described above while IoT handler 432i may interact with the smart thermostat described above.
The natural language classifier 440 is a machine learning tool that derives meaning from natural language by applying rules developed by analyzing a large corpora of real-world examples. As described in more detail below, the corpora may include publicly-available data sets as well as natural language documents collected by the universal interaction platform 400 such as messages output by users, descriptions of the services provided by the service providers, documentation regarding the services and the smart devices 160, etc.
The universal interaction platform 400 enables users to interact with smart devices 160 and service providers via the multi-protocol, multi-format inbox 300 illustrated in
In some instances, the universal interaction platform 400 obtains information regarding a service provider by downloading publicly-available information regarding the service provider and storing that information in a structured data set such that the universal interaction platform 400 is able to respond to user queries.
The universal interaction platform 400 may periodically download information from service providers so as to provide real-time or near real-time information to users. Referring to
In other instances, a service provider may maintain an application programming interface (API) that enables the universal interaction platform 400 to receive information regarding the services offered by the service provider and/or send instructions to the service provider. In this instance, the API documentation may specify the services provided by the service provider and the protocols and formats required by the service provider to request information regarding those services and/or to request those services.
An API may also specify the protocol for receiving information from the service provider and the format of the information output by the service provider.
In other instances, a smart device 160 may have an API that enables the universal interaction platform 400 to receive information stored in the smart device 160 and/or send instructions to the smart device 160 by sending and receiving an object to and from the smart device 160. The object may be, for example, in JavaScript Object Notation (JSON) format, extensible markup language (XML) format, etc. The protocol for communicating with the smart device 160 and the format of the object may be specified by the provider of the smart device 160 in the API documentation.
In order to control the smart thermostat, the thermostat requires an object with the same data values shown in
A client device 130 outputs a message object in step 802. The message object may be, for example, a universal message object in JavaScript Object Notation (JSON) format. The message object may include the body of the message input by the user, for example, via a message interface (e.g., 360, 380) and information indicative of the recipient (i.e., a smart device 160 or service provider) selected by the user via the inbox 300.
The messaging service 420 receives the message object in step 804 and saves a copy of the message in a database in step 806. The database may be stored, for example, in the storage device 122. The messaging service 420 determines the identity of the recipient in step 808. If the recipient is a person (Step 810: Yes), the messaging service 420 outputs the message to the recipient in step 812 (as described, for example, in “the Entefy personal communications patent applications, which are referenced above and incorporated by reference).
If the recipient is a smart device 160 or a service provider (Step 810: No), the natural language classifier 440 parses the message content to determine the intent of the user in step 814. The natural language classifier 440 determines user intent by making a probabilistic determination based on a large data set of information including previous messages output by the user and other users, documentation regarding the selected smart device 160 or service provider, publicly available and other natural language data sets. For each smart device 160 and service provider supported by the universal interaction platform 400, for example, the universal interaction platform 400 may store a map in the storage device 122 that associates natural language expressions previously sent by users (or that could potentially be sent by users) with user intents relevant to the smart device 160 or service provider. The natural language classifier 440 determines the user intent by finding the natural language expression that is most similar to the message sent by the user and selecting the user intent associated with the natural language expression. Using a smart thermostat (e.g., Next) as an example, an exemplary map is shown in Table 1.
If, for example, the universal interaction platform 400 receives a message from a user stating “I'd like it to be 68 in my house,” the natural language classifier 440 compares the message received from the user with the natural language expressions stored in the map and determines the natural language expression (“I'd like it to be <temp> in here”) that is most similar to the message received from the user (“I'd like it to be 68 in my house”) and selects the user intent (in this instance, to set the temperature to 68) associated with the most similar natural language expression.
The universal interaction platform 400 also determines user intent based on other factors, such as the location of the client device 130 and/or the time the message object was output by the client device 130 (or received by the server 121). Returning to the ride service example above, a user may output a message requesting a ride and the universal interaction platform 400 may determine that the user intent is to request a ride from the current location of the client device 130. Returning to the commuter train system example above, a user near may output a message asking about the next train to San Francisco. The universal interaction platform 400 may determine that the client device 130 is closest to Palo Alto and, therefore, may determine that, in this instance, the user intent is to request the departure time for the next train to San Francisco from Palo Alto. Furthermore, the universal interaction platform 400 may determine that the client device 130 is closest to Palo Alto but is not close enough for the user to board the next train to San Francisco. Accordingly, the universal interaction platform 400 may determine that the user intent, in this instance, is to request the departure time for the second train to San Francisco from Palo Alto.
The universal interaction platform 400 also determines user intent based on previous messages output by the user via the inbox 300. Accordingly, universal interaction platform 400 may also store a map of natural language expressions for a particular user. Returning to the smart thermostat example above, an exemplary map of natural language expressions for a particular user is shown in Table 2.
If, for example, the universal interaction platform 400 receives a message from the user associated with Table 2 stating “Make it a bit warmer”, the natural language classifier 440 compares the message received from the user with the natural language expressions stored in Tables 1 and 2, determines the natural language expressions (“Make it <delta_temp> warmer” and “a bit”) that are most similar to the messages received from the user (“Make it a bit warmer”), and determines the intent of the user. In this example, the intent of the user to increase the temperature is found in Table 1 and the exact value of the increase is found in Table 2.
The universal interaction platform 400 develops the map of natural language expressions for a particular user based on previous messages output by the user via the inbox 300. For example, the first time a user outputs a message to the universal interaction platform 400 for transmittal to the smart thermostat and the universal interaction platform 400 cannot determine the user intent based on the map used for all users shown in Table 1 (for example, when the user says “a bit” or “I'm cold” or “Brian is coming over”), the universal interaction platform 400 may ask the user to clarify how the user would like the smart thermostat to respond and then store that user intent for future use when the user outputs another message using similar language. Returning to the ride service example above, if the user requests a ride from “Brian's house”, the universal interaction platform 400 may ask “Where is Brian's house?” (or “Are you currently at Brian's house?”) and then map the user's response (or the location of the client device 130) to “Brian's house” for future use.
Because there is a finite number of relevant instructions for each smart device 160 or service provider, there is a finite number of relevant user intents. Accordingly, the universal interaction platform 400 is able to map each user intent to any number of natural language expressions. The natural language classifier 440 is then able to quickly and accurately determine the user intent based on the natural language expressions used by the user.
The IoT handler 432 associated with the selected service provider or smart device 160 formats an instruction for the smart device 160 or service provider in step 816. The universal interaction platform 400 stores the formats and protocols required by each of the smart devices 160 and service providers supported by the platform in the storage device 122. Additionally, the universal interaction platform stores a map associating each of the user intents described above with a properly formatted instruction for the smart device 160 and service provider.
Using a smart thermostat again as an example, a map associating each of the user intents described above with a properly formatted instruction is shown in exemplary Table 3.
In the smart thermostat example above, the IoT handler 432 associated with the type of smart thermostat formats an instruction to set the temperature to 68 degrees Fahrenheit by building a JSON object with the metadata for the specific smart thermostat and the required data values (for example, device_id: peyiJNo0IldT2Y1IVtYaGQ; locale: en-US; . . . target_temperature_f: 68) and outputting the JSON object for transmittal to the thermostat.
Using the ride service again as an example, the IoT handler 432 associated with the ride service request formats an instruction to request a ride by including all of the required parameters specified in the API documentation (for example, start_latitude: 37.334381; start_longitude: −121.89432; end_latitude: 37.77703; end_longitude: −122.419571; product_id: a1111c8c-c720-46c3-8534-2fcdd730040d) and outputting the instruction to the API endpoint (/v1/requests) specified in the API documentation.
In other instances, the IoT service 430 may determine that the most probable way to interact with a service provider is to send an email to the service provider. For example, a user may input a query requesting a quote for new electric service from XYZ Electric. The universal interaction platform 400, having previously downloaded information from the XYZ website, may not have the information necessary to respond to the user query but may have determined that requests for services may be sent by email to bob@xyzelectric.com. Accordingly, the IoT Service 430 may determine that the method that will most probably return an answer to the user query is to have the messaging service 420 format an email with the fields required for that particular service provider (for example, to: bob@xyzelectric.com) and a body of the email indicative of the user intent (for example, body: “A user has requested a quote for a new electric service.”) and output the email message to the service provider.
If the IoT handler 432 associated with the device 160 or service provider selected by the user is unable to format the instruction for the smart device 160 or service provider (Step 818: No), the natural language classifier 440 parses the message object (received from the user in step 804) to determine the relevant device category or service category in step 822. For example, the relevant category may be transit.
The IoT service 430 determines the IoT handlers 432 that are relevant to the device category or service category in step 824. The relevant IoT handlers 432 may be selected from among the IoT handlers 432 associated with smart devices or services set up by the user.
The IoT service 430 outputs the message object to all of the relevant IoT handlers 432 in step 826. Each of the IoT handlers 834 respond to the message object with a given confidence level that the message object is relevant to the smart device 160 or service provider associated with that IoT handler 432 in step 828. The IoT service 430 selects the IoT service 432 with the highest confidence level in step 830. The selected IoT handler 432 formats an instruction in step 831 as described above with reference to step 816.
The IoT handler 432 (or the messaging service 420) outputs the instruction for transmittal to the smart device 160 or third party service provider device 150 in step 820.
The IoT handler 432 may receive a response from the service provider device 150 or the smart device 160 in step 832. The response may be in the form of an object, an email message, an SMS message, etc. As described above with reference to
The messaging service 420 formats the response message for the user in step 834. The response message is output using natural language. The tone of the natural language may be based on user preferences stored, for example, in the storage device 122. The user preferences may specify a different tone for different services. For example, a user may specify a more formal tone (e.g., “Sir, your car has arrived” and “Thank you. We hope you enjoyed your trip”) when the requested service is a luxury car and a more informal tone (e.g., “Your car is here” and “Trip complete”) when the requested service is not a luxury car.
Additionally or alternatively, the tone of response messages output on behalf of a service provider or a smart device 160 may be specified by the service provider or provider of the smart device. A service provider or provider of a smart device may specify different tones for response messages based on the demographics of the user (age, gender, etc.). The preferences of the service providers and/or providers of smart devices 160 may be stored in the storage device 122.
The universal interaction platform 400 updates the user preferences in the database in step 836.
The messaging service 420 outputs the response message to the user in step 838.
AI-Driven API Analyzer
As described above, the universal interaction platform 400 outputs instructions to third party smart devices 160 and service providers to perform certain functions. The protocols and formats required by a device or service provider to perform those functions may be determined manually. Additionally or alternatively, those protocols and formats may be determined by the API analyzer 460 as described below.
An API endpoint for a service provider or smart device is identified in step 902a. In some instances, a user (for example, a developer for the service provider or provider of the smart device 160) may identify an API endpoint by registering a device 160 or service with the universal interaction platform 400. For example, the API analyzer 460 may include a user interface that allows a user to enter the universal resource locator (URL) of the API endpoint. In other instances, the universal interaction platform 400 may identify the API endpoint, for example by crawling the internet and identifying devices 160 or service providers to add to the universal interaction platform 400.
A service category or device category is determined in step 904a. For example, the service category for a ride service may be “on-demand transit” while the device category for a smart thermostat may be “appliance” or “smart home.” The service category or device category may be selected by the user. For example, the API analyzer 460 may include a user interface that allows a user to select the device category or service category. Additionally or alternatively, the natural language classifier 440 may determine a device category or service category by analyzing documentation (e.g., a website and/or API documentation) regarding the service provider or smart device 160.
A category-specific corpus is selected in step 906a based on the service category or device category determined in step 904a. A corpus is a collection of information about the words and phrases of a language and about the lexical categories to which those words belong. The API analyzer 460 may store a category-specific corpus for each of a plurality of potential service categories and device categories in the storage device 122. For each word and/or phrase in the category-specific corpus, the category-specific corpus may include, for example, definitions, parts of speech, synonyms, lexical connections to other words or phrases, etc. The category-specific corpus may also include weighted probabilities that each word or phrase in the category-specific corpus is related to other words or phrases in the category-specific corpus. As additional service providers and smart devices 160 are added to the universal interaction platform 400 over time, the category-specific corpuses may be modified to include additional words and phrases that are relevant to that particular category. The API analyzer 460 may also store a global corpus that may be used for all smart devices 160 and service providers or for the smart devices 160 and service providers that are unrelated to the category-specific corpuses.
As described in detail below, the API analyzer 460 forms a service-specific or device-specific corpus by appending the category-specific corpus (or global corpus) to include words or phrases relevant to the specific smart device 160 or service.
The user is prompted to manually identify service-specific or device-specific information in step 908. Service-specific or device-specific information may include words or phrases that are relevant to the specific service or smart device 160. The API analyzer 460 may enable the user, for example, to manually describe service-specific or device-specific words or phrases, to upload a service-specific or device-specific dictionary, to identify an object (e.g., a JSON object, an XML object, etc.) that is relevant to the service-specific or device-specific API, etc. If the user manually identifies service-specific or device-specific information (Step: 910: Yes), the API analyzer 460 appends the category-specific corpus (selected in step 906a) to include the service-specific or device-specific information in step 911. Because the category-specific corpus may be appended to include service-specific or device-specific information, it is referred to below as a service-specific or device-specific corpus.
The user is prompted to enter a URL of API documentation in step 912. The API analyzer 460 parses the API documentation in step 914. The API analyzer 460 may determine, for example, the functions available via the API, and the API endpoints, methods, and parameters required to request those functions (as shown, for example, in
The API analyzer 460 tests the API in step 916. Because the API documentation parsed in step 914 may be incomplete or inaccurate, the API analyzer 460 may determine, for example, whether the API may be used to perform additional functions (for example, by including different parameters than are identified in the API documentation). The API analyzer 460 may also receive and parse response messages received from the API.
The API analyzer 460 uses the service-specific or device-specific corpus described above to make probabilistic determinations of the relationships between words and phrases related to the service provider or smart device 160 (e.g., words or phrases used in the API documentation, the API endpoints, the API parameters, the data values of the object, etc.) in step 918. For example, the API analyzer 460 may determine whether a word or phrase is a subset of another word or phrase, whether those words or phrases are synonymous, whether those words or phrases are unrelated, etc.
The API analyzer 460 may prompt the user to verify the previous determinations in step 920. For example, the API analyzer 460 may display the API endpoints, methods, and parameters identified in step 914 via a graphical user interface and prompt the user to confirm whether those API endpoints, methods, and parameters are accurate and complete. The API analyzer 460 may also prompt the user to verify whether the relationships between words and phrases determined in step 918 are accurate. (For example, one API endpoint identified in step 914 may be /calendars/{calendarId}/events/{eventId}. Based on the API endpoint, the API analyzer 460 may determine that ‘event’ (or ‘events’) is not synonymous with ‘calendar’ or ‘calendars’ but is instead a separate referencable object. However, the API analyzer 460 may determine that there still exists a probability that ‘calendar’ and ‘events’ refer to the same thing. Accordingly, the API analyzer 460 may prompt the user to verify the previous determination that ‘calendar’ and ‘events’ are not synonymous but rather are a separate referencable objects.)
If the user indicates that one or more of the previous determinations were invalid (Step 922: No), the API analyzer 460 appends the service-specific or device-specific corpus in step 924 and returns to step 914. The API analyzer 460 may then re-parse the API documentation to determine API functions, endpoints, methods, and parameters (step 914), re-test the API (step 916, if necessary), make additional determinations regarding the relationships between words and phrases related to the service provider or smart device 160 (step 918) and prompt the user to verify the additional determinations (step 920) with the additional information in the device-specific or service-specific corpus.
In step 926a, the API analyzer 460 prompts the user to identify natural language expressions (i.e., instructions and/or queries) that are relevant to the smart device 160 or service provider (as described above, for example, with reference to Table 1) and the API functions and parameters necessary for the smart device 160 or service provider to respond to those instructions or queries (as described above, for example with reference to Table 3).
If the user maps API functions and parameters with natural language expressions (Step 928a: Yes) the API analyzer 460 appends the device-specific or service-specific corpus in step 930a.
In step 932a, the API analyzer 460 maps additional natural language expressions to the API functions and parameters identified in step 914. The additional natural language expressions may be identified by parsing the device-specific or service-specific corpus. The additional natural language expressions may also be natural language expressions that were previously determined to be relevant to the same category of smart device 160 or service provider. Additionally, the additional natural language expressions may be identified based on messages sent by users via the universal interaction platform 400. For example, the universal interaction platform 400 may store messages sent to other users, services or smart devices 160, categorize each message as relevant to certain categories of services or smart devices 160, and identify potentially-relevant natural language expressions from the messages that were identified as relevant to the category of services or smart devices 160. (For the purposes of determining potentially-relevant natural language expressions, the messages are stored without personally identifiable information.)
In step 934a, the API analyzer 460 prompts the user to validate the API mapping performed in step 932a. If any of the mappings were invalid (Step 936a: No), the API analyzer 460 appends the device-specific or service-specific corpus in step 930a and re-runs the API mapping algorithm with the additional information in step 932a. The API analyzer 460 also prompts the user to validate that the API mapping is complete in step 934a. If the user wishes to map additional natural language expressions to API functions (Step 938a: No), the process returns to step 926a. Otherwise (step 938a: Yes), the process is complete.
Websites that describe the services provided by a service provider website are identified in step 902b. The websites may include the service provider website, a third party review website describing the service provider, etc. In some instances, a user (for example, the service provider) may identify the service provider websites via a user interface. In other instances, the universal interaction platform 400 may identify the service provider websites, for example by crawling the internet and identifying service providers to add to the universal interaction platform 400.
A service category is determined (for example, by user selection or by the API analyzer 460 parsing the service provider websites) in step 904b. A category-specific corpus is selected in step 906b based on the service category determined in step 904b. The category-specific corpus is appended to form a service-specific corpus as described below.
The API analyzer 460 parses the content of the service provider websites to extract distinct objects that hold potentially meaningful information in step 942. (If a service provider website says, for example, “We are available from 9 am to 10 pm at 215-569-5500 with friendly service” the API analyzer 460 may determine that ‘9 am,’ 10 pm′ and ‘215-569-5500’ are potentially meaningful whereas “with friendly service” is not an object that includes potentially meaningful information.) The meaningful information may include any information relevant to the service provider or the services offered by the service provider. For example, the API analyzer 460 may parse a restaurant website and extract objects that include information regarding the service provider such as the name, address, telephone number, reviews, hours of operation, etc. as well as objects that include information regarding the services, such as the title, description, and price of each menu item. In the commuter train system example described above with reference to
The API analyzer 460 removes duplicate objects in step 944. The service provider websites may list certain information (for example, an address or phone number) in more than one location. Additionally, some of the same information may be included in both a service provider website and a third-party review website. Accordingly, the API analyzer 460 eliminates the duplicate objects.
The API analyzer 460 determines a probably category for each object by comparing the objects to a database of known data types, structures, and/or keywords in step 946. In the example above, the API analyzer 460 may compare the object ‘215-569-5500’ to the database of known data types, structures, and/or keywords and determine based on the comparison to the database that the most probable category of that object is a phone number.
The API analyzer 460 may prompt the user to validate the categorization of each object in step 948. In the example above, the API analyzer 460 may prompt the user to confirm that ‘215-569-5500’ is a phone number. If the user indicates that any of the categorizations are incorrect (Step 950: No), the service-specific corpus is appended to include this additional information in step 952 and the API analyzer returns to step 946 and re-categorizes the objects.
Once the user indicates that all of the categorizations are correct (Step 950: Yes), the API analyzer 460 determines an optimal API structure in step 954 for users to query the dataset of information regarding the service provider. The optimal API structure is a map linking the objects extracted from the service provider websites with queries for returning that meaningful information. The queries and the objects may have a many-to many relationship. Using the example above, the map may include the example queries and the objects as shown in Table 4.
Using the commuter train system example described above with reference to
The API analyzer 460 may determine the optimal API structure by comparing the dataset for the service provider to other API structures within the universal interaction platform 400 and selecting an API structure (for example, an API structure developed by a third party developer) that is used to output similar types of data. If none of the API structures within the universal interaction platform 400 are used to output similar types of data, the API analyzer 460 may simply store each object separately and create a separate query to return each of the objects. If the universal interaction platform 400 includes multiple APIs that are used to output similar types of data, the API analyzer 460 may select the API structure that is most commonly used to output similar types of data. The universal interaction platform 400 may also include an internal rating of some or all of the API structures and the API analyzer 460 may select the highest rated API structure from among the API structures that are used for similar types of data.
As described above with reference to
In step 926b, the API analyzer 460 prompts the user to map the queries identified by the API analyzer 460 in step 954 with natural language expressions (i.e., queries). Using the examples in Table 4 above, the map may include the example mappings shown in Table 5.
Using the commuter train system example described above, the map may include the example mappings shown in Table 6.
If the user maps API functions and parameters with natural language expressions (Step 928b: Yes) the API analyzer 460 appends the service-specific corpus in step 930b.
In step 932b, the API analyzer 460 maps additional natural language expressions to the queries identified by the API analyzer 460 in step 954. The additional natural language expressions may be identified by parsing the service-specific corpus. The additional natural language expressions may also be natural language expressions that were previously determined to be relevant to the same category of service provider. Additionally, the additional natural language expressions may be identified based on messages sent by users via the universal interaction platform 400. For example, the universal interaction platform 400 may store messages sent to other users, services or smart devices 160, categorize each message as relevant to certain categories of service providers, and identify potentially-relevant natural language expressions from the messages that were identified as relevant to the category of service providers. (For the purposes of determining potentially-relevant natural language expressions, the messages are stored without personally identifiable information.)
In step 934b, the API analyzer 460 prompts the user to validate the mappings performed in step 932b. If any of the mappings were invalid (Step 936b: No), the API analyzer 460 appends the service-specific corpus in step 930b and re-runs the API mapping algorithm with the additional information in step 932b. The API analyzer 460 also prompts the user to validate that the API mapping is complete in step 934b. If the user wishes to map additional natural language expressions to API functions (Step 938b: No), the process returns to step 926b. Otherwise (step 938b: Yes), the process is complete.
The following examples pertain to further embodiments.
Example 1 is a computer-implemented method of determining protocols and formats to interact with a service provider or smart device via an application program interface (API), comprising: identifying an API endpoint for the service provider or smart device; determining a service category or device category for the service provider or smart device; selecting a category-specific corpus based on the service category or device category; forming a service-specific or device-specific corpus by appending information regarding the service provider or smart device to the category-specific corpus; parsing API documentation to determine functions available via the API and parameters required to request the functions available via the API; and mapping the functions and parameters to natural language expressions.
Example 2 includes the subject matter of example 1, wherein the API endpoint and the service category or device category are determined based on user input.
Example 3 includes the subject matter of example 1, further comprising: prompting a user to manually input information regarding the service provider or smart device; and appending the information manually input by the user to the service-specific or device-specific corpus.
Example 4 includes the subject matter of example 1, further comprising: determining relationships between words or phrases included in the API documentation; prompting a user to validate the relationships between the words or phrases included in the API documentation; and appending the service-specific or device-specific corpus based on the user validation.
Example 5 includes the subject matter of example 1, wherein mapping the functions and parameters to natural language expressions comprises: prompting a user to map one or more of the functions and parameters to natural language expressions; appending the service-specific or device-specific corpus based on the user mapping; and mapping additional functions and parameters to natural language expressions or mapping the one or more functions and parameters to additional natural language expressions.
Example 6 includes the subject matter of example 1, wherein mapping the functions and parameters to natural language expressions further comprises: prompting a user to validate the mappings of the functions and parameters to natural language expressions.
Example 7 is an application program interface (API) analyzer, comprising: one or more storage devices that store a plurality of category-specific corpuses; a user-interface; one or more processing units that: identify an API endpoint for the service provider or smart device; determine a service category or device category for the service provider or smart device; select one of the category-specific corpuses based on the service category or device category; form a service-specific or device-specific corpus by appending information regarding the service provider or smart device to the category-specific corpus; store the service-specific or device-specific corpus in the one or more storage devices; parse API documentation to determine functions available via the API and parameters required to request the functions available via the API; map the functions and parameters to natural language expressions; store the mappings in the one or more storage devices.
Example 8 includes the subject matter of example 7, wherein the one or more processing units: prompt a user to manually input information regarding the service provider or smart device via the graphical user interface and append the information manually input by the user to the service-specific or device-specific corpus; determine relationships between words or phrases included in the API documentation, prompt the user to validate the relationships between the words or phrases included in the API documentation via the graphical user interface, and append the service-specific or device-specific corpus based on the user validation; or map the functions and parameters to natural language expressions by prompting a user to map one or more of the functions and parameters to natural language expressions via the graphical user interface, appending the service-specific or device-specific corpus based on the user mapping, and mapping additional functions and parameters to natural language expressions or maps the one or more functions and parameters to additional natural language expressions.
Example 9 includes the subject matter of example 7, wherein the one or more processing units: prompt a user to validate the mappings of the functions and parameters to natural language expressions via the graphical user interface.
Example 10 is a non-transitory computer readable storage medium (CRSM) comprising computer executable instructions stored thereon to cause one or more processing units to: identify an application program interface (API) endpoint for the service provider or smart device; determine a service category or device category for the service provider or smart device; select a category-specific corpus based on the service category or device category; form a service-specific or device-specific corpus by appending information regarding the service provider or smart device to the category-specific corpus; parse API documentation to determine functions available via the API and parameters required to request the functions available via the API; and map the functions and parameters to natural language expressions.
Example 11 includes the subject matter of example 10, wherein the API endpoint and the service category or device category are determined based on user input.
Example 12 includes the subject matter of example 10, wherein the instructions further cause the one or more processing units to: prompt a user to manually input information regarding the service provider or smart device; and append the information manually input by the user to the service-specific or device-specific corpus.
Example 13 includes the subject matter of example 10, wherein the instructions further cause the one or more processing units to: determine relationships between words or phrases included in the API documentation; prompt a user to validate the relationships between the words or phrases included in the API documentation; and append the service-specific or device-specific corpus based on the user validation.
Example 14 includes the subject matter of example 10, wherein the instructions further cause the one or more processing units to map the functions and parameters to natural language expressions by: prompting a user to map one or more of the functions and parameters to natural language expressions; appending the service-specific or device-specific corpus based on the user mapping; and mapping additional functions and parameters to natural language expressions or mapping the one or more functions and parameters to additional natural language expressions.
Example 15 includes the subject matter of example 10, wherein the instructions further cause the one or more processing units to: prompt a user to validate the mappings of the functions and parameters to natural language expressions.
Example 16 is another non-transitory computer readable storage medium (CRSM) comprising computer executable instructions stored thereon to cause one or more processing units to: identify service provider websites; determine a service category for the service provider; select a category-specific corpus based on the service category; form a service-specific corpus by appending information regarding the service provider to the category-specific corpus; parse the service provider websites to extract objects that include meaningful information regarding the service provider or services offered by the service provider; categorize each of the extracted objects by comparing the extracted objects to a database of known data types, structures, and/or keywords and associating each of the extracted objects with a category; map each of the extracted objects to one or more queries; and map each of the queries to natural language expressions.
Example 17 includes the subject matter of example 16, wherein the service provider websites are determined based on user input.
Example 18 includes the subject matter of example 16, wherein the instructions further cause the one or more processing units to prompt a user validate the categorization of each of the extracted objects.
Example 19 includes the subject matter of example 16, wherein the instructions cause the one or more processing units to map each of the queries to natural language expressions by: prompting a user to map one or more of the queries to natural language expressions; appending the service-specific corpus based on the user mapping; and mapping additional queries to natural language expressions or map the one or more queries to additional natural language expressions.
Example 20 includes the subject matter of example 16, wherein the instructions further cause the one or more processing units to: prompt a user to validate the mappings of the queries to the natural language expressions.
In the foregoing description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. It will be apparent, however, to one skilled in the art that the disclosed embodiments may be practiced without these specific details. In other instances, structure and devices are shown in block diagram form in order to avoid obscuring the disclosed embodiments. References to numbers without subscripts or suffixes are understood to reference all instance of subscripts and suffixes corresponding to the referenced number. Moreover, the language used in this disclosure has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter. Reference in the specification to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one disclosed embodiment, and multiple references to “one embodiment” or “an embodiment” should not be understood as necessarily all referring to the same embodiment.
It is also to be understood that the above description is intended to be illustrative, and not restrictive. For example, above-described embodiments may be used in combination with each other and illustrative process steps may be performed in an order different than shown. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the invention therefore should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, terms “including” and “in which” are used as plain-English equivalents of the respective terms “comprising” and “wherein.”
This application claims priority to, and is a continuation of, the commonly-assigned nonprovisional patent application having U.S. patent application Ser. No. 14/986,157, filed Dec. 31, 2015, entitled “Universal Interaction Platform for People, Services, and Devices,” now Pat. Ser. No. 10/353,754, which claims priority to, and is a continuation of, the commonly-assigned nonprovisional patent application having U.S. patent application Ser. No. 14/986,111, filed Dec. 31, 2015, entitled “Universal Interaction Platform for People, Services, and Devices,” now Pat. Ser. No. 10/135,764, both of which are hereby incorporated by reference in their entireties.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5481597 | Given | Jan 1996 | A |
| 5951638 | Hoss | Sep 1999 | A |
| 6101320 | Schuetze | Aug 2000 | A |
| 6950502 | Jenkins | Sep 2005 | B1 |
| 7450937 | Claudatos | Nov 2008 | B1 |
| 7673327 | Polis | Mar 2010 | B1 |
| 7680752 | Clune, III | Mar 2010 | B1 |
| 7886000 | Polis | Feb 2011 | B1 |
| 7908647 | Polis | Mar 2011 | B1 |
| 8090787 | Polis | Jan 2012 | B2 |
| 8095592 | Polis | Jan 2012 | B2 |
| 8108460 | Polis | Jan 2012 | B2 |
| 8112476 | Polis | Feb 2012 | B2 |
| 8122080 | Polis | Feb 2012 | B2 |
| 8156183 | Polis | Apr 2012 | B2 |
| 8281125 | Briceno | Oct 2012 | B1 |
| 8296360 | Polis | Oct 2012 | B2 |
| 8433705 | Dredze | Apr 2013 | B1 |
| 8438223 | Polis | May 2013 | B2 |
| 8458256 | Polis | Jun 2013 | B2 |
| 8458292 | Polis | Jun 2013 | B2 |
| 8458347 | Polis | Jun 2013 | B2 |
| 8468202 | Polis | Jun 2013 | B2 |
| 8527525 | Fong | Sep 2013 | B2 |
| 8954988 | Laredo | Feb 2015 | B1 |
| 8959156 | Polis | Feb 2015 | B2 |
| 9529522 | Barros | Dec 2016 | B1 |
| 20020133509 | Johnston | Sep 2002 | A1 |
| 20020152091 | Nagaoka | Oct 2002 | A1 |
| 20020178000 | Aktas | Nov 2002 | A1 |
| 20020194322 | Nagata | Dec 2002 | A1 |
| 20040117507 | Torma | Jun 2004 | A1 |
| 20040137884 | Engstrom | Jul 2004 | A1 |
| 20040243719 | Roselinsky | Dec 2004 | A1 |
| 20040266411 | Galicia | Dec 2004 | A1 |
| 20050015443 | Levine | Jan 2005 | A1 |
| 20050080857 | Kirsch | Apr 2005 | A1 |
| 20050198159 | Kirsch | Sep 2005 | A1 |
| 20060041423 | Kline | Feb 2006 | A1 |
| 20060193450 | Flynt | Aug 2006 | A1 |
| 20070054676 | Duan | Mar 2007 | A1 |
| 20070073816 | Kumar | Mar 2007 | A1 |
| 20070116195 | Thompson | May 2007 | A1 |
| 20070130273 | Huynh | Jun 2007 | A1 |
| 20070237135 | Trevallyn-Jones | Oct 2007 | A1 |
| 20070299796 | MacBeth | Dec 2007 | A1 |
| 20080062133 | Wolf | Mar 2008 | A1 |
| 20080088428 | Pitre | Apr 2008 | A1 |
| 20080112546 | Fletcher | May 2008 | A1 |
| 20080236103 | Lowder | Oct 2008 | A1 |
| 20080261569 | Britt | Oct 2008 | A1 |
| 20090016504 | Mantell | Jan 2009 | A1 |
| 20090119370 | Stern | May 2009 | A1 |
| 20090177477 | Nenov | Jul 2009 | A1 |
| 20090177484 | Davis | Jul 2009 | A1 |
| 20090177744 | Marlow | Jul 2009 | A1 |
| 20090181702 | Vargas | Jul 2009 | A1 |
| 20090271486 | Ligh | Oct 2009 | A1 |
| 20090292814 | Ting | Nov 2009 | A1 |
| 20090299996 | Yu | Dec 2009 | A1 |
| 20100057872 | Koons | Mar 2010 | A1 |
| 20100210291 | Lauer | Aug 2010 | A1 |
| 20100220585 | Poulson | Sep 2010 | A1 |
| 20100223341 | Manolescu | Sep 2010 | A1 |
| 20100229107 | Turner | Sep 2010 | A1 |
| 20100250578 | Athsani | Sep 2010 | A1 |
| 20100312644 | Borgs | Dec 2010 | A1 |
| 20100323728 | Gould | Dec 2010 | A1 |
| 20100325227 | Novy | Dec 2010 | A1 |
| 20100332217 | Wintner | Dec 2010 | A1 |
| 20110010182 | Turski | Jan 2011 | A1 |
| 20110051913 | Kesler | Mar 2011 | A1 |
| 20110078247 | Jackson | Mar 2011 | A1 |
| 20110078256 | Wang | Mar 2011 | A1 |
| 20110078267 | Lee | Mar 2011 | A1 |
| 20110130168 | Vendrow | Jun 2011 | A1 |
| 20110194629 | Bekanich | Aug 2011 | A1 |
| 20110219008 | Been | Sep 2011 | A1 |
| 20110265010 | Ferguson | Oct 2011 | A1 |
| 20110276640 | Jesse | Nov 2011 | A1 |
| 20110279458 | Gnanasambandam | Nov 2011 | A1 |
| 20110295851 | El-Saban | Dec 2011 | A1 |
| 20120016858 | Rathod | Jan 2012 | A1 |
| 20120209847 | Rangan | Aug 2012 | A1 |
| 20120210253 | Luna | Aug 2012 | A1 |
| 20120221962 | Lew | Aug 2012 | A1 |
| 20120291072 | Maddison | Nov 2012 | A1 |
| 20130018945 | Vendrow | Jan 2013 | A1 |
| 20130024521 | Pocklington | Jan 2013 | A1 |
| 20130067345 | Das | Mar 2013 | A1 |
| 20130097279 | Polis | Apr 2013 | A1 |
| 20130111487 | Cheyer | May 2013 | A1 |
| 20130127864 | Nevin, III | May 2013 | A1 |
| 20130151508 | Kurabayashi | Jun 2013 | A1 |
| 20130197915 | Burke | Aug 2013 | A1 |
| 20130262385 | Kumarasamy | Oct 2013 | A1 |
| 20130262852 | Roeder | Oct 2013 | A1 |
| 20130267264 | Abuelsaad | Oct 2013 | A1 |
| 20130268516 | Chaudhri | Oct 2013 | A1 |
| 20130304830 | Olsen | Nov 2013 | A1 |
| 20130325343 | Blumenberg | Dec 2013 | A1 |
| 20130332308 | Linden | Dec 2013 | A1 |
| 20140020047 | Liebmann | Jan 2014 | A1 |
| 20140032538 | Arngren | Jan 2014 | A1 |
| 20140149399 | Kurzion | May 2014 | A1 |
| 20140270131 | Hand | Sep 2014 | A1 |
| 20140280460 | Nemer | Sep 2014 | A1 |
| 20140297807 | Dasgupta | Oct 2014 | A1 |
| 20150039887 | Kahol | Feb 2015 | A1 |
| 20150095127 | Patel | Apr 2015 | A1 |
| 20150149484 | Kelley | May 2015 | A1 |
| 20150186455 | Horling | Jul 2015 | A1 |
| 20150261496 | Faaborg | Sep 2015 | A1 |
| 20150278370 | Stratvert | Oct 2015 | A1 |
| 20150281184 | Cooley | Oct 2015 | A1 |
| 20150286747 | Anastasakos | Oct 2015 | A1 |
| 20150286943 | Wang | Oct 2015 | A1 |
| 20150339405 | Vora | Nov 2015 | A1 |
| 20160048548 | Thomas | Feb 2016 | A1 |
| 20160057207 | Li | Feb 2016 | A1 |
| 20160078030 | Brackett | Mar 2016 | A1 |
| 20160087944 | Downey | Mar 2016 | A1 |
| 20160092959 | Gross | Mar 2016 | A1 |
| 20160173578 | Sharma | Jun 2016 | A1 |
| 20160379136 | Chen | Dec 2016 | A1 |
| 20170039246 | Bastide | Feb 2017 | A1 |
| 20170039296 | Bastide | Feb 2017 | A1 |
| 20170206276 | Gill | Jul 2017 | A1 |
| 20170220963 | Canaran | Aug 2017 | A1 |
| 20170286155 | Hosabettu | Oct 2017 | A1 |
| 20170364587 | Krishnamurthy | Dec 2017 | A1 |
| 20180048661 | Bird | Feb 2018 | A1 |
| 20180101506 | Hodaei | Apr 2018 | A1 |
| Number | Date | Country |
|---|---|---|
| 9931575 | Jun 1999 | WO |
| 2013112570 | Aug 2013 | WO |
| Entry |
|---|
| Guangyi Xiao et al., “User Interoperability With Heterogeneous IoT Devices Through Transformation,” pp. 1486-1496, 2014. |
| Marr, Bernard, Key Business Analytics, Feb. 2016, FT Publishing International, Ch. 17 “Neural Network Analysis” (Year: 2016). |
| Number | Date | Country | |
|---|---|---|---|
| 20200050502 A1 | Feb 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14986157 | Dec 2015 | US |
| Child | 16511944 | US | |
| Parent | 14986111 | Dec 2015 | US |
| Child | 14986157 | US |
