Patent Application
20250117386
The present disclosure generally relates to natural language processing (NLP), and relates more specifically to applying artificial intelligence (AI) models to personal data.
AI models have become essential tools in numerous applications, including but not limited to automatic translation, chatbots, voice recognition systems, and search engines. Among these, Large Language Models (LLMs) have become prominent due to their ability to generate human-like text based on the context.
However, LLMs and other AI models exhibit significant limitations in the aspects of personalization and personal data search. Currently, LLMs generate data based on a general understanding of language learned from a vast set of training data. This approach results in a “one-size-fits-all” model that may not take into consideration the specific needs, preferences, or contexts of individual users.
Furthermore, LLMs lack the ability to effectively utilize user-specific data. This limitation poses a challenge in situations where users need specific information from their personal data set or history, such as past interactions, preferences, or contextual details. Similarly, in content creation or recommendation systems, the absence of personalization could result in outputs that fail to resonate with the specific tastes and interests of the user. While LLMs can generate contextually accurate content, their inability to personalize output based on individual user data limits their efficacy.
In view of these limitations, there is a need for methods, systems, and techniques for personal data and AI system integration, thereby enabling more user-specific applications, enhancing user experiences, and increasing overall engagement with AI-driven systems.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely based on their inclusion in this section.
The appended claims may serve as a summary.
In the drawings:
While each of the drawing figures illustrates a particular embodiment for purposes of illustrating a clear example, other embodiments may omit, add to, reorder, or modify any of the elements shown in the drawing figures. For purposes of illustrating clear examples, one or more figures may be described with reference to one or more other figures. However, using the particular arrangement illustrated in such other figure/s is not required in other embodiments. Unless explicitly stated otherwise, features of an element described with respect to one figure may apply to similar elements described in other figures.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the subject matter of the present application. It will be apparent, however, to a person of ordinary skill that embodiments may be practiced without incorporating all aspects of the specific details described herein. The detailed description that follows describes exemplary embodiments and the features disclosed are not intended to be limited to the expressly disclosed combination(s). Therefore, unless otherwise noted, features disclosed herein may be combined to form additional combinations that were not otherwise shown for purposes of brevity.
It will be further understood that: the term “or” may be inclusive or exclusive unless expressly stated otherwise; the term “set” may comprise zero, one, or two or more elements; the terms “first”, “second”, “certain”, and “particular” are used as naming conventions to distinguish elements from each other, and does not imply an ordering, timing, or any other characteristic of the referenced items unless otherwise specified; the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items; that the terms “comprises” and/or “comprising” specify the presence of stated features, but do not preclude the presence or addition of one or more other features.
As used herein, the term “component” may be hardware and/or software stored in, or coupled to, a memory and/or one or more processors on one or more computers. As an addition or alternative, a component may comprise specialized circuitry. For example, a component may be hardwired and/or persistently programmed with a set of instructions to perform the functions discussed herein. A component may be a standalone component, work in conjunction with one or more other components, contain one or more other components, and/or belong to one or more other components. A component may be implemented fully and/or partially in one or multiple programs and/or processes. Furthermore, two or more components may be implemented fully and/or partially in the same program and/or process.
As used herein, the term “computer system” refers to one or more computers, such as one or more physical computers, virtual computers, and/or computing devices. For example, a computer system may be, or may include, one or more server computers, desktop computers, laptop computers, mobile devices, special-purpose computing devices with a processor, cloud-based computers, cloud-based clusters of computers, virtual machine instances, and/or other computing devices. A computer system may include another computer system, and a computing device may belong to two or more computer systems. Any reference to a “computer system” may mean one or more computers, unless expressly stated otherwise. When a computer system performs an action, the action is performed by one or more computers of the computer system.
As used herein, the term “device” may be a computer system, hardware, and/or software stored in, or coupled to, a memory and/or one or more processors on one or more computers. As an alternative and/or addition, a device may comprise specialized circuitry. For example, a device may be hardwired or persistently programmed to support a set of instructions to perform the functions discussed herein. A device may be a standalone component, work in conjunction with one or more other devices, contain one or more other devices, and/or belong to one or more other devices.
As used herein, the term “client” refers to a combination of integrated software components and an allocation of computational resources, such as memory, a computing device, and/or processes on a computing device for executing the integrated software components. The combination of the software and the computational resources is configured to interact with one or more servers over a network, such as the Internet. A client may refer to either the combination of components on one or more computers, or the one or more computers (also referred to as “client computing devices”).
As used herein, the term “server” refers to a combination of integrated software components and an allocation of computational resources, such as memory, a computing device, and/or processes on the computing device for executing the integrated software components. The combination of the software and the computational resources is dedicated to providing a particular type of function on behalf of clients of the server. A server may refer to either the combination of components on one or more computing devices, or the one or more computing devices (also referred to as “server system”). A server system may include multiple servers; that is, a server system may include a first computing device and a second computing device, which may provide the same or different functionality to the same or different set of clients.
This document generally describes systems, computer-readable media, methods, and other techniques for personal data and AI system integration. In general, an assistant system generates, maintains, and/or accesses a user database comprising user data associated with a user from one or multiple data sources. The assistant system may be configured to facilitate the collection of the user data from diverse personal data sources, protect access and maintain user control over personal data, and/or integrate AI functionality with user data.
In some embodiments, an assistant agent communicates with a user and an AI system comprising one or more AI agents, such as but not limited to a large language model (LLM) agent. The assistant agent may act as an intermediary between the user and the AI system. The assistant agent provides the AI system with a user prompt description, a database description describing the user database, and instructions to provide a query for relevant data in the user database that the AI system can use to answer the user prompt. The assistant agent system executes the query received from the AI system against the user database, provides the result set to the AI system, and receives an answer from the AI system. The assistant agent uses the answer received from the AI system to generate a response for the user.
While one or more embodiments described herein are described with respect to integrating user data and AI system integration, the embodiments are not limited to the integration of user data and are adaptable to integrate any custom data set with AI systems as described herein. For example, one or more embodiments may be applied to generate a unified database comprising custom data from one or multiple data sources, where the custom data includes information outside of the training data set of an LLM. As an alternative and/or addition, one or more embodiments may be applied to integrate such custom data for use with one or more LLMs and/or other AI models/agents to achieve the results and benefits described herein.
One aspect of the disclosure is directed to a method comprising: determining to respond to a user prompt of a user; submitting, to an artificial intelligence (AI) system comprising at least one AI agent, a user prompt description, a database description describing a user database comprising user data associated with the user, and instructions to provide a query for relevant data in the user database that the AI system can use to answer to the user prompt; receiving the query from the AI system; executing the query against the user database to generate a result set; submitting the result set to the AI system; receiving an answer from the AI system based on the user prompt description and the result set; and generating a response for the user based on the answer from the AI system; wherein the method is performed by one or more computing devices.
In some embodiments, the user prompt description and the result set are submitted to a large language model (LLM) agent of the AI system.
In some embodiments, the database description comprises schema data corresponding to the user database.
In some embodiments, the method includes maintaining an interaction stream for a chat session comprising correspondence with the AI system that is associated with the user.
In some embodiments, the method includes submitting the interaction stream to a summarizer agent of the AI system; and storing a session summary generated by the summarizer agent. As an alternative and/or addition, the method includes submitting one or more session summaries to a short-term memory (STM) agent to determine whether any prior query results are relevant to providing the response; and based on determining that one or more prior query results are relevant, submitting the one or more prior query results to the AI system.
In some embodiments, the method includes submitting the result set to an examiner agent of the AI system to determine whether the result set is sufficient to answer the user prompt; when the result set is not sufficient to answer the user prompt, submitting, to the AI system, instructions to provide a second query for additional relevant data in the user database that the AI system can use to answer the user prompt; executing the second query against the user database to generate a second result set; and submitting the second result set to the AI system.
In some embodiments, the method includes submitting the user prompt description to a classifier agent of the AI system to determine whether the user prompt description falls into one or more or a plurality of categories; and when the prompt falls into a particular category, submitting additional information to the AI system based on a category template corresponding to the particular category.
In some embodiments, at least a portion of the method is performed at a client computing device of the user. As an alternative and/or addition, at least a portion of the method is performed by a browser extension executing at the client computing device.
In some embodiments, at least a portion of the method is performed at an assistant server system.
In some embodiments, the method includes obtaining user data from a plurality of data sources corresponding to a plurality of web server systems that maintain separate user data on behalf of the user; and adding the user data to the user database.
One aspect of the disclosure is directed to a computer system comprising one or more processors; and at least one memory coupled to the one or more processors and storing instructions which, when executed by the one or more processors, cause the one or more processors to perform one or more steps of one or more methods of this disclosure.
One aspect of the disclosure is directed to a non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform one or more steps of one or more methods of this disclosure.
In some implementations, the various techniques described herein may achieve one or more of the following advantages: enabling large language models (LLMs) and/or other AI technology to utilize a user's data; facilitating the collection and integration of user data from diverse data sources into a unified system; maintaining user control over user data and protecting user data, including personal data, in accordance with privacy rules and regulations; enabling diverse implementations ranging from cloud-based solutions to independent self-contained computer environments. Additional features and advantages are apparent from the specification and the drawings.
The web server systems 140-141 may collect and/or process various types of user data corresponding to a user accessing the hosted content and/or services. As used herein, the term “user data” refers to any data that is associated with an individual, such as a user. The user data may include data about the individual from one or multiple web server systems 140-141 and/or other data sources, including data from one or more user accounts maintained by the web server systems 140-141 and/or other data sources. As used herein, the term “user data” includes but is not limited to personal data as defined under one or more privacy rules and/or regulations.
For example, the user data may include IP address information, browser information, and/or computer system information. As an alternative and/or addition, the user data may include information submitted by the user, such as name, email address, phone number, and/or other user-provided content. As an alternative and/or addition, the user data may include usage data, which may encompass the pages visited by the user, the duration of each visit, interactions such as links clicked and files downloaded, user engagement patterns, and navigation paths through a webpage hosted by a web server system 140-141. As an alternative and/or addition, the user data may include demographic information, marketing information, preferences, and/or any other user data associated with the user.
The assistant system 100 may be configured to facilitate the collection of user data from diverse data sources into a unified database, protect access and maintain user control over the collected user data, and/or integrate AI functionality with the collected user data. In some embodiments, the assistant system 100 includes a data gathering system 136, an assistant agent system 138, and an AI system 110.
The data gathering system 136 of the assistant system 100 is configured to obtain the user data. The data gathering system 136 may obtain authorization from the user to obtain the user data on their behalf. The data gathering system 136 may be implemented at one or more computer systems, such as but not limited to a user computing device (e.g., as shown in
In some embodiments, the assistant system 100 handles user data in accordance with one or more privacy rules and/or regulations. For example, the assistant system 100 may ensure that the user retains ownership and/or control of their user data. In some embodiments, a user's data is stored on a computing device owned and/or controlled by the user. As an alternative and/or addition, the user's data may be stored on a server system. The user data stored on a server system may be encrypted at rest, such as for security and/or privacy reasons.
In some embodiments, the data gathering system 136 includes an external data request component 122, a local data collection component 124, and a data processing component 126. As noted with the term “component” generally, the data gathering system 136 and/or its components (such as but not limited to external data request component 122, local data collection component 124, and data processing component 126) may be implemented as one or more dependent or independent processes, and may be implemented on one or multiple computers; the components may be implemented fully and/or partially in one or multiple programs and/or processes; and two or more components shown may be implemented fully and/or partially in the same program and/or process.
The external data request component 122 obtains user data corresponding to the user from one or more data sources that generate and/or store user data corresponding to the user. For example, the external data request component 122 may communicate with one or more data sources storing the user data over a network 150. The data sources may include one or more web server systems 140-141 that host web content 142-143 and make the web content 142-143 available to client computing devices.
In some embodiments, the external data request component 122 obtains user data by obtaining one or more user data files associated with the user. For example, the external data request component 122 may communicate with a data download request component of a data source such as a web server system 140-141. The external data request component 122 may obtain authorization from the user to obtain the user data on their behalf. The external data request component 122 may download one or more user data files from one or more web server systems 140-141. The user data files may be from a data request that was facilitated for the user. This may include a Subject Access Request (SAR) made in accordance with the General Data Protection Regulation (GDPR) to a data controller. In some examples, the external data request component 122 automates one or more SAR requests on behalf of the user after obtaining the user's consent and/or instructions to make such requests from one or multiple data controllers.
As an alternative and/or addition, the external data request component 122 may obtain user data via one or more APIs made available by a data source, such as a web server system 140-141. Such APIs may be used to obtain historic user data and/or newly generated user data. The external data request component 122 may obtain user data from data sources other than web server systems 140-141, such as but not limited to one or more mail server systems. The external data request component 122 may communicate with the mail server system using one or more APIs made available by the mail server system 290, using a protocol for relaying email messages, such as but not limited to Internet Message Access Protocol (IMAP), Messaging Application Programming Interface (MAPI), and Exchange ActiveSync.
In some embodiments, the data gathering system 136 includes a local data collection component 124. The local data collection component 124 obtains user data corresponding to the user that passes through a computer system that the user is using. For example, in
In general, when a user accesses web content 142-143 using a user computing device, the corresponding web server systems 140-141 provide the web content 142-143 to the user computing device. In some embodiments, the local data collection component 124 processes the web content 142-143 at the user computing device to identify and/or collect user data. As an alternative and/or addition, the local data collection component 124 may use an API to collect user data. For example, when the data gathering system 136 is implemented as part of a browser extension, the local data collection component 124 can use the browser API to collect user data authorized by the user. For example, in
In some embodiments, the data gathering system 136 maintains the collected user data in a user database 108. As used herein, the term “database” refers to one or more data stores for at least one set of data. The data store may include one or more tangible and/or virtual data storage locations, which may or may not be physically co-located. A simple example of a database is a text file used to store information about a set of data. Another example of a database is one or more data stores that are maintained by a server. Clients may access the database by submitting requests to the server that cause the database server to perform operations on the database. In some embodiments, the server is a server in a database management system (DBMS).
As used herein, the term “maintain” refers to performing operations on one or more sets of data stored in a database, such as, but not limited to accessing the database, adding data to the database, removing data from the database, modifying data in the database, searching the database, retrieving data from the database, logging events related to set/s of data stored in the database, communicating with clients to provide access to the database, and/or other operations related to the database that is maintained.
The user database 108 may be stored on one or multiple computing devices, such as but not limited to a user computing device (e.g., as shown in
In some embodiments, the user database 108 comprises a universal schema for multiple data sources. As an alternative and/or addition, the user database 108 may include one or more natural language descriptions of data from one or more data sources.
In some examples, the assistant system 100 includes an assistant agent system 138. As used herein, the term “agent” refers to any computational entity that makes decisions, generates responses, and/or takes actions based on its environment and inputs. The term may encompass various implementations, such as but not limited to chatbots, virtual assistants, recommendation engines, control systems, autonomous systems, industrial automation tools, or other implementations. In some examples, the assistant agent system 138 has natural language processing (NLP) functionality, such as receiving inputs and generating outputs in the form of natural language.
The assistant agent system 138 of the assistant system 100 is configured to perform a wide array of tasks on behalf of a user, enhance user experience, and/or streamline operations for the user. The assistant agent system 138 may use user data, such as user data stored in the user database 108, to provide the user with a customized experience. For example, the assistant agent system 138 may answer user questions, provide real-time information, assist in scheduling and calendar management, offer recommendations based on user preferences, facilitate e-commerce transactions, automate repetitive tasks, monitor and analyze data for insights, control smart home devices, aid in language translation, summarize content, generate personalized content, and other functionality. The assistant agent system 138 integrates the AI functionality of the AI system 110 with the user data, such as the user data stored in the user database 108.
In some embodiments, the assistant agent system 138 includes a user interaction component 130. The user interaction component 130 handles interactions with a user. In some embodiments, the user interaction component 130 engages the user in natural language conversations. The user interaction component 130 may include one or more interfaces for interacting with a user, such as but not limited to a text-based chat interface, a voice interface, a graphical user interface (GUI), an augmented reality (AR) interface, a virtual reality (VR) interface, another interface, and/or any combination thereof. As an alternative and/or addition, the user interaction component 130 may be configured to interact with one or more applications, such as by providing a service and/or an application programming interface (API). In some embodiments, the user interaction component 130 comprises a chatbot. As used herein, a chatbot refers to any software and/or hardware components designed to engage in natural language-based conversations with users, such as through text, voice, or other conversational interactions. The chatbot may use one or more of pre-programmed responses, machine learning techniques, context analysis, and/or other techniques to interpret user input and generate output in response to the user input. A chatbot may be implemented in messaging platforms, websites, mobile applications, and/or other digital interfaces.
In some embodiments, the assistant agent system 138 engages in one or more chat sessions with the user. As used herein, the term “chat session” refers to a series of interactions between a user and the assistant agent system 138 over any continuous or intermittent period of time during which communication, in the form of messages, data, or information, is exchanged between the user and the assistant agent system 138. An interaction stream refers to the series of interactions received from or submitted to the user during a chat session. As an alternative and/or addition, the interaction stream may include one or more other components, such as but not limited to one or more series of interactions received from or submitted to one or more AI agents, one or more queries, one or more result sets generated by executing one or more queries, one or more summaries of any such information. The boundary between individual chat sessions may be fluid. In some embodiments, one or more boundaries between individual chat sessions are determined by the assistant agent system 138 and/or an AI agent. For example, the boundary between individual chat sessions may be determined based on topic, which may be determined by an AI agent. In some embodiments, chat sessions may be defined such that one chat session may overlap with another chat session.
In some embodiments, the assistant agent system 138 includes an AI agent interaction component 132. The AI agent interaction component 132 handles interactions with the AI system 110. In some embodiments, the AI agent interaction component 132 acts as an intermediary between a user and the AI system, as described in greater detail with respect to
The AI agent interaction component 132 may interact with one or more AI agents 112-116 of the AI system 110. The AI agent interaction component 132 may include one or more interfaces for interacting with an AI agent. In some embodiments, the AI agent interaction component 132 interacts with one or more AI agents 112-116 that have a natural language-based interface, such as but not limited to a text-based chat interface. As an alternative and/or addition, the AI agent interaction component 132 interacts with one or more AI agents 112-116 that are provided as a service and/or an application programming interface (API
As noted with the term “component” generally, the assistant agent system 138 and/or its components (such as but not limited to user interaction component 130 and AI agent interaction component 132) may be implemented as one or more dependent or independent processes, and may be implemented on one or multiple computers; the components may be implemented fully and/or partially in one or multiple programs and/or processes; and two or more components shown may be implemented fully and/or partially in the same program and/or process.
The AI system 110 of the assistant system 100 provides AI functionality. The AI system 110 may include one or more AI agents, such as but not limited to one or more large language model (LLM) agent/s 112, classifier agent/s 113, summarizer agent/s 114, short-term memory (STM) agent/s 115, and/or examiner agent/s 116. As an alternative and/or addition, the AI system 110 may include any other AI agent designed for any other task, such as sentiment analysis, image recognition, anomaly detection, and/or other functionalities.
As used herein, the term “AI agent” refers to any computational entity with AI functionality that makes decisions, generates responses, and/or takes actions based on its environment and/or inputs. The term may encompass various implementations, such as but not limited to chatbots, virtual assistants, recommendation engines, control systems, autonomous systems, industrial automation tools, or other implementations. An AI agent may utilize machine learning techniques, such as by using one or more underlying machine learning models. As an alternative and/or addition, they can also involve rule-based systems, expert systems, and/or other automated mechanisms that may not rely on machine learning techniques. In some examples, an AI agent has natural language processing (NLP) functionality, such as receiving inputs and generating outputs in the form of natural language. An AI agent can be accessed through various means, such as but not limited to using an application programming interface (API), integrating it directly with other applications, accessing it as a service, accessing it through a user interface, and/or any combination thereof. The AI system 110 may include one or multiple AI agents at one or multiple computers at one or multiple locations, including one or multiple instances executing in cloud infrastructure.
As an alternative and/or addition, the AI system 110 may include one or more AI models and/or an interface for accessing such AI model/s. One or more AI agents may be based on an AI model. For example, an LLM agent 112 is based on a large language model. While one or more embodiments are described with respect to particular AI model/s, unless otherwise specified, such embodiments may be implemented using one or more AI models in place of one or more AI agents without departing from the spirit and scope of this disclosure.
In some embodiments, the AI system 110 includes a large language model (LLM) agent 112. An LLM agent 112 utilizes NLP techniques and deep learning methodologies to comprehend diverse textual inputs. The LLM agent 112 comprises an LLM that is typically implemented as a massive neural network with a large number of parameters that are fine-tuned through training on extensive datasets, allowing the model to capture intricate linguistic nuances and contextual dependencies in human language, thereby enabling the LLM agent 112 to process and generate language. For example, some LLMs have hundreds of billions of parameters, and may be implemented with more. These parameters encode linguistic patterns and contextual relationships, allowing the LLM agent 112 to generate coherent responses to input that resembles human responses. Furthermore, the extensive training effectively internalizes a broad spectrum of factual knowledge, language conventions, and logical reasoning in the LLM. This knowledge, combined with their capacity to consider contextual cues in the input, enables LLMs to often provide accurate and contextually appropriate answers to questions. However, LLMs are limited in their ability to take user data into account given that the underlying training process and/or training dataset did not include the user data. LLMs typically do not learn from new data in real time as they interact with users. LLMs also typically do not have a built-in mechanism to retain information or context from such interactions for future usage. Furthermore, the processing capabilities of LLMs are typically computationally expensive, making it infeasible to process the entirety of the user database 108 to provide each personalized answer.
The AI system 110 may include one or multiple agents and/or models. Furthermore, the AI system 110 may be deployed on one or multiple computer systems, which may include one or more local computers and/or remote computers with respect to the assistant agent system 138 and/or the data gathering system 136. Other example AI agents 113-116 are described in greater detail hereafter with respect to
The assistant system 100 and/or its components described herein (e.g. data gathering system 136, assistant agent system 138, AI system 110, external data request component 122, local data collection component 124, data processing component 126, user interaction component 130, and AI agent interaction component 132, LLM agent 112, classifier agent 113, summarizer agent 114, STM agent 115, examiner agent 116) are presented as individual components for ease of explanation; any action performed by or to one or more components of the assistant system 100 may be considered performed by or to the assistant system 100. The assistant system 100 and/or its components may be implemented as one or more dependent or independent processes, and may be implemented on one or multiple computers; for example, a component may be implemented as a distributed system; alternatively and/or in addition, multiple instances of one or more components may be implemented. In some examples, one or more components of the assistant system 100 may be implemented as a cloud service and/or using one or more cloud service providers. Furthermore, a component may be implemented fully and/or partially in one or multiple programs and/or processes, and two or more components shown may be implemented fully and/or partially in the same program and/or process.
The components of the assistant system 100 may execute on one or multiple computer systems, such as one or more client computing devices, assistant server systems, AI agent server systems, and/or other computing systems. Embodiments that illustrate non-limiting examples are described in greater detail hereinafter.
The assistant system 100 may include one or more features described in U.S. Patent Application No. [APPLICATION NUMBER], titled “[TITLE OF REFERENCED APPLICATION],” filed on [FILING DATE], which is hereby incorporates by reference in its entirety U.S. for all purposes.
The assistant agent system 138 integrates the user data with AI functionality in a variety of ways. A set of nonlimiting examples is provided in the section for explanatory purposes; however, the assistant system 100 may be adapted to integrate the user data with any AI functionality without departing from the spirit or the scope of this disclosure.
Example 1 is a user prompt with an answer that is contained in the user database 108. Although the answer is intrinsic to the user database 108, conventional computing techniques may not readily identify the answer due to the lack of extrinsic knowledge (e.g., which Japanese restaurants are sushi restaurants, which events are with coworkers, when did the user live in New York, etc.). Furthermore, although conventional computing techniques may be implemented to handle such dependencies for a specific user prompt, the universe of potential user prompts in a chat environment with the user is too large to implement such techniques for all potential user prompts.
When the assistant agent system 138 provides the user prompt description to the AI system 110, AI system 110 may return a query that, when executed against the user database 108, identifies a result set comprising relevant data that the AI system 110 can use to identify the sushi restaurant. Although the answer may be intrinsic to the user database 108, the AI system 110 may use extrinsic data to answer the question. As an alternative and/or addition, the AI system may be trained using extrinsic data.
Examples 2-3 are user prompts with answers that are based on information contained in the user database 108, where the answer is extrinsic to the user database 108. The user database 108 may include historical events, locations, actions, behaviors, purchases, sentiment, or other user data that would inform the AI system 110 in making a recommendation.
Example 4 is an interaction initiated by the assistant agent system 138 based on a potential implicit user prompt detected by an assistant system 100. In some embodiments, the assistant system 100 evaluates user data to determine or otherwise detect an implicit user prompt that may be of interest to the user. For example, the assistant system 100 may generate Example 4 based on recent health data. The user interaction component 130 may present the example correspondence to the user in an interface, such but not limited to a chat interface or a GUI.
Example 5 is an interaction initiated by the assistant agent system 138 based on an implicit user prompt detected by and acted on by the assistant system 100. For example, the assistant system 100 may determine and act on an implicit user prompt (e.g., “IS MY RETURN PERIOD CLOSING ON A PURCHASE”) without the user having initiated such action. The AI agent interaction component 132 may interact with the AI system 110 based on this implicit user prompt, and the user interaction component 130 may provide the interaction shown in Example 5 to the user.
Example 6 may be performed upon activation of an interactive UI object, such as but not limited to a button in a GUI. When the user triggers the interactive UI object, the user interaction component 130 interprets a user prompt to provide a customized version of any web content, digital resource, or other content associated with the interactive UI object. For example, the interactive UI object may be associated with search results comprising one or more ordered items. The search results may be from a generic search engine, an e-commerce website, media search engine, database search, social media platform, streaming platform, news aggregator, listing site, or any other source of search results. For example, when the user prompt is interpreted as requesting a customized version of search results from a generic search engine, the assistant agent system 138 may use the AI system 110 to provide personalized search results that include items that are reordered based on the personalization techniques described herein. In some embodiments, the assistant agent system 138 uses the AI system 110 to generate web code for the personalized search results. In some embodiments, the personalized search results comprise web code for display in a browser of a user computing device.
In Example 7, a user asks: “When did I last meet Carolyn? How often have I met her over the past few years?” The assistant agent receives this user prompt from the user. The assistant agent may rewrite the user prompt, enrich the user prompt with relevant context, categorize the user prompt, and/or otherwise process the user prompt. For example, the assistant agent may use one or more AI agents to process the user prompt. The AI agent generates and submits a first system prompt to the LLM agent that includes a user prompt description paraphrasing the user prompt, a database description describing available user data associated with the user, and instructions to provide a query for relevant data. The first system prompt may include additional context about the user, example responses to example user prompts, assistant agent properties, and/or other relevant contextual information about the assistant agent, the user, the user prompt, and the like. In Example 7, before generating the first system prompt, the assistant agent performs the optional step of categorizing the user prompt as a calendar-based question, and generates the first system prompt based on a category template for calendar-based user prompts. As an alternative and/or addition, the first system prompt may request a query on a calendar event database.
First system prompt:
The LLM agent responds with the following query:
The assistant agent executes the query against the user database to generate a result set, generates a second system prompt including the result set, and submits the second system prompt to the LLM agent. The second system prompt may include additional context about the user, example responses to example user prompts, assistant agent properties, and/or other relevant contextual information about the assistant agent, the user, the user prompt, and the like. The second system prompt may be generated using a category template for a second system prompt comprising a result set for calendar events.
Second system prompt:
The LLM agent responds to the paraphrased user prompt as follows:
The assistant agent generates a response for the user based on the LLM agent user prompt. The assistant agent may process the LLM agent response in any manner before generating or sending the response for the user.
In some embodiments, the assistant agent system 252 includes a user interaction component (e.g., user interaction component 130) and an AI agent interaction component (e.g., AI agent interaction component 132). The user interaction component may handle interactions with the user 250, while the AI agent interaction component may handle interactions with the AI system 254.
At block 202, a user prompt is communicated from the user 250 to the assistant agent system 252. As used herein, the term “prompt” refers to any input to which the user expects a response. Prompts can include instructions, questions, or any other type of input. As used herein, the term “user prompt” refers to any such input from a user 250 to the assistant agent system 252. The user prompt may include one or multiple messages or communications from the user 250. The user prompt is differentiated from an “assistant agent prompt” from the assistant agent system 252 to an AI agent of the AI system 254. For example, block 202 involves a user prompt from the user 250, while blocks 204 and 210 involve assistant agent prompts from the assistant agent system 252 to one or more AI agents of the AI system 254.
In some embodiments, the user 250 directly inputs the user prompt using natural language through a user interface of the assistant agent system 252, such as a chatbot interface executing on a user computing device. As an addition and/or alternative, the assistant agent system 252 may detect an implicit user prompt. For example, the assistant agent system 252 may infer that it is desirable to provide the user a response to an implicit user prompt based on the user data and/or other contextual data. Some non-limiting examples of implicit user prompts include an inquiry for information about recent purchases, advice regarding recent health trends, alerts about local weather conditions, and notifications about breaking news. In some embodiments, when the assistant agent system 252 detects or otherwise generates an implicit user prompt, the assistant agent system 252 presents the implicit user prompt to the user in a user interface such that the can select the implicit user prompt to trigger a response to the selected user prompt by interacting with the graphical user interface.
At block 204, the assistant agent system 252 requests a query from the AI system 254 to search a user database comprising the user data. In some embodiments, the assistant agent system 252 submits, to the AI system 254, a user prompt description, a database description describing the user database comprising the user data, and instructions to provide a query for relevant data in the user database that the AI system 254 can use to answer to the user prompt. The user prompt description describes the user prompt. For example, the user prompt description may include the user prompt as input by the user, a modified version of a user prompt input by the user, metadata describing the user prompt, an implicit user prompt inferred by the assistant agent system 252, or any other data related to the user prompt. In some embodiments, the information is submitted to an LLM agent of the AI system 254.
At block 206, the AI system 254 provides the query to the assistant agent system 252. The query describes the information in the user database that is relevant to the user prompt, as determined by the AI system 254. At block 208, the assistant agent system 252 executes the query against the user database to generate a result set.
At block 210, the assistant agent system 252 submits the result set to the AI system 254. As an alternative and/or addition, the assistant agent system 252 may submit additional relevant data to the AI system 254. In some cases, such additional relevant data may also be identified by the AI system 254. In some embodiments, the assistant agent system 252 submits the result set to the same AI agent of the AI system 254, such as but not limited to the LLM agent that received the request for the query. As an alternative and/or addition, the assistant agent system 252 may submit the result set to a different AI agent of the AI system 254. For example, the assistant agent system 252 may submit the user prompt description and the results of the query to a second AI agent. At block 212, the AI system 254 provides an answer to the assistant agent system 252. The answer is generated by the AI system 254 based on the result set and/or other relevant data provided to the AI system 254 by the assistant agent system 252.
At block 214, the assistant agent system 252 provides an answer to the user 250. For example, the assistant agent system 252 may provide the answer through a user interface of the assistant agent system 252, such as a chatbot interface executing on a user computing device of the user 250. The answer provided to the user 250 is based on the answer provided by the AI system 254 at block 212. For example, the assistant agent system 252 may forward at least a portion of the answer received from the AI system 254. As an alternative and/or addition, the assistant agent system 252 may generate an answer based at least in part on the answer received from the AI system 254. In some examples, the answer is in the form of a natural language response to the question. As an alternative or addition, an answer may take a different form, such as but not limited to one or more files, customized search results, customized webpages, or any other form that provides personalized information to the user.
In some embodiments, an assistant agent system (e.g., assistant agent system 138, 252) is implemented as a chat loop. During the chat loop, the assistant agent system waits for the user to provide input. When the user provides input, the assistant agent system 138 processes and generates a response to the user input. For example, each input provided by the user may be treated as a user prompt. As an alternative and/or addition, one or more inputs may be evaluated to determine whether the input/s include a user prompt for which a response should be generated. In some embodiments, the assistant agent system may use one or more AI agents/models to evaluate the input/s.
The assistant agent system may interact with an AI system (e.g., AI system 110, 254) in order to process the user's input and/or generate a response. After providing the response to the user, the assistant agent system waits for the user to provide additional input. The assistant agent system may respond to the user in near real-time. That is, delays are minimized such that conversational flow is maintained so that the user experiences the interactions as responsive.
At block 302, the assistant agent system 138 and the user initiate a chat session. For example, the user may initiate a chat session by opening an assistant application (an example of which is shown in
At block 304, the assistant agent system 138 receives a communication from the user. For example, the communication may include the input comprising a user prompt, such as a question or request that was provided by the user as natural language input. As an alternative and/or addition, the communication may include an indication that the user has selected a user interface element associated with an implicit user prompt that was detected or generated by the assistant agent system 138.
At block 306, the assistant agent system 138 classifies the user prompt. In some embodiments, the assistant agent system 138 classifies the user prompt by submitting the user prompt to a classifier agent 113 of the AI system 110. The classifier agent 113 may be configured to determine whether the user prompt falls into one or more of a plurality of categories. When the prompt falls into a particular category, the assistant agent system 138 may handle the user prompt according to the particular category. In some embodiments, the assistant agent system 138 adds additional information, also referred to as “context”, to an assistant agent prompt based submitted to the AI system 110 at block 310. The additional information may be based on a category template corresponding to the particular category of the user prompt.
At decision block 308, the assistant agent system 138 determines whether the chat history includes sufficient data to respond to the user prompt. For example, one or more prior queries into the user database 108 may include data that is sufficient to respond to the user prompt. In some embodiments, the assistant agent system 138 submits data, such as but not limited to the chat history, one or more result sets, and/or the user prompt description, to an examiner agent 116. The examiner agent 116 may be configured to determine whether the submitted data is sufficient to answer the user prompt. As shown, decision block 308 is first performed before obtaining any query (e.g., block 310) from the AI system 110. This order accounts for the presence of sufficient data, such as from a prior query or prior interaction. As an alternative and/or addition, decision block 308 may be first performed after a first query is obtained and executed (e.g., block 314) and/or after the results of the first query are added to the chat history (e.g., block 316).
As an alternative and/or addition, the assistant agent system 138 may submit data, such as but not limited to the chat history and/or the user prompt description, to a short-term memory (STM) agent 115. The data submitted may include a query, query results of the query, and/or a description of the query and/or the query results. The STM agent 115 may be configured to determine whether any data, such as the prior query results, are relevant to the user prompt. In some embodiments, the assistant agent system 138 may determine that the chat history includes relevant data to answering the user prompt, but that requesting a query from the AI system 110 may be beneficial. In this case, the assistant agent system 138 may include the relevant data in correspondence with the AI system 110. For example, the relevant data may be appended to a request to the AI system 110.
At decision block 308, when the chat history is sufficient to respond to the user prompt, processing continues to block 320, which is described in greater detail below. The chat history may include one or more prior queries and or query results from the chat session or prior chat session. When the chat history is not sufficient, processing continues to block 310.
At block 310, the assistant agent system 138 obtains a plan for responding to the user prompt. For example, the assistant agent system 138 may obtain a plan for responding to the user prompt that includes a query for obtaining information useful for responding to the user prompt. As an alternative and/or addition, the assistant agent system 138 may determine that some user prompts are better addressed using an alternative plan that does not involve a query. In some embodiments, the assistant agent system 138 obtains the plan and/or query from an AI agent, such as an LLM agent 112. The assistant agent system 138 may obtain the plan and/or query by submitting information to the LLM agent 112, such as but not limited to a user prompt description, a database description describing the user database, instructions to provide a query for relevant data in the user database, information describing one or more current data sources or potential data sources, or any other information relevant to generating the plan or and/or query.
At decision block 312, the assistant agent system 138 determines whether the plan for responding to the user prompt includes proceeding with attempting to generate an answer to the user prompt. For example, the plan for responding to the user prompt may include determining not to generate a direct answer to the user prompt. The plan for responding to the user prompt may include one or more reasons for not generating a direct answer, such as but not limited to appropriateness, clarity, feasibility, insufficiency of data, and/or other reasons. When the assistant agent system 138 determines not to proceed with generating an answer, processing continues to block 324.
At block 324, the assistant agent system 138 provides an alternative response to the user. For example, the alternative response may be included in a plan for responding to the user prompt, such as a plan obtained from the LLM agent 112 at block 310. As an alternative and/or addition, the assistant agent system 138 may generate the alternative response. The alternative response may include an explanation for not directly answering the user prompt. For example, the alternative response may indicate that the user prompt is ambiguous, unclear, inappropriate, offensive, harmful, and/or unsafe. As an alternative and/or addition, the alternative response may request further clarification from the user. In some embodiments, the alternative response suggests that the user prompt may be better answered with more information. For example, the alternative response may suggest connecting the assistant agent system 138 to one or more additional data sources to obtain additional user data for the user that is relevant to the user prompt. After the alternative response is provided to the user, process 300 may wait until another message is received from the user. In this case, processing returns to block 304.
Returning to decision block 312, when the plan for responding to the user prompt includes proceeding with generating an answer, processing continues to block 314. At block 314, the assistant agent system 138 executes the query against the user database to generate a result set.
At block 316, the assistant agent system 138 adds the results of the query to the chat history. For example, the result set may be added to the chat history. As an alternative and/or addition, the executed query itself may be added to the chat history. As an alternative and/or addition, a summary of the query and/or result set may be added to the chat history. In some embodiments, the assistant agent system 138 obtains the summary by submitting the query and/or result set to a summarizer agent 114 of the AI system 110. The summarizer agent 114 may be configured to generate a summary of the information provided to the summarizer agent 114. The assistant agent system 138 may instruct the summarizer agent 114 to generate a summary that matches certain parameters, such as to keep the chat history size within a usable size for submission to one or more AI agents.
After block 316, processing returns to decision block 308, where the assistant agent system 138 determines whether the chat history includes sufficient data to respond to the user prompt. For example, when the result set of a first query is not sufficient to answer the user prompt, the assistant agent system 138 may obtain and execute a second query for additional relevant data. Thus, in some embodiments, blocks 308-316 may function as a loop for obtaining sufficient information to generate an answer to the user prompt. Processing exits the loop when a determination is made to not proceed with a direct answer to the user prompt at decision block 312 or when the chat history is sufficient at decision block 308.
When it is determined at decision block 308 that the chat history is sufficient, processing continues to block 320. At block 320, the assistant agent system 138 obtains an answer to the user prompt. In some embodiments, the assistant agent system 138 obtains the answer to the user prompt by submitting the user prompt description, one or more relevant portions of the chat history, and/or one or more result sets for one or more queries to an LLM agent 112 of the AI system 110. The assistant agent system 138 may have previously submitted information to the LLM agent 112, such as the user prompt description (e.g., block 310). The LLM agent 112 may provide the answer to the user prompt to the assistant agent system 138 in response to the submission.
At block 322, the assistant agent system 138 provides the user a response comprising an answer to the user prompt. The response may include or be based on an answer received from the AI system 110 and/or the LLM agent 112. After the response is provided to the user, process 300 may wait until another message is received from the user. In this case, processing returns to block 304.
The assistant application 420 includes an assistant agent system 138. The assistant application 420 may include a user interaction component 430 that handles interactions with the user of the user computing device 402 and/or an agent interaction component 432 that handles interactions with one or more AI agents/models 412-415. The user interactions handled by the user interaction component 430 may be based on one or more AI agent interactions handled by the agent interaction component 432 of the assistant application 420. The AI agents/models 412-415 may include one or more AI agents and/or models that are owned, operated, and/or controlled by the same entity as the assistant application 420, the user computing device 402, one or more third parties, or any combination thereof.
In some embodiments, the assistant application 420 includes one or more of the AI agents/models 415. As an alternative and/or addition, one or more AI agents/models 414 may be provided by an AI system 410C that is distinct from the assistant application 420 that executes on the same user computing device 402.
As an alternative and/or addition, one or more AI agents/models may be provided by a different computing device. For example, one or more AI agents/models 412-413 may be provided by one or more AI agent server systems 410A-B. In some embodiments, the agent interaction component 432 of the assistant application 420 interacts over a network 450 with an LLM agent/model 412 provided by an AI server system 410A. As an alternative and/or addition, the agent interaction component 432 of the assistant application 420 interacts over a network 450 with one or more other AI agents/models 413 provided by one or more other AI server system/s 410B.
The assistant application 420 may include one or more other components of an assistant system as described herein (e.g., assistant system 100), such as but not limited to a data gathering system 436 and/or a user database 408.
In some embodiments, an assistant application is implemented as a browser extension.
The assistant application browser extension 520 may leverage one or more browser extension features to implement assistant application functionality. These functionalities include, but are not limited to, utilizing content scripts to dynamically modify webpage content, employing background scripts, monitoring and/or manipulating network requests, storing local data, providing interactive UI features in the browser interface, managing user authentication and credentials, and/or other features. As an alternative and/or addition, the assistant application browser extension 520 may utilize such functionality to detect and/or generate additional user data. For example, the user may authorize the assistant application browser extension 520 to collect data describing the user's usage of the browser application 521, such as but not limited to the browsing history of the user and/or the user's interactions with one or more web pages.
Embodiments described herein illustrate that an assistant system (e.g., assistant system 100) may operate across multiple computer systems, which may include a user device. In some embodiments, an assistant system may be implemented to operate wholly on a single computer system, such as a user computing device. One benefit of a fully local implementation with respect to a user computing device is that the user is offered more control over their user data, including control and ownership over a physical device where collected user data physically resides.
In some embodiments, an assistant system (e.g., assistant system 100) is implemented to allow a user to access the assistant system functionality across multiple user computing devices.
Each user computing device 602-603 executes an assistant application 620-621. While the assistant application 620-621 may provide any of the functionality of the assistant system 100, the assistant server system 604 may provide at least a subset of the functionality of the assistant system 100. One or more versions of the assistant application 620-621 may be implemented, which may provide varying scopes of assistant system 100 functionality. The assistant applications 620-621 and the assistant server system 604 may communicate over a network 650.
In some embodiments, the assistant server system 604 may include an assistant agent system 638. The assistant agent system 638 may include a user interaction component 630 that provides user interaction functionality for at least a subset of assistant applications 620-621. As an alternative and/or addition, the assistant agent system 638 may include an AI agent interaction component 632 that provides AI agent interaction functionality for at least a subset of assistant applications 620-621. For example, the AI agent interaction component 632 may interact with one or more AI agents 612A implemented at one or more AI agent server system/s 610A outside of the assistant server system 604. As an alternative and/or addition, the AI agent interaction component 632 may interact with one or more AI agents 612B implemented by one or more AI agent systems 610B within the assistant server system 604.
In some embodiments, the assistant server system 604 includes a data gathering system 636 that provides data gathering functionality for at least a subset of assistant applications 620-621. For example, the data gathering system 636 may collect user data on behalf of one or multiple users from one or more web server systems 640 that provide web content 642. As an alternative and/or addition, the data gathering system 636 may collect user data on behalf of one or multiple users from one or more other data sources.
In some embodiments, the data gathering system 636 maintains a user database 609 comprising user data for one or multiple users. The user database 609 may be is encrypted. As an alternative and/or addition, the assistant server system 604 may maintain at least a subset of the user data of one or more users while performing one or more assistant system operations, such as data gathering and/or responding to the user prompt.
At block 702, the assistant agent system 138 determines to respond to a user prompt of a user. For example, the assistant agent system 138 may determine that a user has entered a user prompt, such as through a chat interface or other user interface. As an addition or alternative, the assistant agent system 138 may detect an implicit user prompt.
At block 704, the assistant agent system 138 submits, to an artificial intelligence (AI) system comprising at least one AI agent, a user prompt description, a database description describing a user database comprising user data associated with the user, and instructions to provide a query for relevant data in the user database that the AI system can use to answer to the user prompt. The user prompt description may include the user prompt as input by the user, a modified version of a user prompt input by the user, metadata describing a user prompt, an implicit user prompt, or any other data related to the user prompt. In some embodiments, the assistant agent system 138 submits the information to an LLM agent 112 of the AI system 110.
At block 706, the assistant agent system 138 receives the query from the AI system 110. For example, the assistant system 100 may receive the query from an AI agent, such as but not limited to an LLM agent 112. In some embodiments, the assistant agent system 138 receives the query as part of a plan from the AI system 110 for responding to the user prompt.
At block 708, the assistant agent system 138 executes the query against the user database to generate a result set. In some embodiments, executing the query includes causing execution of the query by another process and/or application. In some embodiments, the user database is a unified database comprising custom data from distinct data sources.
At block 710, the assistant agent system 138 submits the result set to the AI system 110. As an alternative and/or addition, the assistant agent system 138 may submit additional relevant data to the AI system 110. In some cases, such additional relevant data may also be identified by the AI system 110. In some embodiments, the assistant agent system 138 submits the result set to the same AI agent of the AI system 110, such as but not limited to the LLM agent 112 that received the request for the query. As an alternative and/or addition, the assistant agent system 138 may submit the result set to a different AI agent of the AI system 110.
At block 712, the assistant agent system 138 receives an answer from the AI system based on the user prompt description and the result set. The assistant agent system 138 may generate a response for the user based on the answer from the AI system. For example, the assistant agent system 138 may provide the answer through a user interface of the assistant agent system 138, such as a chatbot interface executing on a user computing device of the user.
At block 714, the assistant agent system 138 generates a response for the user based on the answer from the AI system. For example, the assistant agent system 138 may provide the answer through a user interface of the assistant agent system 138, such as a chatbot interface executing on a user computing device of the user.
In some embodiments, after block 714, process 700 returns and/or terminates. For example, process 700 may pass control to a calling process, generate any appropriate record or notification, return after a method or function invocation, process a next user prompt, process another operation initiated by a client device, or terminate.
According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform one or more techniques described herein, including combinations thereof. Alternatively and/or in addition, the one or more special-purpose computing devices may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques. Alternatively and/or in addition, the one or more special-purpose computing devices may include one or more general-purpose hardware processors programmed to perform the techniques described herein pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices, and/or any other device that incorporates hard-wired or program logic to implement the techniques.
The computer system 800 also includes one or more units of main memory 806 coupled to the bus 802, such as random-access memory (RAM) or other dynamic storage, for storing information and instructions to be executed by the processor/s 804. Main memory 806 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor/s 804. Such instructions, when stored in non-transitory storage media accessible to the processor/s 804, turn the computer system 800 into a special-purpose machine that is customized to perform the operations specified in the instructions. In some embodiments, main memory 806 may include dynamic random-access memory (DRAM) (including but not limited to double data rate synchronous dynamic random-access memory (DDR SDRAM), thyristor random-access memory (T-RAM), zero-capacitor (Z-RAM™)) and/or non-volatile random-access memory (NVRAM).
The computer system 800 may further include one or more units of read-only memory (ROM) 808 or other static storage coupled to the bus 802 for storing information and instructions for the processor/s 804 that are either always static or static in normal operation but reprogrammable. For example, the ROM 808 may store firmware for the computer system 800. The ROM 808 may include mask ROM (MROM) or other hard-wired ROM storing purely static information, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically-erasable programmable read-only memory (EEPROM), another hardware memory chip or cartridge, or any other read-only memory unit.
One or more storage devices 810, such as a magnetic disk or optical disk, is provided and coupled to the bus 802 for storing information and/or instructions. The storage device/s 810 may include non-volatile storage media such as, for example, read-only memory, optical disks (such as but not limited to compact discs (CDs), digital video discs (DVDs), Blu-ray discs (BDs)), magnetic disks, other magnetic media such as floppy disks and magnetic tape, solid-state drives, flash memory, optical disks, one or more forms of non-volatile random-access memory (NVRAM), and/or other non-volatile storage media.
The computer system 800 may be coupled via the bus 802 to one or more input/output (I/O) devices 812. For example, the I/O device/s 812 may include one or more displays for displaying information to a computer user, such as a cathode ray tube (CRT) display, a Liquid Crystal Display (LCD) display, a Light-Emitting Diode (LED) display, a projector, and/or any other type of display.
The I/O device/s 812 may also include one or more input devices, such as an alphanumeric keyboard and/or any other keypad device. The one or more input devices may also include one or more cursor control devices, such as a mouse, a trackball, a touch input device, or cursor direction keys for communicating direction information and command selections to the processor 804 and for controlling cursor movement on another I/O device (e.g. a display). A cursor control device typically has degrees of freedom in two or more axes, (e.g. a first axis x, a second axis y, and optionally one or more additional axes z), that allow the device to specify positions in a plane. In some embodiments, the one or more I/O device/s 812 may include a device with combined I/O functionality, such as a touch-enabled display.
Other I/O device/s 812 may include a fingerprint reader, a scanner, an infrared (IR) device, an imaging device such as a camera or video recording device, a microphone, a speaker, an ambient light sensor, a pressure sensor, an accelerometer, a gyroscope, a magnetometer, another motion sensor, or any other device that can communicate signals, commands, and/or other information with the processor/s 804 over the bus 802.
The computer system 800 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware, and/or program logic that causes the computer system 800 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by the computer system 800 in response to the processor/s 804 executing one or more sequences of one or more instructions contained in main memory 806. Such instructions may be read into main memory 806 from another storage medium, such as the one or more storage device/s 810. Execution of the sequences of instructions contained in main memory 806 causes the processor/s 804 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
The computer system 800 also includes one or more communication interfaces 818 coupled to the bus 802. The communication interface/s 818 provide two-way data communication over one or more physical or wireless network links 820 that are connected to a local network 822 and/or a wide area network (WAN), such as the Internet. For example, the communication interface/s 818 may include an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. Alternatively and/or in addition, the communication interface/s 818 may include one or more of: a local area network (LAN) device that provides a data communication connection to a compatible local network 822; a wireless local area network (WLAN) device that sends and receives wireless signals (such as electrical signals, electromagnetic signals, optical signals or other wireless signals representing various types of information) to a compatible LAN; a wireless wide area network (WWAN) device that sends and receives such signals over a cellular network; and other networking devices that establish a communication channel between the computer system 800 and one or more LANs 822 and/or WANs.
The network link/s 820 typically provides data communication through one or more networks to other data devices. For example, the network link/s 820 may provide a connection through one or more local area networks 822 (LANs) to one or more host computers 824 or data equipment operated by an Internet Service Provider (ISP) 826. The ISP 826 provides connectivity to one or more wide area networks 828, such as the Internet. The LAN/s 822 and WAN/s 828 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link/s 820 and through the communication interface/s 818 are example forms of transmission media, or transitory media.
The term “storage media” as used herein refers to any non-transitory media that stores data and/or instructions that cause a machine to operate in a specific fashion. Such storage media may include volatile and/or non-volatile media. Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire, and/or fiber optics, including traces and/or other physical electrically conductive components that comprise the bus 802. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infrared data communications.
Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 804 for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer. The remote computer can load the instructions into its main memory 806 and send the instructions over a telecommunications line using a modem. A modem local to the computer system 800 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on the bus 802. The bus 802 carries the data to main memory 806, from which the processor 804 retrieves and executes the instructions. The instructions received by main memory 806 may optionally be stored on the storage device 810 either before or after execution by the processor 804.
The computer system 800 can send messages and receive data, including program code, through the network(s), the network link 820, and the communication interface/s 818. In the Internet example, one or more servers 830 may transmit signals corresponding to data or instructions requested for an application program executed by the computer system 800 through the Internet 828, ISP 826, local network 822, and a communication interface 818. The received signals may include instructions and/or information for execution and/or processing by the processor/s 804. The processor/s 804 may execute and/or process the instructions and/or information upon receiving the signals by accessing main memory 806, or at a later time by storing them and then accessing them from the storage device/s 810.
Although the concepts herein have been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present disclosure. It is therefore to be understood that numerous modifications may be made to the illustrative embodiments and that other arrangements may be devised without departing from the spirit and scope of the present disclosure as defined by the appended claims.
This application claims the benefit of Provisional Application Ser. No. 63/588,204, filed Oct. 5, 2023, the entire contents of which are hereby incorporated by reference as if fully set forth herein, under 35 U.S.C. § 119(e).
| Number | Date | Country | |
|---|---|---|---|
| 63588204 | Oct 2023 | US |
