Patent Application
20250095834
Clinical environments such as healthcare facilities often include different healthcare providers working together and communicating with one another to treat patients. Documenting patient encounters, capturing information conveyed during those encounters and/or pertaining to events occurring before and/or after the encounters, populating patient records such as electronic health records, and healthcare practice management are integral parts of the practice of many healthcare providers and important to ensuring high-quality healthcare. Traditional means for performing tasks associated with providing healthcare often involve several different disconnected devices such as listening devices, portable electronic devices, workstations, and the like and end users who are equipped with the training, knowledge, experience, and skills to properly utilize these devices and participate in the healthcare process. Relying on separate devices and qualified end users to perform clinical tasks is cumbersome, time and resource intensive, costly, and reduces efficiencies, which may lead to lower-quality healthcare.
Techniques are disclosed herein for improving the efficiency of and reducing the computing resources required to perform various healthcare services in a clinical environment. In certain embodiments, techniques are disclosed for equipping a healthcare provider end user with a clinical software application that can be placed on and utilized from one or both of a mobile computing device and a desktop computing device to facilitate performance of the various tasks typically rendered by a healthcare provider as part of providing healthcare services to patients. Some embodiments provide for the intelligent scheduling of tasks, and for generating and sending notifications upon the occurrence of a criteria for initiating the scheduled task. The notifications can include metadata that is usable to initiate performance of a task associated with a notification based simply on selection by an end user of the displayed notification.
In some embodiments, a method includes accessing one or more utterances and context associated with a user session in which the utterances were generated. A task to be scheduled for the user is then identified by one or more machine-learning models based on the one or more utterances and the context. The method also includes generating, based on the task and the context, a task entry. Generating the task entry comprises creating a description of the task and criteria for initiating the task, generating metadata associated with the task and the context, and storing the description, the criteria, and the metadata as the task entry in a table of a database. A notification configuration entry is then generated for the task entry, and the notification configuration entry is associated with the task entry in the table of the database. The notification configuration entry includes notification instructions for controlling how a notification is to be generated and provided upon satisfaction of the criteria for initiating the task. Upon detecting satisfaction of the criteria for initiating the task, the notification is generated based on the notification instructions, the metadata is appended to the notification to generate a notification message, and the notification message is sent to a client device based on the notification instructions.
In some embodiments, an utterance of the one or more utterances includes a natural language request to schedule a task, and the one or more machine-learning models analyze the natural language request using one or more natural language processing techniques and resultantly extract therefrom the description of the task and the criteria for initiating the task.
In some embodiments, an utterance of the one or more utterances causes the one or more machine-learning models to receive text comprising a transcribed spoken natural language conversation, and the one or more machine-learning models analyze the text using one or more natural language processing techniques, and resultantly predict the task to be scheduled based on dialogue within the text and extract from the text the description of the task and the criteria for initiating the task.
In some embodiments, the one or more utterances are received in messages that further include information comprising an identification of the client device and a channel for communicating with the client device, and the information is used to send the notification message to the client device.
In some embodiments, the metadata further comprises an identification of a selected skill of a plurality of skills for processing steps of the task.
In some embodiments, after sending the notification message to the client device, a response message is received that includes the metadata, and the task is performed by causing the selected skill identified in the metadata to process the steps of the task.
In some embodiments, the notification instructions assign a priority level classification to the notification, and at a time of sending the notification message to the client device, the priority level classification of the notification is compared with a priority level classification of an ongoing interaction of the user. When the priority level classification assigned to the notification is higher than the priority level classification of the ongoing interaction, an instruction is sent to the client device to suspend the ongoing interaction and to immediately display the notification. When the priority level classification of the ongoing interaction is higher than the priority level classification assigned to the notification, an instruction is sent to the client device to defer displaying the notification until the ongoing interaction has ended.
Some embodiments include a system that includes one or more processing systems and one or more computer-readable media storing instructions which, when executed by the one or more processing systems, cause the system to perform part or all of the operations and/or methods disclosed herein.
Some embodiments include one or more non-transitory computer-readable media storing instructions which, when executed by one or more processing systems, cause a system to perform part or all of the operations and/or methods disclosed herein.
The techniques described above and below may be implemented in a number of ways and in a number of contexts. Several example implementations and contexts are provided with reference to the following figures, as described below in more detail. However, the following implementations and contexts are but a few of many.
Features, embodiments, and advantages of the present disclosure are better understood when the following Detailed Description is read with reference to the accompanying drawings.
In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be restrictive. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs.
Providing healthcare to patients typically requires a healthcare provider to repeat a number of common tasks for each patient. For example, regardless of the specific reason for an interaction between a healthcare provider and a patient, or the condition of a given patient, the healthcare provider must typically document the patient interaction. For example, the healthcare provider may record the patient interaction in a subjective, objective, assessment, and plan (SOAP) note, or may enter information gained during the patient interaction into a patient record. The healthcare provider may also engage in various other tasks directly or indirectly related to administering healthcare to the patient, such as requesting additional patient information in the form of charts or images, calling in patient prescriptions, and calendaring future tasks, events, and associated reminders.
Performing such healthcare tasks according to known and commonly used methods can be time consuming. In fact, given the typically high volumes of patient encounters, healthcare providers often spend a considerable portion of their workday documenting patient interactions and associated medical information, which reduces the amount of time available to the healthcare provider to administer actual patient care or perform other more critical tasks. For example, healthcare providers may spend considerable time on a daily basis typing or manually entering patient information into electronic health record (EHR) systems. In addition to being time consuming, this process can be, tedious, and prone to errors such as but not limited to typographical errors, which can result in inaccuracies and inconsistencies in patient records, and can potentially compromise patient safety and the quality of care provided. Traditional EHR systems can also have complex interfaces any may be difficult to navigate, which can increase the time required for healthcare providers to complete such repetitive tasks and generally frustrate the process Traditional EHR system devices may also be cumbersome to operate, and a lack of intercommunication between such devices prevents a healthcare provider from switching between devices while in the process of performing a task even if doing so would be more efficient. These issues may negatively affect patients as well as healthcare providers. For example, patient information may often be retrieved for review or discussion during a patient interaction or recorded during a patient interaction to ensure accuracy. When the process for retrieving or recording such information is inefficient, as is often the case when performed using traditional systems and methods, it can disrupt the natural flow of the patient interaction and may result in a less seamless and less fulfilling experience for the patient. The tedium and time requirements associated with repetitively performing these tasks can also contribute to healthcare provider burnout.
The developed approach described herein addresses these challenges and others by providing techniques for assisting healthcare providers with necessary but time consuming and sometimes tedious tasks. Techniques are disclosed herein for improving the efficiency of and reducing the computing resources required to perform various healthcare services in a clinical environment. In certain embodiments, techniques are disclosed for equipping a healthcare provider end user with a clinical software application that can be installed on and utilized from one or both of a mobile computing device and a desktop computing device to facilitate performance of the various tasks typically rendered by a healthcare provider as part of providing healthcare services to patients. For example, embodiments of the application may enable an end user to easily record conversations with patients, dictate in natural language, generate patient notes, populate patient records, request and receive patient information, fill prescriptions, create or have automatically created task and event reminders, and receive associated notifications. The scheduling of such tasks can be intelligently performed, either at the express direction of an end user or automatically upon analysis of an end user conversation. Notifications can be associated with scheduled tasks, and may be sent upon satisfaction of criteria for initiating the task. The notifications may include metadata that can be used to initiate performance of the task upon end user selection of a displayed notification. The notifications can initiate/drive conversations with a digital assistant of a cloud service provider platform.
The healthcare environment 100 includes a cloud service provider platform 108 that includes capabilities for providing various services to subscribers (e.g., end-users) of the cloud service provider platform 108. The end-users (e.g., clinicians such as doctors and nurses) may utilize the various services provided by the cloud service provider platform 108 to perform various functions involving the treatment, care, observation, and so on of patients. For instance, in the healthcare environment 100, the end-users can utilize the functionality provided by the services to view, edit, or manage a patient's electronic health record, perform administrative tasks such as scheduling appointments, manage patient populations, provide customer service to facilitate operation of the healthcare environment 100, and so on.
The services provided by the cloud service provider platform 108 may include, but are not limited to, digital assistant services, authentication services, user management services, frontend services (e.g., entry point (façade) to all services), and other management services. The various services may be implemented on one or more servers of the cloud service provider platform 108 and may be provided to end-users who subscribe to the cloud services provided by the platform 108. In a certain implementation, the services provided by the cloud service provider platform 108 may represent digital assistant services that may be provided to healthcare providers such as doctors, nurses, technicians, clinicians, medical personnel, and the like. For instance, the service 110a may represent an ambient service, which is an AI-powered, voice-enabled service that automatically documents patient encounters accurately and efficiently at the point of care and provides quick action suggestions. The service 110b may represent a dictation service 110b that allows doctors to generate medical records from voice (e.g., using a Large Language Model (LLM) or pre-seeded templates). The service 110c may represent a clinical automation service where healthcare providers can interact with the digital assistant, and the digital assistant can provide the end user with support for various clinical functional tasks.
Various end-users may interact with the cloud service provider platform 108 using one or more client devices (e.g., 102, 104) that may be communicatively coupled to one or more servers implemented by the services (e.g., 110a, 110b, 110c), via one or more communication channels 106. The client devices (102, 104) may be of various types, including but not limited to, a mobile phone, a tablet, a desktop computer, and the like. The users can interact with the various services via a user interface (UI) of an application installed on the client devices (102, 104) to obtain information about a patient such as medical information from an electronic health record for the patient stored in the electronic health record database 210, collect information relevant to the observation, care, treatment, and/or management of a patient, and so on.
In certain embodiments, the applications installed on the client devices 102, 104 can support multimodal communications between an end user and the client devices 102, 104. For example, the end user can communicate with and utilize the functionality provided by services (110a, 110b and 110c) via audio, voice (natural language), text, or rich user interface controls. For instance, an end user may utilize one or more voice interfaces provided by an application installed on one of the client devices 102, 104 to interact with the services 110a, 110b and 110c. In another example, the end user can interact with the application based on touch input (e.g., tapping, swiping, pinching) and voice input captured by the client device to obtain information about a patient. Voice interactions can be initiated via a wake word or by tapping a dedicated button on screen. The application can interface with the various services which can generate conversational-type responses to the voice-based interactions. In some implementations, the responses can be natural language responses and/or graphical responses. As part of a conversation with the digital assistant, an end user may provide a voice input (e.g., a natural language utterance) to one of the services 110a, 100b, 110c. The platform 108 may include the capability to transcribe the voice input into text using various speech-to-text processing techniques. Components of the platform 108 may then process and determine the meaning of the text by applying natural language understanding (NLU) and/or natural language processing (NLP) techniques thereto, and subsequently provide a response to the user which may be or may include a textual or audible natural language response. In one example, a user may utilize the clinical automation service (110c) to perform various clinical tasks via natural language-based conversations therewith.
The healthcare environment 100 additionally includes an electronic health record database 112. The database 112 may be a storage device managed by a healthcare provider and/or stored remotely such as in a cloud-based server or remote database managed by the cloud service provider platform 108. The database 112 may be configured to store electronic health information related to patients. Each electronic health record associated with a patient can be linked to other electronic health records associated with the patient. For example, one healthcare provider such as a family physician may generate an electronic health record for a patient and store that electronic health record in a local database and another healthcare provider such as a hospital may generate an electronic health record for the patient and store that electronic health record in a cloud-based database. The two electronic health records for the patient can be linked to the patient using an identifier for the patient such as a portion of the patient's personally identifiable information.
The healthcare environment 100 depicted in
In various embodiments, a computer-implemented method includes accessing one or more utterances and context associated with a user session in which the utterances were generated. A task to be scheduled for the user is then identified by one or more machine-learning models based on the one or more utterances and the context. The method also includes generating, based on the task and the context, a task entry. Generating the task entry comprises creating a description of the task and criteria for initiating the task, generating metadata associated with the task and the context, and storing the description, the criteria, and the metadata as the task entry in a table of a database. A notification configuration entry is then generated for the task entry, and the notification configuration entry is associated with the task entry in the table of the database. The notification configuration entry includes notification instructions for controlling how a notification is to be generated and provided upon satisfaction of the criteria for initiating the task. Upon detecting satisfaction of the criteria for initiating the task, the notification is generated based on the notification instructions, the metadata is appended to the notification to generate a notification message, and the notification message is sent to a client device based on the notification instructions.
As used herein, when an action is “based on” something, this means the action is based at least in part on at least a part of the something. As used herein, the terms “similarly,” “substantially,” “approximately” and “about” are defined as being largely but not necessarily wholly what is specified (and include wholly what is specified) as understood by one of ordinary skill in the art. In any disclosed embodiment, the term “similarly,” “substantially,” “approximately,” or “about” may be substituted with “within [a percentage] of” what is specified, where the percentage includes 0.1, 1, 5, and 10 percent.
As described in more detail below, embodiments of the cloud service provider platform 202 can be configured to have an artificial intelligence-driven (AI-driven) conversational-type interface that can conduct conversations with end users (e.g., those using the client devices) and perform functions and/or tasks based on the information conveyed by and/or ascertained from those conversations and other sources. The cloud service provider platform 202 can be configured with and/or configured to access natural language understanding (NLU) capabilities such as natural language processing, named entity recognition, intent classification, and so on. In some implementations, the cloud service provider platform 202 can include a skill-driven digital assistant 206 that includes a plurality of bots that each include one or more skills for conducting conversations and performing functions and/or tasks. In some implementations, the digital assistant 206 can instead be LLM-based and agent-driven, with agent(s) coordinating with LLM(s) for conducting conversations and performing functions and/or tasks. Examples of skill-driven and LLM-based and agent-driven digital assistants are described in U.S. patent application Ser. No. 17/648,376, filed on Jan. 19, 2022, and U.S. patent application Ser. No. 18/624,472, filed on Apr. 2, 2024, each of which are incorporated by reference as if fully set forth herein.
As shown in
The backend 204 of the cloud service provider platform 202 can also include a plurality of capabilities (services) which, in this particular embodiment, include an ambient recording capability 210, a dictation capability 212, and a clinical automation capability 214. The ambient recording service 210 may, for example, be used to record and subsequently transcribe a spoken conversation between a healthcare provider and a patient into text. The text can then be used, for example, to generate textual patient notes (e.g., SOAP notes) for a healthcare provider. The dictation service 212 may be usable to, for example, dictate terms that can be used to automatically populate various forms, such as healthcare forms. The clinical automation service 214 can be used to facilitate natural language conversations between a healthcare provider and the digital assistant 208, such as but not limited to for purposes of retrieving patient information. Clinical automation conversations between an end user and the digital assistant 208 may occur via a web channel 216, as shown. In some embodiments, each of the services 210, 212, 214 can be facilitated by a corresponding skill of the plurality of skills 208.
The backend 204 of the cloud service provider platform 202 may further include a task manager and notification service layer 218. The task manager and notification service layer 218 may allow an end user to create, or have automatically created, reminders of events or tasks, and to subsequently receive notifications upon the occurrence of a time of an event or a time to perform a task. The task manager and notification service layer 218 can include a scheduler 220 service that operates to schedule the events or reminders and a task manager service 222 that includes various task management APIs for performing functions such as creating, updating, and deleting tasks. The task manager service 222 may also allow each consuming service to configure a channel by which to receive notifications. A notification service 224 of the task manager and notification service layer 218 is also present and can operate to manage the generation of notification messages when an event or a task becomes due. Upon occurrence of a criteria for initiating a given scheduled task, the notification service 224 can communicate with the digital assistant 206 via an event channel 226 and through an event service 228, such that a notification can be generated by a skill 208 of the digital assistant 206 that is appropriate to the given task and subsequently transmitted to a client device for display to an end user. The task manager and notification service layer 218 may also include one or more e management APIs 250 that may perform management/administrative functions related to, for example and without limitation, global application settings, feature flag configurations, and metadata management.
As referenced above and described in more detail below, the application 230 displays on the client device 232 at the direction of one or more of the plurality of skills 208 of the backend 204 of the cloud service provider platform 202, various user interfaces via which an end user healthcare provider can interact with one or more of the skills 208 to perform various clinical tasks. In this particular embodiment of the application 230, an application framework includes a presentation layer 234, a data model layer 236, and a media layer 238. The presentation layer 234 can interpret messages sent by the clinical automation service 214. The data model layer 236 can function as a binding model between user interface (UI) fields and data contained in response messages sent to the client device 232 by the backend skills 208. The media layer 238 can be used, at least from a clinical automation perspective, for voice, audio and event support.
The cloud service provider platform 202 also includes a shared services component 240 that facilitates, among other things, authorizing client devices to access the cloud service provider platform 202, and an exchange of messages between client devices executing client applications and the skills 208 of the cloud service provider platform 202 backend 204. To this end, the shared services component 240 may include an external API facade 242. Except for reminders associated with the task manager and notification service layer 218, all API calls may use the external API facade 242 as a shared API service. The external API facade 242 can abstract clinical automation from an EHR solution (e.g., Cerner). The external API facade can also protect the clinical automation capability from API changes that occur outside of the cloud service provider platform 202 infrastructure, and may provide a zero-query access to information about a current healthcare provider, appointments, patients associated with those appointments, locations, and other information that can be used to populate a client application home screen that may provide a healthcare provider end user with a quick overview of important information (according to the healthcare provider).
The shared services component 240 can also include a frontend service 244 that is exposed to the client device 232. The frontend service 244 is an API service that can implement REST APIs for client applications (e.g., for configuring preferences, activating features, etc.). The frontend service 244 is also a shared service that dispatches messages and streams directed from the client application 230 to the capabilities 210, 212, 214 of the cloud service provider platform 202 backend 204. The frontend service 244 can serve as the entry point to all system services. For example, the client application 230 may go through the frontend service 244 to avail itself of any service of the cloud service provider platform 202. The frontend service 244 may include an authorization service 246. The authorization service 246 can facilitate authorization of the client device 232 (and the end user) to access and utilize the services of the cloud service provider platform 202. The frontend service 244 and the associated authorization service 246 can ensure that end user sessions get created and that the sessions are authenticated and authorized. For example, the frontend service 244 can ensure that a clinical automation operation is provided a valid session with all the information required to pass along when invoking protected services through the external API facade 242, which can relieve the clinical automation operation from the requirement to handle user login and session management.
The shared services component 240 is also shown to include a unified events channel 248, which is a cloud service of the cloud service provider platform 202 and can facilitate the aforementioned bidirectional transfer of messages between client devices/applications and the skills 208 of the cloud service provider platform 202 backend 204. The unified events channel 248 can provide a WebSocket interface by which audio, conversational text, and event traffic can flow to and from the client application 230. For example, audio and conversation text received from the applications can be routed by the unified events channel 248 to the ambient 210, dictation 212, and clinical automation 214 services of the cloud service provider platform 202.
All communications from the client application 230 to the cloud service provider platform 202 and from the cloud service provider platform 202 (e.g., from a skill 208) to the client application 230 are in the form of messages. The messages may be in a conversation message model (CMM) format. The CMM format can represent a message to or from a bot and can serve as a unified representation of messages that can flow between the client application 230 and the cloud service provider platform 202. CMM messages can convey basic requests and responses, and can also convey information defining complex user interfaces. The CMM format can also define the various message types that a skill can send to a client application. These message types may include, for example, Text Message Payload, Postback Message Payload, Form Message Payload, Edit Form Message Payload, and Inbound Event Message Payload message types.
Applications installed on client devices (i.e., “client applications”) are AI-enabled healthcare applications that are designed to help healthcare providers more efficiently perform the various repetitive tasks typically associated with delivering healthcare services to patients. For example, through use of a client application, a healthcare provider can quickly and efficiently access patient information on an as-needed basis (e.g., during a patient interaction), record details of patient interactions, generate patient notes, populate patient records, receive patient appointment schedules, schedule tasks and events, receive notifications about tasks and events that need to be performed, request and receive patient information including lab results, imaging results, lists of medications taken and medication allergies, and perform clinical functions such as ordering tests or prescribing medications. In some embodiments, such functions may be enabled via a natural language conversation with a digital assistant.
In various embodiments, a client application can be installed on a mobile computing device of an end user (as a mobile client application), a desktop computing device of an end user, (as a desktop client application), or on both a mobile computing device and a desktop computing device of an end user. The end user may be a healthcare provider. In some embodiments, the mobile computing device and the associated mobile client application may serve as the primary mechanism by which the healthcare provider interacts with the skills and services associated with a cloud service provider platform, including interactions with a digital assistant, while the desktop client application executing on the desktop computing device may serve as a companion application to the mobile client application.
As used herein, the term “mobile computing device” is intended to encompass handheld portable computing devices such as but not limited to smartphones and tablets that can easily be carried by a healthcare provider and used at virtually any location of the healthcare provider. Such mobile computing devices may execute mobile operating systems such as the iOS operating system from AppleTV or the Android operating system from Google™, although other mobile operating systems may instead be present on other mobile computing devices. As used herein, the term “desktop computing device” is intended to encompass computing devices such as personal computers, laptop computers, or similar computing devices that, even if technically mobile, are nonetheless of a size or configuration that would be too cumbersome to transport or would otherwise discourage a healthcare provider from carrying the computing device from one location to another. Such desktop computing devices may execute desktop operating systems such as Microsoft's Windows operating system, although other desktop operating systems may instead be present on other desktop computing devices. Client applications may be distributed to client devices through various channels, the type of which depends on whether the client application is a mobile client application or a desktop computing application. For example, mobile client applications may be distributed and updated through “app stores,” such as Google Play™ for mobile computing devices equipped with an Android™ operating system and the Apple App Store™ for mobile computing devices equipped with an iOS™ operating system. Desktop client applications may be distributed, according to one embodiment, through a Citrix environment.
Whether installed on a mobile computing device or a desktop computing device, a client application can utilize various functionality afforded by the mobile computing device or the desktop computing device and its associated operating system. For example, when the client application is a mobile client application installed on a mobile computing device, the mobile client application may operate a sound recording device (e.g., microphone) and/or an image capture device (e.g., camera) of the mobile computing device. Likewise, when the client application is a desktop client application installed on a desktop computing device, the desktop client application may take advantage of the likely larger display of the desktop computing device for presenting patient imaging information (e.g., x-rays) to the healthcare provider.
Both a mobile client application and a desktop client application can be controlled by various skills of a digital assistant (e.g., the digital assistant 206 of
The cloud service provider platform 302 can include a frontend service 314 that is exposed to the client devices 308, 312. The frontend service 314 may operate in the same or a similar manner to the frontend service 244 described above with respect to the cloud service provider platform 202 of
The cloud service provider platform 302 may also include an authorization service 322. The authorization service 322 can facilitate authorization of the client devices 308, 312 to access and utilize the services of the cloud service provider platform 302. In at least some embodiments, the authorization service 322 may be configured to work with supported Identity Providers (IDPs).
As shown, the services of the cloud service provider platform 302 may further include an API shim service 324 and a user service 326. The API shim service 324 can offer all required provider APIs required for building use cases, thereby eliminating any direct dependency on custom, healthcare provider-specific API interfaces. The user service 326 may provide end user management services for the cloud service provider platform 302, such as end user profile management services, user preference settings services, and so on. A management API 328 may also be present, and may perform management/administrative functions related to, for example, global application settings, feature flag configurations, and metadata management. A front-facing administration console 328 can provide an administrative user interface for managing, for example, end user sessions with the client applications 306, 310, metadata, client application settings (e.g., application feature management, end user profile/preferences). In some embodiments, the administration console 328 can be launched either directly in a web browser by entering a URL, or by selecting an administrator console user interface link presented by the client applications 306, 310 on the respective client devices 308, 312. In at least some embodiments, both end users and platform administrators may have access to the administration console 328. The administration console 328 may also provide a user interface by which the number of concurrent end user sessions with client applications can be managed. The concurrent session management mechanism of the administration console 328 can limit the number of available client application sessions a particular end user may have for a given tenancy.
Access to particular features and services of the cloud service provider platform 302 may be based on end user roles and capabilities of the client devices 308, 312. In order to implement such access control, the client applications 306, 310 may make calls to the user service 326 during a bootstrapping operation to determine end user capabilities (e.g., features accessible to the end user). Upon determining the end user capabilities, the client applications 306, 310 can render user interfaces on the client devices 308, 312 based on the end user capabilities. For example, if a standard user interface for end user interaction with the ambient recording service 318 includes selectable buttons, and the end user does not have a button selection capability, the ambient recording service user interface can be displayed on the client devices 308, 312 without buttons. In some embodiments, the client applications 306, 310 may also be configured with the ability to selectively restrict end user access to particular features and services of the cloud service provider platform 302.
This embodiment of the mobile client application framework 400 includes an application framework 402 and an associated mobile application layer 404. The application framework 402 can include several core components. As shown and described with respect to
The framework embodiment of
As shown in
A mobile client application can be built by adding the mobile application layer 404 on top of the application framework 402. As a result, the mobile application inherits all the capabilities offered by the application framework 402. The mobile application may be built for particular use and compatibility with Apple IOS and Google Android mobile device operating systems. As shown in
A desktop client application can be built by adding the desktop application layer 504 on top of the application framework 502. As a result, the desktop application inherits all the capabilities offered by the application framework 502. In some embodiments, the desktop client application may be built for particular use and compatibility with a Windows operating system. More specifically, some embodiments of the desktop client application can be built using the Windows Presentation Framework (WPF), which is a graphical user interface for Windows dot NET. The WPF application uses Microsoft Edge WebView2 control to embed the web view built using the application framework 502 container. In other words, the WebView2 control can integrate an application framework 502 created using JavaScript with the Windows native application runtime.
As shown in
As represented in
The configuration of the home screen 614 may vary based on, for example, the end user and the specific (e.g., clinical) setting in which the client computing device 608 and associated client application 602 are being used. In some embodiments, the home screen 614 can include multiple widgets via which the end user can view new, important and time sensitive information at a glance. As used herein, the term “widget” is intended to describe a small user interface element that displays important and concise information. In some embodiments, a widget can be a client application user interface construct comprising read only data along with optional buttons or links. Some embodiments of the client application home screen 614 may include several types of widgets. Examples of such widgets may include a header, conversation starters area 616, and a key items display area 618 (which may provide information unique to the end user).
If the end user chooses to interact with a given widget, such as by engaging an action button 620, 630 of the conversation starters area 616, the widget can expand into a multi-modal conversational view (e.g., task panel) to show more information or perform an action. The task panel may be an interactive modality of the skills 612 and may consist of various areas for providing information or facilitating end user interaction with the cloud service provider platform 606. In one embodiment, the task panel may include a task header, a task response area, and a task content area. Other task panel configurations are also possible.
Embodiments of the client application 602 can support multimodal end user input. Therefore, once the home screen 614 of the client application 600 has been rendered with widgets, data, etc., the end user can interact in various ways with the home screen 614 and other user interfaces and associated widgets, etc., subsequently displayed on the client computing device 608 to send messages carrying task requests to the cloud service provider platform 606. For example, the end user may engage in gesture-based interactions with the home screen 614 or another user interface screen displayed on the client computing device 608 by the client application 602. The gesture-based interactions may include, for example, a tap or click 622 of the action button 620, the action button 630, or another feature on the displayed user interface. The gesture-based interactions may also include typing 624 in a designated area of the user interface, such as within an area bounded by the widget 618.
Alternatively, the end user may engage in utterance-based interactions 626 with the home screen 614, or with another user interface screen displayed on the client computing device 608 by the client application 602 subsequent to the home screen 614. The end user may initiate an utterance-based interaction 626 with the client application 602 through voice or text such as, for example, by speaking a designated wake word/phrase (e.g., “Hey, Oracle”). The wake word may be spoken alone or the wake word may be included with a task request (e.g., “Hey, Oracle, order medication for patient X”). The spoken wake word is heard by a microphone of the client computing device 608, which may be configured in the settings to be always listening. The client application can utilize a native speech AI library to detect the wake word within an utterance of the end user, and can thereafter direct the utterance to a speech service of the cloud service provider platform 606 via a frontend service thereof. The speech service may employ automatic speech recognition (ASR) and/or other techniques to translate the utterance into text. When the utterance includes more than just the wake word, the translated text can be sent back to the client application 602. For example, during further speech by the end user, partial transcriptions may be continuously sent to a command text area of a client application 602 user interface displayed on the client computing device 608 to update the displayed transcription until completed transcription is finally displayed.
The end user can interact with the home screen 614 (or other user interfaces) presented by the client application 602 on the client computing device 608 to send task request messages to the cloud service provider platform 606. Therefore, the data flow 700 may further include receiving from the client computing device 608, a task request message associated with a task to be performed by the cloud service provider platform 606. The task request message may be generated by the client computing device 608 in response to an end user interaction with the first graphical user interface page or a subsequent graphical user interface page. As shown in
Generally speaking, the payload content of a response message can inform a client application what to display and what to say. In addition to causing a client application to display a new (e.g., second) graphical user interface page on a client device, the payload content of a response message can also cause a corresponding change in state of the client application, where the new application state corresponds to and allows the end user to appropriately interact with the information displayed in the new graphical user interface page.
As a result of receiving the response message at the client computing device 804, the state of the client application 802 in
Referring again to
When a new conversation (e.g., task request) is started with the cloud service provider platform 606 via a gesture-based or utterance-based interaction of the end user with the client applications 602, and a new data flow results therefrom, resulting updates to the user interface (e.g., home screen 614) on the client computing device 608 may be handled in a variety of ways. For example, a task overlay procedure may be utilized when displaying new user interfaces associated with responses to new task requests. According to the task overlay procedure, a new task panel can be launched by the client application 602 and overlayed on top of an existing task panel to present new content associated with responses to new task requests or other conversations with the cloud service provider platform 606. Thus, for example, when a new response message is received from the cloud service provider platform 606 while the end user is still interacting with payload content received with a previous response message, the current interaction may be suspended and the payload content of the new response message may be displayed in a new (e.g., overlayed) task panel. Alternatively, if the current interaction is determined to be more important/of higher priority than the payload content of the new response message, it is possible that display of the payload content of the new response message can be delayed until the current end user interaction is completed.
The task overlaying procedure may occur successively. This means that payload content associated with each of a number of subsequent task requests by the end user may be received and presented in a series of overlayed task panels that form a stack and wherein, within the stack, the home screen 614 is at the bottom and the task overlay presenting the most recently received response message payload content is at the top. Each time a new task panel is added to the stack, the flow associated with response message payload content presented in the prior top task panel can be suspended and the current flow can become the flow associated with the response message payload content presented in the new task panel. Thus, the task overlay procedure can be configured such that the flow associated with the response message payload content of the top task panel can always be the current flow and, likewise, the state of the client application may always be set to correspond to the current flow. The state of the client application may also update with each overlay of a new task panel. When the end user terminates interaction with the current flow and the top-most task panel is correspondingly closed, the immediately preceding task panel in the stack becomes the top task panel and the previously suspended flow associated with the response message payload content thereof becomes the current flow. The client application state may automatically update accordingly. This process of adding and removing task panels from a task panel stack may continue until the task panel of the home screen 614 becomes the top task panel.
In some embodiments, the cloud service provider platform 606 may send messages to the client computing device 608 in blocks (e.g., blocks of three or more messages). At least some of the messages in a block may be related, and in some cases more than one of the messages may be required to render a complete view on the client computing device 608. As the blocks traverse the pipeline of the cloud service provider platform 606, the blocks may be chunked into smaller blocks/sets of messages. Thus, a complete message block, as originally designated by the cloud service provider platform 606, may not arrive as one message block at the client application 602. It is also possible in some embodiments, that the cloud service provider platform 606 will only send one message at a time to the client computing device 608. Given that the view rendered on the client computing device 608 by the client application 602 can become unstable if the view relies on multiple messages and only some of the messages are received, the client application 602 may require that all messages related to a given view be received and parsed before the client application 602 will render the view associated with the messages. To this end, the client application 602 may “debounce” incoming messages-meaning that the client application 602 may aggregate incoming messages for some period of time or until some threshold or other condition is met. For example, the client application 602 may aggregate incoming messages until there is a lull (e.g., 100 milliseconds) between receiving new messages. All the aggregated messages may then be processed in one group, which increases the likelihood that all the messages required to render a particular view on the client computing device 608 will have been received and processed by the time the view is to be rendered.
In embodiments where, for example, the client computing device 608 receives messages from the cloud service provider platform 606 in blocks or the messages are otherwise aggregated and then processed as described above, multiple task panels of a stack of task panels already displayed on the client computing device 608 may be simultaneously replaced upon receipt of a block of messages or upon processing of an aggregated group of messages. In this regard, each block of messages or processed aggregated group of messages may have associated therewith, a screen identification parameter that identifies the view to be rendered by the payload content of the messages (e.g., “screen.id: start screen”). When the new block of messages is received at the client device or when an aggregated group of messages is processed at the client device, the screen identification parameters associated with the currently displayed task overlays can be analyzed and compared to the screen identification parameter for the new block of messages or the processed aggregated group of messages. When there is a match of screen identification parameters, the currently displayed task panels from the topmost task panel to, inclusively, the task panel having the matching screen identification parameter, can be removed, and replaced with a task panel rendered according to the payload content of the new block of messages or the processed aggregated group of messages. If no screen identification parameter match is found, then a task panel rendered according to the payload content of the new block of messages or processed aggregated group of messages can be overlayed on the current topmost task panel and the view is updated to correspond to the content of the new task panel. In cases, however, where an auto-close operation of the current topmost task panel has already been triggered, the current topmost task panel may be removed prior to overlaying the new task panel on the stack, as the triggering of the auto-close operation indicates that the skill flow corresponding to that task has already ended and there is no possibility of returning thereto.
A communication between a client application and a cloud service provider platform can fall into one of several categories. For example, and without limitation, a communication between a client application and a cloud service provider platform can be related to a dictation capability, an ambient recording capability, or a clinical automation capability, each of which may be implemented through communications between the client application and the cloud service provider platform. Other categories of communications related to other capabilities are also possible.
In some embodiments, a task request message from a client application to a cloud service provider platform can activate a dictation service to perform a dictation operation, whereby a healthcare provider can generate electronic healthcare records using voice inputs. Creating various electronic healthcare records is a necessary task, but can also be repetitive, tedious, and time-consuming. For example, a typical physician may spend approximately two hours on electronic healthcare record-related tasks every day. The dictation service implemented according to the disclosure through cooperation of a client application with a cloud service provider platform can reduce the time required to generate electronic healthcare records, which can allow healthcare providers more time to focus on core functions.
The dictation service may utilize or may be based on an artificial intelligence (AI) speech service associated with the cloud service provider platform. For example, the dictation service may utilize a speech-to-text service 902 of the cloud service provider platform 202 to transcribe dictated speech to text for insertion into electronic healthcare record documents. In some embodiments, and as shown in
In some embodiments, the dictation service may be most conveniently activated and utilized from a mobile client application, as most if not all mobile computing devices will have a microphone. The mobile client devices are also highly portable. Nonetheless, the dictation service can also be utilized using a desktop client application installed on a desktop computing device. For example, with an end user logged in to both a mobile client application on a mobile computing device and a companion desktop client application on a desktop computing device, the end user may issue a command to switch the desktop computing device to a dictation mode. This can occur, for example, by way of an end user interaction with a widget on a user interface of the desktop client application, or alternatively, by way of a spoken command such as “Hey Oracle, pair with desktop.” In either case, the mobile client application can detect the command and can, using the microphone of the mobile computing device, cause an audio stream of the speech to be sent to the cloud service provider platform. The cloud service provider platform will subsequently process the speech, and respond to the desktop client application rather than the mobile client application.
The dictation service may be implemented using a collection of components of the cloud service provider platform. For example, at a high level, the cloud service provider platform may include a dictation sub-service(s) that implements and orchestrates a dictation use case (e.g., activate, deactivate, manage workflow, send notifications, and deliver speech-to-text conversion results). In some embodiments, the audio stream conveying the dictated speech may be sent to a dictation engine (server) 908. The dictation server 908 may, for example, encompass all the components necessary to manage a complete dictation operation lifecycle and to interact with outside systems. The dictation server 908 may manage both inbound and outbound streaming within a session, may store and forward the speech audio for processing, and may send the result in a response message to the client device(s). The dictation service components may also include, for example, processing elements (workers) that receive a message from an inbound queue, process it and pushes the message to an outbound queue (if its eligible for further processing); a session manager that can manage user connection mapping, track user session status changes, update the session status, store session metadata, and notify the client application(s) of state changes; an audio stream handler that can manage incoming client connection requests, forward audio, and respond with dictation results; a dictation worker that can publish the transcription result for further post processing; a post dictation worker that can process the transcribed text and identify/enrich the transcription; a database worker that can create session metadata and may update as the session status changes; and one or more management APIs 910. Various other components may also be present, and other cloud service provider platform embodiments may have different configurations. In any case, use of the dictation service can meaningfully improve the efficiency electronic healthcare record creation, thereby unburdening healthcare providers to perform other functions that can have a greater impact on the patient healthcare experience.
In some embodiments, a task request message from a client application to a cloud service provider platform can cause an ambient recording service to perform an ambient recording operation whereby a spoken conversation, such as a conversation between a healthcare provider and a patient, can be recorded, transcribed to text, and used for various purposes including creating clinical notes. Creating clinical notes is an essential but time consuming process. For example, it is reported that healthcare providers typically spend approximately two to three hours per day creating clinical notes. The ambient recording service implemented according to the disclosure through cooperation of a client application with a cloud service provider platform can significantly reduce the time required to generate clinical notes.
Referring again to
A healthcare provider may, in some embodiments, utilize a mobile client application installed on a mobile computing device to stream a conversation to the cloud service provider platform using a microphone of the mobile computing device. The conversation may be streamed to the cloud service provider platform via a WebSocket connection (e.g., provided by the universal event channel 248 of
It may also be possible in other embodiments to utilize a desktop client application installed on a desktop computing device to activate and interact with the ambient recording service. The process of using a desktop client application may be accomplished in a similar manner to that described above with respect to the dictation service. When a desktop client application and associated desktop computing device is used to perform an ambient recording operation, a mobile client application to which an end user (e.g., healthcare provider) is also logged in can detect an ambient recording service activation command from the end can an audio stream of the conversation to be sent to the cloud service provider platform. The cloud service provider platform will subsequently process the conversation as described above, and respond to the desktop client application rather than the mobile client application.
Once a clinical note (e.g., SOAP note) has been created, the clinical note may be sent to the client computing device and displayed thereon by the client application for healthcare provider review and possible revision. The healthcare provider can then sign the clinical note once approved. The ambient recording service can thus eliminate the need for healthcare providers to create clinical notes from scratch, thus increasing the amount of time that healthcare providers can spend with patients by reducing the amount of time required to be spent on documentation.
In some embodiments, a task request message from a client application to a cloud service provider platform can cause an clinical automation service to perform a clinical task. Various types of clinical tasks can be performed by the clinical automation service. For example, the clinical automation service can be utilized to request and obtain patient information such as charts, records, and images (e.g., x-rays, CT scans). The clinical automation service may also be utilized to perform tasks such as ordering patient medication (e.g., calling in prescriptions). Utilizing the clinical automation service can reduce the time required to perform clinical tasks such as those described above.
Generally speaking, the clinical automation service is implemented by a multimodal digital assistant that provides support for the various clinical functional tasks. Healthcare providers can interact with the digital assistant through, for example, user interface controls, text, and audio and voice inputs.
In some embodiments, a healthcare provider can engage with the clinical automation service using a mobile client application installed on a mobile computing device and/or using a desktop client application installed on a desktop computing device, in the same or a similar manner as that described above relative to the dictation service and the ambient recording service.
As is the case with other task requests and corresponding tasks, the data flow associated with use of the clinical automation service involves an exchange of messages between the client application 1004 (e.g., mobile client application) the cloud service provider platform 1002. More particularly, task request messages sent from the client application 1004 relative to tasks to be performed by the clinical automation service are sent to the digital assistant 1006 (also see
The selected skill may be chosen, for example, by utilizing components 1012 that may apply natural language processing, natural language understanding, named entity recognition, and/or other artificial intelligence technologies to analyze the natural language utterance provided in a payload of a task request message and to thereby determine the intent of the request (i.e., to identify the task). The response generated by the selected skill and sent back to the client device 1004 includes a payload having a content that is responsive to the requested task. For example, the payload may include a natural language response to a query (e.g., a request for information), which may initiate further conversation between the healthcare provider and the digital assistant 1006. The payload content may also include requested information—e.g., John Smith's chart relative to the example task request described above. In either case, receipt of the response message by the client device on which the client application 1004 is installed will cause the client application 1004 to display the payload content on a display of the client computing device and will update the state of the client application 1004 accordingly.
The view presented can be defined by configuration parameters in the response message. In some embodiments, the view may be a task panel overlay, as previously described. In some embodiments, where a task panel is displayed upon receipt of the response message as an overlay or otherwise, the task panel may include, for example a shared context area that can provide various types of contextual information to the healthcare provider end user, including information about a patient (e.g., patient name) or information about the healthcare provider. The task panel may also include a dialog area where the actual response message (e.g., text) or a prompt can be displayed. When a prompt is displayed, the prompt may be accompanied by an audio (verbal) response that is spoken to the healthcare provider. The task panel may also display a modal area where rich user interface content such as forms, tables, cards, or similar layouts may be displayed. The model area may display, for example, the details of a given task (e.g., in case of a medication order, information about the medication being ordered, ordering options, etc.). In any case, use of the clinical automation service can simplify and render more efficient a number of clinical tasks that are commonly performed by healthcare providers, which again can provide healthcare providers with more time to interact with patients or perform other non-administrative more directly related to patient care.
An end user (e.g., health care provider) facility may be equipped with both mobile computing devices and desktop computing devices via which the health care provider can interact with a cloud service provider platform configured to assist the health care provider in performing various clinical functions. A mobile client application may be installed on each mobile computing device and a desktop client application may be installed on each desktop computing. The mobile computing devices and the desktop computing devices may have different capabilities. For example, the mobile computing devices may have microphones, while the desktop computing devices may have no microphones or may have microphones of lower quality. On the other hand, the desktop computing devices may be equipped with displays that are larger and of higher resolution than the displays of the mobile computing devices, and/or may be equipped with speakers that are of higher quality than the built-in speakers of the mobile computing devices. Thus, it may be preferred to use a mobile computing device and its associated mobile client application for some tasks, and a desktop computing device and its associated desktop client application for other tasks.
Additionally, in at least some cases, performance of a clinical task may be rendered more efficient or otherwise may be better performed if the health care provider can begin the task on one of a mobile computing device or a desktop computing device and complete the task on the other of the mobile computing device and the desktop computing device. For example, it may be easier for a health care provider to use a mobile computing device and a voice command to request retrieval of a patient x-ray by the cloud service provider platform, but it may be preferred to view the x-ray on the larger and higher resolution display of a desktop computing device. Consequently, it is desirable that a health care provider or another end user of a clinical assistance cloud service provider platform can seamlessly transition one or more times between a mobile computing device and a desktop computing device when utilizing the cloud service provider platform to perform a given clinical task.
Accordingly, a mobile client application and a desktop client application that are concurrently associated with a same user session with the cloud service provider platform, can be paired with one another so that the mobile client application and the desktop client application can operate in concert, under the control of the cloud service provider platform, to provide an end user with a single seamless experience when the end user switches between client devices while performing a task. Likewise, with the mobile client application and the desktop client application paired, an end user can still perform a task when one of the client devices is unavailable or when, for example, one of the client devices (e.g., the desktop computing device) is not conveniently accessible based on the end user's current location.
In some embodiments, the mobile computing device and the mobile client application installed thereon may be considered to be the primary client device/client application, and the desktop computing device/desktop client application may be considered to be the secondary client device/client application. For example, the desktop client application may, in some embodiments, be considered to be a companion application to the mobile client application and the capabilities of the desktop client application and the desktop computing device can be used to support use of the mobile client application and the mobile computing device.
A better understanding of a paired client application control scheme 1100 for use in a system for facilitating healthcare provider tasks, such as the a system 200, can be gained by reference to
As is further represented in
It may also be understood from
As illustrated in
Pairing the mobile client application 1102 and the desktop client application 1106 may involve registering each client application with the unified event channel 1120 of the cloud service provider platform 1110. When an end user initially logs in to the cloud service provider platform 1110 using a computing device in some embodiments, the unified event channel 1120 may receive a registration request from the client application executing on the computing device. Thus, when an end user initially signs in to the cloud service provider platform 1110 using the mobile client application 1102 on the mobile computing device 1104, the unified event channel 1120 may receive a registration request from the mobile client application 1102. The registration request message may include information that identifies the mobile computing device 1104 and capabilities of the mobile computing device 1104 to the cloud service provider platform 1110. For example, the mobile client application 1102 may inform the unified event channel 1120 that it is installed on a “mobile” device that includes a high-quality microphone. Likewise, when an end user initially signs in to the cloud service provider platform 1110 using the desktop client application 1106 of the desktop computing device 1108, the unified event channel 1120 can receive a registration request from the desktop client application 1106. The registration request message may include information that identifies the desktop computing device 1108 and capabilities of the desktop computing device 1108 to the cloud service provider platform 1110. For example, the desktop client application 1106 may inform the unified event channel 1120 that it is installed on a “desktop” device that includes a display capable of displaying large images. The client device identifications and capabilities may be saved upon registering the mobile computing device 1104 and the desktop computing device 1108 with the cloud service provider platform 1110.
In some embodiments, the unified event channel 1120 may, relative to the mobile client application 1102 and the desktop client application 1106, provide a capability that is akin to an enhanced “chat room.” Each client computing device that is assigned to a “chat room” and each client computing device in the same room can see the messages associated with the other client computing devices. Each client computing device may also stream audio to the room and receive audio streams from other client computing devices. The unified event channel 1120 can also allow each client computing device in a room to see what other computing devices are in the room (i.e., registered with the cloud service provider platform 1110) and their capabilities. Unlike combined application behavior, which is typically determined by the combination of front-end and back-end system components, the behavior of an intelligent paired application control scheme as described herein is determined by the combination of front-end systems and the capabilities of the client computing devices registered with the unified event channel 1120.
A mobile client application and a desktop client application can be paired in different ways. For example, the mobile client application 1102 and the desktop client application 1106 can be paired according to a “single sign on” application pairing procedure where the end user first signs in to the desktop client application 1106 on the desktop computing device 1108 and then pairs the mobile client application 1102 with the desktop client application 1106. For example, once the end user has signed in to the desktop client application 1106, the end user may, in some embodiments, be able to subsequently pair the mobile client application 1102 with the desktop client application 1106 by using a camera of the mobile computing device 1104 to read a bar code, a QR code, or another action initiating object displayed on a display of the desktop computing device 1108. Alternatively, a temporary alphabetic, alphanumeric, or numeric code may be obtained from the desktop computing device 1108 and entered into the mobile client application 1102.
Alternatively, the mobile client application 1102 and the desktop client application 1106 can be paired according to a “separate sign on” application pairing procedure where the user independently enters sign-in information into the desktop client application 1106 and the mobile client application 1102. The registration requests received by the unified event channel 1120 during each of the separate sign-ins serves to pair the mobile client application 1102 with the desktop client application 1106.
Upon registration of the mobile client application 1102 with the cloud service provider platform 1110, the mobile client application 1102 may immediately or soon thereafter request from the unified event channel 1120 and identification of other client devices that are registered with the cloud service provider platform 1110. Likewise, upon registration of the desktop client application 1106 with the cloud service provider platform 1110, the desktop client application 1106 may immediately or soon thereafter request from the unified event channel 1120 and identification of other client devices that are registered with the cloud service provider platform 1110. When a mobile computing device is identified as being registered with the cloud service provider platform 1110, a mobile microphone option may appear on a user interface of the desktop client application 1106 and selecting the mobile microphone option can allow the desktop computing device 1108 to record audio using the microphone of the mobile computing device 1104.
In some embodiments, neither concurrent user sessions with multiple mobile computing devices nor concurrent user sessions with multiple desktop computing devices are permitted. In such an embodiment, a sign in to a second mobile computing device while already signed in with the mobile computing device 1104 may terminate the previous user mobile computing device session or a sign in to a second desktop computing device while already signed in with the desktop computing device 1108 may terminate the previous desktop computing device user session.
During use of the mobile computing device 1104 and the desktop computing device 1108 to receive assistance from the services or skills of the cloud service provider platform 1110 with clinical tasks, the unified event channel 1120 receives tasks request messages from the mobile computing device 1104 and/or the desktop computing device 1108. For example, the unified event channel 1120 may receive from the mobile computing device 1104 in response to an end user interaction with the mobile client application 1102, a task request message corresponding to a related task to be performed by the cloud service provider platform 1110. The task request message may also identify a state of the mobile client application 1102. Upon receiving the state of the mobile client application 1102, the cloud service provider platform 1110 can synchronize a state of the desktop client application 1106 with the state of the mobile client application 1102. The cloud service provider platform 1110 can then determine a context associated with the task request message and, based at least in part on the context, can route the task request message to a selected task flow of a plurality of task flows associated with the various services or skills of the cloud service provider platform 1110. As a result, the unified event channel 1120 can receive a response message from the cloud service provider platform 1110 that was generated in response to the task being executed by processing steps of the task. The response message can have a payload content that includes a set of configuration parameters that define a layout of the payload content when displayed on the user interface 1134 of the mobile computing device 1104. The response message is then provided to both the mobile computing device 1104 and the desktop computing device 1108, whereafter the payload content of the response message is displayed on a display of one or both of the mobile computing device 1104 and the desktop computing device 1108 according to the configuration parameters and the capabilities of the mobile computing device 1104 and the desktop computing device 1108. In other example data flows, the task request message may be received from the desktop computing device 1108 instead of the mobile computing device 1104. The task request message received by the unified event channel 1120 may result from an end user interaction with a widget or another interface object displayed on the mobile computing device 1104 or the desktop computing device 1108, or by the end user issuing a spoken command.
A number of different end user-system sequence flows may be associated with engaging the various services and skills offered by the cloud service provider platform 1110 using the mobile computing device 1104 or the desktop computing device 1108. For example, when the mobile client application 1102 is in a standby mode (rather than having a muted microphone, or being in an active audio mode), an end user may utter a wake word (e.g., “Hey, Oracle”) followed by a query to activate and enter the clinical automation (assistant) mode of operation and send the query to the digital assistant 1114. In such a scenario, the mobile client application 1102 can run the audio feed associated with the utterance through the wake word component 1126, and the wake word component 1126 can recognize the wake word, notify the mobile client application 1102, and specify an audio offset immediately following receipt of the wake word. The mobile computing device 1104 can then begin streaming audio to the unified event channel 1120, starting at the specified offset. The audio stream can be directed by the unified event channel 1120 to the real-time speech API 1118 of the cloud service provider platform 1110. At the same time, the mobile client application 1102 can send a message to all related client devices registered with the cloud service provider platform 1110 (i.e., all client devices in the same virtual “room”) informing the client devices that a client automation (assistant) operation has begun. The desktop client application 1106 of the desktop computing device 1108 receives this message and, at a minimum, may resultantly update a related status icon of the current desktop client application 1106 user interface 1136. With the audio stream being delivered to the real-time speech API 1118, the real-time speech service of the cloud service provider platform 1110 can begin returning partial transcripts to the mobile computing device 1104 and the desktop computing device 1108 via the unified event channel 1120. In some embodiments, the mobile client application 1102 may display partial transcripts for end user feedback, such as in a text box of the user interface 1134. The desktop client application 1106 may, in some embodiments, ignore all partial transcripts received by the desktop computing device 1108, or may display partial transcripts, such as in a bubble notification or a command box 1138. The real-time speech service eventually returns a final transcript addressed to both the mobile computing device 1104 and the desktop computing device 1108. In some embodiments, the mobile client application 1102 may display the final transcript in a text box. The final text of the final transcript is also forwarded to the digital assistant 1114. This process can repeat until the end user conversation with the digital assistant 1114 is terminated.
Another sequence flow can result from an end user activating and entering the clinical automation (assistant) mode of operation from the desktop computing device 1108. For example, the end user may click on a clinical automation (assistant) widget (not shown in
Another sequence flow can result from an end user activating and entering the dictation mode of operation from the desktop computing device 1108. For example, the end user may engage with a user interface widget or may type a hotkey combination to start dictation, and then begin to speak. In this case, the desktop client application 1106 is aware of the widget interaction or the of the hotkey event and send a message to all related client devices registered with the cloud service provider platform 1110 (i.e., all client devices in the same virtual “room”) informing the client devices that a dictation operation has begun. The desktop client application 1106 can also activate a dictation plugin 1140, which may in some embodiments, perform at least some dictation preparation work. The mobile client application 1102 receives the message sent by the desktop client application 1106 and can begin streaming the audio associated with the speech of the end user to the unified event channel 1120, which can direct the audio stream to the dictation service 1116 of the cloud service provider platform 1110. The mobile client application 1102 may also update the user interface 1134 of the mobile computing device 1104 to reflect activation of the dictation mode. The dictation service can begin to return various responses, which are addressed to both the mobile computing device 1104 and the desktop computing device 1108. In this example, the mobile client application 1102 may ignore the responses (message type), while the desktop client application 1106 may pass all such messages to the dictation plugin 1140 to be acted upon. This sequence flow can continue until the dictation operation is terminated.
Another sequence flow can result from an end user activating and entering the dictation mode of operation from the mobile computing device 1104. For example, the end user may click on a dictation widget (e.g., action button) on the user interface 1134 of the mobile computing device 1104. In this case, it is mobile client application 1102 that sends the message to the registered client devices to inform the client devices that a dictation operation has begun prior to beginning streaming the audio of the end user's speech. The desktop client application 1106 receives the message sent by the mobile client application 1102, and may update its user interface accordingly. The desktop client application 1106 may also activate the dictation plugin 1140. Dictation service responses may then begin to be returned and may acted upon as described above relative to activating the dictation service from the desktop computing device 1108, until the dictation operation is terminated. Another sequence flow can result from an end user using the mobile computing device 1104 to activate and enter the dictation mode of operation via the clinical automation (assistant) mode. For example, the end user may utter a dictation activation phrase that includes a wake word (e.g., “Hey Oracle, start dictation”) while in standby mode, or just an dictation activation phrase (e.g., “start dictation”) when already in the clinical automation (assistant) mode. In this case, the clinical automation (assistant) functionality of the mobile client application 1102 detects the command and proceeds as described in the example sequence flow described immediately above, starting with the second step.
In the above sequence flow examples, the participating client computing devices (e.g., the mobile computing device 1104 and the desktop computing device 110) handle the events in the virtual “room” in a manner that is appropriate to their client application states and device capabilities. For example, when dictation is initiated, whether on the mobile computing device 1104 or on the desktop computing device 1108, the desktop client application 1106 may react to the dictation start event by displaying a the “dictation box” user interface component 1138 for collecting dictated text. If a dictation operation starts when the desktop computing device 1108 is not present, the mobile client application 1102 may display such a dictation box user interface object instead. In any case, many other sequence flows and associated user interface views are possible in other embodiments. For example, with respect to audio capture and wake word detection, the mobile client application 1102 can include additional widgets (e.g., action buttons) for controlling actions such as, without limitation, starting and stopping dictation, navigating form fields, pasting dictation box contents while in dictation mode, starting the clinical automation (assistant) mode by clicking a button rather than uttering a wake word; and muting the microphone. Additionally, some embodiments of the mobile client application 1102 can include a mobile dictation box, regardless of whether the desktop computing device 1108 is present, to facilitate continued dictation while an end user walks or otherwise moves from one location to another, or at least while being away from the desktop computing device 1108. In such an example, dictated text may be accumulated in the mobile dictation box and synchronized between the mobile client application 1102 and the desktop client application 1106. As the end user gets closer to the desktop computing device 1108 or arrives at another location and signs in to a different desktop computing device, the accumulated dictated text can be transferred to (pasted) into a document, etc., such as a document of an electronic health record system.
In terms of distribution of functionality between the mobile client application 1102 and the desktop client application 1106, there are a number of possible options. For example, much of the functionality can be distributed (made available) through one of the mobile client application 1102 or the desktop client application 1106, or it can be duplicated (made available) through both the mobile client application 1102 and the desktop client application 1106. The paired mobile client application 1102 and desktop client application 1106 may have different hierarchies in different embodiments. For example, in a minimal functionality configuration of the mobile client application 1102, the mobile client application 1102 may only effectuate audio capture and wake word detection. Beyond these functions, there may be no mobile client application run-time user experience at all once the two app instances are paired up. In such a minimal functionality configuration, the mobile computing device 1104 can function as a regular networked microphone (without custom buttons). To start, stop and control a dictation operation, for example, an end user may have to engage widgets on the desktop computing device user interface 1136, use keyboard shortcuts, or use voice commands.
In another example, the desktop client application 1106 may have a minimal functionality configuration. In such a configuration, the desktop client application 1106, when paired, may only delegate audio capture and wake word detection to the mobile client application 1102. All other functionality may remain available on the desktop computing device 1108 regardless of whether it is paired. The mobile client application 1102 may also provide duplicate functionality. For example, an end user can choose to start dictation by engaging a widget on the mobile client device user interface 1134 or by engaging a widget of a desktop client application 1106 toolbar (or full-size window, etc.). Additionally, if dictation and clinical automation (assistant) top-level commands are provided on the mobile client application 1102, a desktop toolbar mode (when paired) can be disabled or reduced to a read-only status display in some embodiments. In some embodiments of such a configuration, a full-size clinical automation (assistant) view provided by the mobile client application 1106 may be reduced or disabled when the client applications are paired since the clinical automation (assistant) is already implemented by the mobile client application 1102.
The paired mobile client application 1102 and desktop client application 1106 can also handle output data in different ways. As an example, consider a case where an end user issues a voice command to open an x-ray image. The captured audio associated with the voice command is sent to the digital assistant 1114 via the unified event channel 1120, where the voice command is analyzed and the intent of the voice command is determined to be a request to open the x-ray image. The command is then published back to the unified event channel 1120 (virtual room) and the desktop client application 1106, if the present, can pick up the command and the desktop computing device 1108 can be used to display the desired x-ray image, as the display of the desktop computing device 1108 is preferred for this purpose based on the display capability identified in the related earlier desktop computing device 1108 registration request message. If the desktop client application 1106 is not present, then the mobile client application 1102 can pick up the command and open the x-ray, such as by using an embedded viewer.
According to various embodiments, a root session may be maintained and may act as a parent session for each user login from various client applications. When an end user logs in to a mobile client application or a desktop client application, a session manager can determine if a root session exists. If no root session exists, the session manager can create a root session, followed by session for the current client application login. Every login for the same user will then be a child of root session. When a service needs to share state across multiple logins (desktop or mobile) for an end user, the root session can be used for saving the state.
It can be understood from the foregoing disclosure, that pairing a mobile client application and a desktop client application and controlling both applications from a cloud service provider platform can provide several unique and valuable functions. For example, when the client applications are paired as described herein, an end user working on a desktop computing device can use the mobile client application to stream audio (e.g., voice) using a microphone of the mobile computing device, and the associated cloud service provider platform will route processed results (e.g., a transcript) of the audio stream to the desktop client application. Additionally, an end user can input voice commands to the mobile client application to navigate the desktop client application. The user state can be carried across both the mobile client application and the desktop client application. For example, an end user can sign in to both the mobile client application and the desktop client application and share the same state across logged-in applications running on different platforms with various capabilities.
Healthcare providers commonly see numerous patients each day. The cumulative number of core and administrative tasks that may need to be performed relative to the patients treated by a healthcare provider may be significant. Some of these tasks may be tasks that are performed in during a patient visit, while others may not. Rather, some tasks may need to be performed prior to a patient visit and other tasks may need to be performed at some point after a patient visit. For example, in addition to scheduling a patient visit, a healthcare provider may need to obtain a patient record, an x-ray or another patient image, or other patient-related information prior to the patient visit. During the patient visit, the healthcare provider may wish to, for example, check the patient's blood pressure, collect blood for lab work, and/or conduct other tests or have certain discussions with the patient. After the patient visit, the healthcare provider may need to order the lab work, call in a prescription, or order other testing for the patient. Healthcare providers may benefit from the ability to easily and accurately generate calendar events for such events and tasks and to associate therewith future reminders (notifications) that can be provided to the healthcare providers at or around a time at which an event is set to occur or a task is to be performed. System embodiments according to the disclosure can provide a digital assistant for efficiently scheduling healthcare-related events and tasks and related techniques for generating and sending timely notifications for the scheduled events and tasks. In some embodiments of the cloud service provider platforms described herein, the digital assistant thereof may be associated with a task manager and notification service component (e.g., task manager and notification service component 218 of
A better understanding of various mechanisms by which a digital assistant of a cloud service provider platform can perform scheduling and notification functions may be obtained by reference to, for example, the architectural diagram of
The data flow 1200 illustrated by the data flow diagram of
In the example of
In one embodiment, an end user can generate a task entry for a future task, note, etc., through a conversation with the digital assistant 1202. For example, the end user may establish a conversation with the digital assistant 1202 by selecting a clinical automation (assistant) mode widget on the client application of the client computing device 1208 or by uttering a wake word (or a wake word along by a command) as described above with respect to
The task entry may include various information. For example, the task entry may include a description of the task (which may include a title), the due date or other criteria for initiating the task (e.g., time at which the task is to be performed). The task description that is part of a given task entry can help to uniquely identify the task. For example, if a task entry is generated for a task of calling in a patient prescription to a pharmacy, the task description in the task entry can be used to differentiate the task from another scheduled task for a different patient. Generating the task entry may further comprise generating metadata associated with the task and with a context of the end user conversation with the digital assistant 1202. In some embodiments, the metadata may further comprise an identification of a selected skill of a plurality of skills of the digital assistant 1202 for processing steps of the task. The description, the task initiating criteria, and the metadata may be stored as the task entry in a table of a database, such as the database 1226.
In another embodiment, a task entry may be generated without any scheduling input form the end user. For instance, a task entry may be generated automatically by the digital assistant 1202 based on a conversation of the end user. For example, when a healthcare provider speaks with a patient and the ambient recording service is used to record and transcribe the conversation, the text of the transcription may be analyzed by the aforementioned one or more machine learning models of or associated with the digital assistant 1202, and one or more tasks to be scheduled may be identified as a result. The identified tasks can be saved to the database 1226 of the task manager and notification service layer 1204, and task entries for the tasks may be created by invoking the task manager APIs 1214 and engaging the scheduler service 1210 as described above. Such task entries created automatically at the instruction of the digital assistant 1202 may include the same information described above with respect to task entries created at the express instruction of the end user, including metadata.
Irrespective of the mechanism by which task entries are generated, the task manager and notification service layer 1204 may be responsible for deduplicating tasks and sorting tasks based on priority and triggering criteria (e.g., due date). If similar tasks are proposed by the digital assistant 1202, the associated description of the task may be analyzed and used to reject duplicate tasks. If an end user asks the digital assistant 1202 to list the tasks associated with all the stored task entries, the digital assistant 1202 can obtain a list of the tasks by querying the task manager and notification service layer 1204. The tasks may then be presented to the end user on a user interface of the client computing device 1208. In some embodiments, the presented tasks may be sorted based on triggering criteria (e.g., due date) and priority by default. In some embodiments, the end user can optionally delete a task from the list, mark a task as completed, mark a task for later completion, or change a single occurrence task to a recurring task.
As previously mentioned, a feature of the task manager and notification service layer 1204 is generating notifications and notification messages. For example, notification service 1218 of the task manager and notification service layer 1204 may generate a notification configuration entry for each task entry, and may associate the notification configuration entry with the task entry in the table of the database 1226. The notification configuration entry can include, for example, notification instructions for controlling how a notification is to be generated and later provided to the client computing device 1208. Upon detecting satisfaction of the criteria for initiating the task as identified in the task entry, the notification service 1218 can generate the notification based on the notification instructions, append the metadata to the notification to generate a notification message, and send the notification message 1224 to the client computing device 1208 based on the notification instructions.
A notification for a task can be generated in multiple ways. For example, a notification for a task may be automatically generated by the notification service 1218 of the task manager and notification service layer 1204 as described above—i.e., upon occurrence of the identified criteria for initiating the associated task. Alternatively, a notification for a task may be generated when a service or skill of the cloud service provider platform 1206 needs to notify the end user about a status of some operation related to the task (e.g., generation of or a recognized failure to generate a SOAP note). Thus, the notification messages generated by the notification service 1218 may be related to: notifications that are triggered by the digital assistant 1202 (e.g., alerting the doctor about health conditions of patient when lab result is ready); notifications that indicate a state change (e.g., when a SOAP note is ready for review for the patient); notifications for tasks that are set by the end user (e.g., call in a prescription refill for a patient); and notifications generated by rules, which are notifications set by the end user or configured by the digital assistant 1202 from past end user behavior (e.g., at 6 AM every morning, show a given healthcare provider his/her patient appointments for the day).
The notification messages generated by the notification service 1218 are sent to the client computing device 1208 at the appropriate times. Notification messages 1224 can be delivered directly to the client application of the client computing device 1208 using the same (CMM) and payload format as any other messages transmitted from the cloud service provider platform 1206 to the client computing device 1208, and vice versa. In some embodiments, the cloud service provider platform 1206 caches a token and channel metadata provided by the client computing device 1208 when an end user initiates a conversation with the cloud service provider platform 1206 (e.g., at registration of the client computing device 1208). When it becomes necessary to send a notification to the client computing device 1208, the cached token and channel metadata can be used by the cloud service provider platform 1206 to identify the client computing device 1208 and the channel for communication therewith, ensuring proper delivery of generated notifications to the client computing device 1208.
In some embodiments, the notification messages can be sent to the client computing device 1208 push notifications (e.g., APNS for an iOS operating system or FCM for an Android operating system). In some embodiments, in-app notifications are also supported and may appear as an in-line popup on a user interface of the client computing device 1208. In some embodiments, notification messages may be overlayed on one another on a user interface of a client computing device, as previously described regarding task panel overlays.
In contrast to a typical mobile, web, or desktop application where a user manually an exclusively sets preferences, embodiments described herein can offer AI-powered user settings based on user context. More specifically, cloud service provider platform embodiments can leverage digital assistant recommendations to provide an optimized end user experience by automatically adjusting certain configurations based on the end user context. For instance, turning off low-priority user notifications if a healthcare provider end user is busy speaking to a patient and an ambient recording operation is ongoing, or turning on a do not disturb feature based on a location of the end user (e.g., deferring all low priority notifications if the healthcare provider is in a medical lab).
In some embodiments, a notification message can be categorized or assigned a priority level classification, such as by the notification instructions of the notification configuration entry. The priority level classification can be subsequently used to determine whether a notification of a notification message sent to the client computing device 1208 should be displayed immediately or if display of the notification should be deferred until a later time. For example, at a time of sending a notification message 1224 to the client computing device 1208, a priority level classification of the associated notification can be compared with a priority level classification (or category, etc.) of an ongoing interaction of an end user. When the priority level classification assigned to the notification is higher than the priority level classification of the ongoing interaction, an instruction may be sent to the client computing device 1208 to suspend the ongoing interaction and to immediately display the notification. Contrarily, when the priority level classification of the ongoing interaction is higher than the priority level classification assigned to the notification, an instruction may be sent to the client computing device 1208 to defer displaying the notification until the ongoing interaction has ended. In some embodiments, a notification message conveying a low priority notification may be marked or otherwise include information indicating that the message is, for example, “deferrable.” In some embodiments, a notification message conveying a high priority notification may be similarly appropriately marked. When display of a notification is deferred, the associated notification message can be queued, such as by the notification manager 1220 of the task manager and notification service layer 1204 until the ongoing interaction of the end user has completed. Upon completion of the ongoing interaction, the queued notification is presented to the end user (e.g., in the client application and/or by a device-specific push notification). Other methods for determining whether to display or delay displaying of a notification are possible in other embodiments.
Examples of situations in which it may be desirable to either immediately display or temporarily defer displaying a task notification are provided below. In one example of a situation in which it may be desirable to immediately display a task notification, the task notification may inform a healthcare provider that awaited diagnostic test results for a patient are available or are retrievable for viewing and, for example, the underlying test is related to a potentially serious condition of the patient and/or the healthcare provider is currently interacting with the patient and the test results are relevant to the interaction. In another example, a healthcare provider may be engaged in a low priority conversation with the digital assistant 1202 of the cloud service provider platform 1206 when the cloud service provider platform receives a notification message from the notification service 1218 indicating that a SOAP note is ready for a patient. In this case, the notification can be immediately displayed without negatively affecting the ongoing interaction of the healthcare provider with the cloud service provider platform 1206, as the healthcare provider can simply continue the ongoing interaction and whenever ready, tap on notification to view the details thereof. Situations in which it may be desirable to defer displaying a task notification until a current interaction of an end user is completed includes a situation where immediately displaying the notification could undesirably impact an ongoing interaction of the end user with skill or related to the context of the notification. For example, it may be undesirable to immediately display a notification when a healthcare provider is in the process of using an ambient service of a cloud service provider platform to record a conversation with a patient.
When an end user taps on or otherwise selects or engages a displayed notification, a message is returned to the cloud service provider platform 1206 indicating that the end user has elected to review the notification. The return message includes the metadata from the notification message. Upon receipt of the return message, the digital assistant 1202 can route the task associated with the notification to the selected skill identified in the metadata. In some embodiments, the task may be assigned a category prior to generation of the metadata, and the selected skill identified in the metadata may be a skill of the same category.
In some implementations, the notifications can be used to trigger or start a contextually relevant conversation between the end user and the digital assistant 1202. The context for the conversation may be derived, at least in part, from the metadata associated with the notification. For example, upon a healthcare provider engaging a notification generated for a task comprising calling in a refill prescription for a patient, the digital assistant 1202 can launch a conversation that queries the healthcare provider about the type and quantity of the medication being refilled. Upon engaging a displayed notification, the client application of the client computing device 1208 may also navigate the end user to a desired state in a conversation. For example, this could be either a user interface where a healthcare provider can view the details of the notification or it may be some actions suggested by the digital assistant 1202 based on the notification.
Upon receiving a return message from the client computing device 1208 (which may be configured as an event message) with the metadata as at least part of the message payload, and routing the task to the selected skill identified in the metadata, the selected skill can perform the task by processing steps of the task using a task flow of the skill. In this regard, any ongoing conversational flow between the end user and the selected skill can be suspended before the selected skill performs the task. In response to performing the task by the selected skill, the cloud service provider platform 1206 may send a response message to the client computing device 1208. The response message may include a payload content responsive to the task performed (e.g., the payload content may include a patient x-ray retrieved from an electronic health records store by the digital assistant 1202. Receipt of the response message by the client computing device 1208 may cause the client application to render the payload content of the response message on a user interface of a display of the client computing device 1208. The end user may interact with the user interface and further interaction between the end user and the cloud service provider platform 1206 may result.
At block 1302, one or more utterances and context associated with a user session in which the utterances were generated are accessed. The utterances may be utterances associated with a conversation between an end user and a digital assistant, and the context may be a request by the end user to schedule a task. The conversation may include necessary details about the task. Alternatively, the utterances may be part of a spoken conversation between two parties, such as a medical context conversation between a healthcare provider and a patient.
At block 1304, one or more machine-learning models identify a task to be scheduled for the user based on the one or more utterances and the context. The one or more machine-learning models can be part of or associated with a cloud service provider platform, such as with a digital assistant of the cloud service provider platform. The one or more machine-learning models may analyze the utterances (which may be in the form of transcribed text) using one or more natural language processing techniques. The one or more machine-learning models may resultantly extract the description of the task and the criteria for initiating the task.
At block 1306, a task entry is generated based on the task and the context. The task entry is effectively a calendar entry. Generating the task entry comprises creating a description of the task and criteria for initiating the task, generating metadata associated with the task and the context, and storing the description, the criteria, and the metadata as the task entry in a table of a database. The description of the task may include a title of the task. The criteria for initiating the task may be a time for performing the task. The time may be, for example, a date and a time of day. Alternatively, the criteria for initiating the task may be the completion of an operation related to the task, or a realized failure of an operation related to the task. The metadata may include various information related to the context. In some embodiments, the metadata may also include an identification of a selected task flow of a plurality of skills for processing steps of the task.
At block 1308, a notification configuration entry for the task entry is generated, and the notification configuration entry is associated with the task entry in the table of the database. The notification configuration entry includes notification instructions for controlling how a notification is to be generated and provided upon satisfaction of the criteria for initiating the task. For example, the notification configuration entry may assign a priority level classification to the notification and, at the time of sending the notification message to a client device, the priority level classification of the notification can be compared with a priority level classification of an ongoing interaction of the user to determine whether to immediately display the notification or to defer displaying the notification until the ongoing interaction of the user is completed.
At block 1310, upon detecting satisfaction of the criteria for initiating the task, the notification is generated based on the notification instructions, the metadata is appended to the notification to generate a notification message, and the notification message is sent to a client device based on the notification instructions. Once received by the client device, the notification can be displayed (either immediately or on a deferred basis). End user interaction with the notification can initiate a conversation with a digital assistant of the cloud service provider platform. The conversation may cause the task to be performed by the selected task. The conversation may include a query by the digital assistant for additional information.
In some implementations, the techniques disclosed herein for improving the efficiency of and reducing the computing resources required to perform various healthcare services in a clinical environment may be provided according to one or more of the following examples:
In one example, a method includes accessing one or more utterances and context associated with a user session in which the utterances were generated. A task to be scheduled for the user is then identified by one or more machine-learning models based on the one or more utterances and the context. The method also includes generating, based on the task and the context, a task entry. Generating the task entry comprises creating a description of the task and criteria for initiating the task, generating metadata associated with the task and the context, and storing the description, the criteria, and the metadata as the task entry in a table of a database. A notification configuration entry is then generated for the task entry, and the notification configuration entry is associated with the task entry in the table of the database. The notification configuration entry includes notification instructions for controlling how a notification is to be generated and provided upon satisfaction of the criteria for initiating the task. Upon detecting satisfaction of the criteria for initiating the task, the notification is generated based on the notification instructions, the metadata is appended to the notification to generate a notification message, and the notification message is sent to a client device based on the notification instructions.
In another example of the method, an utterance of the one or more utterances includes a natural language request to schedule a task, and the one or more machine-learning models analyze the natural language request using one or more natural language processing techniques and resultantly extract therefrom the description of the task and the criteria for initiating the task.
In another example of the method, an utterance of the one or more utterances causes the one or more machine-learning models to receive text comprising a transcribed spoken natural language conversation, and the one or more machine-learning models analyze the text using one or more natural language processing techniques, and resultantly predict the task to be scheduled based on dialogue within the text and extract from the text the description of the task and the criteria for initiating the task.
In another example of the method, token and channel metadata received from the client device are cached, and the cached token and channel metadata is used to identify the client device and the channel for communication with the client device when the notification is sent.
In another example of the method, the metadata further comprises an identification of a selected skill of a plurality of skills for processing steps of the task.
In another example of the method, after sending the notification message to the client device, a response message is received that includes the metadata, and the task is performed by causing the selected skill identified in the metadata to process the steps of the task.
In another example of the method, the notification instructions assign a priority level classification to the notification, and at a time of sending the notification message to the client device, the priority level classification of the notification is compared with a priority level classification of an ongoing interaction of the user. When the priority level classification assigned to the notification is higher than the priority level classification of the ongoing interaction, an instruction is sent to the client device to suspend the ongoing interaction and to immediately display the notification. When the priority level classification of the ongoing interaction is higher than the priority level classification assigned to the notification, an instruction is sent to the client device to defer displaying the notification until the ongoing interaction has ended.
In one example, a system includes one or more processing systems and one or more computer readable media storing instructions which, when executed by the one or more processing systems, cause the system to perform operations. The operations comprise accessing one or more utterances and context associated with a user session in which the utterances were generated. A task to be scheduled for the user is then identified by one or more machine-learning models based on the one or more utterances and the context. The method also includes generating, based on the task and the context, a task entry. Generating the task entry comprises creating a description of the task and criteria for initiating the task, generating metadata associated with the task and the context, and storing the description, the criteria, and the metadata as the task entry in a table of a database. A notification configuration entry is then generated for the task entry, and the notification configuration entry is associated with the task entry in the table of the database. The notification configuration entry includes notification instructions for controlling how a notification is to be generated and provided upon satisfaction of the criteria for initiating the task. Upon detecting satisfaction of the criteria for initiating the task, the notification is generated based on the notification instructions, the metadata is appended to the notification to generate a notification message, and the notification message is sent to a client device based on the notification instructions.
In another example of the system, an utterance of the one or more utterances includes a natural language request to schedule a task, and the operations further comprise analyzing the natural language request by the one or more machine-learning models using one or more natural language processing techniques, and resultantly extracting from the natural language request the description of the task and the criteria for initiating the task.
In another example of the system, the operations further comprise receiving at the one or more machine-learning models, as a result of an utterance of the one or more utterances, text comprising a transcribed spoken natural language conversation, and analyzing the text by the one or more machine-learning models using one or more natural language processing techniques, and resultantly predicting the task to be scheduled based on dialogue within the text and extracting from the text the description of the task and the criteria for initiating the task.
In another example of the system, cached token and channel metadata previously received from the client device is usable to identify the client device and the channel for communication with the client device at a time for sending a notification.
In another example of the system, the metadata further comprises an identification of a selected skill of a plurality of skills for processing steps of the task.
In another example of the system, the operations further comprise, after sending the notification message to the client device, receiving a response message that includes the metadata, and performing the task by causing the selected skill identified in the metadata to process the steps of the task.
In another example of the system, the notification instructions include a priority level classification for the notification, and the operations further comprise, at a time of sending the notification message to the client device, comparing the priority level classification of the notification with a priority level classification of an ongoing interaction of the user. When the priority level classification assigned to the notification is higher than the priority level classification of the ongoing interaction, an instruction is sent to the client device to suspend the ongoing interaction and to immediately display the notification. When the priority level classification of the ongoing interaction is higher than the priority level classification assigned to the notification, an instruction is sent to the client device to defer displaying the notification until the ongoing interaction has ended.
In one example, one or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause a system to perform operations. The operations comprise accessing one or more utterances and context associated with a user session in which the utterances were generated. A task to be scheduled for the user is then identified by one or more machine-learning models based on the one or more utterances and the context. The method also includes generating, based on the task and the context, a task entry. Generating the task entry comprises creating a description of the task and criteria for initiating the task, generating metadata associated with the task and the context, and storing the description, the criteria, and the metadata as the task entry in a table of a database. A notification configuration entry is then generated for the task entry, and the notification configuration entry is associated with the task entry in the table of the database. The notification configuration entry includes notification instructions for controlling how a notification is to be generated and provided upon satisfaction of the criteria for initiating the task. Upon detecting satisfaction of the criteria for initiating the task, the notification is generated based on the notification instructions, the metadata is appended to the notification to generate a notification message, and the notification message is sent to a client device based on the notification instructions.
In another example of the one or more non-transitory computer-readable media, an utterance of the one or more utterances includes a natural language request to schedule a task, and the operations further comprise analyzing the natural language request by the one or more machine-learning models using one or more natural language processing techniques, and resultantly extracting from the natural language request the description of the task and the criteria for initiating the task.
In another example of the one or more non-transitory computer-readable media, the operations further comprise receiving at the one or more machine-learning models, as a result of an utterance of the one or more utterances, text comprising a transcribed spoken natural language conversation, and analyzing the text by the one or more machine-learning models using one or more natural language processing techniques, and resultantly predicting the task to be scheduled based on dialogue within the text and extracting from the text the description of the task and the criteria for initiating the task.
In another example of the one or more non-transitory computer-readable media, the metadata further comprises an identification of a selected skill of a plurality of skills for processing steps of the task.
In another example of the one or more non-transitory computer-readable media, the operations further comprise, after sending the notification message to the client device, receiving a response message that includes the metadata, and performing the task by causing the selected skill identified in the metadata to process the steps of the task.
In another example of the one or more non-transitory computer-readable media, the notification instructions include a priority level classification for the notification, and the operations further comprise, at a time of sending the notification message to the client device, comparing the priority level classification of the notification with a priority level classification of an ongoing interaction of the user. When the priority level classification assigned to the notification is higher than the priority level classification of the ongoing interaction, an instruction is sent to the client device to suspend the ongoing interaction and to immediately display the notification. When the priority level classification of the ongoing interaction is higher than the priority level classification assigned to the notification, an instruction is sent to the client device to defer displaying the notification until the ongoing interaction has ended.
The term cloud service is generally used to refer to a service that is made available by a cloud service provider (CSP) to users (e.g., cloud service customers) on demand (e.g., via a subscription model) using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure are separate from the user's own on-premise servers and systems. Users can thus avail themselves of cloud services provided by the CSP without having to purchase separate hardware and software resources for the services. Cloud services are designed to provide a subscribing user easy, scalable access to applications and computing resources without the user having to invest in procuring the infrastructure that is used for providing the services.
There are several cloud service providers that offer various types of cloud services. As discussed herein, there are various types or models of cloud services including infrastructure as a service (IaaS), software as a service (SaaS), platform as a service (PaaS), and others. A user can subscribe to one or more cloud services provided by a CSP. The user can be any entity such as an individual, an organization, an enterprise, and the like. When a user subscribes to or registers for a service provided by a CSP, a tenancy or an account is created for that user. The user can then, via this account, access the subscribed-to one or more cloud resources associated with the account.
As noted above, IaaS is one particular type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In an IaaS model, a cloud computing provider can host the infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., a hypervisor layer), or the like). In some cases, an IaaS provider may also supply a variety of services to accompany those infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Thus, as these services may be policy-driven, IaaS users may be able to implement policies to drive load balancing to maintain application availability and performance.
In some instances, IaaS customers may access resources and services through a wide area network (WAN), such as the Internet, and can use the cloud provider's services to install the remaining elements of an application stack. For example, the user can log in to the IaaS platform to create virtual machines (VMs), install operating systems (OSs) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and even install enterprise software into that VM. Customers can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, managing disaster recovery, etc.
In most cases, a cloud computing model will require the participation of a cloud provider. The cloud provider may, but need not be, a third-party service that specializes in providing (e.g., offering, renting, selling) IaaS. An entity might also opt to deploy a private cloud, becoming its own provider of infrastructure services.
In some examples, IaaS deployment is the process of putting a new application, or a new version of an application, onto a prepared application server or the like. It may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is often managed by the cloud provider, below the hypervisor layer (e.g., the servers, storage, network hardware, and virtualization). Thus, the customer may be responsible for handling (OS), middleware, and/or application deployment (e.g., on self-service virtual machines (e.g., that can be spun up on demand) or the like.
In some examples, IaaS provisioning may refer to acquiring computers or virtual hosts for use, and even installing needed libraries or services on them. In most cases, deployment does not include provisioning, and the provisioning may need to be performed first.
In some cases, there are two different challenges for IaaS provisioning. First, there is the initial challenge of provisioning the initial set of infrastructure before anything is running. Second, there is the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.) once everything has been provisioned. In some cases, these two challenges may be addressed by enabling the configuration of the infrastructure to be defined declaratively. In other words, the infrastructure (e.g., what components are needed and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., what resources depend on which, and how they each work together) can be described declaratively. In some instances, once the topology is defined, a workflow can be generated that creates and/or manages the different components described in the configuration files.
In some examples, an infrastructure may have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., a potentially on-demand pool of configurable and/or shared computing resources), also known as a core network. In some examples, there may also be one or more inbound/outbound traffic group rules provisioned to define how the inbound and/or outbound traffic of the network will be set up and one or more virtual machines (VMs). Other infrastructure elements may also be provisioned, such as a load balancer, a database, or the like. As more and more infrastructure elements are desired and/or added, the infrastructure may incrementally evolve.
In some instances, continuous deployment techniques may be employed to enable deployment of infrastructure code across various virtual computing environments. Additionally, the described techniques can enable infrastructure management within these environments. In some examples, service teams can write code that is desired to be deployed to one or more, but often many, different production environments (e.g., across various different geographic locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some instances, the provisioning can be done manually, a provisioning tool may be utilized to provision the resources, and/or deployment tools may be utilized to deploy the code once the infrastructure is provisioned.
The VCN 1406 can include a local peering gateway (LPG) 1410 that can be communicatively coupled to a secure shell (SSH) VCN 1412 via an LPG 1410 contained in the SSH VCN 1412. The SSH VCN 1412 can include an SSH subnet 1414, and the SSH VCN 1412 can be communicatively coupled to a control plane VCN 1416 via the LPG 1410 contained in the control plane VCN 1416. Also, the SSH VCN 1412 can be communicatively coupled to a data plane VCN 1418 via an LPG 1410. The control plane VCN 1416 and the data plane VCN 1418 can be contained in a service tenancy 1419 that can be owned and/or operated by the IaaS provider.
The control plane VCN 1416 can include a control plane demilitarized zone (DMZ) tier 1420 that acts as a perimeter network (e.g., portions of a corporate network between the corporate intranet and external networks). The DMZ-based servers may have restricted responsibilities and help keep breaches contained. Additionally, the DMZ tier 1420 can include one or more load balancer (LB) subnet(s) 1422, a control plane app tier 1424 that can include app subnet(s) 1426, a control plane data tier 1428 that can include database (DB) subnet(s) 1430 (e.g., frontend DB subnet(s) and/or backend DB subnet(s)). The LB subnet(s) 1422 contained in the control plane DMZ tier 1420 can be communicatively coupled to the app subnet(s) 1426 contained in the control plane app tier 1424 and an Internet gateway 1434 that can be contained in the control plane VCN 1416, and the app subnet(s) 1426 can be communicatively coupled to the DB subnet(s) 1430 contained in the control plane data tier 1428 and a service gateway 1436 and a network address translation (NAT) gateway 1438. The control plane VCN 1416 can include the service gateway 1436 and the NAT gateway 1438.
The control plane VCN 1416 can include a data plane mirror app tier 1440 that can include app subnet(s) 1426. The app subnet(s) 1426 contained in the data plane mirror app tier 1440 can include a virtual network interface controller (VNIC) 1442 that can execute a compute instance 1444. The compute instance 1444 can communicatively couple the app subnet(s) 1426 of the data plane mirror app tier 1440 to app subnet(s) 1426 that can be contained in a data plane app tier 1446.
The data plane VCN 1418 can include the data plane app tier 1446, a data plane DMZ tier 1448, and a data plane data tier 1450. The data plane DMZ tier 1448 can include LB subnet(s) 1422 that can be communicatively coupled to the app subnet(s) 1426 of the data plane app tier 1446 and the Internet gateway 1434 of the data plane VCN 1418. The app subnet(s) 1426 can be communicatively coupled to the service gateway 1436 of the data plane VCN 1418 and the NAT gateway 1438 of the data plane VCN 1418. The data plane data tier 1450 can also include the DB subnet(s) 1430 that can be communicatively coupled to the app subnet(s) 1426 of the data plane app tier 1446.
The Internet gateway 1434 of the control plane VCN 1416 and of the data plane VCN 1418 can be communicatively coupled to a metadata management service 1452 that can be communicatively coupled to public Internet 1454. Public Internet 1454 can be communicatively coupled to the NAT gateway 1438 of the control plane VCN 1416 and of the data plane VCN 1418. The service gateway 1436 of the control plane VCN 1416 and of the data plane VCN 1418 can be communicatively coupled to cloud services 1456.
In some examples, the service gateway 1436 of the control plane VCN 1416 or of the data plane VCN 1418 can make application programming interface (API) calls to cloud services 1456 without going through public Internet 1454. The API calls to cloud services 1456 from the service gateway 1436 can be one-way: the service gateway 1436 can make API calls to cloud services 1456, and cloud services 1456 can send requested data to the service gateway 1436. But, cloud services 1456 may not initiate API calls to the service gateway 1436.
In some examples, the secure host tenancy 1404 can be directly connected to the service tenancy 1419, which may be otherwise isolated. The secure host subnet 1408 can communicate with the SSH subnet 1414 through an LPG 1410 that may enable two-way communication over an otherwise isolated system. Connecting the secure host subnet 1408 to the SSH subnet 1414 may give the secure host subnet 1408 access to other entities within the service tenancy 1419.
The control plane VCN 1416 may allow users of the service tenancy 1419 to set up or otherwise provision desired resources. Desired resources provisioned in the control plane VCN 1416 may be deployed or otherwise used in the data plane VCN 1418. In some examples, the control plane VCN 1416 can be isolated from the data plane VCN 1418, and the data plane mirror app tier 1440 of the control plane VCN 1416 can communicate with the data plane app tier 1446 of the data plane VCN 1418 via VNICs 1442 that can be contained in the data plane mirror app tier 1440 and the data plane app tier 1446.
In some examples, users of the system, or customers, can make requests, for example create, read, update, or delete (CRUD) operations, through public Internet 1454 that can communicate the requests to the metadata management service 1452. The metadata management service 1452 can communicate the request to the control plane VCN 1416 through the Internet gateway 1434. The request can be received by the LB subnet(s) 1422 contained in the control plane DMZ tier 1420. The LB subnet(s) 1422 may determine that the request is valid, and in response to this determination, the LB subnet(s) 1422 can transmit the request to app subnet(s) 1426 contained in the control plane app tier 1424. If the request is validated and requires a call to public Internet 1454, the call to public Internet 1454 may be transmitted to the NAT gateway 1438 that can make the call to public Internet 1454. Metadata that may be desired to be stored by the request can be stored in the DB subnet(s) 1430.
In some examples, the data plane mirror app tier 1440 can facilitate direct communication between the control plane VCN 1416 and the data plane VCN 1418. For example, changes, updates, or other suitable modifications to configuration may be desired to be applied to the resources contained in the data plane VCN 1418. Via a VNIC 1442, the control plane VCN 1416 can directly communicate with, and can thereby execute the changes, updates, or other suitable modifications to configuration to, resources contained in the data plane VCN 1418.
In some embodiments, the control plane VCN 1416 and the data plane VCN 1418 can be contained in the service tenancy 1419. In this case, the user, or the customer, of the system may not own or operate either the control plane VCN 1416 or the data plane VCN 1418. Instead, the IaaS provider may own or operate the control plane VCN 1416 and the data plane VCN 1418, both of which may be contained in the service tenancy 1419. This embodiment can enable isolation of networks that may prevent users or customers from interacting with other users', or other customers', resources. Also, this embodiment may allow users or customers of the system to store databases privately without needing to rely on public Internet 1454, which may not have a desired level of threat prevention, for storage.
In other embodiments, the LB subnet(s) 1422 contained in the control plane VCN 1416 can be configured to receive a signal from the service gateway 1436. In this embodiment, the control plane VCN 1416 and the data plane VCN 1418 may be configured to be called by a customer of the IaaS provider without calling public Internet 1454. Customers of the IaaS provider may desire this embodiment since database(s) that the customers use may be controlled by the IaaS provider and may be stored on the service tenancy 1419, which may be isolated from public Internet 1454.
The control plane VCN 1516 can include a control plane DMZ tier 1520 (e.g., the control plane DMZ tier 1420 of
The control plane VCN 1516 can include a data plane mirror app tier 1540 (e.g., the data plane mirror app tier 1440 of
The Internet gateway 1534 contained in the control plane VCN 1516 can be communicatively coupled to a metadata management service 1552 (e.g., the metadata management service 1452 of
In some examples, the data plane VCN 1518 can be contained in the customer tenancy 1521. In this case, the IaaS provider may provide the control plane VCN 1516 for each customer, and the IaaS provider may, for each customer, set up a unique compute instance 1544 that is contained in the service tenancy 1519. Each compute instance 1544 may allow communication between the control plane VCN 1516, contained in the service tenancy 1519, and the data plane VCN 1518 that is contained in the customer tenancy 1521. The compute instance 1544 may allow resources, that are provisioned in the control plane VCN 1516 that is contained in the service tenancy 1519, to be deployed or otherwise used in the data plane VCN 1518 that is contained in the customer tenancy 1521.
In other examples, the customer of the IaaS provider may have databases that live in the customer tenancy 1521. In this example, the control plane VCN 1516 can include the data plane mirror app tier 1540 that can include app subnet(s) 1526. The data plane mirror app tier 1540 can reside in the data plane VCN 1518, but the data plane mirror app tier 1540 may not live in the data plane VCN 1518. That is, the data plane mirror app tier 1540 may have access to the customer tenancy 1521, but the data plane mirror app tier 1540 may not exist in the data plane VCN 1518 or be owned or operated by the customer of the IaaS provider. The data plane mirror app tier 1540 may be configured to make calls to the data plane VCN 1518 but may not be configured to make calls to any entity contained in the control plane VCN 1516. The customer may desire to deploy or otherwise use resources in the data plane VCN 1518 that are provisioned in the control plane VCN 1516, and the data plane mirror app tier 1540 can facilitate the desired deployment, or other usage of resources, of the customer.
In some embodiments, the customer of the IaaS provider can apply filters to the data plane VCN 1518. In this embodiment, the customer can determine what the data plane VCN 1518 can access, and the customer may restrict access to public Internet 1554 from the data plane VCN 1518. The IaaS provider may not be able to apply filters or otherwise control access of the data plane VCN 1518 to any outside networks or databases. Applying filters and controls by the customer onto the data plane VCN 1518, contained in the customer tenancy 1521, can help isolate the data plane VCN 1518 from other customers and from public Internet 1554.
In some embodiments, cloud services 1556 can be called by the service gateway 1536 to access services that may not exist on public Internet 1554, on the control plane VCN 1516, or on the data plane VCN 1518. The connection between cloud services 1556 and the control plane VCN 1516 or the data plane VCN 1518 may not be live or continuous. Cloud services 1556 may exist on a different network owned or operated by the IaaS provider. Cloud services 1556 may be configured to receive calls from the service gateway 1536 and may be configured to not receive calls from public Internet 1554. Some cloud services 1556 may be isolated from other cloud services 1556, and the control plane VCN 1516 may be isolated from cloud services 1556 that may not be in the same region as the control plane VCN 1516. For example, the control plane VCN 1516 may be located in “Region 1,” and cloud service “Deployment 7,” may be located in Region 1 and in “Region 2.” If a call to Deployment 7 is made by the service gateway 1536 contained in the control plane VCN 1516 located in Region 1, the call may be transmitted to Deployment 7 in Region 1. In this example, the control plane VCN 1516, or Deployment 7 in Region 1, may not be communicatively coupled to, or otherwise in communication with, Deployment 7 in Region 2.
The control plane VCN 1616 can include a control plane DMZ tier 1620 (e.g., the control plane DMZ tier 1420 of
The data plane VCN 1618 can include a data plane app tier 1646 (e.g., the data plane app tier 1446 of
The untrusted app subnet(s) 1662 can include one or more primary VNICs 1664(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1666(1)-(N). Each tenant VM 1666(1)-(N) can be communicatively coupled to a respective app subnet 1667(1)-(N) that can be contained in respective container egress VCNs 1668(1)-(N) that can be contained in respective customer tenancies 1670(1)-(N). Respective secondary VNICs 1672(1)-(N) can facilitate communication between the untrusted app subnet(s) 1662 contained in the data plane VCN 1618 and the app subnet contained in the container egress VCNs 1668(1)-(N). Each container egress VCNs 1668(1)-(N) can include a NAT gateway 1638 that can be communicatively coupled to public Internet 1654 (e.g., public Internet 1454 of
The Internet gateway 1634 contained in the control plane VCN 1616 and contained in the data plane VCN 1618 can be communicatively coupled to a metadata management service 1652 (e.g., the metadata management system 1452 of
In some embodiments, the data plane VCN 1618 can be integrated with customer tenancies 1670. This integration can be useful or desirable for customers of the IaaS provider in some cases such as a case that may desire support when executing code. The customer may provide code to run that may be destructive, may communicate with other customer resources, or may otherwise cause undesirable effects. In response to this, the IaaS provider may determine whether to run code given to the IaaS provider by the customer.
In some examples, the customer of the IaaS provider may grant temporary network access to the IaaS provider and request a function to be attached to the data plane app tier 1646. Code to run the function may be executed in the VMs 1666(1)-(N), and the code may not be configured to run anywhere else on the data plane VCN 1618. Each VM 1666(1)-(N) may be connected to one customer tenancy 1670. Respective containers 1671(1)-(N) contained in the VMs 1666(1)-(N) may be configured to run the code. In this case, there can be a dual isolation (e.g., the containers 1671(1)-(N) running code, where the containers 1671(1)-(N) may be contained in at least the VM 1666(1)-(N) that are contained in the untrusted app subnet(s) 1662), which may help prevent incorrect or otherwise undesirable code from damaging the network of the IaaS provider or from damaging a network of a different customer. The containers 1671(1)-(N) may be communicatively coupled to the customer tenancy 1670 and may be configured to transmit or receive data from the customer tenancy 1670. The containers 1671(1)-(N) may not be configured to transmit or receive data from any other entity in the data plane VCN 1618. Upon completion of running the code, the IaaS provider may kill or otherwise dispose of the containers 1671(1)-(N).
In some embodiments, the trusted app subnet(s) 1660 may run code that may be owned or operated by the IaaS provider. In this embodiment, the trusted app subnet(s) 1660 may be communicatively coupled to the DB subnet(s) 1630 and be configured to execute CRUD operations in the DB subnet(s) 1630. The untrusted app subnet(s) 1662 may be communicatively coupled to the DB subnet(s) 1630, but in this embodiment, the untrusted app subnet(s) may be configured to execute read operations in the DB subnet(s) 1630. The containers 1671(1)-(N) that can be contained in the VM 1666(1)-(N) of each customer and that may run code from the customer may not be communicatively coupled with the DB subnet(s) 1630.
In other embodiments, the control plane VCN 1616 and the data plane VCN 1618 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 1616 and the data plane VCN 1618. However, communication can occur indirectly through at least one method. An LPG 1610 may be established by the IaaS provider that can facilitate communication between the control plane VCN 1616 and the data plane VCN 1618. In another example, the control plane VCN 1616 or the data plane VCN 1618 can make a call to cloud services 1656 via the service gateway 1636. For example, a call to cloud services 1656 from the control plane VCN 1616 can include a request for a service that can communicate with the data plane VCN 1618.
The control plane VCN 1716 can include a control plane DMZ tier 1720 (e.g., the control plane DMZ tier 1420 of
The data plane VCN 1718 can include a data plane app tier 1746 (e.g., the data plane app tier 1446 of
The untrusted app subnet(s) 1762 can include primary VNICs 1764(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1766(1)-(N) residing within the untrusted app subnet(s) 1762. Each tenant VM 1766(1)-(N) can run code in a respective container 1767(1)-(N), and be communicatively coupled to an app subnet 1726 that can be contained in a data plane app tier 1746 that can be contained in a container egress VCN 1768. Respective secondary VNICs 1772(1)-(N) can facilitate communication between the untrusted app subnet(s) 1762 contained in the data plane VCN 1718 and the app subnet contained in the container egress VCN 1768. The container egress VCN can include a NAT gateway 1738 that can be communicatively coupled to public Internet 1754 (e.g., public Internet 1454 of
The Internet gateway 1734 contained in the control plane VCN 1716 and contained in the data plane VCN 1718 can be communicatively coupled to a metadata management service 1752 (e.g., the metadata management system 1452 of
In some examples, the pattern illustrated by the architecture of block diagram 1700 of
In other examples, the customer can use the containers 1767(1)-(N) to call cloud services 1756. In this example, the customer may run code in the containers 1767(1)-(N) that requests a service from cloud services 1756. The containers 1767(1)-(N) can transmit this request to the secondary VNICs 1772(1)-(N) that can transmit the request to the NAT gateway that can transmit the request to public Internet 1754. Public Internet 1754 can transmit the request to LB subnet(s) 1722 contained in the control plane VCN 1716 via the Internet gateway 1734. In response to determining the request is valid, the LB subnet(s) can transmit the request to app subnet(s) 1726 that can transmit the request to cloud services 1756 via the service gateway 1736.
It should be appreciated that IaaS architectures 1400, 1500, 1600, 1700 depicted in the figures may have other components than those depicted. Further, the embodiments shown in the figures are only some examples of a cloud infrastructure system that may incorporate an embodiment of the disclosure. In some other embodiments, the IaaS systems may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration or arrangement of components.
In certain embodiments, the IaaS systems described herein may include a suite of applications, middleware, and database service offerings that are delivered to a customer in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by the present assignee.
Bus subsystem 1802 provides a mechanism for letting the various components and subsystems of computer system 1800 communicate with each other as intended. Although bus subsystem 1802 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1802 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, which can be implemented as a Mezzanine bus manufactured to the IEEE P1386.1 standard.
Processing unit 1804, which can be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of computer system 1800. One or more processors may be included in processing unit 1804. These processors may include single core or multicore processors. In certain embodiments, processing unit 1804 may be implemented as one or more independent processing units 1832 and/or 1834 with single or multicore processors included in each processing unit. In other embodiments, processing unit 1804 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.
In various embodiments, processing unit 1804 can execute a variety of programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some, or all of the program code to be executed can be resident in processor(s) 1804 and/or in storage subsystem 1818. Through suitable programming, processor(s) 1804 can provide various functionalities described above. Computer system 1800 may additionally include a processing acceleration unit 1806, which can include a digital signal processor (DSP), a special-purpose processor, and/or the like.
I/O subsystem 1808 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, pointing devices such as a mouse or trackball, a touchpad or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and/or gesture recognition devices such as the Microsoft Kinect® motion sensor that enables users to control and interact with an input device, such as the Microsoft Xbox® 360 game controller, through a natural user interface using gestures and spoken commands. User interface input devices may also include eye gesture recognition devices such as the Google Glass® blink detector that detects eye activity (e.g., ‘blinking’ while taking pictures and/or making a menu selection) from users and transforms the eye gestures as input into an input device (e.g., Google Glass®). Additionally, user interface input devices may include voice recognition sensing devices that enable users to interact with voice recognition systems (e.g., Siri® navigator), through voice commands.
User interface input devices may also include, without limitation, three dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, and audio/visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser rangefinders, and eye gaze tracking devices. Additionally, user interface input devices may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, position emission tomography, medical ultrasonography devices. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments and the like.
User interface output devices may include a display subsystem, indicator lights, or non-visual displays such as audio output devices, etc. The display subsystem may be a cathode ray tube (CRT), a flat-panel device, such as that using a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, and the like. In general, use of the term “output device” is intended to include all possible types of devices and mechanisms for outputting information from computer system 1800 to a user or other computer. For example, user interface output devices may include, without limitation, a variety of display devices that visually convey text, graphics, and audio/video information such as monitors, printers, speakers, headphones, automotive navigation systems, plotters, voice output devices, and modems.
Computer system 1800 may comprise a storage subsystem 1818 that provides a tangible non-transitory computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software can include programs, code modules, instructions, scripts, etc., that when executed by one or more cores or processors of processing unit 1804 provide the functionality described above. Storage subsystem 1818 may also provide a repository for storing data used in accordance with the present disclosure.
As depicted in the example in
System memory 1810 may also store an operating system 1816. Examples of operating system 1816 may include various versions of Microsoft Windows®, Apple Macintosh®, and/or Linux operating systems, a variety of commercially-available UNIX® or UNIX-like operating systems (including without limitation the variety of GNU/Linux operating systems, the Google Chrome® OS, and the like) and/or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS operating systems. In certain implementations where computer system 1800 executes one or more virtual machines, the virtual machines along with their guest operating systems (GOSs) may be loaded into system memory 1810 and executed by one or more processors or cores of processing unit 1804.
System memory 1810 can come in different configurations depending upon the type of computer system 1800. For example, system memory 1810 may be volatile memory (such as random access memory (RAM)) and/or non-volatile memory (such as read-only memory (ROM), flash memory, etc.) Different types of RAM configurations may be provided including a static random access memory (SRAM), a dynamic random access memory (DRAM), and others. In some implementations, system memory 1810 may include a basic input/output system (BIOS) containing basic routines that help to transfer information between elements within computer system 1800, such as during start-up.
Computer-readable storage media 1822 may represent remote, local, fixed, and/or removable storage devices plus storage media for temporarily and/or more permanently containing, storing, computer-readable information for use by computer system 1800 including instructions executable by processing unit 1804 of computer system 1800.
Computer-readable storage media 1822 can include any appropriate media known or used in the art, including storage media and communication media, such as but not limited to, volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storage and/or transmission of information. This can include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other tangible computer readable media.
By way of example, computer-readable storage media 1822 may include a hard disk drive that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive that reads from or writes to a removable, nonvolatile magnetic disk, and an optical disk drive that reads from or writes to a removable, nonvolatile optical disk such as a CD ROM, DVD, and Blu-Ray® disk, or other optical media. Computer-readable storage media 1822 may include, but is not limited to, Zip® drives, flash memory cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD disks, digital video tape, and the like. Computer-readable storage media 1822 may also include, solid-state drives (SSD) based on non-volatile memory such as flash-memory based SSDs, enterprise flash drives, solid state ROM, and the like, SSDs based on volatile memory such as solid state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory based SSDs. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computer system 1800.
Machine-readable instructions executable by one or more processors or cores of processing unit 1804 may be stored on a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium can include physically tangible memory or storage devices that include volatile memory storage devices and/or non-volatile storage devices. Examples of non-transitory computer-readable storage medium include magnetic storage media (e.g., disk or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard drives, floppy drives, detachable memory drives (e.g., USB drives), or other type of storage device.
Communications subsystem 1824 provides an interface to other computer systems and networks. Communications subsystem 1824 serves as an interface for receiving data from and transmitting data to other systems from computer system 1800. For example, communications subsystem 1824 may enable computer system 1800 to connect to one or more devices via the Internet. In some embodiments communications subsystem 1824 can include radio frequency (RF) transceiver components for accessing wireless voice and/or data networks (e.g., using cellular telephone technology, advanced data network technology, such as 3G, 4G or EDGE (enhanced data rates for global evolution), WiFi (IEEE 802.12 family standards, or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and/or other components. In some embodiments communications subsystem 1824 can provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.
In some embodiments, communications subsystem 1824 may also receive input communication in the form of structured and/or unstructured data feeds 1826, event streams 1828, event updates 1830, and the like on behalf of one or more users who may use computer system 1800.
By way of example, communications subsystem 1824 may be configured to receive data feeds 1826 in real-time from users of social networks and/or other communication services such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and/or real-time updates from one or more third party information sources.
Additionally, communications subsystem 1824 may also be configured to receive data in the form of continuous data streams, which may include event streams 1828 of real-time events and/or event updates 1830, that may be continuous or unbounded in nature with no explicit end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measuring tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.
Communications subsystem 1824 may also be configured to output the structured and/or unstructured data feeds 1826, event streams 1828, event updates 1830, and the like to one or more databases that may be in communication with one or more streaming data source computers coupled to computer system 1800.
Computer system 1800 can be one of various types, including a handheld portable device (e.g., an iPhone® cellular phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system.
Due to the ever-changing nature of computers and networks, the description of computer system 1800 depicted in the figure is intended only as a specific example. Many other configurations having more or fewer components than the system depicted in the figure are possible. For example, customized hardware might also be used and/or particular elements might be implemented in hardware, firmware, software (including applets), or a combination. Further, connection to other computing devices, such as network input/output devices, may be employed. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will appreciate other ways and/or methods to implement the various embodiments.
Although specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also encompassed within the scope of the disclosure. Embodiments are not restricted to operation within certain specific data processing environments but are free to operate within a plurality of data processing environments. Additionally, although embodiments have been described using a particular series of transactions and steps, it should be apparent to those skilled in the art that the scope of the present disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above-described embodiments may be used individually or jointly.
Further, while embodiments have been described using a particular combination of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of the present disclosure. Embodiments may be implemented only in hardware, or only in software, or using combinations thereof. The various processes described herein can be implemented on the same processor or different processors in any combination. Accordingly, where components or services are described as being configured to perform certain operations, such configuration can be accomplished, e.g., by designing electronic circuits to perform the operation, by programming programmable electronic circuits (such as microprocessors) to perform the operation, or any combination thereof. Processes can communicate using a variety of techniques including but not limited to conventional techniques for inter process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.
The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that additions, subtractions, deletions, and other modifications and changes may be made thereunto without departing from the broader spirit and scope as set forth in the claims. Thus, although specific disclosure embodiments have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.
The use of the terms “a” and “an” and “the” and similar referents in the context of describing the disclosed embodiments (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms “comprising,” “having,” “including,” and “containing” are to be construed as open-ended terms (i.e., meaning “including, but not limited to,”) unless otherwise noted. The term “connected” is to be construed as partly or wholly contained within, attached to, or joined together, even if there is something intervening. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., “such as”) provided herein, is intended merely to better illuminate embodiments and does not pose a limitation on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.
As used herein, when an action is “based on” something, this means the action is based at least in part on at least a part of the something. As used herein, the terms “substantially,” “approximately” and “about” are defined as being largely but not necessarily wholly what is specified (and include wholly what is specified) as understood by one of ordinary skill in the art. In any disclosed embodiment, the term “substantially,” “approximately,” or “about” may be substituted with “within [a percentage] of” what is specified, where the percentage includes 0.1, 1, 5, and 10 percent.
Disjunctive language such as the phrase “at least one of X, Y, or Z,” unless specifically stated otherwise, is intended to be understood within the context as used in general to present that an item, term, etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and/or Z). Thus, such disjunctive language is not generally intended to, and should not, imply that certain embodiments require at least one of X, at least one of Y, or at least one of Z to each be present.
Preferred embodiments of this disclosure are described herein, including the best mode known for carrying out the disclosure. Variations of those preferred embodiments may become apparent to those of ordinary skill in the art upon reading the foregoing description. Those of ordinary skill should be able to employ such variations as appropriate and the disclosure may be practiced otherwise than as specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein.
All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.
In the foregoing specification, aspects of the disclosure are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above-described disclosure may be used individually or jointly. Further, embodiments can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are, accordingly, to be regarded as illustrative rather than restrictive.
The present application claims the benefit of and priority to U.S. Provisional Application No. 63/583,214, filed Sep. 15, 2023, the entire contents of which are incorporated herein by reference for all purposes.
| Number | Date | Country | |
|---|---|---|---|
| 63583214 | Sep 2023 | US |
