Patent Grant
11438435
With the increased prominence and variety of computing devices, a user may have multiple devices with which a task can be performed. However, some computing devices may be better-suited for some tasks than other computing devices. Further, in some instances, a computing device that is well-suited for one step of a task may not be as well-suited for another step of the task.
It is with respect to these and other general considerations that the aspects disclosed herein have been made. Also, although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving the specific problems identified in the background or elsewhere in this disclosure.
The present disclosure provides systems and methods for user interaction and task completion using multiple devices. In examples, a task is comprised of one or more steps. Accordingly, a set of devices may be used to perform a task, such that different devices may perform different steps. A device management service may update state information at each device of the set, thereby enabling a user to interact with any of the computing devices when performing steps of the task. For example, information entered or changed by the user at one device is made available to the other devices, thereby enabling the user to complete aspects of the task at any of the other devices.
A device management service may also automatically determine which device should be used by the user, based on task or step requirements, device characteristics, and device capabilities, among other examples. As a result, the computing device that received the initial user input relating to a step may not be the computing device used to interact with the user as the user performs the step. Thus, rather than being required to complete a task using only the same device (even when the device is not well-suited for the current step or task), the user is provided with the option to use and, in some instances, is automatically transitioned to use, different devices within the set.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and/or advantages of examples will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the disclosure.
Non-limiting and non-exhaustive examples are described with reference to the following figures.
Various aspects of the disclosure are described more fully below with reference to the accompanying drawings, which form a part hereof, and which show specific example aspects. However, different aspects of the disclosure may be implemented in many different forms and should not be construed as limited to the aspects set forth herein; rather, these aspects are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the aspects to those skilled in the art. Aspects may be practiced as methods, systems or devices. Accordingly, aspects may take the form of a hardware implementation, an entirely software implementation or an implementation combining software and hardware aspects. The following detailed description is, therefore, not to be taken in a limiting sense.
Traditionally, users use a single computing device to initiate and subsequently perform a task to completion. For example, a user may use a voice command computing device to converse with a voice assistant in order to obtain information, send a message, or set a timer, among other examples. During the conversation, the user may answer voice prompts and receive information in the form of auditory responses from the voice assistant. As another example, a user may use a laptop computing device or tablet computing device to interact with graphical elements when performing a task, for example to select seats for a movie showing, make reservations at a geographically convenient and/or well-reviewed restaurant for dinner, or shop for items from an online retailer.
However, given that voice inputs are traditionally used as the primary or only input method to control a voice command computing device, the voice command computing device may not be well-suited to perform tasks that would typically be performed using a computing device having a display, touchscreen, mouse, and/or keyboard. As a result, performing such tasks using the voice command computing device may be frustrating, difficult, and time-consuming. Conversely, there may be tasks for which the voice command computing device is better-suited, such that using the voice command computing device may avoid user frustration, reduce task complexity, or decrease task duration. In other examples, a subset of steps associated with a task may be better-suited for one device, whereas a different subset of steps may be better-suited for a different device, as a result of differing device characteristics and capabilities, among other examples. It will be appreciated that the above combination of computing devices is provided as an example, and arrangements of devices with varying capabilities may be used without departing from the aspects described herein.
Accordingly, the present disclosure provides systems and methods for user interaction and task completion musing multiple devices. As used herein, a task is comprised of one or more steps. A task may comprise a set of discrete steps or, in other examples, steps of a task may be implicit or inferred. In examples, a set of devices is used to perform a task, wherein a device management service evaluates a given step of the task to determine which device or devices should be used to interact with the user. As an example, the device management service evaluates various factors, including, but not limited to, task or step capability requirements, device characteristics, device capabilities, device proximity to the user, user preferences, historical device usage or success rates for performing similar steps, and/or the ease with which a step may be completed using a given device. Additionally, when a user interacts with one of the devices in the set, the device management service may generate a state information update that is provided to each of the other devices. For example, information entered or changed by the user at one device is made available to the other devices, thereby enabling the user to perform different aspects of the task at any of the other devices. Thus, rather than being required to complete a task using only the same device (even when the device is not well-suited for the current step or task), the user is provided with the option to use and, in some instances, is automatically transitioned to use, different devices within the set. In addition to using a different computing device, it will be appreciated that any number of devices may be used to complete a step or task. As an example, the device management service may select a set of devices having the requisite capabilities to complete a task, such that the set of devices in aggregate provide the necessary capabilities, even though no one device has all of the necessary capabilities.
In examples, a set of devices is associated using a task session. The task session may be initialized when a user begins a task and may associate the set of devices while the user completes the task. As an example, each device in the set may be associated with an identifier, thereby enabling a device management service to identify devices within a set accordingly. Example identifiers include, but are not limited to, a user account identifier (e.g., username, email address, account number, etc.), one or more an interne protocol (IP) or media access control (MAC) addresses, a set of device hardware characteristics and/or device capabilities, a set of device names (e.g., as may be defined by a user, automatically generated, etc.), or a globally unique identifier (GUID). It will be appreciated that the identifier may be pre-associated with a device, or may be associated with the device as part of the initialization process. For example, a Quick Response (QR) code may be used to provide an identifier to devices that should be part of a set during the initialization process. Other techniques may be used to determine the set of devices in addition to or as an alternative to the techniques described herein, including, but not limited to, radio device detection (e.g., Bluetooth, Wi-Fi, etc.), network broadcasting/scanning, or using audio and/or light signals. While the examples are described in the context of session initialization, it will be appreciated that similar techniques may be used to add a device to an existing session.
Before a task session is initialized, a device may provide pre-session contextual information to a device management service. The device management service may use such information when initializing a session or during a session, among other examples. For example, contextual information regarding a webpage accessed by a computing device or a skill accessed by a voice command computing device may be provided to the device management service. Example contextual information includes, but is not limited to, a title (e.g., of a webpage, of a skill, etc.), a description, a Uniform Resource Locator (URL) associated with accessed content, and/or data provided to or received from the user. As a result, when a session is initialized, the device management service may use the contextual information to identify one or more devices to include in the set of devices for the session, as the device management service is able to automatically determine that such computing devices have previously been used in a manner that is related to the task. As another example, the device management service may use the contextual information to identify a task for which the session is initialized and/or pre-populate data within a task. It will be appreciated that pre-session contextual information may be received from any number of computing devices, such that the device management service may aggregate and process such information accordingly.
As used herein, a service provider provides functionality or content usable by a computing device. As an example, a service provider distributes an application that may be executed by a computing device. In another example, a service provider provides a skill processing engine, such that a smart speaker computing device (which may or may not be manufactured by the service provider) is able to interact with the skill processing engine in order to access content or functionality of the service provider. Other examples include, but are not limited to, web applications, smart home services, content streaming services, as well as content creation and/or distribution platforms.
In examples, device management service 110 facilitates user interaction and task completion using computing devices 102 and 104 according to aspects described herein. In an example, computing device 104 enables a user to access a service provided by service provider 108. As another example, a user interacts with skill processing engine 106 using computing device 102 to access functionality provided by service provider 108. Traditionally, such user interactions are not associated, such that a user that begins a task at computing device 102 would be unable to perform aspects of the task at computing device 104, or vice versa. By contrast, device management service 110 communicates state information to computing devices 102 and 104 relating to user interactions with service provider 108, and, in some instances, determines which device (e.g., either computing device 102 or computing device 104) should be used to interact with the user. In other examples, user is able to provide an indication as to which device should be used . As another example, the user interacts with either device at the user's discretion. As a result, the user is able to perform aspects of a task on either computing device and, in some instances, the device that is well-suited for the task may be automatically determined for the user.
As illustrated, computing device 102 comprises microphone 112 and speaker 114. In some examples, computing device 102 is a voice command computing device, wherein a primary method of user input is using voice commands received using microphone 112 and a primary method of output is using auditory responses provided using speaker 114. For example, computing device 102 may be a smart speaker computing device, a smart remote of a television, a controller of a video game console computing device, a smartwatch, or any of a variety of other computing devices. In some instances, a voice command computing device comprises a display and/or a touch screen, thereby enabling alternative mechanisms to convey information to a user and/or receive user input. While computing device 102 is illustrated as comprising one microphone 112 and one speaker 114, it will be appreciated that any number of such elements and other hardware characteristics may be used.
Computing device 102 communicates with skill processing engine 106 to process user inputs and provide responses. As used herein, a skill enables a user to interact with functionality provided by a service provider. For example, a skill may comprise a voice user interface (VUI) with which a user is able to request information and answer prompts, thereby controlling computing operations using auditory interactions. In examples, third-party developers may provide a skill using an application programming interface (API) provided by a service provider. In some examples, skill processing engine 106 communicates with service provider 108 to access data, perform remote computations, and/or manipulate user data, among other examples. In other examples, skill processing engine 106 need not communicate with service provider 108 to process a request from computing device 102. It will be appreciated that, in some instances, at least a part of the processing performed by skill processing engine 106 described herein may instead be performed by a different element of system 100, such as computing device 102, service provider 108, and/or device management service 110.
Computing device 104 is illustrated as comprising display 116 and physical input device 118. Computing device 104 may be a mobile device, a tablet device, a desktop computing device, a laptop computing device, a smart home device, or any of a variety of other devices. Computing device 104 is provided as an example of a different computing device having different device characteristics and different capabilities as compared to computing device 102. For example, where computing device 102 provides a VUI with which to interact with a user, computing device 104 may instead provide a graphical user interface (GUI) as a primary user interaction mechanism. As an example, the user may interact with a GUI displayed by computing device 104 on display 116 using physical input device 118. Example physical input devices include, but are not limited to, a keyboard, a mouse, a touchscreen, one or more hardware buttons, and components for sensing gestures or other actions of the user (e.g., using computer vision techniques, proximity sensors, etc.). In an example, computing device 104 comprises a web browser application, which enables a user to access Internet content (e.g., as may be provided by service provider 108). In another example, computing device 104 executes a native application, which may be provided by service provider 108.
While computing devices 102 and 104 are discussed with respect to example device types (e.g., a smart speaker computing device, a smartwatch, a laptop computing device, a device having a physical input device, a device without a screen, etc.), it will be appreciated that, in other examples, any of a variety of other computing devices may be used, each of which may have varying device characteristics and/or capabilities. Example computing devices include, but are not limited to, a mobile device, a tablet device, a desktop computing device, a laptop computing device, a server computing device, a distributed computing device, a smart watch computing device, a smart home or Internet of Things (IoT) computing device, a smart speaker computing device, or a smart television, a video game console computing device, among other examples.
Additionally, it will be appreciated that example device characteristics are described herein, and other examples of device characteristics include, but are not limited to, storage space, quantity of random access memory (RAM), the availability and/or type of specialized processing hardware (e.g., a graphics processing unit, a field-programmable gate array, neural network hardware, a hardware cryptography engine, video encoding/decoding hardware, etc.), available communication modules (e.g., Wi-Fi, Bluetooth, Ethernet, USB, etc.), display characteristics (e.g., presence or absence of a display, display size, pixel density, display technology type, etc.), the presence or quantity of available speakers, microphones, and/or cameras, and/or the presence or type of available input devices (e.g., physical input devices, voice input, etc.). Example device capabilities include, but are not limited to, an ability to communicate information discreetly or privately to a user, an ability to move within a user's environment, an ability to playback video content, an ability to execute a machine learning model, an ability to identify a hot word or other invoking action and process subsequent input accordingly, or an ability to provide notifications to a user, an ability to control a user's environment (e.g., manage one or more lights, adjust room temperature, etc.). Thus, as compared to device characteristics, device capabilities instead relate to the functionality offered by a device rather than the device characteristics used to provide such functionality. Information relating to device characteristics and/or device capabilities may be stored in a device profile, as described herein.
Service provider 108 provides a service that is accessed by computing device 102 and computing device 104, as described above. As an example, service provider 108 may provide content and functionality via the Internet, which may be accessed using a web browser, a native application, or a VUI (e.g., via skill processing engine 106), among other techniques. Service provider 108 comprises request processing engine 120 and user data store 122. In an example, request processing engine 120 processes requests from computing device 104 and skill processing engine 106. For example, request processing engine 120 may process requests from computing device 102 and computing device 104 (e.g., in the form of requests for web content, requests from a native application, requests via skill processing engine 106, etc.), and may communicate with user data store 122 (e.g., to access, modify, delete, and/or add data stored therein). User data store 122 may store user information, as may be received or generated by request processing engine 120. Examples include, but are not limited to, account information (e.g., usernames, email addresses, hashed passwords, etc.), user-generated content, and other data used by service provider 108 in providing the service.
As illustrated, device management service 110 is comprised of session management engine 124, session data store 126, device selection engine 128, and device information data store 130. Session management engine 124 manages a set of computing devices associated with one or more tasks according to aspects described herein. As an example, session management engine 124 receives state updates from computing device 102 and/or computing device 104, such that updated state information is provided or otherwise made available to other devices in the set accordingly. In some examples, contextual information relating to computing device 102 and/or computing device 104 may be received prior to the initialization of a session, such that the contextual information may later provide context for a session. In another example, session management engine 124 initializes and manages a task session that associates a set of devices (e.g., computing device 102 and computing device 104).
Accordingly, session management engine 124 enables a user to use the set of computing devices to perform different steps of the task. As described herein, devices may be selected automatically, at the direction of the user, or the user may select a device, among other examples. In some examples, session management engine 124 communicates with device selection engine 128 to determine a device that the user interacts with to perform a step of a task, as discussed in greater detail below. As another example, a user may be using an initial device to perform a step, while device management service 110 may determine that a different device should be used. Accordingly, an indication may be provided to the user at the initial device, such that the user may choose to switch to the different device. In some instances, rather than switching from the initial device to the different device, both devices may be used. If device management service 110 identifies a device type that the user does not currently own, a recommendation to purchase such a device may be presented to the user. While example task sessions may be described with respect to a single task, in other examples it will be appreciated that a task session may associate a set of devices for multiple tasks.
Session data store 126 may be used by session management engine 124 to store contextual information, state information, and information relating to potential, active, and/or past task sessions. For example, when a session state update is received from a computing device, session management engine 124 may store information relating to the session state update in session data store 126. Such state information may then be used to determine a task associated with a task session, to pre-populate data needed for a task, and to provide state updates to other computing devices, among other examples. Example session state information includes, but is not limited to, a session or user account identifier for a service provider (e.g., service provider 108), at least a part of a user input received at a computing device, at least a part of a response generated based on a user input, or data relating to a user action (e.g., a recently accessed web page, a recently used skill, a recently contacted contact, a missed phone call, etc.).
Device selection engine 128 evaluates a set of factors to determine one or more devices from a set of devices that are used to interact with a user as part of a task. In examples, device selection engine 128 and session management engine 124 communicate, such that a device determination made by device selection engine 128 is used by session management engine 124 to assign a specific computing device to a given step of a task. Thus, while examples herein are described with respect to selecting and assigning a single device to a step of a task, it will be appreciated that, in other examples, multiple devices are selected such that each device of the selected device is used to perform different aspects of the step. Example factors include, but are not limited to, task or step capability requirements (e.g., the availability of a display, the ability to receive voice input, the ability to provide video playback, etc.), computing device characteristics, computing device capabilities, device proximity to the user, user preferences, historical device usage or success rates for performing similar steps, and/or the ease with which a step may be completed using a given device. In some examples, a task or a step may provide an indication as to a type of device, device characteristics, device capabilities, and/or available capabilities that are required and/or recommended.
Device information data store 130 stores information relating to the factors described herein. For example, device information data store 130 stores user preference information, which may indicate one or more mappings between a task type and a general type of computing device or a computing device having certain capabilities. In some examples, a mapping indicated by a user preference may be between a task type and a specific computing device. It will be appreciated that, in other examples, a mapping may associate multiple devices with a task and/or step. For example, the devices associated with a task requiring user input may comprise one device having a display to in order to present information to the user, as well as another device implementing a VUI with which to receive the user input, such that both devices are used when completing the task. As another example, device information data store 130 stores one or more device profiles. As used herein a device profile comprises information relating to a computing device, such as device characteristics, device capabilities, proximity to the user, and/or battery charge state (e.g., a percentage of charge, charging, draining, not present, etc.), among other examples.
In other examples, device information data store 130 stores historical device usage or success rate information, wherein the historical device usage information may relate to a specific user and/or may relate to a larger population of users. As an example, population device usage information may be used when user-specific usage information is unavailable (e.g., as may be the case when the user is new, has recently purchased a new device, etc.). Success rate information describe how effective (e.g., based on user satisfaction, completion time, accuracy, etc.) a type of device is for performing a type of task. Such information may be collected based on an analysis of user survey information and/or usage data provided from computing devices on which a user has opted-in to such data collection.
At operation 204, at least one other computing device is determined based on the received identifier. In examples, pre-session contextual information received from a computing device may be evaluated to identify an association between a computing device and the identifier received at operation 202. As discussed herein, pre-session contextual information may be stored by a session data store, such as session data store 126 in
Moving to operation 206, a task session is generated that associates the computing device from operation 202 and the at least one other computing device determined at operation 204 with a task. In examples, information relating to the session (e.g., the set of associated devices, one or more tasks associated with the session, etc.) is stored in a session data store, such as session data store 126 in
Flow progresses to operation 208, where a step of the task associated with the user input is determined. In some examples, the indication received at operation 202 comprises an indication as to the step associated with the user input. In other instances, the step may be determined by communicating with a service provider, may be determined based on an analysis of the user input, or may be determined using state information stored in a session data store, among other examples.
At operation 210, the determined step is evaluated to generate a set of requirements. As described herein, in some instances a task and/or step may provide an indication as to required and/or recommended capabilities. In other examples, the set of requirements is determined based on evaluating actions, potential user inputs, potential outputs, and/or other information associated with the step. In such examples, the evaluation may comprise evaluating mappings between types of actions, inputs, outputs, and other information and preferred or required device capabilities. In other examples, historical device usage information may be evaluated.
Moving to operation 212, one or more processing computing devices for performing the step are determined. In some examples, aspects of operation 212 are performed by a device selection engine of a device management service, such as device selection engine 128 of device management service 110 in
At operation 214, an indication is provided to the determined device (or, in some instances, the determined devices) to perform the step. In some instances, the indication comprises at least a part of the information received at operation 202 (e.g., user input, a task, a step, etc.). The indication may be provided directly to the computing device or may be provided via an intermediary (e.g., via a service provider, a skill processing engine, etc.), among other examples. It will be appreciated that, in some examples, multiple indications may be provided. For example, if it is determined that a step should be processed using a browser on a computing device, an indication may be provided to the computing device to open the browser to a webpage, while a second indication may be provided to a service provider, such that the service provider has information relating to the step, task, and/or session. For example, the webpage may be populated with data previously received from a user. As another example, a skill processing engine may receive state information, such that a user using a voice command computing device is able to reference the state information when conversing with the voice command computing device.
Eventually, flow may progress to operation 216, where another indication of user input is received, as is the case when the user performs another step in the task session. Accordingly, flow returns to operation 208, where a step associated with the user input is determined as discussed above. Thus, flow loops through operations 208, 210, 212, 214, and 216 as the user performs steps of the task using the set of devices with which the session is associated. Ultimately, flow terminates at operation 214.
At operation 224, device profiles are accessed. For example, a device profile for each device of a set of devices associated with a session may be accessed. Device profiles may be accessed from a device information data store, such as device information data store 130 in
At operation 226, device hardware characteristics are evaluated. Example device hardware characteristics include, but are not limited to, storage space, quantity of RAM, the availability and/or type of specialized processing hardware, available communication modules, display, the presence or quantity of available speakers, microphones, and/or cameras, and/or the presence or type of available input devices. Such hardware characteristics are evaluated in view of the step requirements received at operation 222 to determine the extent to which the available devices satisfy the set of step requirements.
Flow progresses to operation 228, where device capabilities are evaluated. Example device capabilities include, but are not limited to, an ability to communicate information discreetly or privately to a user, an ability to move within a user's environment, an ability to playback video content, an ability to execute a machine learning model, an ability to identify a hot word or other invoking action and process subsequent input accordingly, an ability to provide notifications to a user, an ability to control a user's environment (e.g., manage one or more lights, adjust room temperature, etc.). Thus, as compared to hardware characteristics, device capabilities instead relate to the functionality offered by a device rather than the hardware characteristics used to provide such functionality. Such device capabilities are evaluated in view of the step requirements received at operation 222 to determine the extent to which the available devices satisfy the set of step requirements.
At operation 230, historical device usage information is evaluated. In examples, historical device usage information is accessed from a device information data store, such as device information data store 130 in
Moving to operation 232, a ranking is generated for the set of devices based on the factors evaluated at operations 226-230. In examples, the factors may be weighted or prioritized when ranking the set of devices. It will be appreciated that while example factors and evaluations have been described with respect to method 220, one or more evaluations may be omitted, or alternative or additional evaluations may be used. At operation 234, a device is selected to perform a step based on the generated ranking. In some examples, a set of device is selected, wherein the set comprises devices that rank above a predetermined threshold. Flow terminates at operation 234.
Process flow 300 begins at operation 312, where a user input relating to a step of a task is received at computing device 302. As an example, computing device 302 may receive the user input in the form of a user interacting with a physical input device or issuing a voice command, among other examples. It will be appreciated that, while
At operation 314, an indication of the user input is generated and provided to management service 310. In examples, the indication may comprise an identifier associated with computing device 302. As discussed herein, the identifier may be associated with an ongoing session. Thus, using the identifier, management service 310 identifies the set of session devices associated with the task at operation 316.
Moving to operation 318, step requirements are determined for the step of the task that the user is performing. In examples, the determination comprises performing aspects of operation 210 discussed above with respect to
At operation 320, a processing computing device to perform the step is determined from the set of devices associated with the session. In an example, the determination comprises performing aspects of operation 212 discussed with respect to method 200 of
As illustrated, the indication is provided to service provider 308, such that service provider 308 processes the user input at operation 324. In examples, the indication comprises session state information and/or at least a part of the user input, among other information. As an example, processing the user input may comprise manipulating data, generating a webpage, and/or generating a voice response, among other examples. Thus, process flow 300 is illustrated as providing the indication to service provider 308 rather than computing device 302, as computing device 302 provided the user input to service provider 308 at operation 312, such that service provider 308 is able to generate a response and respond accordingly. It will be appreciated that, in other examples, an indication is provided to a computing device, thereby causing the computing device to take action instead of or in addition to service provider 308. For example, sensitive data (e.g., camera input, financial information, passwords, etc.) may be stored locally by the computing device, such that only the computing device is able to process the sensitive data. Such an example is discussed below with respect to
At operation 326, a response to the user input is provided to computing device 302, which is received by computing device 302 at operation 328. As an example, receiving the response may comprise processing received webpage data in a web browser, using a speaker to play a received voice response, or updating the state of a native application, among other examples.
At operation 330, an indication of the current session state (e.g., as may have changed as a result of processing the user input at operation 324) is provided to management service 310. Accordingly, at operation 332, management service 310 generates a session state update notification, which is provided to other devices of process flow 300, which, as illustrated, is computing device 304. In some examples, management service 310 updates a session data store, such as session data store 126 in
Thus, computing device 304 processes the user input at operation 342. In examples, the indication received from management service 310 comprises session state information and/or at least a part of the user input, among other information. Accordingly, computing device 304 may process the user input in view of the current state of the session and information relating to past steps that were previously performed by the user.
At operation 344, computing device 304 outputs a response to the user input. As an example, outputting the response may comprise updating the state of a native application, outputting a voice response using a speaker, or providing haptic feedback, among other examples.
At operation 346, an indication of the current session state (e.g., as may have changed as a result of processing the user input at operation 342) is provided to management service 310. Accordingly, at operation 348, management service 310 generates a session state update notification, which is provided to other devices of process flow 340, which, as illustrated, is service provider 308. In some examples, management service 310 updates a session data store, such as session data store 126 in
Process flow 360 begins at operation 362, where a user input is received. As an example, computing device 302 may receive the user input in the form of a user interacting with a physical input device or issuing a voice command, among other examples. The arrow from operation 362 to operation 364 indicates that information relating to the user input is provided to service provider 308. As an example, the user input may be the submission of a webpage to process using a request processing engine, or may be a voice utterance that is provided to service provider 308 to process using a skill processing engine. It will be appreciated that, in some examples, the user input be processed at least in part by computing device 302 prior to communicating the user input to service provider 308.
At operation 364, service provider 308 processes the user input. In examples, session provider 308 uses session state information (e.g., as may be received from a device management service according to aspects described herein). As an example, processing the user input may comprise manipulating data, generating a webpage, and/or generating a voice response, among other examples. Accordingly, at operation 366, a response to the user input is provided to computing device 302, which is received by computing device 302 at operation 368. As an example, receiving the response may comprise processing received webpage data in a web browser, using a speaker to play a received voice response, or updating the state of a native application, among other examples.
At operation 370, an indication of the current session state (e.g., as may have changed as a result of processing the user input at operation 364) is provided to management service 310. Accordingly, at operation 372, management service 310 identifies session devices to which updated state information should be provided. According to aspects described herein, management service 310 may identify a set of devices associated with the session using an identifier, such as a user account identifier, one or more IP or MAC addresses, a set of device hardware characteristics and/or device capabilities, a set of device names, or a GUID, among other examples.
Moving to operation 374, management service 310 generates a session state update notification, which is provided to other devices of process flow 300, which, as illustrated, is computing device 304. In some examples, management service 310 updates a session data store, such as session data store 126 in
Process flow 380 begins at operation 382, where a user input is received. As an example, computing device 304 may receive the user input in the form of a user interacting with a physical input device or issuing a voice command, among other examples. At operation 384, computing device 304 processes the user input. In examples, computing device 304 uses session state information (e.g., as may be received from a device management service according to aspects described herein).
Accordingly, at operation 386, computing device 304 outputs a response to the user input. As an example, outputting the response may comprise updating the state of a native application, outputting a voice response using a speaker, or providing haptic feedback, among other examples.
At operation 388, an indication of the current session state (e.g., as may have changed as a result of processing the user input at operation 384) is provided to management service 310. Accordingly, at operation 390, management service 310 identifies session devices to which updated state information should be provided. According to aspects described herein, management service 310 may identify a set of devices associated with the session using an identifier, such as a user identifier, an IP or MAC address, a set of device hardware characteristics and/or device capabilities, a device name, or a GUID, among other examples.
Moving to operation 392, management service 310 generates a session state update notification, which is provided to other devices of process flow 300, which, as illustrated, is service provider 308. In some examples, management service 310 updates a session data store, such as session data store 126 in
As stated above, a number of program modules and data files may be stored in the system memory 404. While executing on the processing unit 402, the program modules 406 (e.g., application 420) may perform processes including, but not limited to, the aspects, as described herein. Other program modules that may be used in accordance with aspects of the present disclosure may include electronic mail and contacts applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided application programs, etc.
Furthermore, embodiments of the disclosure may be practiced in an electrical circuit comprising discrete electronic elements, packaged or integrated electronic chips containing logic gates, a circuit utilizing a microprocessor, or on a single chip containing electronic elements or microprocessors. For example, embodiments of the disclosure may be practiced via a system-on-a-chip (SOC) where each or many of the components illustrated in
The computing device 400 may also have one or more input device(s) 412 such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or swipe input device, etc. The output device(s) 414 such as a display, speakers, a printer, etc. may also be included. The aforementioned devices are examples and others may be used. The computing device 400 may include one or more communication connections 416 allowing communications with other computing devices 450. Examples of suitable communication connections 416 include, but are not limited to, radio frequency (RF) transmitter, receiver, and/or transceiver circuitry; universal serial bus (USB), parallel, and/or serial ports.
The term computer readable media as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, or program modules. The system memory 404, the removable storage device 409, and the non-removable storage device 410 are all computer storage media examples (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other article of manufacture which can be used to store information and which can be accessed by the computing device 400. Any such computer storage media may be part of the computing device 400. Computer storage media does not include a carrier wave or other propagated or modulated data signal.
Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term “modulated data signal” may describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.
One or more application programs 566 may be loaded into the memory 562 and run on or in association with the operating system 564. Examples of the application programs include phone dialer programs, e-mail programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, Internet browser programs, messaging programs, and so forth. The system 502 also includes a non-volatile storage area 568 within the memory 562. The non-volatile storage area 568 may be used to store persistent information that should not be lost if the system 502 is powered down. The application programs 566 may use and store information in the non-volatile storage area 568, such as e-mail or other messages used by an e-mail application, and the like. A synchronization application (not shown) also resides on the system 502 and is programmed to interact with a corresponding synchronization application resident on a host computer to keep the information stored in the non-volatile storage area 568 synchronized with corresponding information stored at the host computer. As should be appreciated, other applications may be loaded into the memory 562 and run on the mobile computing device 500 described herein.
The system 502 has a power supply 570, which may be implemented as one or more batteries. The power supply 570 might further include an external power source, such as an AC adapter or a powered docking cradle that supplements or recharges the batteries.
The system 502 may also include a radio interface layer 572 that performs the function of transmitting and receiving radio frequency communications. The radio interface layer 572 facilitates wireless connectivity between the system 502 and the “outside world,” via a communications carrier or service provider. Transmissions to and from the radio interface layer 572 are conducted under control of the operating system 564. In other words, communications received by the radio interface layer 572 may be disseminated to the application programs 566 via the operating system 564, and vice versa.
The visual indicator 520 may be used to provide visual notifications, and/or an audio interface 574 may be used for producing audible notifications via the audio transducer 525. In the illustrated embodiment, the visual indicator 520 is a light emitting diode (LED) and the audio transducer 525 is a speaker. These devices may be directly coupled to the power supply 570 so that when activated, they remain on for a duration dictated by the notification mechanism even though the processor 560 and other components might shut down for conserving battery power. The LED may be programmed to remain on indefinitely until the user takes action to indicate the powered-on status of the device. The audio interface 574 is used to provide audible signals to and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 525, the audio interface 574 may also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. In accordance with embodiments of the present disclosure, the microphone may also serve as an audio sensor to facilitate control of notifications, as will be described below. The system 502 may further include a video interface 576 that enables an operation of an on-board camera 530 to record still images, video stream, and the like.
A mobile computing device 500 implementing the system 502 may have additional features or functionality. For example, the mobile computing device 500 may also include additional data storage devices (removable and/or non-removable) such as, magnetic disks, optical disks, or tape. Such additional storage is illustrated in
Data/information generated or captured by the mobile computing device 500 and stored via the system 502 may be stored locally on the mobile computing device 500, as described above, or the data may be stored on any number of storage media that may be accessed by the device via the radio interface layer 572 or via a wired connection between the mobile computing device 500 and a separate computing device associated with the mobile computing device 500, for example, a server computer in a distributed computing network, such as the Internet. As should be appreciated such data/information may be accessed via the mobile computing device 500 via the radio interface layer 572 or via a distributed computing network. Similarly, such data/information may be readily transferred between computing devices for storage and use according to well-known data/information transfer and storage means, including electronic mail and collaborative data/information sharing systems.
As will be understood from the foregoing disclosure, one aspect of the technology relates to a system comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, causes the system to perform a set of operations, the set of operations comprising: receiving an indication of user input at a first computing device associated with a step of a task session, wherein the indication comprises an identifier associated with the task session; determining, using the identifier, a set of computing devices associated with the task session, wherein the set of computing devices comprises the first computing device and a second computing device; determining a processing computing device to perform the step of the task session from the set of computing devices; and providing, to the processing computing device, an indication to process the user input associated with the step of the task session. In an example, the set of operations further comprises: receiving, from the processing computing device, a session state indication comprising session state information; when the processing computing device is the first computing device, providing a state update indication to the second computing device based on the session state information; and when the processing computing device is the second computing device, providing a state update indication to the first computing device based on the session state information. In another example, determining the processing computing device comprises evaluating at least one device characteristic of the first computing device and at least one device characteristic of the second computing device based on a requirement associated with the step to automatically determine the processing computing device. In a further example, determining the processing computing device comprises evaluating at least one device capability of the first computing device and at least one device capability of the second computing device based on a requirement associated with the step to automatically determine the processing computing device. In yet another example, providing the indication to process the user input associated with the step of the task session further comprises providing an indication to a third computing device that the processing computing device is processing the user input. In a further still example, the processing computing device is the second computing device, thereby causing the user to interact with a different computing device than received the user input to perform the step. In another example, the set of computing devices further comprises a third computing device, and the set of operations further comprises: determining the third computing device to perform the step of the task session, thereby causing the user to interact with both the second computing device and the third computing device to perform the step.
In another aspect, the technology relates to a method for initializing a task session associated with a set of computing devices. The method comprises: receiving, from a first computing device of a user, an indication of input associated with a step of a task, wherein the indication comprises an identifier; identifying, based on the identifier, a second computing device associated with the user; generating a task session associated with a set of computing devices, wherein the set of computing devices comprises the first computing device and the second computing device; determining a processing computing device to perform the step from the set of computing devices to process the input; and providing, to the processing computing device, an indication to process the input associated with the step of the task session. In an example, the method further comprises: receiving, from the processing computing device, a session state indication comprising session state information; when the processing computing device is the first computing device, providing a state update indication to the second computing device based on the session state information; and when the processing computing device is the second computing device, providing a state update indication to the first computing device based on the session state information. In another example, the method further comprises: receiving, from the second computing device, pre-session contextual information, wherein the pre-session contextual information comprises the identifier; and storing the pre-session contextual information. In a further example, the second computing device is identified based on the stored pre-session contextual information, and generating the task session further comprises analyzing the pre-session contextual information. In yet another example, the second computing device is identified by receiving an input comprising the identifier from the second computing device after receiving the indication from the first computing device. In a further still example, determining the processing computing device comprises evaluating a set of factors for each computing device based on a requirement associated with the step, wherein the set of factors are selected from the group consisting of: a computing device characteristic; a computing device capability; proximity of the computing device to the user; user preference information; and historical device usage information.
In a further aspect, the technology relates to another method for determining a computing device to perform a step of a task session. The method comprises: receiving an indication of user input at a first computing device associated with the step of the task session, wherein the indication comprises an identifier associated with the task session; determining, using the identifier, a set of computing devices associated with the task session, wherein the set of computing devices comprises the first computing device and a second computing device; determining a processing computing device to perform the step of the task session from the set of computing devices; and providing, to the processing computing device, an indication to process the user input associated with the step of the task session. In an example, the method further comprises: receiving, from the processing computing device, a session state indication comprising session state information; when the processing computing device is the first computing device, providing a state update indication to the second computing device based on the session state information; and when the processing computing device is the second computing device, providing a state update indication to the first computing device based on the session state information. In another example, determining the processing computing device comprises evaluating at least one device characteristic of the first computing device and at least one device characteristic of the second computing device based on a requirement associated with the step to automatically determine the processing computing device. In a further example, determining the processing computing device comprises evaluating at least one device capability of the first computing device and at least one device capability of the second computing device based on a requirement associated with the step to automatically determine the processing computing device. In yet another example, providing the indication to process the user input associated with the step of the task session further comprises providing an indication to a third computing device that the processing computing device is processing the user input. In a further still example, the processing computing device is the second computing device, thereby causing the user to interact with a different computing device than received the user input to perform the step. In another example, the processing device is determined based on at least one of historical device usage information and user preference information.
Aspects of the present disclosure, for example, are described above with reference to block diagrams and/or operational illustrations of methods, systems, and computer program products according to aspects of the disclosure. The functions/acts noted in the blocks may occur out of the order as shown in any flowchart. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved.
The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure as claimed in any way. The aspects, examples, and details provided in this application are considered sufficient to convey possession and enable others to make and use the best mode of claimed disclosure. The claimed disclosure should not be construed as being limited to any aspect, example, or detail provided in this application. Regardless of whether shown and described in combination or separately, the various features (both structural and methodological) are intended to be selectively included or omitted to produce an embodiment with a particular set of features. Having been provided with the description and illustration of the present application, one skilled in the art may envision variations, modifications, and alternate aspects falling within the spirit of the broader aspects of the general inventive concept embodied in this application that do not depart from the broader scope of the claimed disclosure.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6314447 | Lea | Nov 2001 | B1 |
| 6927770 | Ording | Aug 2005 | B2 |
| 7925022 | Jung et al. | Apr 2011 | B2 |
| 7970870 | Hinckley et al. | Jun 2011 | B2 |
| 8164461 | Bischoff | Apr 2012 | B2 |
| 8332755 | Zhang | Dec 2012 | B2 |
| 8650280 | Hu et al. | Feb 2014 | B2 |
| 9386264 | Dureau | Jul 2016 | B2 |
| 9801219 | Sonnino et al. | Oct 2017 | B2 |
| 20050049850 | Porter | Mar 2005 | A1 |
| 20070168471 | Childress | Jul 2007 | A1 |
| 20090122333 | Kato | May 2009 | A1 |
| 20110055380 | Yockey et al. | Mar 2011 | A1 |
| 20120254965 | Parker | Oct 2012 | A1 |
| 20150153571 | Ballard et al. | Jun 2015 | A1 |
| 20170237820 | Scarborough | Aug 2017 | A1 |
| 20170300365 | Liu et al. | Oct 2017 | A1 |
| 20170357534 | Gupta et al. | Dec 2017 | A1 |
| 20180373568 | Frost | Dec 2018 | A1 |
| 20190286480 | Park | Sep 2019 | A1 |
| 20190294452 | Herring, Jr. | Sep 2019 | A1 |
| 20190373102 | Weinstein | Dec 2019 | A1 |
| 20190387820 | Gellis | Dec 2019 | A1 |
| 20200042323 | Sano | Feb 2020 | A1 |
| 20200067999 | Vianson | Feb 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 2012087767 | Jun 2012 | WO |
| 2013132144 | Sep 2013 | WO |
| Entry |
|---|
| “Pair Google Home Max speakers”, Retrieved From: https://support.google.com/googlehome/answer/7559493?hl=en, Retrieved Date: Jan. 14, 2019, 3 Pages. |
| Fedewa, Joe, “Insignia Smart Speaker gets Speaker Pairing like the Google Home Max”, Retrieved From: https://www.xda-developers.com/insignia-speaker-pairing-google-home/, Published Date: Oct. 22, 2018, 11 Pages. |
| Wagoner, Ara, “How to use Tasker to automate Bluetooth connections”, Retrieved From: https://www.androidcentral.com/moto-g6-shouldve-been-android-one-device, Published Date: Sep. 14, 2017,11 Pages. |
| “International Search Report and Written Opinion Issued in PCT Patent Application No. PCT/US20/017110”, dated May 8, 2020, 15 Pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20200280616 A1 | Sep 2020 | US |
