Patent Application
20190087205
Mobile computing devices such as smartphones and tablets are typically configured with graphical icons on their graphical user interfaces (GUI), which can provide a variety of useful and convenient features for computing device users.
A digital assistant or other suitable functionality supported on a computing device such as a smartphone, tablet, personal computer (PC), media player, wearable computing device including smartwatches and head-mounted display (HIVID) devices, and the like is configured to automatically adjust a modality of a display responsively to context and/or interactions with a device user. A modality may be a configuration of graphical elements such as icons as well as a configuration of the overall graphical user interface (GUI) supported on a display on a user's device. Factors that may affect a modality may include one or more of type, size, color, position, appearance, orientation, and animation of graphical elements. The digital assistant may analyze applicable context associated with a user and/or one or more devices to determine a modality that enhances the coherence and utility of a given user experience. Various adjustments made to icon types and characteristics (e.g., size, shape, etc.) can further increase the user's ease in operating a device. In addition, disfavored modalities can be expeditiously transitioned out of user experiences based on crowd-sourced data from a group of unique users. Varying the modality to fit the applicable context can further enable limited resources such as a battery power and network bandwidth to be conserved to thereby improve device operation.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. It will be appreciated that the above-described subject matter may be implemented as a computer-controlled apparatus, a computer process, a computing system, or as an article of manufacture such as one or more computer-readable storage media. These and various other features will be apparent from a reading of the following Detailed Description and a review of the associated drawings.
Like reference numerals indicate like elements in the drawings. Elements are not drawn to scale unless otherwise indicated.
The modality of user experiences with mobile devices can be varied to enhance ease of use and improve the device performance in a variety of different scenarios. In one illustrative scenario, the digital assistant may observe, with notice to the user and user consent, calendar and location data associated with the user. Based on the observations, the digital assistant determines that the user has a flight scheduled at the airport and the user is currently driving toward the airport. The digital assistant may automatically pre-load a new modality that shows various travel icons available to the user, such as applications related to travel and the user's ticket. The pre-loaded modality may be exposed on the device when the user arrives at the airport. As an alternative, the digital assistant may immediately load and display the new modality upon arrival.
The new modality may be a completely new GUI experience on the user's display in some cases. For example, the new modality may divide the device's display screen into one or more regions, such as an active region and a classic region. The digital assistant may position the new modality in an active region of the display and position unrelated icons in the classic region. The classic region of the display can depict a generic configuration of the GUI (e.g., the typical GUI selected by the user). Since the active region includes a modality based on the context data, the active region may alternatively be designated as a context-based region. When the digital assistant creates and loads the new modality into the active (or context-based) region, the active region may be positioned in front of the classic region or otherwise be given more prominence than the classic region.
Continuing with this illustrative scenario, prior to the airport arrival, the user may request from the digital assistant information regarding services such as food and concessions that are available at the airport. In response, the digital assistant may provide the user with responsive service information, through voice, sounds, graphics, or text. Based on the user interaction, the digital assistant may supplement and adjust the pre-loaded modality that originally only included travel and ticket applications. For example, the digital assistant may adjust the pre-loaded modality to include a food and dining application and links to one or more websites for available restaurants.
The digital assistant may also adjust the presentation of the modality based on the context data. For example, if available context indicates low temperatures from cold weather, the digital assistant may logically infer that the user may be wearing gloves. The digital assistant may increase the sizes of the displayed icons to accommodate the increased surface area of the gloves and make it easier for the user to make selections. Such changes to the GUI may be considered a new modality since each modality is based on different context.
Subsequent to or contemporaneously with providing the user with the new modality, the digital assistant may continue to pre-load various modalities throughout the day. By intelligently loading and pre-loading useful icons in accommodating configurations based on a given scenario, users are provided with greater ease of use and convenience when using their mobile device. Users can also typically spend less effort navigating through the GUI on their device to find the icons of interest which can reduce device resource utilization to thereby make device operations more efficient.
In an illustrative embodiment, the user may provide feedback to the digital assistant regarding a modality. The digital assistant can then analyze and apply the feedback to other applicable modalities to further tailor the modality to particular context and use scenarios. Changes or adjustments in configuration may include, for example, changes in type of icons, color, position, size, shape, etc. Thus, if a user chooses to adjust a particular modality, such as shrink or expand the size of a given icon, then the digital assistant may identify other similar or applicable modalities (created previously or in the future), and apply that adjustment of size to the other modalities.
In another illustrative embodiment, a digital assistant service supported on a remote server may interoperate with a plurality of devices associated with a group or universe of unique users. Each of the unique users may adjust a given modality characteristic, such as icon size, which is transmitted to the digital assistant service. When the service identifies that a threshold number of users have made corresponding adjustments to the GUI or elements therein, the service may store an update to that modality to reflect such adjustment. In addition, the service may implement that adjustment across other similar or applicable modalities that it supports. Even further, the service can implement the adjustment to new users or existing users who have not provided feedback. Such crowd-sourced data from the universe of other users can thus be used to provide finely-tuned and relevant user experiences across a range of different users.
Turning now to the drawings,
For example, applications represented by graphical elements such as icons 120 across the displays 115 of devices 110 in
The applications and GUI interfaces may interoperate with the OS layer 215. The OS layer can manage the system and resources 240, provide the GUI 245 for the user, and control operation of applications 250. Thus, if a user selects an application, the OS layer can execute the functions and processes associated with the selected application, such as enabling the calculator, maps, or digital assistant to execute on the device. In addition, the OS layer may interoperate with the hardware layer 220 and manage the various hardware components. For example, the hardware layer can include abstractions of one or more processors 255 such as central processing units (CPU) and graphic processing units (GPU), memory 260 (e.g., hard-disk drive, flash memory, etc.), and also user input/output devices such as a pointing device (e.g., mouse) 265, or microphone 270. Other input/output devices (not shown) may include a keyboard, touch screen display, and speakers.
The digital assistant may include a modality module 320 that is configured to store, create, add, delete, and adjust various modalities to display on a user's device. The modality module may be a component of the digital assistant that operates locally and/or remotely.
Alternatively, the modality module may be separately instantiated from the device and digital assistant, for example, by independent operation on a server, as illustratively depicted by the remote modality service 325. Therefore, the adjustment, creation, deletion, etc. of modalities may be implemented by any one or combination of the local or remote digital assistants along with the local or remote modality service, as may be needed to a suit a particular embodiment.
The taxonomy of modality adjustments may be part of instructions 470 stored in memory 465 of the device 110, which is executable by one or more processors (
In other exemplary embodiments, the digital assistant may create a single region that reconfigures a default modality such as the typical modality selected by the user. For example, the new modality may position relevant icons in a prominent manner within the default modality, and use additional space for the icons in the default modality.
The letters (i.e., W, E, T, and A) depicted in the active regions of the example modalities 515 and 520 represent different categories that may be grouped together in certain embodiments. For example,
For example, as user 130 is a passenger at an airport, the digital assistant creates a particular modality with various travel icons in an active region 505 of the display, as shown in
The device associated with user 905 depicts various communication icons, such as a messenger and phone. The user 905 is picking up a passenger so therefore the applicable context analyzed by the digital assistant is different. In this scenario, the communication applications may be displayed to support convenient communications between the user 905 and the flight passenger that the user awaits to pick up.
The illustrative modality shown in
For example, as shown in the modality 1805 in
Modality 1810 in
The adaptive varying of modalities based on context as shown in
In an embodiment, virtual objects imposed by the HIVID device in a real world canvas may vary the size of the objects depending on the time of day based on the understanding that users become tired as the day progresses. For example, virtual objects that are relatively smaller early in the morning and in the afternoon may be rendered relatively larger as the evening approaches to accommodate a user's potentially tired eyes. In another example, if sensor data detects that the user is walking or running, virtual objects may be changed or adjusted to accommodate the moving conditions. The HIVID device may enlarge the rendered virtual objects, emphasize the virtual object (e.g., highlight, change color, animate, bold), move the virtual object to the center of the display so the user does not need to search for the object, and the like.
When the service determines that the threshold is met, the service transmits the update (e.g., feedback) to devices of the other unrelated users 2405. The unrelated users may be users who have not utilized the adjusted modality or users who have not provided feedback regarding the adjusted modality. In addition, the service may apply the feedback to related modalities (e.g., similar size, position, or color of icon) as well. When the service adjusts related modalities then the related modalities may also be transmitted to the devices of the other unrelated users.
At step 2520, the service may adjust the modality associated with the corresponding feedback when the pre-set threshold is satisfied. At step 2525, the service may adjust applicable modalities with the corresponding feedback. Applicable modalities may include modalities that have similar features as the adjusted modality, such as shape, color, size, or position of icon. At step 2530, the service transmits the adjusted modalities to the group of users and other users unassociated with the feedback. The users unassociated with the feedback may be users who have not yet provided feedback for that modality or have not yet utilized that modality.
In step 2605, a classic region on a display is rendered, in which the classic region is populated with icons in a GUI. In step 2610, interactions between a user and a digital assistant are monitored. In step 2615, the digital assistant periodically collects context data associated with the user or the computing device. The context data can at least partially be collected from the interactions between the digital assistant and the user. In step 2620, a modality is identified based on the monitored context data. The modality includes a configuration of the GUI that is different from the classic region. In step 2625, the modality is pre-loaded. In step 2630, an active region is rendered on the display using the pre-loaded modality.
A number of program modules may be stored on the hard disk 2928, magnetic disk 2930, optical disk 2938, ROM 2917, or RAM 2921, including an operating system 2955, one or more application programs 2957, other program modules 2960, and program data 2963. A user may enter commands and information into the computer system 2900 through input devices such as a keyboard 2966 and pointing device 2968 such as a mouse. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, trackball, touchpad, touchscreen, touch-sensitive device, voice-command module or device, user motion or user gesture capture device, or the like. These and other input devices are often connected to the processor 2905 through a serial port interface 2971 that is coupled to the system bus 2914, but may be connected by other interfaces, such as a parallel port, game port, or universal serial bus (USB). A monitor 2973 or other type of display device is also connected to the system bus 2914 via an interface, such as a video adapter 2975. In addition to the monitor 2973, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. The illustrative example shown in
The computer system 2900 is operable in a networked environment using logical connections to one or more remote computers, such as a remote computer 2988. The remote computer 2988 may be selected as another personal computer, a server, a router, a network PC, a peer device, or other common network node, and typically includes many or all of the elements described above relative to the computer system 2900, although only a single representative remote memory/storage device 2990 is shown in
When used in a LAN networking environment, the computer system 2900 is connected to the local area network 2993 through a network interface or adapter 2996. When used in a WAN networking environment, the computer system 2900 typically includes a broadband modem 2998, network gateway, or other means for establishing communications over the wide area network 2995, such as the Internet. The broadband modem 2998, which may be internal or external, is connected to the system bus 2914 via a serial port interface 2971. In a networked environment, program modules related to the computer system 2900, or portions thereof, may be stored in the remote memory storage device 2990. It is noted that the network connections shown in
The architecture 3000 illustrated in
The mass storage device 3012 is connected to the CPU 3002 through a mass storage controller (not shown) connected to the bus 3010.The mass storage device 3012 and its associated computer-readable storage media provide non-volatile storage for the architecture 3000.
Although the description of computer-readable storage media contained herein refers to a mass storage device, such as a hard disk or CD-ROM drive, it may be appreciated by those skilled in the art that computer-readable storage media can be any available storage media that can be accessed by the architecture 3000.
By way of example, and not limitation, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. For example, computer-readable media includes, but is not limited to, RAM, ROM, EPROM (erasable programmable read only memory), EEPROM (electrically erasable programmable read only memory), Flash memory or other solid state memory technology, CD-ROM, DVDs, HD-DVD (High Definition DVD), Blu-ray, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the architecture 3000.
According to various embodiments, the architecture 3000 may operate in a networked environment using logical connections to remote computers through a network. The architecture 3000 may connect to the network through a network interface unit 3016 connected to the bus 3010. It may be appreciated that the network interface unit 3016 also may be utilized to connect to other types of networks and remote computer systems. The architecture 3000 also may include an input/output controller 3018 for receiving and processing input from a number of other devices, including a keyboard, mouse, or electronic stylus (not shown in
It may be appreciated that the software components described herein may, when loaded into the CPU 3002 and executed, transform the CPU 3002 and the overall architecture 3000 from a general-purpose computing system into a special-purpose computing system customized to facilitate the functionality presented herein. The CPU 3002 may be constructed from any number of transistors or other discrete circuit elements, which may individually or collectively assume any number of states. More specifically, the CPU 3002 may operate as a finite-state machine, in response to executable instructions contained within the software modules disclosed herein. These computer-executable instructions may transform the CPU 3002 by specifying how the CPU 3002 transitions between states, thereby transforming the transistors or other discrete hardware elements constituting the CPU 3002.
Encoding the software modules presented herein also may transform the physical structure of the computer-readable storage media presented herein. The specific transformation of physical structure may depend on various factors, in different implementations of this description. Examples of such factors may include, but are not limited to, the technology used to implement the computer-readable storage media, whether the computer-readable storage media is characterized as primary or secondary storage, and the like. For example, if the computer-readable storage media is implemented as semiconductor-based memory, the software disclosed herein may be encoded on the computer-readable storage media by transforming the physical state of the semiconductor memory. For example, the software may transform the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. The software also may transform the physical state of such components in order to store data thereupon.
As another example, the computer-readable storage media disclosed herein may be implemented using magnetic or optical technology. In such implementations, the software presented herein may transform the physical state of magnetic or optical media, when the software is encoded therein. These transformations may include altering the magnetic characteristics of particular locations within given magnetic media. These transformations also may include altering the physical features or characteristics of particular locations within given optical media to change the optical characteristics of those locations. Other transformations of physical media are possible without departing from the scope and spirit of the present description, with the foregoing examples provided only to facilitate this discussion.
In light of the above, it may be appreciated that many types of physical transformations take place in the architecture 3000 in order to store and execute the software components presented herein. It also may be appreciated that the architecture 3000 may include other types of computing devices, including handheld computers, embedded computer systems, smartphones, PDAs, and other types of computing devices known to those skilled in the art. It is also contemplated that the architecture 3000 may not include all of the components shown in
The illustrated device 110 can include a controller or processor 3110 (e.g., signal processor, microprocessor, microcontroller, ASIC (Application Specific Integrated Circuit), or other control and processing logic circuitry) for performing such tasks as signal coding, data processing, input/output processing, power control, and/or other functions. An operating system 3112 can control the allocation and usage of the components 3102, including power states, above-lock states, and below-lock states, and provides support for one or more application programs 3114. The application programs can include common mobile computing applications (e.g., image-capture applications, e-mail applications, calendars, contact managers, web browsers, messaging applications), or any other computing application.
The illustrated device 110 can include memory 3120. Memory 3120 can include non-removable memory 3122 and/or removable memory 3124. The non-removable memory 3122 can include RAM, ROM, Flash memory, a hard disk, or other well-known memory storage technologies. The removable memory 3124 can include Flash memory or a Subscriber Identity Module (SIM) card, which is well known in GSM (Global System for Mobile communications) systems, or other well-known memory storage technologies, such as “smart cards.” The memory 3120 can be used for storing data and/or code for running the operating system 3112 and the application programs 3114. Example data can include web pages, text, images, sound files, video data, or other data sets to be sent to and/or received from one or more network servers or other devices via one or more wired or wireless networks.
The memory 3120 may also be arranged as, or include, one or more computer-readable storage media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, Flash memory or other solid state memory technology, CD-ROM (compact-disc ROM), DVD, (Digital Versatile Disc) HD-DVD (High Definition DVD), Blu-ray, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the device 110.
The memory 3120 can be used to store a subscriber identifier, such as an International Mobile Subscriber Identity (IMSI), and an equipment identifier, such as an International Mobile Equipment Identifier (IMEI). Such identifiers can be transmitted to a network server to identify users and equipment. The device 110 can support one or more input devices 3130—such as a touchscreen 3132; microphone 3134 for implementation of voice input for voice recognition, voice commands and the like; camera 3136; physical keyboard 3138; trackball 3140; and/or proximity sensor 3142; and one or more output devices 3150- such as a speaker 3152 and one or more displays 3154. Other input devices (not shown) using gesture recognition may also be utilized in some cases. Other possible output devices (not shown) can include piezoelectric or haptic output devices. Some devices can serve more than one input/output function. For example, touchscreen 3132 and display 3154 can be combined into a single input/output device.
A wireless modem 3160 can be coupled to an antenna (not shown) and can support two-way communications between the processor 3110 and external devices, as is well understood in the art. The modem 3160 is shown generically and can include a cellular modem for communicating with the mobile communication network 3104 and/or other radio-based modems (e.g., Bluetooth® 3164 or Wi-Fi 3162). The wireless modem 3160 is typically configured for communication with one or more cellular networks, such as a GSM network for data and voice communications within a single cellular network, between cellular networks, or between the device and a public switched telephone network (PSTN).
The device can further include at least one input/output port 3180, a power supply 3182, a satellite navigation system receiver 3184, such as a GPS receiver, an accelerometer 3186, a gyroscope (not shown), and/or a physical connector 3190, which can be a USB port, IEEE 1394 (FireWire) port, and/or an RS-232 port. The illustrated components 3102 are not required or all-inclusive, as any components can be deleted and other components can be added.
A graphics processing unit (GPU) 3208 and a video encoder/video codec (coder/decoder) 3214 form a video processing pipeline for high speed and high resolution graphics processing. Data is carried from the GPU 3208 to the video encoder/video codec 3214 via a bus. The video processing pipeline outputs data to an A/V (audio/video) port 3240 for transmission to a television or other display. A memory controller 3210 is connected to the GPU 3208 to facilitate processor access to various types of memory 3212, such as, but not limited to, a RAM.
The multimedia system 110 includes an I/O controller 3220, a system management controller 3222, an audio processing unit 3223, a network interface controller 3224, a first USB (Universal Serial Bus) host controller 3226, a second USB controller 3228, and a front panel I/O subassembly 3230 that are preferably implemented on a module 3218. The USB controllers 3226 and 3228 serve as hosts for peripheral controllers 3242(1) and 3242(2), a wireless adapter 3248, and an external memory device 3246 (e.g., Flash memory, external CD/DVD ROM drive, removable media, etc.). The network interface controller 3224 and/or wireless adapter 3248 provide access to a network (e.g., the Internet, home network, etc.) and may be any of a wide variety of various wired or wireless adapter components including an Ethernet card, a modem, a Bluetooth® module, a cable modem, or the like.
System memory 3243 is provided to store application data that is loaded during the boot process. A media drive 3244 is provided and may comprise a DVD/CD drive, hard drive, or other removable media drive, etc. The media drive 3244 may be internal or external to the multimedia system 110. Application data may be accessed via the media drive 3244 for execution, playback, etc. by the multimedia system 110. The media drive 3244 is connected to the I/O controller 3220 via a bus, such as a Serial ATA bus or other high speed connection (e.g., IEEE 1394).
The system management controller 3222 provides a variety of service functions related to assuring availability of the multimedia system 110. The audio processing unit 3223 and an audio codec 3232 form a corresponding audio processing pipeline with high fidelity and stereo processing. Audio data is carried between the audio processing unit 3223 and the audio codec 3232 via a communication link. The audio processing pipeline outputs data to the A/V port 3240 for reproduction by an external audio player or device having audio capabilities.
The front panel I/O subassembly 3230 supports the functionality of the power button 3250 and the eject button 3252, as well as any LEDs (light emitting diodes) or other indicators exposed on the outer surface of the multimedia system 110. A system power supply module 3239 provides power to the components of the multimedia system 110. A fan 3238 cools the circuitry within the multimedia system 110.
The CPU 3201, GPU 3208, memory controller 3210, and various other components within the multimedia system 110 are interconnected via one or more buses, including serial and parallel buses, a memory bus, a peripheral bus, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures can include a Peripheral Component Interconnects (PCI) bus, PCI-Express bus, etc.
When the multimedia system 110 is powered ON, application data may be loaded from the system memory 3243 into memory 3212 and/or caches 3202 and 3204 and executed on the CPU 3201. The application may present a graphical user interface that provides a consistent user experience when navigating to different media types available on the multimedia system 110. In operation, applications and/or other media contained within the media drive 3244 may be launched or played from the media drive 3244 to provide additional functionalities to the multimedia system 110.
The multimedia system 110 may be operated as a standalone system by simply connecting the system to a television or other display. In this standalone mode, the multimedia system 110 allows one or more users to interact with the system, watch movies, or listen to music. However, with the integration of broadband connectivity made available through the network interface controller 3224 or the wireless adapter 3248, the multimedia system 110 may further be operated as a participant in a larger network community.
When the multimedia system 110 is powered ON, a set amount of hardware resources are reserved for system use by the multimedia system operating system. These resources may include a reservation of memory (e.g., 16 MB), CPU and GPU cycles (e.g., 5%), networking bandwidth (e.g., 8 kbps), etc. Because these resources are reserved at system boot time, the reserved resources do not exist from the application's view.
In particular, the memory reservation preferably is large enough to contain the launch kernel, concurrent system applications, and drivers. The CPU reservation is preferably constant such that if the reserved CPU usage is not used by the system applications, an idle thread will consume any unused cycles.
With regard to the GPU reservation, lightweight messages generated by the system applications (e.g., pop-ups) are displayed by using a GPU interrupt to schedule code to render pop-ups into an overlay. The amount of memory needed for an overlay depends on the overlay area size and the overlay preferably scales with screen resolution. Where a full user interface is used by the concurrent system application, it is preferable to use a resolution independent of application resolution. A scaler may be used to set this resolution such that the need to change frequency and cause a TV re-sync is eliminated.
After the multimedia system 110 boots and system resources are reserved, concurrent system applications execute to provide system functionalities. The system functionalities are encapsulated in a set of system applications that execute within the reserved system resources described above. The operating system kernel identifies threads that are system application threads versus gaming application threads. The system applications are preferably scheduled to run on the CPU 3201 at predetermined times and intervals in order to provide a consistent system resource view to the application. The scheduling is to minimize cache disruption for the gaming application running on the multimedia system.
When a concurrent system application requires audio, audio processing is scheduled asynchronously to the gaming application due to time sensitivity. A multimedia system application manager (described below) controls the gaming application audio level (e.g., mute, attenuate) when system applications are active.
Input devices (e.g., controllers 3242(1) and 3242(2)) are shared by gaming applications and system applications. The input devices are not reserved resources, but are to be switched between system applications and the gaming application such that each will have a focus of the device. The application manager preferably controls the switching of input stream, without knowledge of the gaming application's knowledge and a driver maintains state information regarding focus switches.
The display system 3300 may further include a gaze detection subsystem 3310 configured for detecting a direction of gaze of each eye of a user or a direction or location of focus, as described above. Gaze detection subsystem 3310 may be configured to determine gaze directions of each of a user's eyes in any suitable manner. For example, in the illustrative example shown, a gaze detection subsystem 3310 includes one or more glint sources 3312, such as infrared light sources, that are configured to cause a glint of light to reflect from each eyeball of a user, and one or more image sensors 3314, such as inward-facing sensors, that are configured to capture an image of each eyeball of the user. Changes in the glints from the user's eyeballs and/or a location of a user's pupil, as determined from image data gathered using the image sensor(s) 3314, may be used to determine a direction of gaze.
In addition, a location at which gaze lines projected from the user's eyes intersect the external display may be used to determine an object at which the user is gazing (e.g. a displayed virtual object and/or real background object). Gaze detection subsystem 3310 may have any suitable number and arrangement of light sources and image sensors. In some implementations, the gaze detection subsystem 3310 may be omitted.
The display system 3300 may also include additional sensors. For example, display system 3300 may comprise a global positioning system (GPS) subsystem 3316 to allow a location of the display system 3300 to be determined. This may help to identify real-world objects, such as buildings, etc. that may be located in the user's adjoining physical environment.
The display system 3300 may further include one or more motion sensors 3318 (e.g., inertial, multi-axis gyroscopic, or acceleration sensors) to detect movement and position/orientation/pose of a user's head when the user is wearing the system as part of an augmented reality or virtual reality HMD device. Motion data may be used, potentially along with eye-tracking glint data and outward-facing image data, for gaze detection, as well as for image stabilization to help correct for blur in images from the outward-facing image sensor(s) 3306. The use of motion data may allow changes in gaze direction to be tracked even if image data from outward-facing image sensor(s) 3306 cannot be resolved.
In addition, motion sensors 3318, as well as microphone(s) 3308 and gaze detection subsystem 3310, also may be employed as user input devices, such that a user may interact with the display system 3300 via gestures of the eye, neck and/or head, as well as via verbal commands in some cases. It may be understood that sensors illustrated in
The display system 3300 can further include a controller 3320 having a logic subsystem 3322 and a data storage subsystem 3324 in communication with the sensors, gaze detection subsystem 3310, display subsystem 3304, and/or other components through a communications subsystem 3326. The communications subsystem 3326 can also facilitate the display system being operated in conjunction with remotely located resources, such as processing, storage, power, data, and services. That is, in some implementations, an HIVID device can be operated as part of a system that can distribute resources and capabilities among different components and subsystems.
The storage subsystem 3324 may include instructions stored thereon that are executable by logic subsystem 3322, for example, to receive and interpret inputs from the sensors, to identify location and movements of a user, to identify real objects using surface reconstruction and other techniques, and dim/fade the display based on distance to objects so as to enable the objects to be seen by the user, among other tasks.
The display system 3300 is configured with one or more audio transducers 3328 (e.g., speakers, earphones, etc.) so that audio can be utilized as part of an augmented reality or virtual reality experience. A power management subsystem 3330 may include one or more batteries 3332 and/or protection circuit modules (PCMs) and an associated charger interface 3334 and/or remote power interface for supplying power to components in the display system 3300.
It may be appreciated that the display system 3300 is described for the purpose of example, and thus is not meant to be limiting. It may be further understood that the display device may include additional and/or alternative sensors, cameras, microphones, input devices, output devices, etc. than those shown without departing from the scope of the present arrangement. Additionally, the physical configuration of a display device and its various sensors and subcomponents may take a variety of different forms without departing from the scope of the present arrangement.
Various exemplary embodiments of the present varying modality of user experiences with a mobile device based on context are now presented by way of illustration and not as an exhaustive list of all embodiments. An example includes one or more hardware-based computer-readable memory devices storing instructions which, when executed by one or more processors disposed in a computing device, cause the computing device to: render a classic region on a display, the classic region being populated with icons in a graphical user interface (GUI); and monitor a digital assistant that interacts with a user, wherein the digital assistant is configured to: periodically collect context data associated with the user or the computing device, and wherein the context data is at least partially collected from the interactions between the digital assistant and the user, identify a modality based on the monitored context data, wherein the modality includes a configuration of the GUI that is different from the classic region, pre-load the modality, and render an active region using the pre-loaded modality on the display.
In another example, the classic region is displayed while the modality is pre-loaded. In another example, the context data includes one or more of sensor data, calendar data, contact data, communication data, location data, user interaction with the digital assistant, user feedback, or third-party feedback. In another example, the user interaction with the digital assistant includes one or more of gathering information, answering questions for the user, performing tasks, providing reminders to the user, or searching files. In another example, the digital assistant is further configured to: create an icon based on the user interactions with the digital assistant; and populate the active region with the created icon. In another example, the created icon is website accessible over the Internet. In another example, the digital assistant is further configured to: when a current modality of the active region is rendered, identify a new modality based on the collected context data, wherein the new modality includes a configuration of the GUI that is different from both the active region and the classic region; remove the current modality from the active region of the display; and render the display with the new modality. In another example, the new modality is defined by one or more changes of type, position, size, color, or animation of icons.
A further example includes a method to dynamically update graphical user interface (GUI) configurations of an electronic device, comprising: storing a plurality of modalities associated with a user of the electronic device, wherein each of the plurality of modalities is mapped to a respective contextual scenario associated with the user, and wherein each modality defines a unique combination of GUI elements; upon an occurrence of a contextual scenario, transmitting a communication to the electronic device to responsively select a corresponding mapped modality with which to configure the GUI; receiving user input indicating feedback for at least a portion of the selected modality; identifying non-selected modalities of the plurality of modalities to which the feedback applies; and modifying the identified non-selected modalities using the feedback.
In another example, the feedback comprises user adjustment of one or more of a size, shape, position, color, animation, or description of a GUI element. In another example, the method further comprises: searching for non-selected modalities associated with an unrelated user; and applying the feedback to the non-selected modalities of the unrelated user. In another example, the user is in a group of users, each member of the group being associated with respective one or more unique devices, and the method further comprising: receiving user input from the group of users describing feedback for the selected modality; identifying a pattern in the feedback; and when a pattern is identified, providing the feedback to devices associated with one or more users that are not in the group. In another example, the identified pattern is provided to the devices when a number of users meeting a predetermined threshold have submitted feedback.
A further example includes a mobile computing device configured for utilization by a user, comprising: one or more processors; a display device supporting a graphical user interface (GUI); and one or more hardware-based memory devices storing a plurality of applications and further storing computer-readable instructions which, when executed by the one or more processors cause the mobile computing device to: identify categories for icons based on applications with which respective icons are associated; configure the GUI rendered on the display device into a context-based region and a non-context-based region; develop contextual data using responsive interactions of the digital assistant with the user; determine a subset of icons based on the developed contextual data, wherein the subset includes icons from different categories; present the subset of icons in the context-based region of the display device; and hold remaining icons not in the subset in the non-context-based region of the display device in a configuration that remains static as the contextual data is developed.
In another example, the categories include one or more of fitness, news, entertainment, finance, food, business, travel, sports, photo and video, shopping, or weather. In another example, the developed contextual data is based on the user interacting with the digital assistant using voice, gestures, or physical interaction. In another example, the executed instructions further cause the mobile computing device to iteratively change presentation of the subset of icons according to changes in the developed contextual data, in which each change of presentation of the subset of icons comprises a new modality. In another example, the changes in presentation include a change of icons in the subset or one or more icons within the subset changing in one or more of size, color, position, orientation, or animation. In another example, the executed instructions further cause the mobile computing device to add an icon into the context-based region upon an occurrence of an event, wherein the event is unrelated to the contextual data. In another example, the event is one or more of receiving a message or receiving a notification associated with an application.
The subject matter described above is provided by way of illustration only and is not to be construed as limiting. Various modifications and changes may be made to the subject matter described herein without following the example embodiments and applications illustrated and described, and without departing from the true spirit and scope of the present invention, which is set forth in the following claims.
