In existing computing interfaces, the output modality is typically fixed, and needs to be explicitly chosen or changed by the user, such as to adjust to a changed need or situation. For example, if a person is on a private phone call in which the audio comes through a computer's speakers, and another person walks in, the user must take an action to protect privacy, such as to mute the sound or switch to using a headset. In general, the system has no ability to automatically provide the user with options or change the output method.
As users deal with more and more computing concepts and resources, such as cloud computing, devices with more computing power and more sensing technologies, and more interactive displays in public places, users have more options than ever for receiving output. At different times, users want to use different output modalities and have output customized for a desirable end-user experience. However, known technologies do not help users with the selection. What is desirable is output that naturally adapts to a user's implicit or explicit intent and/or a user's real-time environmental conditions (e.g., noise level, light level, presence of other people, presence of other output devices, location and so forth).
This Summary is provided to introduce a selection of representative concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in any way that would limit the scope of the claimed subject matter.
Briefly, various aspects of the subject matter described herein are directed towards a technology by which output data in the form of audio, visual, haptic, and/or other output is automatically selected and tailored by a system, including adapting in real time, to address one or more users' specific needs, context and implicit/explicit intent. To this end, state data along with preference data may be input into a real time adaptive output system. The system selects an output modality or modalities based upon the state data and the preference data, and outputs data corresponding to the selected modality or modalities. The system monitors for a change to the state data to adapt the output modality or modalities based upon a suitable change to the state data.
The output may be rendered on a single output mechanism to a single user, may be a computed consensus output modality based upon the preferences of a plurality of users, or may be rendered via different output mechanisms to different users. To adapt the output modality, the system may change (e.g., add, remove and/or switch) one or more output mechanisms that are in use, and/or may change rendering characteristics (e.g., volume, display properties and so forth) of the output data.
In one implementation, an adaptive output system includes a plurality of output mechanisms that are each available to render output data corresponding to an output modality, an output processor configured to process raw data into output data corresponding to one or more output modalities, and a recommendation engine configured to adaptively determine, based upon state data, one or more intended output modalities according to which the output data is rendered. An intended output modality may correspond to a local or remote output mechanism, and both local and remote output mechanisms may be used. A personalization engine may provide personalization information that the recommendation engine uses in determining the intended output modality. A conversion mechanism may be used to convert raw data or output data in one format to output data in another format.
Other advantages may become apparent from the following detailed description when taken in conjunction with the drawings.
The present invention is illustrated by way of example and not limited in the accompanying figures in which like reference numerals indicate similar elements and in which:
Various aspects of the technology described herein are generally directed towards a technology by which computer sensors detect various conditions (e.g., state data), which are then processed (typically along with other data) to determine what output modality or modalities are intended/desired by a user or set of users. In this manner, the output provided by one or more computing systems automatically and fluidly adapts to the user's or users' current situation.
It should be understood that any of the examples herein are non-limiting. For example, while a computer system, mobile device (e.g., Smartphone), telephone devices and so forth are described as examples of systems that can implement real-time adaptive output, devices other than those exemplified may benefit from the technology described herein. As such, the present invention is not limited to any particular embodiments, aspects, concepts, structures, functionalities or examples described herein. Rather, any of the embodiments, aspects, concepts, structures, functionalities or examples described herein are non-limiting, and the present invention may be used various ways that provide benefits and advantages in computing and user interfaces in general.
As described below, based upon this state data and possibly other information, the adaptive output system 102 selects one or more available output mechanisms 1081-108n, and sends data thereto to output to the user or users. For example, whether a user is typing or speaking may be used as factors in determining the output, as may whether one user or multiple users are to be presented with the output. The rendering of the output may include appropriate conversion or other formatting of the data and other characteristics of the output (e.g., audio volume), if necessary, as represented by conversion mechanism(s) 109. Note that which output mechanisms 1081-108n are currently available is also part of the selection process.
The sensors 1061-106m provide environmental data and/or other sensed data to the adaptive output system 102. More particularly, a computer system has the ability to gather detailed information about its environment and the user or users interacting with the system. Examples of ways to sense such data include via computer vision, microphones, accelerometers, compasses, clocks, GPS, thermometers, humidity sensors, light sensors, infrared sensors, depth sensors, entity extraction and so forth. These can sense environmental and other data, such as current room and/or device temperature, whether the user is moving and at what speed, whether the user is alone or with someone else, the amount of ambient light, computer-related or output device-related data (e.g., device battery life, available power, running programs and services), and so forth.
Further, systems can detect information about their own states and the presence and states of other devices via internal or other systems, as represented in
In addition to state data, a system can also use custom and/or existing or predefined (e.g., default) user profile information, represented in
In general, the adaptive output system 102 processes the various input data and the user preference data 112 to regularly (e.g., continuously) determine the intent of the user or users as to what output mechanism or mechanisms 1081-108n of those available are most desirable to the user or users. Example output mechanisms correspond to a display screen, projection, holographic video or I/O display, speech or other audio through speakers, headset or bone conduction, haptics, smell, position/kinetics (e.g., of robot), proximity, asynchronous communications such as email or text message. Multiple simultaneous outputs (e.g., visual/haptic, visual/aural) or any other combination may be used.
Further, in addition to output mechanism selection, the adaptive output system 102 can determine how to render the output on a given mechanism, e.g., to tailor the output characteristics to current conditions and/or situations. Output rendering options include selecting display colors and/or brightness, resolution, aspect ratio, the volume of audio output, language, literacy level (e.g., different vernacular, phrasing, dictionary and the like), speed of output (e.g., speaking more slowly to a child), and so forth.
Consider the above example of a person on a private phone call on speakerphone in her office. If another person walks in, the adaptive output system 102 detects this state change, and can adapt the call by automatically muting it or switching it from speakerphone to a headset or earpiece, such as according to a user-specified preference or default preference as maintained in the preference data 112. As another example, if a person reading content on a tablet changes his or her state to drive a car, and wants to continue to interact with the content while driving, the adaptive output system 102 may automatically switch to using speech instead of a visual display of the content. If a user has low literacy skills, the system can automatically adapt the content of news articles or websites to the user's reading level.
Another aspect of the adaptive output system 102 is directed towards adapting output to address the needs of more than one user (e.g., consensus output). For example, the system may calculate an optimal output for each individual user, take into account the tasks being done at the time, and apply an algorithm to find an optimal solution that maximizes (or e.g., averages) preference indices across the group constrained by the tasks being done. One user may receive one type of output, e.g., text, while another user may receive another type of output, e.g., speech. The output may be over local and/or remote devices, e.g., one user may read test output on a local display while another user hears the text-to-speech equivalent of that output on a remote Smartphone. In an alternative scenario, for tasks that are driven by one leader, the system may ask the group to designate a leader, or may suggest a group leader. If there are independent tasks being done by individuals in the same location, the system may employ a time and space sharing algorithm that takes into account the equality or inequality of each of the participants and their tasks, and load balances the optimal outputs accordingly. Each of these may be considered consensus output.
Note that existing systems allow for users to take manual actions to make a change to output, or allow for preset accessibility options which the system will apply to a logged in user, but these settings are chosen beforehand. They are thus not automated, or are all-or-nothing configuration settings that are not adaptive to state data changes in real time. Moreover, they do not migrate to other systems based on the user profile.
Also represented in
A recommendation engine 226 processes the various input information/state data and makes a recommendation as to what each user likely intends with respect to what output modality or modalities are desired. An output processor 228, which may be hard-coded to an extent and/or include plug-ins 2291-2295 for handling the output data of various modalities, processes the raw data 218 into formatted output data, as needed, which may be queued in an output event queue 230 for synchronization purposes. As shown in
As part of the processing and queuing, the output processor 228 may communicate with a personalization engine 234, which, as described above, may access preference data 112 and other information to select an output mechanism or mechanisms, as well as format the data for a user or multiple users. This may include changing the output characteristics to provide the desired output modality or modalities to the selected output mechanism or mechanisms. The output may be to one user, over one or more modalities, and may be to multiple users, including over different modalities to different users.
The conversion mechanism 109 may include a text-to-speech engine, speech-to-text engine, dictionaries, entity extraction engines (e.g., to process still images, video, or 3D visuals to convert what is being shown to text or speech), and so forth to format/convert the raw data 220 to the desired output format. For example, the personalization engine 234 may access a custom speech dictionary that is used by the conversion mechanism 109 to convert audio data to text, with the text then queued for output.
Note that the recommendation engine 226 may not necessarily determine the output directly, but in one alternative, may process the data in the output queue 230 only to make recommendations to a program that consumes the queued data. For example, the output data may be provided in a variety of formats and types to a program, with the recommendation engine 226 only suggesting which type of output modality is likely desired. This allows a program or device to override a suggestion, such as on specific request of the user through that program or device.
Note that the system or a receiving program may forward the recommendations to another program. For example, the receiving program may route the recommendations to a remote device, which may choose to use them or not. Consider an example where text and speech is routed to a cell phone that is in a silent mode, with a recommendation that the speech be output; because of the silent mode, the cell phone will only output the text, and buffer or discard the speech data.
Thus, the system enables applications or helps applications to manage multiple output modalities to one or more output mechanisms, to one or more users or groups of users. The output devices may be physically attached or remote, and may be different for different users.
At step 404, the system accesses the available state data and the user preference data. For example, the adaptive output system may first determine the input modalities currently being used, and use existing user profile data to calculate a first guess at what the preferred or most appropriate output or outputs are most likely to be. For example, if a user is typing and there is a desktop monitor and speakers present, it is likely that the visual display and speakers will be the best output.
Based on the initial data, in one implementation the system forms a baseline model, referred to herein as the “user-environment-device-intent landscape” (step 406). There may be a baseline model per user. As described above, this may include the social situation/presence of others, e.g., whether the user is alone or not, or with family, friends, co-workers or unknown people, (such as determined by facial recognition, for example.) This may also include the user's current environment, e.g., in a room, outdoors, on a train, at home, at work, at school, in a hospital, and so forth. The user's geospatial location, explicit and implicit inputs (e.g., what input devices are nearby and being used or not used), the task at hand, the motion (moving or not), time of day, language data may further be evaluated. Still further, other external variables may be considered, such as in a library system that has a no talking rule.
From this information, the adaptive output system creates the user-environment-device-intent landscape that represents each user's intent, as determined by the recommendation engine and/or inferred by user activities, applications, and communications. The system then adapts the output modalities and rendering to match the user's or users' likely intent with respect to receiving output information. Note that the intent may be based on a consensus computation or the like as described above.
At step 410, the system thereafter monitors the state data of the internal and external information sources to update the landscape model in real-time, using reasoning (e.g., based on training) to adapt output modalities including device selection and rendering as needed to optimize the user experience. Note that in the flow diagram, step 412 represents determining whether a change is sufficient enough to return to step 406 to re-determine the intent. For example, a driver may slow down, but this state change is not sufficient to re-compute the landscape model for that user.
In this manner, a computing system may determine one or more optimal output modalities in real-time, across devices, users, and situations. This is based upon a variety of information about the user, such as the user's environment, where the user is looking, devices nearby, existing input and available output modalities, social situation, geospatial location, presence of others, task at hand, physical motion, time of day, language, and other external variables, which are processed to determine the most optimal output modality or modalities (mechanisms and modes for rendering) from the system. The system can adaptively render the output to the optimal locations in the optimal way for ease of use, as appropriate to each user's situation, and adapt as the situation and users or other state changes in real-time.
The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to: personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and so forth, which perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in local and/or remote computer storage media including memory storage devices.
With reference to
The computer 510 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computer 510 and includes both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by the computer 510. Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above may also be included within the scope of computer-readable media.
The system memory 530 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 531 and random access memory (RAM) 532. A basic input/output system 533 (BIOS), containing the basic routines that help to transfer information between elements within computer 510, such as during start-up, is typically stored in ROM 531. RAM 532 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 520. By way of example, and not limitation,
The computer 510 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media, described above and illustrated in
The computer 510 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 580. The remote computer 580 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 510, although only a memory storage device 581 has been illustrated in
When used in a LAN networking environment, the computer 510 is connected to the LAN 571 through a network interface or adapter 570. When used in a WAN networking environment, the computer 510 typically includes a modem 572 or other means for establishing communications over the WAN 573, such as the Internet. The modem 572, which may be internal or external, may be connected to the system bus 521 via the user input interface 560 or other appropriate mechanism. A wireless networking component such as comprising an interface and antenna may be coupled through a suitable device such as an access point or peer computer to a WAN or LAN. In a networked environment, program modules depicted relative to the computer 510, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
While the invention is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the invention to the specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention.