Patent Application
20090177601
In contemporary computing, many user interface outputs are not particularly user-friendly with respect to how and when they appear. In general, such user interfaces are system (or machine) driven instead of user-centric.
By way of example, there are currently many kinds of information delivered to computer users from time to time without considering whether the user and his and/or her machines are currently busy. For example, when a user is busy composing a document, a computer system may download security updates and remind the user to update the system with them. When in a business meeting or giving a presentation, instant messages and other personal and/or irrelevant reminders may keep popping up. This is not only distracting, but also may lead to personal information being viewable by others.
This Summary is provided to introduce a selection of representative concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in any way that would limit the scope of the claimed subject matter.
Briefly, various aspects of the subject matter described herein are directed towards a technology by which state data including user presence and/or user attention data is used to determine whether a set of information is to be output as corresponding data to a selected candidate output modality (e.g., of a plurality of output mechanisms). In this way, personal information is intelligently managed with respect to when and how it is output to a user.
Information may be received from a remote source, from a reminder source, and may have been previously cached for later possible output. The user presence and attention data may comprise audio signals, video signals, keyboard and/or mouse activity signals and/or calendar data. Output types corresponding to output modalities may include a display for text/graphics/video/animation, a speaker for audio output (e.g., text to speech, tones and so forth), a vibration mechanism, a storage, another device, and so forth.
In one aspect, a set of information is processed to determine whether to output corresponding data, including by computing a cost associated with outputting the data to a selected candidate modality. A lowest cost path analysis may be used. Further, various costs may be computed and compared, e.g., a first information cost and a second cost of outputting the data on a selected candidate modality, to determine whether to output the data to that modality. User state, other state, conversion cost (e.g., text to speech), and user preferences and other data may be considered in the computation. If output is determined, the output data may be modified based on a current output environment as determined via the state data.
In one example implementation, a manager component processes sets of information and obtains state data including user presence and attention data. An output mechanism coupled to the manager component is selected and the manager component determines whether to output data corresponding to a set of incoming information to that selected output mechanism based on the current user state as determined from the presence and attention data. The manager component may be coupled to logic that routes information to at least one of the output mechanisms and converts the information into a suitable output type as appropriate for each output mechanism. The manager component may be coupled to further logic that determines from the state data whether, and if so when, to output the data corresponding to the information to each output mechanism. The manager component may compute costs related to outputting data corresponding to the information to the output mechanisms.
Other advantages may become apparent from the following detailed description when taken in conjunction with the drawings.
The present invention is illustrated by way of example and not limited in the accompanying figures in which like reference numerals indicate similar elements and in which:
Various aspects of the technology described herein are generally directed towards managing information coming into a computer system and providing a corresponding output through one or more appropriate modalities and at an appropriate time. The management is based upon a user's current state (status), and possibly other data (e.g., user preference data, importance of the incoming information, and so forth). There is thus provided status-aware personal information management that operates in what is perceived to be an intelligent, or smart, manner. In general, instead of a system-driven message/information broadcast system, there is provided a more friendly, presence-aware and user-centric communication and information transfer mechanism and process.
In one aspect, smart personal information management generally acts as an information filter. However, beyond merely removing truly unnecessary information such as spam and other junk email messages, smart personal information management further delivers filtered information to a user in an appropriate time with an appropriate modality according to the users' current status. By operating in such a user-centric manner, a more user friendly experience is realized instead of disturbing the user with unnecessary, irrelevant, untimely or otherwise inappropriate information.
While many of the examples herein are described with respect to a personal computer, it is understood that virtually any computing device that is coupled to one or more sensors or can otherwise detect user state information is capable of implementing smart personal information management. For example, a GPS-enabled mobile computing device can determine state information by changing coordinates, e.g., when a device is moving it is known that the device is likely present with the user even at times when the user is not interacting with the device. At such a time, audio and/or vibration may likely provide a better notification output than visible output, whereas visible output may be more appropriate at times when the user is known to be interacting with the device.
As such, the present invention is not limited to any particular embodiments, aspects, concepts, structures, functionalities or examples described herein. Rather, any of the embodiments, aspects, concepts, structures, functionalities or examples described herein are non-limiting, and the present invention may be used various ways that provide benefits and advantages in computing and information management in general.
Turning to
As incoming information 106 arrives, the status aware personal information manager 102 processes the incoming information 106 in conjunction with user preference data (and/or default settings) 108. If not filtered out completely, e.g., as are known junk emails, the status aware personal information manager 102 uses the state data 104 to compute whether (and if so, via what modality or modalities) to generate appropriate state-based output 110 that corresponds to the incoming information and output mechanisms' capabilities, e.g., to output the information 106 (such as text) itself, convert the information 106 to another format (such as text-to-speech) for the output 110, convert the information 106 to one or more types of notifications for the output 110, and so forth.
Further, the status aware personal information manager 102 processes the incoming information 106, the user preference data (and/or default settings) 108 and the state data 104 to determine a time (or a set of times) for the output or outputs. For example, output may be immediate, or may be buffered or otherwise persisted for later output, e.g., as a notification the next time the user is detected as interacting with the computing device. Multiple outputs and times may be generated, e.g., an immediate audible notification may be generated, with a visible output for the same incoming information triggered by some later event.
As generally represented in
Thus, in general, the status aware personal information manager 102 can determine a current status of the user, such as busy (on phone, working on computer, having a discussion in the room with others), not busy (in room alone, off computer), or totally unavailable (away, offline). Note that even when determined to be totally unavailable, the status aware personal information manager 102 may still take some action when incoming data is received, for example to automatically forward data (a message and/or a notification) to a user's mobile device.
By way of other examples, when via the state data 104 the user is deemed busy, the status aware personal information manager 102 can report a user's busy status to other computers, block any incoming information to avoid bothering the user, and/or postpone any messages/reminders until sometime later when the user is available and not as busy. In contrast, when via the state data 104 the user is deemed present but not busy nor viewing the display, the status aware personal information manager 102 may provide audible notifications and/or other distantly noticeable notifications (e.g., by lighting an LED) as information is received.
Also represented in
To this end, in one example implementation the status aware personal information manager 102 includes routing/modalities logic 228 for processing and/or routing incoming data to one or more appropriate output channels so that the appropriate output is received by the output mechanisms. More particularly, in addition to routing the data to the correct output, the logic 228 considers modalities as to how the data is to be output with respect to current state data. For example, the input information may be in the form of text, which can be output as the actual text itself, converted (by text-to-speech, or TTS technology) to speech and output as speech, converted to a notification that indicates text is available, and/or other modes of output. A notification may be in the form of other text or graphics, an illuminated set of one or more indicators such as LEDs (with variable color, flash patterns, intensities and the like), an audible beep (with variable pitches, lengths, intensities and the like), a vibration (with variable vibration patterns and the like), and so forth. If an output is deemed appropriate based on the current user state (and possibly the importance of the information and user preferences), the routing/modality logic 228 selects one or more outputs and converts modalities as appropriate, and routes the corresponding appropriate output to the corresponding output mechanism or mechanisms.
As can be readily appreciated, the state data 104 may change during output, resulting in a change in output modalities. For example, information in the form of an ongoing text chat can be transferred to mobile device via TTS technology when a user leaves the computer but wants to keep the discussion going.
Moreover, output may be conditioned to fit a current environment as detected via current state data 104. For example, a loud environment may be detected, whereby a louder audio output may be generated by the modalities logic 228. Similarly, a bright environment may require a change in backlighting, for example. If a user is currently looking away from the display, a notification may pop up longer than if the user currently looking, and may even remain popped-up until the user looks at the display. However, note that if the user is deemed busy, the notification may not appear at all to avoid distraction, regardless of whether the user is looking at the display. Still further, the type and/or importance of the information may weigh on the decision, e.g., a calendar reminder may pop up, whereas a stock price update may not.
Timing logic 230 is also used to decide when (and whether) to output data. For example, based on detected state that indicates that the user is in the room but not in front of the computer, the timing logic 230 may immediately output an audible sound and illuminate an LED, while buffering display monitor output (e.g., video, text and/or graphics, animation) for display at a time when the user is looking at the display. If a user is not present at all, no output may be generated (although some indication may be output as a failsafe), or output data (or data that will generate an output) may be forwarded to another device.
With respect to software updates, updating a computer may be considered as another form of outputting data (e.g., to a hard drive), and thus may be controlled by the timing logic 230. For example, downloading and/or installing an update or set of updates may be postponed until a time when the user is detected by the presence/attention detection mechanism as not busy or not present or both, possibly after waiting for a certain length of time. Time of day may be a factor, e.g., updates may be postponed until a user is not only not busy and/or present, but is also not likely to resume working, such as at midnight.
Multiple types of output may be generated for the same piece of information, at possibly different times. For example, an audible notification may be output indicating that a text message has been received, with the text of the message later output when the user is looking at the screen. A notification may appear indicating that content is ready for download, with an automatic download performed at a later time. Any system reboot that is required will not automatically occur at a time when the user is interacting with the system.
Turning to
Step 302 represents computing a cost based on the importance and/or relevance of a selected set of information to be processed for possible output, where a lower cost corresponds to more important and/or relevant information; (a higher “weight” is equivalent to a lower cost, and vice-versa). For example, via the preference data 108 a user may indicate that email messages received from a supervisor are highly important/relevant, whereby a low cost may be associated with messages from that supervisor. At the same time, known junk email may be computed to have a very high cost, whereby such a message will be discarded, or alternatively kept but only output to a junk mail folder with no corresponding notification, for example. Such cost data may be computed dynamically or may be maintained in the system, such as part of the preference data 108.
As can be readily appreciated, a cost of a set of information may be computed in virtually any way, e.g., by summing costs determined according to various criteria, including according to user preference data. For example, a computed cost for a set of information may consider its type (e.g., email versus an instant message versus a stock quote versus calendar data), its source (e.g., a supervisor), how old it is (which may be relative to its type), and so forth.
Step 304 represents evaluating whether to discard the selected information, such as based on its cost versus a maximum allowed cost. In this manner, for example, there is avoided further unnecessary computation for very costly information, such as for once-buffered but now expired notifications, junk messages that are automatically discarded, and so forth. If discarded, the example branches to step 324 to looks for whether there is another set of incoming or previously buffered information to process.
If not discarded by step 304, output costs are computed for various output types at step 306 in this example. This may include a cost for switching modalities. For example, there is a computational expense associated with converting text to speech, which thus corresponds to a higher cost than simply leaving text as is for output, which has essentially no cost. There is also a cost associated with converting specific information, such as text, to a simpler type of notification, such as an LED signal, in that only the simpler notification rather than the specific information may be seen by the user, at least for some time, whereby the amount of information that is conveyed is reduced.
Step 308 represents adjusting the information and/or output costs based on the current state data and/or user preference data. For example, if a user is detected as being in the room but not currently at the computer, the cost of displaying output on the monitor may be significantly increased because the user will not likely see it. However, in such a state, the cost of sounding an audible notification will decrease (or remain the same), as will the cost of driving an LED because the user may view the LED from a distance. The cost of routing the information to a mobile device may likewise decrease or remain the same. As another example of state data, when the user is giving a presentation, the cost of displaying visible data text or graphics may be set to the maximum, while the cost of routing the information to a mobile device may be reduced. As a result, the user's mobile device may buzz, but the presentation will not be overlaid with any visible output.
Further, other user preference data may factor into the computed output costs. For example, a user may specify that no instant message notifications are to appear when the user is in a busy state, e.g., as determined by presence/attention data such as a certain amount of interaction with the keyboard and/or mouse sensor (block 2223) in conjunction with being in front of the display sensor (block 2222). The user preference data thus may be used to increase the cost of instant message notifications when in a busy state.
As can be readily appreciated, given a set of costs corresponding to the various criteria, a lattice may be formed, through which a lowest cost path algorithm or the like may be used to select the lowest output cost or costs, with appropriate output generated. However, multiple types of outputs may be desired, as may buffering input information (or corresponding output data) for later output. The remainder of
Step 310 selects a first type of output mechanism from those available, and step 312 represents evaluating the computed cost for the information against the cost of outputting data corresponding to that information on the selected type of output mechanism. Note that user preference data may factor into the computed costs. If the cost comparison results in output being routed to the selected output mechanism, step 314 is executed where the information is output as appropriate, including any modality conversion and/or modification for the current environment (e.g., loud audio in a noisy environment).
By way of example, a text/graphics display may be selected as a candidate output type at step 310, and the computed and adjusted cost for that mechanism compared against the cost computed for the type of information. If the information is already text or graphics there is no modality conversion cost factor, and, if the user is present but not busy, then based on the cost comparison, it is likely that the information will be output as text/graphics (at step 314). However, if the information is speech that needs to be converted to text, and the user is deemed busy, then there is a lesser likelihood that the information will be output as text or graphics. If the next selected output type candidate (step 322) is audio output, however, the conversion cost factor is less and thus there is a greater likelihood that it will be output.
If not output to the selected candidate type, step 316 represents determining whether the information is of a type that is to be buffered for later possible output. Note that such buffering is in addition to conventional program output; for example, an email message that is received is already maintained by the email program, and it is unlikely that a user wants a notification at a later time that it was received awhile ago. However, a user may want a list of instant message notifications that came in but were blocked while the user was absent or busy. Step 318 represents saving information (e.g., the input information, corresponding output data or data from which corresponding output data may be generated) for incoming information that was not output on a particular type of output mechanism but still may be output at a later time. Thus, for example, an LED may flash for information while the user is not in front of the display, indicating that new information is available, with the text of that information buffered for later display (e.g., if still sufficiently timely such that its cost does not grow too high).
Steps 320 and 322 repeat the input information processing for each different type of possible output. Note that although not explicitly shown in
When there are no more types of output to be considered, step 320 branches to step 324 where the process is repeated until no more information remains to be processed. In this manner, sets of new information and/or buffered information are each processed to determine whether and how to output notifications and/or data, as well as whether to buffer information for later possible output. When such information has been processed, the example steps of
The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to: personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and so forth, which perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in local and/or remote computer storage media including memory storage devices.
With reference to
The computer 410 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computer 410 and includes both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by the computer 410. Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer-readable media.
The system memory 430 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 431 and random access memory (RAM) 432. A basic input/output system 433 (BIOS), containing the basic routines that help to transfer information between elements within computer 410, such as during start-up, is typically stored in ROM 431. RAM 432 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 420. By way of example, and not limitation,
The computer 410 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media, described above and illustrated in
The computer 410 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 480. The remote computer 480 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 410, although only a memory storage device 481 has been illustrated in
When used in a LAN networking environment, the computer 410 is connected to the LAN 471 through a network interface or adapter 470. When used in a WAN networking environment, the computer 410 typically includes a modem 472 or other means for establishing communications over the WAN 473, such as the Internet. The modem 472, which may be internal or external, may be connected to the system bus 421 via the user input interface 460 or other appropriate mechanism. A wireless networking component 474 such as comprising an interface and antenna may be coupled through a suitable device such as an access point or peer computer to a WAN or LAN. In a networked environment, program modules depicted relative to the computer 410, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
An auxiliary subsystem 499 (e.g., for auxiliary display of content) may be connected via the user interface 460 to allow data such as program content, system status and event notifications to be provided to the user, even if the main portions of the computer system are in a low power state. The auxiliary subsystem 499 may be connected to the modem 472 and/or network interface 470 to allow communication between these systems while the main processing unit 420 is in a low power state.
While the invention is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the invention to the specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention.
