Patent Application
20050015731
This invention generally relates to systems, methods, user interfaces, and computer-readable media for handling, entering, or manipulating data across different portions or regions of a computer desktop. For example, at least some examples of this invention include the ability to alter the content of a first portion or region of a desktop from a second, independent portion or region of the desktop.
As a result of technological advancements and consumer demand, personal computer systems today are more flexible, versatile, and user-friendly than ever. Today, computer users are able to simultaneously run multiple application programs, open and use multiple electronic documents, and move between different programs and/or documents with a simple mouse click or other input action.
Because computers are able to simultaneously run multiple application programs and allow interaction with multiple electronic documents, the ability to organize and present the available information to users in a convenient and useful manner is an important aspect of a computer system. Some computer systems utilize an “electronic desktop” to organize information available on a computer and present this information to the computer user.
Because a mouse is a relative pointing device, it can be lifted and moved during the procedure described above in order to make long movements across the desktop (if necessary). Not all computers, however, use a mouse type pointing device. Rather, recently, pen-based computing systems have become popular. Pen-based computing systems (like those described in more detail below in conjunction with
Difficulties arise, however, when users attempt to use pen-based computing systems in multi-monitor mode like that described above in conjunction with
Another interesting use also applies to traditional mice. Conventional multi-monitor traversal occurs by “warping” from the edge of one desktop/monitor pair to the edge of another desktop/monitor pair. This gives the illusion of panning smoothly across multiple monitors that are, for instance, arranged side-by-side. Panning to the right edge of the monitor on the left “warps” the mouse to the left edge of the monitor on the right. This behavior works because of an explicit action taken by the user to configure the operation system to understand the physical relationship of the monitors. If the monitors do not live in the same planar relationship, like a laptop in a conference room with a 2nd desktop projecting on the conference room screen, it is impossible to create the same illusion since the monitors have a 3-dimensional relationship rather than a flat planar relationship.
Accordingly, at least some aspects of the present invention seek to enable data entry, manipulation, and handling in pen-based computing systems operating an electronic desktop in a multi-monitor mode. Additionally, advantageous aspects of this invention may be applied to computer systems that operate with user input devices other than electronic pens, such as mice, trackballs, touchpads, rollerballs, eraser heads, keyboards, and the like. At least some aspects of the present invention allow a more natural user interface for manipulating multiple desktops that are being projected to non-planar monitors when using any type of user input devices, including traditional relative pointing devices, such as a mouse.
Aspects of the present invention generally relate to systems, methods, user interfaces, and computer-readable media for entering, handling, or manipulating data on a computer or electronic desktop. As one example, aspects of this invention may include systems, methods, and user interfaces that: (a) provide a first viewable region capable of displaying a first portion of a desktop on a display device; and (b) provide a second viewable region capable of displaying a second portion of the desktop on the display device, wherein a portion of the first viewable region redirects data input to and associates the data input with the second portion of the desktop. Other example aspects of this invention relate to systems, methods, and user interfaces that include: (a) maintaining a first portion of a desktop; (b) maintaining a second portion of the desktop, wherein the second portion of the desktop includes a region representing the first portion of the desktop; and (c) altering content of the first and/or second portion of the desktop in at least some instances based on data input directed to the region. Still additional example aspects of this invention relate to systems, methods, and user interfaces that include: (a) displaying a first portion of a desktop by a first display device; (b) displaying a second portion of the desktop by a second display device, wherein at least a portion of a display by the second display device includes a region representing the first display device; and (c) altering content displayed by the first display device in at least some instances based on data input directed to the region representing the first display device. Example aspects of this invention also relate to computer-readable media having stored thereon computer-executable instructions for performing various methods, including the methods described above and including methods for operating systems and generating user interfaces like those described above.
The foregoing summary of aspects of the invention, as well as the following detailed description of various examples, is better understood when read in conjunction with the accompanying drawings, which are included by way of example, and not by way of limitation with regard to the claimed invention. In the figures:
a through 6c illustrate various features available in at least some examples of systems and methods according to the invention;
a and 7b illustrate additional features available in at least some examples of systems and methods according to the invention;
a and 8b illustrate additional features available in at least some examples of systems and methods according to the invention;
a and 9b illustrate additional features available in at least some examples of systems and methods according to the invention;
a and 10b illustrate an example of movement of data from one portion of a desktop to another in at least some examples of systems and methods according to the invention; and
Various specific examples of the invention are described in detail below in conjunction with the attached drawings. To assist the reader, this specification is broken into various subsections, as follows: Terms; General Description of Handling Data Across Different Portions or Regions of a Desktop; Example Hardware Useful with the Invention; Specific Examples of the Invention; and Conclusion.
A. Terms
The following terms are used in this specification, and unless otherwise noted or clear from the context, these terms have the meanings provided below.
Desktop—An arrangement or on-screen display of icons or other representations of various application programs, electronic documents, and the like that are stored on and/or available through a computer system. In some instances, an individual display device may display only a portion of an electronic desktop, and in some instances, various independent portions of the desktop may be displayed by multiple display devices operated by a common computer system.
Viewable region—An area or portion of a monitor or other display device that displays or provides user access to all or some of the desktop.
User interface—The combination of menus, screen design, user input commands, command language, help screens, and/or the like that create the way in which a user interacts with a computer system. User input devices (such as pens, mice, touch screens, touchpads, keyboards, rollerballs, trackballs, and the like) also may be included.
Jump pane—A portion of a user interface providing a portal that allows users to enter, move, or otherwise manipulate or handle data located in one portion of a desktop from a second, independent portion of the desktop.
Display device—Any device that displays or renders information and/or generates data that enables a display or rendering of information. Display devices include, but are not necessarily limited to monitors, projectors, screens, and the like.
“Render” or “Rendered” or “Rendering”—The process of determining how information (including text, graphics, and/or electronic ink) is to be displayed, whether on a screen, printed, or output in some other manner.
“Computer-Readable Medium”—Any available media that can be accessed by a user on a computer system. By way of example, and not limitation, “computer-readable media” may include computer storage media and communication media. “Computer storage media” includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. “Computer storage media” includes, but is not limited to: RAM, ROM, EEPROM, flash memory or other memory technology; CD-ROM, digital versatile disks (DVD) or other optical storage devices; magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices; or any other medium that can be used to store the desired information and that can be accessed by a computer. “Communication media” typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of “computer-readable media.”
B. General Description of Hanlding Data Across Different Portions or Regions of a Desktop
In general, this invention relates to systems, methods, user interfaces, and computer-readable media for handling data with respect to a computer desktop. One more specific aspect of this invention relates to methods for providing user interfaces. As one example, methods according to this aspect of the invention may include: providing a first viewable region capable of displaying a first portion of a desktop on a display device; and providing a second viewable region capable of displaying a second portion of the desktop on the display device, wherein a portion of the first viewable region redirects data input to and associates the data input with the second portion of the desktop. In some examples of methods according to this aspect of the invention, user input is accepted, and at least some of the user input includes the data input that is redirected to the second portion of the desktop.
Another example aspect of this invention relates to methods that include: displaying a first portion of a desktop using a first display device; displaying a second portion of the desktop using a second display device, wherein at least a portion of a display by the second display device includes a region representing the first display device; and altering content displayed by the first display device in at least some instances based on data input directed to the region representing the first display device.
Still another example aspect of this invention relates to methods that include: maintaining a first portion of a desktop; maintaining a second portion of the desktop, wherein the second portion of the desktop includes a region representing the first portion of the desktop; and altering content of the first and/or second portions of the desktop in at least some instances based on data input directed to the region.
Additional aspects of this invention relate to systems for handling data on a computer desktop, including, for example, systems for performing the various methods described above. Examples of such systems may include: a first display device displaying a first portion of a desktop; a second display device displaying a second portion of the desktop, wherein at least a portion of a display by the second display includes a region representing the first display device; and a processor programmed and adapted to alter content displayed by the first display device in at least some instances based on data input directed to the region representing the first display device. Other examples of such systems may include: a receiver constructed and adapted to receive input; and a processor programmed and adapted to: (a) maintain a first portion of a desktop; (b) maintain a second portion of the desktop, wherein the second portion of the desktop includes a region representing the first portion of the desktop; and (c) altering content of the first and/or second portion of the desktop in at least some instances based on data input directed to the region.
Still additional aspects of this invention relate to user interfaces for interacting with a desktop on a computer screen or other display device. For example, user interfaces according to some examples of this invention may be displayed by a display device and include: a first region representing a first portion of a desktop; a second region representing a second portion of the desktop; and a data transfer path or portal that allows data to be transferred between the first region and the second region.
Input can be introduced into systems and methods according to examples of this invention in any suitable manner. For example, in at least some instances, the data input transferred across different portions or regions of the desktop may constitute user input introduced via a suitable user input device, such as a pen (or stylus), a mouse, a trackball, a rollerball, a touch pad, a touch screen, a keyboard, or the like.
As generally described above, at least some aspects of this invention relate to systems and methods wherein data is movable between different regions or portions of a desktop such that one portion or region on the desktop may be altered by a user through another portion or region of the desktop. Altering the content in a first desktop portion or region may include, for example, altering the content displayed by a display device for the first desktop portion or region or otherwise altering the stored content associated with the first desktop portion or region. Altering the content displayed by a display device may include, for example, determining at least a first coordinate of a second desktop portion or region associated with the data input to be redirected to the first desktop portion or region, and remapping the first coordinate to a corresponding coordinate of the first desktop portion or region.
Additional aspects of this invention relate to computer-readable media having stored thereon computer-executable instructions for performing various methods, including the methods described above and including suitable methods for operating systems and generating user interfaces like those described above.
Various aspects and examples of the present invention will be described in detail below in conjunction with the attached figures. The description and figures should be construed as examples of the invention and not as limitations on the invention.
C. Example Hardware Useful with the Invention
A basic input/output system 160 (BIOS) containing the basic routines that help to transfer information between elements within the computer 100, such as during start-up, is stored in the ROM 140. The computer 100 also includes a hard disk drive 170 for reading from and writing to a hard disk (not shown), a magnetic disk drive 180 for reading from or writing to a removable magnetic disk 190, and an optical disk drive 191 for reading from or writing to a removable optical disk 192, such as a CD ROM or other optical media. The hard disk drive 170, magnetic disk drive 180, and optical disk drive 191 are connected to the system bus 130 by a hard disk drive interface 192, a magnetic disk drive interface 193, and an optical disk drive interface 194, respectively. The drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules, and other data for the personal computer 100. It will be appreciated by those skilled in the art that other types of computer-readable media that can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, punch cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROMs), and the like, also may be used in the example operating environment without departing from the invention.
A number of program modules can be stored on the hard disk drive 170, magnetic disk 190, optical disk 192, ROM 140 or RAM 150, including an operating system 195, one or more application programs 196, other program modules 197, and program data 198. A user can enter commands and information into the computer 100 through input devices, such as a keyboard 101 and a pointing device 102. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices often are connected to the processing unit 110 through a serial port interface 106 that is coupled to the system bus, but they may be connected by other interfaces, such as a parallel port, game port, a universal serial bus (USB), or the like. Further still, these devices may be coupled directly to the system bus 130 via an appropriate interface (not shown). A monitor 107 or other type of display device also is connected to the system bus 130 via an interface, such as a video adapter 108. In addition to the monitor 107, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. In one example, a pen digitizer 165 and accompanying pen or stylus 166 are provided in order to digitally capture freehand electronic ink input. Although a direct connection between the pen digitizer 165 and the serial port interface 106 is shown, in practice, the pen digitizer 165 may be coupled to the processing unit 110 directly, to a parallel port, to another interface, and to the system bus 130, as known in the art. Furthermore, although the digitizer 165 is shown apart from the monitor 107, the usable input area of the digitizer 165 may be co-extensive with the display area of the monitor 107. Further still, the digitizer 165 may be integrated in the monitor 107, or may exist as a separate device overlaying or otherwise appended to the monitor 107.
The computer 100 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer 109. The remote computer 109 can be a server, a router, a network PC, a peer device, or other common network node, and it typically includes many or all of the elements described above relative to the computer 100, although only a memory storage device 111 has been illustrated in
When used in a LAN networking environment, the computer 100 may be connected to the local network 112 through a network interface or adapter 114. When used in a WAN networking environment, the personal computer 100 typically includes a modem 115 or other means for establishing a communications over the wide area network 113, such as the Internet. The modem 115, which may be internal or external to the computer 100, maybe connected to the system bus 130 via the serial port interface 106. In a networked environment, program modules depicted relative to the personal computer 100, or portions thereof, may be stored in the remote memory storage device 111.
It will be appreciated that the network connections shown are illustrative and other techniques for establishing a communications link between the computers can be used. The existence of any of various well-known protocols such as TCP/IP, UDP, Ethernet, FTP, HTTP, and the like is presumed, and the system can be operated in a client-server configuration to permit a user to retrieve web pages from a web-based server. Any of various conventional web browsers can be used to display and manipulate data on web pages.
The stylus 204 may be equipped with one or more buttons or other features to augment its capabilities. In one example, the stylus 204 could be implemented as a “pencil” or “pen,” in which one end constitutes a writing portion and the other end constitutes an “eraser” end that, when moved across the display, indicates portions of the display to be erased. Other types of input devices, such as a mouse, a trackball, or the like could be used. Additionally, a user's own finger could be the stylus 204 and used for selecting or indicating portions of the displayed image on a touch-sensitive or proximity-sensitive display. Consequently, the term “user input device,” as used herein, is intended to have a broad definition and encompasses many variations on well-known input devices, such as stylus 204. Region 205 shows a feedback region or contact region permitting the user to determine where the stylus 204 has contacted the display surface 202.
In various examples, the system provides an ink platform as a set of COM (component object model) services that an application program can use to capture, manipulate, and store ink. The ink platform also may include a mark-up language including a language like the extensible markup language (XML). Further, the system may use DCOM as another implementation. Yet further implementations maybe used including the Win32 programming model and the .Net programming model from Microsoft Corporation. These platforms are commercially available and known in the art.
The invention now will be described in conjunction with the remaining figures, which illustrate various examples of the invention and information to help explain the invention. The specific figures and information contained in this detailed description should not be construed as limiting the invention.
D. Specific Examples of the Invention
As described above, aspects of this invention relate to entering, moving, or otherwise manipulating data or content in a first portion or region of a desktop from a second, independent portion or region of the desktop.
a-6c illustrate an example system 400 in which a first display device 402 of the system forms a portion of a pen-based computing system 410, such as one like that described above in conjunction with
In the example illustrated in
Changes to data and information contained within or directed to display area or viewable region 416 can be made in any suitable manner. For example, a user could enter new data, delete existing data, and/or modify existing data within display area or viewable region 416 using an input device, such as a pen 406, a mouse type device (not shown), or a keyboard (not shown). As another example a user could move data into display area or viewable region 416 from display area or viewable region 414 (or another portion or region of the desktop) using a “dragging” action (e.g., pen tap or selection and drag and/or mouse click or selection and drag). This feature is described in more detail in conjunction with
a and 6c illustrate another convenient feature useful in some examples of this invention. For example, the first portion or viewable region 414 of the desktop may include an area in which a user can enter data, such as an electronic notepad 422. Using a pen (or another suitable input device), a user can enter notes on the notepad 422. In at least some examples of the invention, as noted above, the data relating to these notes will not be located in the second portion of the desktop 416, and therefore, this information will not transfer to or appear on the other display device 404. However, if the user wanted the information in the notes 422 to appear on the other display device 404, the desired data could be moved to the second portion of the desktop 416, for example, by a drag and drop operation, by a copy operation, or in any other suitable manner, without departing from the invention.
a and 7b illustrate a “dragging” operation useful to move data from the first portion of the desktop to the second portion of the desktop and vice versa. As shown in
Of course, rather than remove content from the first portion of the desktop to place in the second portion of the desktop (e.g., through the above-described type “move” action), content in the first portion of the desktop could be cut or copied and then pasted into the second portion of the without departing from the invention. New data also could be entered directly into the second portion of the desktop through region 416. Also, in a manner similar to that illustrated in
Any size “portal” could be made available to allow transfer of data between the two portions of the desktop. For example, in some instances, the entire viewable region 416 can be considered a “jump pane,” and data manipulation of any kind that occurs within the jump pane effectively occurs within the second portion of the desktop. As another example, data crossing into or out of the second region 416 may be allowed to take place only along one or more predetermined locations or edges of region 416 and/or along only a portion of one or more of these locations or edges. Additionally, the ability to use the jump pane and/or portal may be activated in any suitable manner without departing from the invention. For example, the portal and/or jump pane may be activated automatically any time a user enters a multi-monitor mode and/or extended desktop mode. As another example, if desired, the user may selectively activate the jump pane or portal by interacting with a user interface item, such as a button or menu item. Other suitable ways of activating and using the jump pane and/or portal also are possible without departing from the invention.
a and 8b illustrate additional features available in at least some examples of systems and methods according to the invention.
In many instances, pen-based computing systems like those described above contain relatively small display devices, in order to keep the computing system as small, lightweight, and mobile as possible. Small display devices of this type can be difficult for some users to see, particularly in the situation illustrated in which only a portion 416 of the display device 402 is intended to represent and display the entire content 418 of another display 404. In these situations, the content 430 on the represented display portion 416 can become quite small and difficult to understand. Accordingly, some example systems, methods, and user interfaces according to the invention may include a magnifier that enlarges at least some of the represented display portion. For example, as illustrated in
a and 9b illustrate still additional features present in at least some examples of the invention. In some examples of the invention, like those illustrated in
It is not necessary, however, in every instance, to fully mirror the content of the display portion 418 of display device 404 within the represented display portion 416 of display device 402. Rather, as illustrated in
In some examples of the system of
While the various examples described above specifically illustrate use of the invention with an electronic pen and one monitor provided as part of a pen-based computing system, these features are not required. For example, a mouse, touchpad, trackball, rollerball, eraser head, joystick, or other suitable user input devices could be used to enter, manipulate, and/or delete the data in the same general manner described above without departing from the invention. Additionally, a keyboard also could be used, in at least some examples and some situations, for user input entry without departing from the invention.
In some situations, however, it may be preferable to limit use of the jump pane or portal to only pen events. As described above, because an electronic pen is an absolute pointing device, the jump pane or portal according to examples of the invention advantageously allow data to be transferred to different portions of a desktop from the portion of the desktop displayed by a device displaying the first portion of the desktop. User input devices like a mouse, touchpad, rollerball and trackball, however, are relative pointing devices, not absolute pointing devices like a pen. Accordingly, on some systems in which a jump pane or portal is available, a mouse (or the like) can be used in the same manner described in
a and 10b illustrate an example of a situation in which a mouse may be used to move data between portions of the desktop when the jump pane is open and available. Specifically, as illustrated in
If the user desires to move data structure 510 from the first portion 502 of the desktop to the second portion using a mouse 512, he/she may do so by first moving the mouse 512 to the right as far as necessary to move the data structure 510 to the desired location. For example, in a first move, as shown in
If necessary, these steps may be repeated as often as necessary to place the visual representation of data structure 510 in the desired location on the desktop. Also, the direction of mouse and data structure movement may be different and freely selected (not necessarily left to right) without departing from the invention.
As the procedure starts (S600), the coordinates of the secondary or external display device (e.g., display device 404 from
Based on these coordinates determined at S602, a jump pane window (e.g., area 416) representing the secondary or external display is opened S604 within the device displaying a first portion of the desktop (e.g., display device 402 and desktop portion 414 in
In the present example, only pen-based events may use the jump pane (i.e., in this example, mouse, trackball, touchpad, rollerball, or keyboard action cannot make use of the jump pane). Accordingly, at S608, systems and methods according to this example of the invention determine whether a user has terminated an input session (e.g., closed a document, timed out, quit, or otherwise stopped an application program or an input session). If “Yes,” the procedure ends (S610). If “No” at S608, systems and methods according to this example of the invention determine whether an input event is occurring and whether it involves user input through a pen (S612). If “No,” the user input hook processor is made available for other processing (S614), and the procedure returns to S608. In this example, attempts to use a mouse type device to move data through the jump pane can be treated in any suitable manner without departing from the invention. For example, the attempt can be ignored, and the visual representation of the data (if any) can be returned to its original location on the originating portion of the desktop.
If at S612, however, the system determines that the detected event is a pen-based event (answer “Yes”), the system next determines whether the event is located within the jump pane area of the system (e.g., the system determines whether the event is located within area 416 from
Once the coordinates of the event on the external display device are determined at S618, the processor may inject the event into the external display device (e.g., display device 404 using the remapped coordinates), and if desired, into its representation in the jump pane (e.g., in area 416 of display device 402) (S620). All additional processing associated with the event (if any) may then be completed (S622) and the procedure will return to S608 (e.g., wait for the next event and/or a continuation of the present event).
Many variations on the above-described procedure may be made without departing from the spirit and scope of the invention. For example, the order of various steps may be changed, additional steps may be added, and certain steps may be modified and/or omitted without departing from the invention. As one more specific example, if all user input device events are able to use the jump pane (not just pen events), then S608 may be omitted.
As additional examples of variations, systems, methods, and user interfaces according to the invention could be used to maintain, view, and interact with more than two independent portions of the desktop. For example, a user could be working with three or even more portions of the desktop, optionally using three or more independent display devices, without departing from the invention. As another example variation, the invention may be used over suitable remote connections such that the display device displaying the first portion of the desktop is provided at a different location from the display device displaying the second portion of the desktop (which also displays the region representing the first portion of the desktop).
Another example use might include use of systems and methods according to the invention for a slide presentation wherein slides are presented to an audience on one display device and another display device (e.g., for the speaker) includes a visual representation of the specific slide being displayed and an area on which the speaker can make notes that are not displayed on the slide display device.
Finally, the present invention also relates to computer-readable media including computer-executable instructions stored thereon for performing the various methods and/or for use in the various systems described above. The computer-readable media may constitute computer-executable instructions stored on the various specific examples described above.
E. Conclusion
Various examples of the present invention have been described above, and it will be understood by those familiar with this art that the present invention includes within its scope all combinations and subcombinations of these examples. Additionally, those familiar with the art will recognize that the above examples simply exemplify various aspects of the invention. Various changes and modifications may be made without departing from the spirit and scope of the invention, as defined in the appended claims.
