Patent Application
20070013722
Contemporary computer systems offer users the ability to magnify a portion of their display screen, typically for providing visually-impaired users with better accessibility to computers, although also offering mainstream users the benefit of magnification in certain contexts. The portion of the display screen that is magnified can be selectively based on where the mouse is located, where the keyboard is focused, and/or where the user is otherwise editing, e.g., when selecting content such as for cutting and pasting.
In general, when the user runs a magnifier, a magnification window is opened and the magnified content displayed in that window. The user can adjust the magnification window in size and position, adjust the magnification level, and possibly choose other effects, e.g., color inversion. One such magnifier program operates by driver hooking, to intercept graphics calls sent towards the video driver, essentially in a screen capture operation, where the magnifier program then scales up a corresponding graphics image in the magnification window.
Because with any magnification factor greater than 1×, the magnified image is larger than the source image. As a result, it is often necessary for the user to navigate to locate a desired portion of the source region for magnification in the magnification window. When working with a large source region, it is fairly easy for users to lose their orientation with respect to the source image, that is, the user sees a magnified image in the magnification window but cannot determine what portion of the source is being magnified.
Briefly, various aspects of the present invention are directed towards a context map that assists a user in determining a current location within the source region that is being magnified. The context map is valuable with large magnification windows, such as when the magnification window covers most or all of the entire desktop.
When displaying magnified content, such as in a magnification window, a context map, comprising a visible image representing a location of the unmagnified content that is being magnfied, is displayed. The visible image may be an indicator, a reduced-size representation of the entire desktop with an indicator therein, or virtually anything that serves as a map for the user. The visible image may be made to selectively appear and/or disappear, such as by user command.
The magnified content may be displayed on one monitor, and the context map on another monitor or display device. The context map may be on the same monitor as the magnified content, possibly superimposed over some part of the magnified content.
The magnification window may be a floating window that moves with a mouse or the like, or keyboard focus, and the context map may be docked, or displayed over at least part of the floating window such that the context map also moves with it.
Other advantages will become apparent from the following detailed description when taken in conjunction with the drawings.
The present invention is illustrated by way of example and not limited in the accompanying figures in which like reference numerals indicate similar elements and in which:
Exemplary Operating Environment
The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to: personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and so forth, which perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in local and/or remote computer storage media including memory storage devices.
With reference to
The computer 110 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computer 110 and includes both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by the computer 110. Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer-readable media.
The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation,
The computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media, described above and illustrated in
The computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 110, although only a memory storage device 181 has been illustrated in
When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user input interface 160 or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
Magnification Engine and API
Various aspects of the technology described herein are directed towards providing a context map by which a user may maintain a sense of orientation of the source when viewing magnified content. In one example implementation, this is accomplished by a magnification program that works with a magnification engine and application program interface (API), wherein the engine provides much of the functionality needed to couple the magnification application program to a computer system, such as via operating system components. One such magnification engine/API are described in copending U.S. patent application entitled “Magnification Engine and Interface for Computers” attorney docket number 5620/313231, filed concurrently herewith, assigned to the assignee of the present invention, and hereby incorporated by reference.
In one example implementation described herein, the magnification engine and interface are described in the form of a Win32 control that may be hosted by any application program, or run independently (e.g., via an operating system shell). However, as will be understood, the present invention is not limited to controls in general or a Win32 control, let alone any particular implementation or examples described herein. Instead, numerous ways to implement the present invention are feasible, and as such, the present invention is not limited to any of the particular examples used herein, but rather may be used various ways that provide benefits and advantages in computing in general.
Turning to
Once created, the program calls an initialize function of a magnification engine/API (via an application program interface), providing the handle. This is generally represented in
Step 208 represents obtaining the transform that is (typically) used for the magnification level, e.g., 2×, 3×, and so forth. Note that the transform can designate something other than magnification, e.g., skew, and that if magnification is used, the magnification may be 1× (no magnification) or some fraction smaller than 1×, e.g., a reduced size image may be shown. Thus, although the present invention is primarily described herein with reference to magnification, any transform, including an identity transform that does not technically magnify or reduce, may be applied.
Step 210 represents the concept of selective window magnification, or filtering. More particularly, not everything in a designated region needs to be magnified. Instead, the magnification engine/API allows the program to designate which windows to magnify, and which ones should be excluded from the final magnified output. Note that the magnifier window never magnifies itself, although it would be feasible to do so. Filtering may be via inclusion or exclusion. Thus, a program may specify a list of windows to include (but not the magnification window) or elect to magnify all windows except those specifically excluded (the magnification window will be excluded). In this manner, a program has significant flexibility as to what gets magnified.
Once the magnification engine/API 310 is configured,
In order to composite immediate mode graphics and retained mode graphics, a redirection layer 312 essentially sends the GDI graphics to a surface, as if the immediate mode graphics were retained mode instructions. A graphics server 314, typically including a rasterizer or the like for handling GDI primitives, provides a graphics stream 316 containing one or more bitmaps and/or primitives from the various surfaces to a composition engine 318. The composition engine then composites the various output from the surfaces, and sends composited graphics data to a graphics card to render the graphical output.
In one implementation, for each set of GDI data, the graphics stream 316 may comprise a rasterized bitmap. This is because in this implementation, the composition engine 318 only includes a rasterizer 320 for the presentation subsystem's primitives (that correspond to a rendering tree), and does not rasterize GDI primitives.
Note, however, for GDI data that is to be magnified, the GDI primitives may be sent in the stream for processing by a magnification engine/API 322 and its connected components, which will rasterize the primitives prior to sending to the composition engine 320. More particularly, one way to magnify visible content is to scale the content using its rasterized bitmap, as is done is conventional magnification. However, this generally provides a low quality and low performance result, but nevertheless is reasonable for existing immediate mode programs. Alternatively, by changing data of the primitives, e.g., doubling font size, line lengths and so forth for 2× magnification, magnification with better quality and performance is obtained.
Scaling may be accomplished by a plug-in client scaling component 322. Alternatively, or in addition to, magnification and other effects may be accomplished by a separate magnification effects program, referred to herein as a “third party” program 324, even though there is no intent to limit such a program to any particular vendor, and indeed, the provider of the magnification engine/API may also provide such a program. This provides extensibility, for example, as better magnification/effect programs become available, such as better quality bitmap magnification programs. Alternatively, an application that wants to implement its own StretchBlt algorithm (possibly with a different smoothing algorithm) can do so by setting the magnification transform to the desired transform, and then performing magnification in its own filter. Setting the transform is necessary for the presentation subsystem (e.g., Avalon/Media Integration Layer) primitive scaling which is not handled by the extensibility point. In one implementation, applications handle their own magnification via a SetMagnifierImageFilterCallback function. In general, at least one component for performing magnification is available.
For the magnified part of the stream, the magnification engine 312 thus may process the primitives and/or bitmaps as configured, and return the modified data to the graphics server and/or composition engine as appropriate. If still further (e.g., non-scaling) effects such as color effects are desired, the graphics server 314 and/or composition engine 318 may again call the magnification engine/API 312 for routing to the appropriate effects code. This is represented in
With respect to the rendering tree primitives, magnification or the like is straightforward, as a transform node may be added to the rendering tree by modifying the primitives. As mentioned above, GDI primitives may be transformed prior to rasterization to accomplish magnification and the like, or the GDI primitives may be rasterized into a bitmap, with the bitmap magnified in a known manner, e.g., as conventionally accomplished. In any event, the graphics stream 316 ultimately contains the various surfaces comprising primitives and/or bitmaps, including the data corresponding to the magnified content.
A graphics stream reader component 432 works with a selective window filtering component 434 so that only GDI data and rendering tree data from the graphics server 314 that correspond to included, or non-excluded, windows are processed for magnification. In one implementation, the graphics stream reader component 432 is a client of a Graphics Pipe (stream) API, and receives the raw packets from the graphics stream and hands them off to the selective window filtering component (HWND packet filter) 434. The selective window filtering component 434 filters the packets from the graphics pipe reader component based on the HWND associated with each packet. Magnifier HWNDs are always filtered out, and the inclusive or exclusive list of HWNDs is used to further filter the packets.
For relevant messages, that is, for source windows (e.g., based on their HWNDS) that are to be magnified, an image filter 436 works with the scaling component 322 and/or a third party magnifier 324 to scale the image, in the example of
GDI content is then rasterized by a content rasterizer 438 prior to passing to a graphics stream writer 440, while rendering tree data is passed in its modified form to the graphics stream writer 440, where it will be rasterized. The content rasterizer 438 is responsible for rasterizing and compositing the packets. This component exposes three significant properties, namely transform, clip region (combines the source rect with the output clip from the output window) and output surface, where the output surface size is based on the clip region and transform.
The content rasterizer 438 is coupled to the graphics stream writer (composition engine writer) 440 to communicate with the graphics server and compose the magnified region of the desktop with the rest of the desktop. In this manner, the magnified data reaches the composition engine 318 in the appropriate graphics stream.
The magnified data may be returned to the magnification engine/API 312 for post-processing to add any effects, shown via effects component 442. In general, the effects component 442 implements the mechanism for other (e.g., third party) code to implement custom color effects. Note that although not specifically shown as such in
In this manner, magnification programs may be easily coupled to the computer system, while because the magnification API enables a magnifier to be implemented as a Win32 control, the magnifier control may be dropped into any suitable application program. It is possible to build a magnification client that enlarges a control and refreshes as the user moves from control to control.
Context Map
The magnification program, which may comprise the plug-in 322 or similar component, may provide a number of magnification modes. Once such mode corresponds to a docked or floating magnification window that tracks mouse position and/or keyboard focus; as the mouse/focus moves, the magnified content changes to correspond to the new position. Note that the magnification program supports multi-monitor configurations.
Another mode is a floating lens magnification mode, in which the floating lens magnifier is a floating magnification window that tracks the mouse cursor. The magnification window always contains the mouse cursor. In one implementation, the mouse cursor is always anchored in the center of the lens. This feature generally assists with the reading of text and magnification or specific UI elements.
Another mode is full screen magnification mode. In full screen magnification mode, the size of the magnifier's client area is equal to the physical viewable area of the display. In this mode, the entire desktop is magnified by a user specified scale factor. The user is then presented with a full-screen view of a portion of the magnified desktop. The user can work with the user interface displayed within that view. To work with another portion of the desktop, the user can pan the view, e.g., in eight directions (every 45 degrees). For desktops spanning multiple displays, a magnifier magnifies the entire desktop. Thus, the magnified view of the desktop with which the user interacts, or can pan around, can span several displays.
At any time, but in particular in the full-screen mode, it is relatively easy for a user to lose the current orientation with respect to the entire desktop. Various mechanisms provide a context map to assist the user in knowing the current location.
For example,
For a single monitor situation, various location mechanisms are feasible, and may be selectively rendered. For example, a keyboard key or key combination may cause a rectangle to flash to correspond to the area that is being magnified, or some other image such as arrows may appear. Alternatively, such an indicator may be visible at all times. In one implementation, if a user has a single display or does not wish to devote an entire display to a context map, a smaller context map may appear, e.g., in the bottom corner of the screen. When displayed, this smaller context map is always on top. Other indicators are feasible.
By way of example,
While the invention is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the invention to the specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention.
