Patent Application
20070288640
In order to describe the manner in which the above-recited and other advantageous features of the invention can be obtained, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered to be limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
The present invention extends to methods, systems, and computer program products for extending remote session protocols by enabling legacy clients to simultaneously display multiple cursors. The embodiments of the present invention may comprise a special purpose or general-purpose computer including various computer hardware or modules, as discussed in greater detail below.
As previously mentioned, current remote session protocols (e.g., remote desktop protocol (RDP)) modify the position and/or appearance of a mouse cursor on the viewer's display using a mouse cursor update or control packet. More specifically, the remote protocol sends cursor instructions as packet data units (PDUs) for synchronizing both the sharer's and viewer's mouse pointers. In screen sharing or remote assistance scenarios, however, there are multiple users involved; thus generating a need to simultaneously display multiple mouse cursors as they actually appear.
Note that one solution to the above problem might be to modify the remote session protocol to include configuration settings for multiple cursors. Such solution, however, would not work for legacy clients that only have the notion of receiving data instructions for syncing a single mouse cursor. Accordingly, embodiments provide for a mechanism that extends the protocol of a remote session by enabling legacy clients (as well as other clients) to simultaneously display multiple cursors, without having to modify data instructions for the protocol that controls legacy client's cursor position and/or appearance. In such an embodiment, each viewer will have the ability to control its own mouse pointer, which will not be automatically synchronized in shape, size, and/or position with the sharer's computer. As such, the viewer's mouse pointer will be meaningful to that viewer and will be used to track intended modifications for the mouse position. The actual mouse shape, size, and/or position (as appearing from the sharer's or some other remote source) will come from the sharer and can be the result of input coming from multiple users.
In others words, embodiments advantageously provide that the viewer's user interface will display at least two mouse cursors. The first cursor will be the viewer's cursor as described above; the second cursor is the sharer's cursor (or some other clients mouse pointer), which will typically match the actual position and/or shape of the cursor in the remote session. In order to support legacy clients by not changing the underlining remote session protocol (e.g., RDP), embodiments render the sharer's cursor (or other pointer as the case may be) as part of the protocol's graphic stream.
For example, in one embodiment the sharer's cursor is sent to the client as plain or standard bitmap updates (i.e., bitmaps that are not normally cached). Such bitmap representation is the most generic way to send the cursor to the client; at the same time keeping compatibility with older, legacy and/or micro-clients (i.e., clients that support only plain bitmap updates or a very limited set of update types). In such case, however, the cursor is sent in a very insufficient way typically not suitable for slow link connections. Nevertheless, if the client is a micro, such bitmap or screen data updates are typically the only way that the sharer's cursor may be sent, which is typically not a problem considering that most micro-clients are usually designed to run in high bandwidth configurations.
Problems may arise, however, in the screen sharing or remote assistance scenarios where the sharer does not include a high speed connection. In such case, mouse updates may take an unacceptably high amount of bandwidth and the update rate will hurt the user experience. In such situation, embodiments provide a better solution by sending the share's mouse cursor in the form of protocol orders, as will be described in greater detail below.
Although more specific reference to advantageous features are described in greater detail below with regards to the Figures, embodiments within the scope of the present invention also include computer-readable media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of computer-readable media.
Computer-executable instructions comprise, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
As used herein, the term “module” or “component” can refer to software objects or routines that execute on the computing system. The different components, modules, engines, and services described herein may be implemented as objects or processes that execute on the computing system (e.g., as separate threads). While the system and methods described herein are preferably implemented in software, implementations in hardware or a combination of software and hardware are also possible and contemplated. In this description, a “computing entity” may be any computing system as previously defined herein, or any module or combination of modulates running on a computing system.
As mentioned above, in screen sharing, remote assistance, and other similar scenarios, it is desirable for the sharer's mouse pointer 155 or cursor to be simultaneously displayed on the viewer's computing device 190. Typical remote protocols (e.g., RDP), however, only have the notion of a single mouse cursor. As such, they send mouse cursor control packets 180 for manipulating the cursor appearance (e.g., size, shape, etc.) and/or position of a viewer's 190's mouse pointer 195 when syncing it with the sharer's 150's cursor 145.
In order to support legacy clients 190, rather than modifying data instructions for the protocol that controls the viewer's 190's cursor position and/or appearance (e.g., mouse cursor control packets 180), embodiments generate a graphical representation of the mouse cursor 145. Such graphics data 175 can then be sent across a graphics stream 170 of a remote session protocol (e.g., RDP). Note that embodiments still support modification to a viewer's 190 cursor using the mouse cursor control packets 180; however, typically the viewer's 190's cursor 195 will be held constant or in a fixed shape and/or size during the remote session. In other words, the viewer 190 typically has full control over its mouse cursor 195, which will normally have a different appearance than other mouse cursors simultaneously displayed therewith for ease in identification. Nevertheless, other embodiments allow the modification of the viewer's mouse cursor 195 using cursor instructions 180, e.g., prior to holding the viewer's mouse cursor 195 constant. For example, such modification may be done to visually indicate to the viewer 190 that its cursor 195 is different than the sharer's 150's mouse cursor 145, which is used to track the local cursor 145 position. Note that in such instance, the cursor change may be to a non-standard cursor shape or some other dramatic change; however, all of these changes are optional.
As mentioned above, mouse pointers (e.g., cursor 110 shown as a cross) other than the sharer's 145 may be simultaneously displayed with the viewer's cursor 195. For example, other clients 105 cursors 110 may be part of a screen sharing or remote session (e.g., a presentation, lesson, etc.). Such cursors 110 may also simultaneously be displayed along with the viewer's cursor 195 in accordance with embodiments described herein. Accordingly, any specific cursor (e.g., sharer's cursor 145) simultaneously displayed with the viewer's 195 (as well as any appearance thereof) is used herein for illustrative purposes only and is not meant to limit the scope of embodiments herein unless otherwise explicitly claimed. Nevertheless, it should be noted that in the event that the other cursors 110 are displayed, such cursors are not the result of painting the cursor 110 on the sharer's 150's screen that is distributed as normal graphics data. Instead, they are inserted as cursor orders in the graphics stream 170 as will be described in greater detail below.
Regardless of which cursor 110, 145 is sent as a graphical representation, the viewer 190 receives such data 175 (e.g., a bitmap, primitive, sprite, or other graphical representation) across the graphic stream 170, which it can then use to simultaneously display with its own pointer 195. This graphics data 175 may be generated each time a move of the sharer's mouse cursor 145 is detected or at appropriate deltas in order to reduce bandwidth required to send multiple updates. Note that in the event that the graphical representation or graphics data 175 of the cursor 145 is sent as full screen data bitmaps, little if any erasing logic may be needed when sending updates—as will be described in greater detail below for other embodiments. In other words, screen data PDUs and other plain bitmaps are typically opaque so that it may not be necessary to erase a previous mouse pointer in cases that the updates intersect the current position of the mouse pointer. All that may be needed is to ensure that painting of the mouse pointer 145 occurs for the graphics data 175 prior to sending such across the graphic stream 170 to the client or viewer 190.
In other embodiments, however, some problems arise in screen sharing and other remote session scenarios where the sharer 150 does not include a high speed connection. In such instances, mouse updates of plain update bitmaps may take unacceptably high amounts of bandwidth and the update rate can hurt the user experience. For such scenarios, embodiments can send the sharer's 150's mouse pointer 145 in the form of rendering orders 120. For instance, updates can be stored as cursor bitmaps 160 or sprites (e.g., AND and XOR bitmaps) within a temporary 155 or other cache. Upon detecting a move (or other action) of the sharer's 150's mouse pointer 145 (or other mouse cursor 110 as the case may be), rendering orders 125, 130, 135 are inserted into the orders stack 115, which are sent as part of the normal graphics stream 170 when making the appropriate drawing and erasing functions as needed.
More specifically, the sharer 150 encoder used in remote sessions can reserve a portion of a frame buffer (on the viewer 190) for storing cursor bitmaps 160. Accordingly, updating the sharer's 150's (or other's 105's) cursor shape 145 (110) means caching 155 the bitmaps 160 (e.g., AND and XOR bitmaps) used by a cursor 145. Note that although the temporary cache 155 is shown on the viewer 190 side, this cache 155 mechanism may be on either or both sides of the distributed system 100. In other words, as one would appreciate, the cursor bitmaps 160 can first be stored in a temporary cache 155 on the sharer's 150's side. Eventually, however, the cursor bitmaps 160 will be sent as graphics data 175 in the graphics stream 170, which may then be stored on the viewer side 190 in a frame buffer. As will be described in greater detail below, update orders 120 can then (or simultaneously) be sent as part of the graphics stream 170 in rendering of the cursor 145, 110 updates on the viewer 190 display. Of course, the system is not limited to any particular type of configuration and use of temporary cache 155 (e.g., a frame buffer or other storage mechanism on either the sharer 150 or viewer 190); and therefore, any specific aesthetic layout or use thereof is for illustrative purposes only and is not meant to limit or narrow the scope of embodiments described herein.
Nevertheless, it should be noted that typically in remote session protocols (e.g., RDP) the frame buffers on the sharer 150 and viewer 190 are kept in sync. Due to the rendering of the sharer's 150's cursor 145 as orders 120 of the graphics stream 170, however, these buffers will not include identical data. More specifically, the cursor bitmaps 160 will typically not be included on the sharer's 150's frame buffer 155, since the sharer's 150's cursor 145 is not rendered as graphics data on its own screen. Nevertheless, as will be described in greater detail below, there may be instances where syncing of the two frame buffers 155 is needed—for example, as a mechanism for erasing the rendered sharer's 150 or other clients 105 cursor 110, 145.
Regardless of where or how the temporary cache 155 is used, upon detection of a needed sharer's 150's cursor 145 update (due to a move or other action or screen modification that causes the cursor 110, 145 to change in position and/or appearance), the update shape 160 with the orders or primitives used in the screen sharing or other remote session will be stored. In other words, the orders to draw mouse cursor 135 are inserted as part of the orders stack 115 typically at the end for ensuring that the pointer 145, 110 is displayed over other screen bitmaps 165 rendered using other orders 130 within the stack 115. Note that in order to draw the cursor 145, 110 on the viewer side 190, the cached 155, e.g., AND and XOR bitmaps 160 will typically be used in a quaternary raster operation—similar to a MaskBlt—to render he cursor 145, 110 to the viewer 190.
More specifically, the cursor 145, 110 can be rendered by drawing at coordinates specified by a cursor 145, 110 position and an offset of the cursor 145, 110 hotspot. Typical protocols, however, do not support quaternary raster operations, so the actual drawing of the cursor 145 may be split into two equivalent ternary raster operations. Of course, there are other mechanisms for rendering primitives or cached bitmaps 160 on the viewer's 190's display, and therefore the above use of raster operations are for illustrative purpose only and are not meant to limit or narrow the scope of embodiments unless otherwise explicitly claimed.
Note that before the mouse pointer 145 is drawn, it may be necessary to make sure that the mouse pointer 145 can be erased. Embodiments accomplish this task in various ways. For example, the cursor bitmaps 160 and other screen portions 165 may be saved (e.g., using SaveScreenBits or memblt) to an off screen cache 155 using similar operations as described in the above rendering operations. Another way may be to save the mouse pointer 145, 110, but draw the mouse pointer 145, 110 in an inverted way. If the screen content (e.g., screen bitmaps 165) were saved using an offscreen or other similar bitmap operation, erasing the mouse cursor 145 may imply restoring the screen bits 165 from temporary cache 155. For example, we can restore the screen bits by syncing the content of the sharer's 150's frame buffer with the viewer's 190's in order to render the application 140 without the cursor 145 (as previously mentioned). Alternatively, the bits may be resorted by save screen orders (e.g., RestoreSavedScreen) in the stack 115 that restore screen bitmaps 165 that are saved on the viewer 190 side.
On the other hand, if the drawing of the cursor 145 on the viewer side 190 was not saved but instead inverted (as shown in
Note that the operation for the above drawing and erasing (as well as the replacement of full bitmaps as graphics data 175) can be used when a mouse pointer move or other operation is detected. In one embodiment, the mouse pointer 145 should typically be erased at the beginning of an update—as shown in order stack 115 as erase mouse cursor order 125—which will then be redrawn at the end of the order stack 115 as previously described. One problem associated with such embodiment, however, is that it might cause flickering on the viewer 190. Accordingly, another embodiment provides for detecting if an order 130 intersects a rectangle occupied by the mouse pointer 145 position. If this is the case, a mouse erase order 125 may be encoded in the order update packet followed by the mouse draw orders 135 at the end of the order update packet. In other words, the erase mouse cursor 125 at the beginning of the order stack 115 is optional. Also note that this erase 125 may not be needed if it is determined that other orders 130 fully overwrite a current mouse 145 position. Such orders 130 (including those mentioned above), however, will typically need to be opaque. If the orders 130 are transparent or the destination of the rectangle does not completely replace the mouse cursor 145 position, the cursor 145 may have to be erased as described above.
Note that the cursors instructions 180 sent over the control channel 185 may still modify portions of the viewer's mouse 195. For example, as described above, these may be used to set the viewer's mouse 195 to a fixed shape or size for the duration of the remote session. In addition, for non-legacy clients that support multiple cursor instructions 180, such mechanisms may also be employed with embodiments described herein. Nevertheless, the graphical representation or graphics data 175 of the cursor 145 for the sharer 150 or any other client 105 may be generated as described above and sent across the graphic stream 170 for display on the viewer 190.
Also note that the graphics data 175 can be a full bitmap, cursor bitmap 160, or other orders 120 or graphics data 175 as previously described. In addition note that although typically the viewer's mouse cursor 190 will be different (in shape, size, position, etc.) from a sharer's mouse cursor 145 and/or other client's mouse cursors 110, such embodiment is not is typically used for convenient purposes but is not necessary for all scenarios. Also note that in the case where there is a slow link or other mechanism that causes flickering, bitmaps may or may not be sent at each particular move operation for the cursor 145, but rather at intermediate intervals or deltas in the remote session.
The present invention may also be described in terms of methods comprising functional steps and/or non-functional acts. The following is a description of steps and/or acts that may be performed in practicing the present invention. Usually, functional steps describe the invention in terms of results that are accomplished, whereas non-functional acts describe more specific actions for achieving a particular result. Although the functional steps and/or non-functional acts may be described or claimed in a particular order, the present invention is not necessarily limited to any particular ordering or combination of steps and/or acts. Further, the use of steps and/or acts in the recitation of the claims—and in the following description of the flow diagram for FIG. 2—is used to indicate the desired specific use of such terms.
As previously mentioned,
Accordingly, data instructions may be sent as mouse cursor control packets 180 for making an initial change to the appearance and/or position of the viewer's 190's mouse cursor 195. Note, however, that after the initial change to the viewer's 190 mouse cursor 195, usually no further data instructions 180 are used to change the viewer's 190's mouse cursor 195 in order to fix its appearance for the duration of the remote session and allow the viewer full control thereof.
Method 200 also includes a step for rendering 230 an additional mouse cursor as a bitmap. Step for 230 includes an act of determining 210 that an additional mouse cursor is to be displayed at a viewer's computing device during a remote session. For example, sharer 150 can determine that at least one additional mouse cursor 145, 110 is to be displayed at the viewer's computing device 190 during the remote session. Note that such cursor 145 may also be a cursor 110 from another client 105 or maybe the sharer's 150's own mouse cursor 145.
Based on the determination, step for 230 further includes an act of generating 220 a graphical representation of the additional mouse cursor. In other words, graphics data 175 which includes a rendering of the cursor 110, 145 may be generated. Note that such graphics data 175 may include any combination of orders 120, full or plain bitmaps, cursor bitmaps 160, screen bitmaps 165, or other data normally sent over the graphics stream 170—the form of such data typically depends on what type of data the client or viewer 190 supports. As such, step for 230 includes an act of sending 225 the graphic representation over a graphics stream to the viewer's computing device. That is, the graphics stream 170 sends the graphics data 175 including graphics representation of cursor 110, 115 to the viewers 190 display in order to simultaneously display the additional mouse cursor 110, 145 with the viewer's 190 mouse cursor 195 during the remote session.
As noted above, the graphical representation 175 of the addition mouse cursor may be a graphics order 120 (125, 130, 135) of an order stack 115, and/or a graphical bitmap (e.g., AND and XOR bitmaps 160, or full screen bitmaps). Also note that the application executed 140 maybe a remote desktop, screen sharing application, or some remote assistance program. Of course, other programs are also contemplated herein. Note that if a position of the mouse cursor 145 is changed, a second graphical representation of the mouse cursor 145 in the changed position may be generated and sent to the remote device or viewer 190 for simultaneously displaying with the viewer's mouse cursor 195.
The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.
