Method and system for generating a graphical display for a remote terminal session

Abstract

In one aspect the invention relates to a method for generating a graphical display for a remote terminal session. The method includes monitoring output produced by an application program executing on a server, identifying a bitmap representation within the output produced by the application program and determining a check value for the bitmap representation. The method also includes retrieving a compressed data format of the bitmap representation based at least in part on the check value and transmitting to the remote terminal session the compressed data format in place of the bitmap representation.

Description

BACKGROUND

1. Field of Invention

The invention generally relates to remote sessions, and, more particularly, to generating a graphical display for a remote terminal session.

2. Description of Prior Art

A thin-client protocol is used for displaying output, produced by an application running on a server, on a client with limited processing capabilities. Two exemplary thin client protocols are ICA, Independent Computing Architecture from Citrix Systems, Inc., Ft. Lauderdale, Fla. and RDP, Remote Desktop Protocol from Microsoft, Inc., Redmond, Wash. The client is also sometimes referred to as a remote terminal session. One thin-client protocol intercepts commands by the application program to the server operating system (“OS”) to draw to a display screen. The intercepted or similar commands are transmitted to the remote session, using for example, the presentation layer packet. When the remote session (e.g., thin-client) receives the command, the remote session passes the received commands to the remote session OS. The thin-client draws the application program output on its display using the received commands as if the application program were executing on the thin-client.

Typically, when the application program draws images to the display screen, the image is represented as a bitmap. A bitmap format of an image is generally a very large data set. Thus the thin-client protocol must transmit over the network the bitmap representation of an image, which is a large amount of data, along with the applicable commands on how to display the bitmap representation. For networks of low bandwidth, this results in a large time delay before the complete image is received and displayed on the client. This can result in inconvenience and unhappiness for the user of the client. Also, if the user is paying directly for bandwidth used, for example in a wireless network, transmission of these large bitmap formats results in large costs associated with each transmission.

SUMMARY OF THE INVENTION

The invention lowers the time and cost of transmission of an image and other non-textual elements with a large bitmap representation by substituting an available compressed format of the image for the bitmap format. The compressed format can typically reduce the necessary bandwidth by a factor of 3 to 10. The client performs the decompression of received data using available libraries. The client then substitutes the decompressed image for the bitmap representation using, for example, modified thin-client protocol commands with other identifying data.

In one aspect the invention relates to a method for generating a graphical display for a remote terminal session. The method includes monitoring output produced by an application program executing on a server, identifying a bitmap representation within the output produced by the application program and determining a check value for the bitmap representation. The method also includes retrieving a compressed data format of the bitmap representation based at least in part on the check value and transmitting to the remote terminal session the compressed data format in place of the bitmap representation.

In one aspect the invention relates to a method for generating a graphical display for a remote terminal session. The method includes monitoring output produced by an application program executing on a server and identifying a textual element and a non-textual element of the output. The method also includes retrieving a compressed data format associated with the non-textual element and transmitting to the remote terminal session the textual element and the compressed data format in place of the non-textual element.

In one embodiment, the method further includes receiving the compressed data format and the textual element and generating a display using the textual element and the compressed data format. In another embodiment, the method further includes transmitting the compressed data format and the textual element using a presentation layer protocol packet. In another embodiment, the method further includes transmitting the presentation layer protocol packet using a command for transmitting a file in its native format.

In another embodiment, the method further includes conforming the presentation layer protocol packet to a remote access protocol, a thin-client protocol and/or a presentation protocol. In another embodiment, the method further includes transmitting further comprises replacing the bitmap representation with the compressed data format, wherein the non-textual element is a bitmap representation. In another embodiment, the method further includes determining whether the remote terminal session can render the non-textual element using the compressed data format. In another embodiment, the method further includes, upon determination that the remote terminal session cannot render the non-textual element using the compressed data format, transmitting an image-rendering library capable of rendering the non-textual element using the compressed data format.

In another embodiment, the method further includes intercepting the output and inspecting the intercepted output for a bitmap representation of the non-textual element. In another embodiment, the method further includes calculating a first check value for a bitmap representation of the non-textual element and searching an image store for the compressed data format with a second check value identical to the first check value.

In another aspect, the invention relates to a system for generating a graphical display for a remote terminal session. The system includes an output filter module and a server agent. In another embodiment, the system includes a client agent. In another embodiment, the system further includes a server node, wherein the server node including the server agent and the output filter module. In another embodiment, the system further includes a client node, wherein the client node includes the client agent and a display. In another embodiment, the system further includes a network.

In another aspect the invention relates to an article of manufacture having computer-readable program means embodied therein for generating a graphical display for a remote terminal session. The article includes computer-readable program means for performing any of the aforementioned methods.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and further advantages of the invention may be better understood by referring to the following description taken in conjunction with the accompanying drawing, in which:

FIG. 1 is a block diagram of an illustrative embodiment of a system to generate a display for a remote terminal session in accordance with the invention;

FIG. 2 is a flow diagram of an illustrative embodiment of a process to generate a display for a remote terminal session in accordance with the invention.

DETAILED DESCRIPTION

In broad overview, FIG. 1 illustrates a system 100 to generate a display for a remote terminal session that includes a first computing system (“client node”) 105 in communication with a second computing system (“server node”) 110 over a network 115. For example, the network 115 can be a local-area network (LAN), such as a company Intranet, a wide area network (WAN) such as the Internet or the World Wide Web or the like. A user of the client node 105 can be connected to the network 115 through a variety of connections including standard telephone lines, LAN or WAN links (e.g., T1, T3, 56 kb, X.25), broadband connections (ISDN, Frame Relay, ATM), wireless connections and the like. The client node 105 includes a client transceiver 130 to establish communication with the network 115. The server node 110 includes a server transceiver 135 to establish communication with the network 115. The connections can be established using a variety of communication protocols (e.g., ICA, RDP, HTTP, TCP/IP, IPX, SPX, NetBIOS, Ethernet, RS232, direct asynchronous connections and the like).

The server node 110 can be any computing device capable of providing the requested services of the client node 105. Particularly, this includes generating and transmitting commands and data to the client node 105 that represents the output being produced by an application program 140 executing on the server 110. The server node 110 includes the server transceiver 135, the executing application program 140, a server agent 150, an output filter module 155 and an image store 160. The server agent 150 corresponds to a module that interfaces with a client agent 175 and other components of the server node 110 to support the remote display and operability of the application program 140. The server agent module 150 and all modules mentioned throughout the specification are implemented as a software program and/or a hardware device (e.g., ASIC, FPGA and the like)

For clarity, all of these components are shown on server node 110. It is to be understood that the server node 110 can represent a single server or can represent several servers in communication with each over the network 115 or another network (not shown). In multiple server embodiments, the functionality of the components can be distributed over the available servers. For example, in one embodiment with multiple servers, the transceiver 135 is on a web server, the application program 140, the server agent 150 and the output filter module 155 are on an application server and the image store 160 is on a storage device, such as a disk in a RAID system.

The client node 105 can be any computing device (e.g., a personal computer, set top box, wireless mobile phone, handheld device, personal digital assistant, kiosk, etc) used to provide a user interface to the application program 140 executing on the server node 110. The client node 105 includes the client transceiver 130, a display 145, a client agent 175 and a graphics library 180 (also referred to as an image-rendering library). The client agent 175 corresponds to a module, implemented as a software program and/or a hardware device (e.g., ASIC, FPGA and the like) that receives commands and data from the server node 110 and from a user (not shown) of the client node 105. The client agent 175 uses the received information when interacting with other components of the client node 105 (e.g., when directing the operating system to output data onto the display 145). The client agent 175 also transmits requests and data to the server node 110 in response to server-issued commands or user actions at the client node 105.

The server node 110 hosts one or more application programs 140 that can be accessed by the client nodes 105. Examples of such applications include word processing programs such as MICROSOFT Word and spreadsheet programs such as MICROSOFT Excel, both manufactured by Microsoft Corporation of Redmond, Wash. Other examples include financial reporting programs, customer registration programs, programs providing technical support information, customer database applications, application set managers and the like. For clarity, another example of an application program is Internet Explorer, manufactured by Microsoft Corporation of Redmond, Wash., and will be used as the exemplary application program 140. It is understood that other application programs can be used.

During execution of the application program 140, for example Internet Explorer, the server node 110 communicates with the client node 105 over a transport mechanism. In one embodiment, the transport mechanism provides multiple virtual channels 185 through the network 115 so the server agent 150 can communicate with the client agent 175. One of the virtual channels 185 provides a protocol for transmitting graphical screen data from the server node 110 to the client node 105. The server 110 executes a protocol driver, in one embodiment as part of the server agent 150, that intercepts graphical display interface commands generated by the application program 140 and targeted at the server's operating system. The protocol driver translates the commands into a protocol packet suitable for transmission over the transport mechanism.

The application program 140, in this example Internet Explorer, executing on the server 110, retrieves a web page. As explained above, the application program 140 generates graphical display commands to the server operating system, as if it was going to display the output at the server node 110. The server agent 150 intercepts these commands and transmits them to the client agent 175. The client agent 175 issues the same or similar commands to the client operating system to generate output for the display 145 of the client node 105.

In one embodiment, a web page has both textual elements (e.g., titles, text, ASCII characters and the like) and non-textual elements (e.g., images, photos, icons, splash screens and the like) incorporated therein. The non-textual elements are sometimes transmitted to the Internet Explorer application program 140 from a web server (not shown) in a compressed data format (e.g., file, data stream and the like), also referred to as the non-textual element's native format. Examples of compressed formats are JPEG, GIF, PNG and the like. The non-textual element represented in a compressed data format may be, for example, 20 kilobytes in size. That same non-textual element decompressed into a bitmap representation is, for example, 300 kilobytes in size.

The application program 140, when generating the display of the web page, retrieves, for example, a JPEG data format of a non-textual element and decompresses the JPEG data format into a bitmap for display. The output filter module 155 determines that the bitmap representation is from a compressed format and obtains the corresponding compressed format of the non-textual element from the image store 160, as explained in more detail below. In one embodiment, the image store 160 is persistent storage. In other embodiments, the image store 160 is temporary storage, cache, volatile memory and/or a combination of temporary and persistent storage.

The server agent 150 replaces the bitmap representation of the non-textual element with the non-textual element in the compressed data format that the output filter module 155 retrieved from the image store 160. The server agent 150 transmits the non-textual element in the compressed format, along with the graphical display interface commands associated with the bitmap representation, to the client node 105. In one embodiment the server agent 150 uses a unique protocol command that identifies a transmission of a non-textual element that is not in bitmap representation, even though the associated commands are applicable to a bitmap representation of a non-textual element. In other embodiments other identifying techniques can be used. For example, the protocol command can have a modifier comment, or a command switch. The command can also use a change of context, a combination of multiple commands and the like.

The client agent 175 receives the transmission of the non-textual element file in the compressed data format, along with the graphical display interface commands associated with the bitmap representation of the non-textual element. The client agent 175 determines that the non-textual element is in the compressed data format and not the bitmap representation. In one embodiment, the client agent 175 makes this determination because the non-textual element in compressed format is transmitted using a unique protocol command. In another embodiment, the size of the non-textual element data and/or other characteristics about the non-textual element included with the associated graphical display interface commands are enough to enable the client agent 175 to make the determination.

The client agent 175 determines whether the client node 105 contains the necessary library 180 to decompress the compressed format of the non-textual element. If the client node 105 has the appropriate graphics library(ies) 180 installed to perform the decompression algorithms, the client agent 175 uses the library 180 to decompress the compressed format of the non-textual element into its bitmap representation. The client agent 175 performs the received associated graphical display interface commands on the bitmap representation to generate the non-textual element of the output of the application program 140 on the client display 145.

In one embodiment, the client agent 175 does not contain all the decompressing algorithms to decompress the non-textual element from a compressed format into a bitmap representation. If the client node 105 does not have the appropriate graphics library(ies) 180 installed to perform the decompression algorithms, the client agent 175 requests the needed graphics library from the server node 110. In another embodiment, the client agent 175 determines if a predetermined set of the most widely used graphics libraries 180 are installed on the client node 105 prior to receiving any non-textual elements from the server node 110. If the most widely used graphics libraries 180 are not installed on the client node 105, the client agent 175 requests the missing libraries from the server node 110 prior to receiving any non-textual elements from the server node 110.

In yet another embodiment, the client agent 175 determines which graphics libraries 180 the client node 105 includes and transmits that library information to the server agent 150. In this embodiment, when the server agent 150 receives the compressed data format of a bitmap representation from the output filter module 155, the server agent 150 determines, using the transmitted library information, whether the client node 105 can render the compressed data format. If the server agent 150 determines that the client node 105 has the applicable library, the server agent 150 substitutes the compressed data format for the bitmap representation of the non-textual element. If the server agent 150 determines that the client node 105 does not have the applicable library, the server agent 150 does not substitute the compressed data format for the bitmap representation of the non-textual element and instead transmits the bitmap representation to the client 105.

For the server agent 150 to replace the bitmap representation of the non-textual element with the non-textual element in the compressed format, the output filter module 155 determines that the bitmap representation is from a compressed format contained in the image store 160. To make this determination, the output filter module 155 calculates one or more check values for the bitmap representation. For example, the output filter module 155 can calculate a single check value for the entire bitmap representation and/or the output filter module 155 can calculate four check values, one for each for the entire bitmap representation. In another example, the output filter module 155 can calculate N check values, one for each of the N lines in the bitmap representation. A check value is the result of an algorithm that generates a substantially unique value for different arrangements of data. The check value is, for example a Cyclic Redundancy Code (“CRC”), a check sum, a result of a hashing function and the like. The check value is based on the bitmap representation and not the data as arranged in a compressed data format. However, when the compressed data format is stored in the image store 160, it is stored with a check value attribute that corresponds to the one or more check values of the bitmap representation of the compressed data when decompressed.

The output filter module 155 searches the image store 160 for a non-textual element in compressed data format that has a check value attribute that is the same one or more check values the output filter module 155 calculates for the bitmap representation. The output filter module 155 retrieves the compressed format of non-textual element with the same check value attribute as the one or more check values and sends the compressed format of the non-textual element to the server agent 150 for transmittal to the client agent 175 in place of the bitmap representation.

The server node 110 stores compressed formats of non-textual elements in the image store 160 the first time the application program 140 calls a graphics library (not shown) to create a bitmap representation from a compressed format file. The output filter module 155 calculates the associated check value of the bitmap representation as the application program 140 decompresses the compressed format and generates the bitmap representation. As described above, the output filter module 155 can calculate the check value when the bitmap representation is complete, when a quadrant of the bitmap representation is complete, when a line of the bitmap representation is complete and the like. The server 110 stores the compressed format file and the associated check value attribute in the image store 160 and retrieves the compressed format file the first and any subsequent times the application program 140 generates the associated non-textual element.

Whether the server 110 stores the compressed format file and its associated check value attribute(s) in the image store 160 in a temporary portion (e.g., RAM memory buffer, cache and the like) or a persistent portion (e.g., disk, non-volatile memory buffer, and the like) is based at least in part on design and hardware limitations (e.g., the size of the persistent storage). One exemplary criterion used to make that determination is the number of times the application program 140 generates the non-textual element. For example, if the application program 140 generates a particular non-textual element more than a predetermined number of times, the server 110 stores the compressed format file and its associated check value attribute(s) corresponding to that particular non-textual element persistently in the image store 160.

In other embodiments, the server 110 stores the non-textual element if it is static, complex and the like. For example, if the application program 140 always generates a splash screen at initialization, the server 110 stores the compressed format file corresponding to that splash screen in the persistent portion of the image store 160. In another embodiment, if the non-textual element is complex, static and/or generated repeatedly but does not have a corresponding compressed format file, the output filter module 155 generates a compressed format file for that non-textual element, in a standards-based or proprietary-based format. In any subsequent transmissions, the server agent 150 transmits the generated compressed format file in place of the bitmap representation. If the compressed format is a proprietary-based format, the server agent 150 determines whether the client node 105 includes the applicable proprietary-based graphics library to decompress the compressed format file into a bitmap representation. If not included on the client node 105, the server agent 150 transmits the applicable library to the client node 105 for installation.

Although the illustrated embodiment depicts the image store 160 on the server node 110, in an alternate embodiment, at least a portion of the image store (not shown) is on the client node 105. In this alternate embodiment, the output filter module 155 calculates the one or more check values of the bitmap representation and transmits the one or more check values to the server agent 150. The server agent 150 transmits these one or more check values to the client agent 175. The client agent 175 searches the portion of the image store on the client node 105 for a compressed data format stored with an identical one or more check values attribute. The client agent 175 transmits the results of this search to the server agent 150.

If the compressed data format for the non-textual element exists on the client node 105, the server agent 150 does not have to send either the compressed data format or the bitmap representation over the network 115. The server agent 150 only transmits the graphical display interface commands associated with the bitmap representation of the non-textual element. If the compressed data format for the non-textual element does not exist on the client node 105, the output filter module 155 obtains the corresponding compressed format of the non-textual element from the image store 160. The server agent 150 replaces the bitmap representation of the non-textual element with the non-textual element in the compressed data format that the output filter module 155 retrieved from the image store 160. The server agent 150 transmits the non-textual element in the compressed format, along with the graphical display interface commands associated with the bitmap representation, to the client node 105.

FIG. 2 illustrates an exemplary embodiment of a process 200 to generate a display for a remote terminal session, using the exemplary embodiment of FIG. 1. The output filter module 155 monitors the output of the application program 140 by monitoring calls made to the operating system of the server node 110. When the output filter module 155 detects (step 205) a display command from the application program 140, the output module 155 determines (step 210) whether the application program 140 is generating a bitmap representation of a non-textual element.

If the application program 140 is not generating a bitmap representation of a non-textual element, the output filter module 155 transmits (step 215) the display command to the server agent 150, which transmits that command, or a representative command defined in the protocol, to the client agent 175. If the application program 140 is generating a bitmap representation of a non-textual element, the output filter module 155 calculates (step 220) one or more check values corresponding to the bitmap representation of the non-textual image.

Using the one or more calculated check value(s), the output filter module 155 searches the image store 160 to determine (step 225) whether a compressed data format with identical check value attribute(s) exists. If there is a compressed data format in the image store 160 with check value attribute(s) identical to the one or more check values the output filter module 155 calculates, the output module 155 replaces (step 230) the bitmap representation of the non-textual element with the compressed data format. The output module 155 transmits (step 230) the compressed data format to the server agent 150 for eventual transmission to the client agent 175. The output module 155 also transmits all of the commands associated with the replaced bitmap representation along with the compressed data format.

If there is not a compressed data format with identical one or more check value attributes in the image store 160, the output module 155 determines (step 235) whether the bitmap representation of a non-textual element corresponding to the compressed data format meets a predetermined criterion for persistent storage (e.g., any of the criteria described above). If the output module 155 determines (step 235) that the predetermined criterion is met, the output module 155 stores (step 240) the compressed data format and the corresponding check value attribute, identical to the one or more calculated check values, in the persistent portion of the image store 160. If the output module 155 determines (step 235) that the predetermined criterion is not met, the output module 155 stores (step 245) the compressed data format and the corresponding check value attribute, identical to the one or more calculated check values, in the temporary portion of the image store 160.

Once the output module 155 stores (step 240 or 245) the compressed data format and the corresponding check value attribute, identical to the one or more calculated check values, in the image store 160, the output module 155 replaces (step 230) the bitmap representation of the non-textual element with the compressed data format. The output module 155 transmits (step 230) the compressed data format to the server agent 150 for eventual transmission to the client agent 175. The output module 155 continues monitoring the output generated by the application program 140 until the output module 155 detects (step 205) another display command from the application program 140.

EQUIVALENTS

The invention can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The foregoing embodiments are therefore to be considered in all respects illustrative rather than limiting on the invention described herein. Scope of the invention is thus indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.

Claims

1. A method for generating a graphical display, the method comprising: analyzing display output via an output filter executing on a first computing device, the display output comprising bitmap display data and textual data generated by an application executing on the first computing device, and displayed on a display device connected to a second computing device;identifying, by the output filter, within the display output bitmap display data corresponding to a bitmap representation of a non-textual element;calculating, by the output filter, a check value representative of the bitmap representation of the non-textual element;determining, by the first computing device, whether the calculated check value matches a second check value on a repository of the second computing device, the second check value corresponding to the bitmap representation of the non-textual element; andif the calculated check value does not match the second check value, retrieving, by the output filter from a repository of the first computing device, a compressed format data element of the bitmap display data based on the calculated check value matching a first check value in the repository of the first computing device and transmitting, by the first computing device to the second computing device, the retrieved compressed format data element in place of the bitmap representation of the non-textual element,else, transmitting, by the first computing device to the second computing device, graphical display interface commands associated with the bitmap display data along with the textual data.

2. The method of claim 1, wherein identifying bitmap display data further comprises identifying one or more non-textual elements.

3. The method of claim 1, wherein identifying bitmap display data further comprises identifying one or more non-textual elements transmitted as display image data.

4. The method of claim 1, wherein calculating further comprises calculating a string representative of the bitmap display data.

5. The method of claim 1, wherein calculating further comprises calculating a plurality of check values representative of the bitmap display data.

6. The method of claim 1, further comprising searching the repository of the first computing device.

7. The method of claim 1, further comprising searching the repository of the second computing device.

8. The method of claim 1, further comprising transmitting to the second computing device the textual data along with the compressed format data element.

9. The method of claim 1, further comprising searching a repository installed on a third computing device.

10. The method of claim 1, further comprising searching a repository having a portion installed on a third computing device.

11. The method of claim 1, further comprising: determining a rendering component does not exist on the second computing device in communication with the first computing device; andtransmitting the bitmap display data from the first computing device to the second computing device.

12. A system for generating a graphical display, the system comprising an output filter executing on a first computing device to: analyze display output comprising bitmap display data and textual data, the display output generated by an application executing on the first computing device, and displayed on a display device connected to a second computing device,identify within the display output bitmap display data corresponding to a bitmap representation of a non-textual element,calculate, by the output filter, a check value representative of the bitmap representation of the non-textual element,wherein the first computing device determines whether the calculated check value matches a second check value on a repository of the second computing device, the second check value corresponding to the bitmap representation of the non-textual element, and if the calculated check value does not match the second check value, the output filter retrieves from a repository of the first computing device a compressed format data element of the bitmap display data based on the calculated check value matching a first check value in the repository of the first computing device and the first computing device transmits to the second computing device the retrieved compressed format data element in place of the bitmap representation of the non-textual element,else the first computing device transmits to the second computing device graphical display interface commands associated with the bitmap display data along with the textual data.

13. The system of claim 12, wherein the output filter identifies one or more non-textual elements.

14. The system of claim 12, wherein the output filter identifies one or more non-textual elements transmitted as display image data.

15. The system of claim 12, wherein the output filter calculates a string representative of the bitmap display data.

16. The system of claim 12, wherein the output filter calculates a check value representative of the bitmap display data.

17. The system of claim 12, wherein the output filter searches the repository of the first computing device.

18. The system of claim 12, wherein the output filter searches a repository of the second computing device.

19. The system of claim 12, wherein the first computing device transmits to the second computing device the textual data along with the compressed format data element.

20. The system of claim 12, wherein the output filter searches a repository installed on a third computing device.

21. The system of claim 12, wherein the output filter searches a repository having a portion installed on a third computing device.

22. The system of claim 12, wherein the output filter: determines a rendering component does not exist on the second computing device in communication with the first computing device; andtransmits the bitmap display data from the first computing device to the second computing device.