Patent Application
20110202673
1. Field of the Invention
Concepts described herein relate generally to data compression, and more particularly, to a dictionary based compression scheme in a wide area network optimization device.
2. Description of Related Art
Geographically distributed entities, such as a business with multiple local offices in different locations, frequently desire to maintain a logically unified proprietary network that is accessible by all of the geographically distributed entities. To accomplish this, a wide area network (WAN) may be used to link the different locations.
Information technology managers within geographically distributed entities face the conflicting requirements of keeping costs in check while providing users with new and faster applications. This conflict is most acute on the WAN, where costs can be high and getting applications to perform well is often hampered by limited bandwidth and high latency.
Solutions are known to help improve performance over the WAN portion of a network. These solutions may be based on a number of technologies, including data compression, data prefetching and caching, application-specific acceleration, and policy-based bandwidth allocation.
Data caching and compression, in particular, can be used to reduce the perceived latency (response time) of data accesses over a WAN. Compression and caching techniques may be disk-based, as opposed to semiconductor memory based, due to orders-of-magnitude larger storage capacity of typical hard disk drives.
Dictionary-based compression techniques can be particularly effective in compressing large file transfers. With this type of compression technique, data may be stored at compression devices at opposite ends of the WAN. When a block of data that has been previously sent is again received by the compression device, the compression device does not need to send the actual block of data. Instead, it may simply send a reference to the block of data to the device on the other end of the WAN.
Unfortunately, as more and more types and sizes of information are transmitted across a WAN, the data stored for execution of the compression technique grows as well, thereby decreasing the efficiency and increasing the time and processing cost associated with the compression operation.
In one aspect, a method may include receiving a stream of data; identifying a selected compression dictionary from among a number of compression dictionaries based on the stream of data; notifying a receiving device about the selected compression dictionary; compressing the stream of data using the selected compression dictionary; and transmitting the compressed stream of data to the receiving device.
In another aspect, a device for compressing data over a network may include logic to identify a compression dictionary associated with a received data stream, where the identified compression dictionary is one of a number of available compression dictionaries; logic to notify a receiving device regarding the identified compression dictionary; logic to compress the received data stream using the identified compression dictionary to generate a compressed data stream; and logic to transmit the compressed data stream to the receiving device over a network.
In yet another aspect, a system may include a transmitting network device to receive a data stream from a first local network and transmit a compressed data stream via a wide area network and a receiving network device to receive the compressed data stream from the wide area network. The transmitting network device may include logic to identify a selected compression dictionary from a number of compression dictionaries based on the received data stream; logic to output a notification message to the receiving network device, where the notification message includes at least an identification of the selected compression dictionary; logic to receive an acknowledgement message from the receiving network device indicating receipt of the notification message; logic to compress the data stream using the selected compression dictionary; and logic to transmit the compressed data stream to the receiving network device via the wide area network. The receiving network device may include logic to receive the notification message from the transmitting network device; logic to identify the selected compression dictionary from the notification message; logic to transmit the acknowledgement message to the transmitting network device; logic to receive the compressed data stream; and logic to decompress the compressed data stream using the selected compression dictionary.
In still another aspect, a method for decompressing a compressed data stream may include receiving a compression dictionary notification message from a transmitting network device via a network; identifying a selected compression dictionary from the compression dictionary notification message; receiving the compressed data stream; and decompressing the compressed data stream using the selected compression dictionary.
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate one or more embodiments of the invention and, together with the description, explain the invention. In the drawings,
The following detailed description of the invention refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements. Also, the following detailed description does not limit the invention. Instead, the scope of the invention is defined by the appended claims and equivalents.
As described herein, a network optimization device may operate to increase application performance over a wide area network. The network optimization device may use disk-based compression to achieve the performance increase. The disk-based compression may be performed using an efficient indexing scheme.
WAN 170 may include a public network, such as the Internet. In some implementations, WAN 170 may be implemented by privately owned or leased data lines between private networks 180, or by a combination of a public network and privately owned or leased data lines. In one implementation, data traffic between private networks 180 through WAN 170 may be characterized by one or more of increased cost, higher latency, or lower bandwidth than data traffic within a single corporate network 180. Accordingly, network applications accessing resources between a number of private networks 180 and across WAN 170 may tend to run less optimally than an application running locally at a private network 180.
Private networks 180 may each include a number of computing devices, such as, for example, client computing stations 110, network security device(s) 115, servers 120, and databases 125. Client computing stations 110 may include computing devices of end-users, such as desktop computers or laptops. Network security devices 115 may include, for example, firewalls or other devices relating to network security. Servers 120 may include one or more devices designed to interact with and deliver information to other devices, such as client computing stations 110. Databases 125 may include information repositories available on, for example, network 180-C. Servers 120 may, for example, serve data files or network applications to computing stations 110 over WAN 170. Servers may interact with databases 125 when providing information to client computing stations 110.
In the exemplary system shown in
Consistent with an aspect of the invention, private networks may be associated with network optimization devices (NODs) 150-A through 150-C (collectively referred to as network optimization devices 150). As shown in
Processor 220 may include a processor, microprocessor, ASIC, FPGA, or processing logic that may interpret and execute instructions. Main memory 230 may include a random access memory (RAM) or another type of dynamic storage device that may store information and instructions for execution by processor 220. ROM 240 may include a ROM device or another type of static storage device that may store static information and instructions for use by processor 220. Storage device 250 may include a magnetic storage medium such as a hard disk drive.
Input device 260 may include a mechanism that permits an operator to input information to the device, such as a keyboard, a mouse, a pen, voice recognition and/or biometric mechanisms, etc. Output device 270 may include a mechanism that outputs information to the operator, including a display, a printer, a speaker, etc. Communication interface 280 may include any transceiver-like mechanism that enables the device to communicate with other devices and/or systems.
Device 200 may perform certain operations, as described in detail below. The device may perform these operations in response to processor 220 executing software instructions contained in a computer-readable medium, such as memory 230. A computer-readable medium may be defined as a physical or logical memory device.
The software instructions may be read into memory 230 from another computer-readable medium, such as data storage device 250, or from another device via communication interface 280. The software instructions contained in memory 230 may cause processor 220 to perform processes that will be described later. Alternatively, hardwired circuitry may be used in place of or in combination with software instructions to implement processes consistent with the principles of the invention. Thus, implementations consistent with the principles of the invention are not limited to any specific combination of hardware circuitry and software.
Before describing the operation of network optimization devices 150 in additional detail, it will be helpful to generally describe compression over a wide area network.
Assume that client computing station 110, in private network 180-A, would like to transfer a file or a portion of a file to file server 120, in private network 180-C. The file transfer is being made across WAN 170. Network optimization devices 150-A and 150-C may operate together to increase the perceived performance of file transfers for client computing station 110 by, among other things, using data compression when transferring data over WAN 170.
One form of data compression is known as symbol or dictionary-based compression. With this form of data compression (examples include the Lempel-Ziv (LZ) family of compression algorithms, etc.), pairs of network optimization devices 150 may store patterns of data, such as patterns from previously exchanged data, as strings in memory or on a hard disk drive. Each string of data may be associated with an identifier or codeword, such as a location on the hard disk drive or a position in an index. Each string should ideally be much longer than the corresponding codeword. When a string to be compressed is subsequently encountered, only the codeword corresponding to the string is sent, rather than the string itself. Dictionary-based compression techniques are often referred to as lossless compression, because no data is lost during compression and decompression of the data.
As a simplified example of the operation of data compression for transmitted data, assume that client computing station 110 initiates transmission of data to file server 120, shown as arrow 310 in
To efficiently implement a dictionary-based compression scheme, it is desirable to be able to efficiently identify data that was previously seen or sent. Unfortunately, as noted above, as increasing quantities and types of information are transmitted over WAN 170, the size of the compression dictionary will similarly increase. As the size of the compression dictionary increases, the efficiency with which previously identified strings may be identified is reduced.
In accordance with one implementation described herein, enhanced compression speed and efficiency may be obtained by maintaining and propagating multiple different compression dictionaries. The selection or usage of a particular compression dictionary may be based on network-specific information associated with the data to be compressed, such as a network protocol associated with the data, a port or range of ports associated with the data, or a combination of network characteristics. In this manner, a size of a compression dictionary to be searched during the compression of a data stream may be minimized. Further, by restricting an applied compression dictionary to relate more specifically to a type of data being transmitted, additional compression efficiencies may be realized.
Compression component 420 and decompression component 430 may perform dictionary-based compression and decompression of data for entities transferring data through network optimization device 150. An entity, as used herein, may refer to any distinguishable unit that uses network optimization device 150, such as a computer, a process, or a group of computers/processes. Although shown as a single component in
In one implementation consistent with embodiments described herein, compression component 420 and decompression component 430 may create and/or utilize one of compression dictionaries 440-A to 440-N in compressing data being sent to the WAN or decompressing data received from the WAN. For example, compression component 420 may receive data from a local network or local device, such as client computing station 110. In one implementation, compression component 420 may receive a TCP (transmission control protocol) data stream from a local network. The received data stream may include various types of data, such as http (hypertext transfer protocol) data, sip (session initiation protocol) data, ftp (file transfer protocol) data, etc. The received data stream may include packet information associated with the data being transmitted.
In accordance with embodiments described herein, data sharing predefined network characteristics may be compressed using a common dictionary that is not applied to unrelated or dissimilar types of information. By restricting a compression dictionary to utilize only data meeting predetermined criteria, the size of the dictionary may be kept small, thereby increasing the speed and efficiency with which it is used. In one implementation, compression dictionary identifier logic 425 may analyze the received data and may identify a compression dictionary (e.g., compression dictionaries 440-A to 440-N) to use based on predefined or automatic data compression rules.
For example, as shown in
In one implementation consistent with aspects described herein, the compression rules in rules engine 450 may be received from users via a user interface associated with network optimization device 150. In other implementations, rules engine 450 may be configured to automatically create compression rules based on network conditions associated with the data passing through network optimization device 150. For example, rules engine 450 may be configured to automatically monitor the data for traffic-related criteria, such as a threshold amount of data associated with one or more network characteristics, e.g., a flow, a protocol, etc., received within a predetermined amount of time. When it is determined that a threshold amount of data has been met, rules engine 450 may be configured to create one or more compression rules based on the identified network characteristic.
Once rules engine 450 of compression dictionary identifier logic 425 identifies a compression rule associated with received data, compression component 420 may determine whether one of existing compression dictionaries 440-A to 440-N corresponds to the identified compression rule. For example, compression dictionary identifier logic 425 may determine whether either the rule being applied is a newly created rule or whether the initial data associated with a rule is initially received. If an existing compression dictionary does not correspond to the identified compression rule (e.g., in storage device 250), compression component 420 may create a new compression dictionary in compression component 420 (e.g., in storage device 250). As described above in relation to known dictionary-based compression systems, each compression dictionary 440-A to 440-N may be dynamically created as data associated with the particular compression dictionary is received by network optimization device 150. As more data associated with the compression dictionary is received, the compression dictionary becomes more and more populated. However, if an existing compression dictionary is identified as corresponding to the data, the identified compression dictionary may be used to compress the received data, with additional unique data being added to the dictionary in a known manner.
Because each transmitting network optimization device 150-A may create and update a number of different compression dictionaries 440 for an outbound data stream, network optimization device 150-A may also be provided with a mechanism for notifying a receiving network optimization device 150-C of which compression dictionaries have been created and which compression dictionary is being used to compress a current data stream. In one implementation consistent with aspects described herein, the notification of a compression dictionary being created and/or used may be provided via an out-of-band message, sent to receiving network optimization device 150-C. For example, the notification message may include at least a sequence number associated with the beginning of the compressed data and information regarding which rule has been applied and/or which dictionary has been used to compress the data. The notification message may be transmitted through WAN 170 via an out-of-band compression dictionary message formatted in any suitable format, such as extensible markup language (XML) or simple network management protocol (SNMP) message. Transmission of such a message is illustrated in
In an alternative implementation, the notification of a compression dictionary being created and/or used may be provided via in-band signaling, where the dictionary identification and/or compression rule information may be injected into an initial portion of the data stream being compressed. Upon receipt of this in-band signaling information, receiving network optimization device 150-C may determine the appropriate dictionary to either create or apply to the received data stream. As with the above-described out-of-band implementation, transmitting network optimization device 150-A may require an acknowledgement message from receiving network optimization device 150-C prior to usage of the identified dictionary.
Each dictionary 440 may include an index of the historical data seen by compression component 420 and decompression component 430 and identified as relating to a rule associated with that dictionary. By limiting the contents of each dictionary 440 to only entries identified by a rule, a size of each dictionary may be reduced. Each dictionary 440 may be stored in, for example, a storage device, such as main memory 230 or storage device 250. In some exemplary implementations, each dictionary 440 may be refreshed or reinitiated in response to a user command or in response to predefined criteria, such as the passage of a set length of time, or reaching a set size limit. By providing such refreshing, a speed with which compression is accomplished may be maximized.
When acting as a receiving network optimization device 150, a compression dictionary notification message may be received via either in-band or out-of-band signaling. As described above, the compression dictionary notification message may include a sequence number associated with the beginning of the compressed data as well as an indication of a compression dictionary and/or compression rule used to compress the data.
Upon receipt of the compression dictionary notification message, decompression component 430 may return an acknowledgement message to transmitting network optimization device 150A. decompression dictionary identifier logic 435 may then examine the compression dictionary notification message and determine whether an existing dictionary corresponds to the received message. If not, decompression component 430 may establish a new dictionary corresponding to the rule identified in the compression dictionary notification message. In one implementation consistent with embodiments described herein, the rule corresponding to the established dictionary may be stored in rules engine 450 for use in compressing subsequent outgoing data.
Once an appropriate dictionary has been either identified or established, compressed data received from transmitting network optimization device 150-A via WAN 170 may be decompressed using the identified compression dictionary. The decompressed data may then be forwarded to local entities, applications, or devices.
Once received, compression dictionary identifier logic 425 may compare the data with one or more compression rules maintained and/or configured by rule engine 450 to determine whether a particular compression dictionary is to be used to compress the data (block 505). As described above, rule engine 450 may be configured to automatically create a rule and/or compression dictionary based on the satisfaction of various network-related characteristics.
If no rule applying to the received data is identified (block 505—NO), the data may be compressed using a default or standard dictionary (block 510). However, if a rule is identified (or created) that applies to the received data (block 505—YES), compression dictionary identifier logic 425 may determine whether a corresponding compression dictionary associated with the rule has been previously created (block 515). If no corresponding compression dictionary has been previously created (block 515—NO), a compression dictionary corresponding to the identified rule may be generated (block 520) and the received data may be compressed using the generated compression dictionary (block 525).
Returning to block 515, when it is determined that a compression dictionary corresponding to the identified rule exists or has been previously established (block 515—YES), the received data may be compressed using the previously established compression dictionary (block 530).
Following compression of the data using either the default compression dictionary, a newly created compression dictionary, or a previously established compression dictionary, a compression dictionary notification message may be transmitted to the receiving network optimization device (block 535). As described above, the compression dictionary notification message may include an out-of-band message transmitted to the receiving network optimization device that includes at least a sequence number associated with the compressed data and an identification of the compression dictionary and/or rule used to compress the data. In other implementations, the compression dictionary notification message may be transmitted in-band.
An acknowledgement of receipt of the compression dictionary notification message may be received from the receiving network optimization device (block 540). This acknowledgement may be in the form of an in-kind acknowledgment message (ACK). For example, an out-of-band compression dictionary notification message may be acknowledged via a return out-of-band acknowledgement message. Similarly, an in-band compression dictionary notification message may be acknowledged via a return in-band acknowledgement message.
Once the acknowledgement message has been received, network optimization device 150-A may output the compressed data via WAN 170 to receiving network optimization device 150-C (block 545).
Decompression dictionary identifier logic 435 of decompression component 430 may identify either a previously established compression dictionary to use or the basis for a newly created compression dictionary based on the received compression dictionary notification message (block 605). If a new compression dictionary is to be created in response to the received compression dictionary notification message, the compression dictionary may be created at block 610.
An acknowledgement of receipt of the compression dictionary notification message may be transmitted to transmitting network optimization device 150-A (block 615). This acknowledgement may be in the form of an in-kind acknowledgment message. For example, an out-of-band compression dictionary notification message may be acknowledged via a return out-of-band acknowledgement message. Similarly, an in-band compression dictionary notification message may be acknowledged via a return in-band acknowledgement message.
Compressed data may be received from transmitting network optimization device 150-A (block 620). The received data may be decompressed based on the dictionary identified by or created in response to the received compression dictionary notification message (block 625). The decompressed data may then be forwarded to a destination device or application (block 630).
In the manner described above, data 705 may be received by network optimization device 150-A prior to being transmitted across WAN 170. Upon receipt by network optimization device 150-A, dictionary identifier logic 425 may determine which compression dictionary should be used to compress the data. To accomplish this, compression dictionary identifier logic 425 may consult rules engine 450 for determining whether any existing or automatically generated rules exist which apply to received data 705. For the purposes of this example, assume that rules engine 450 identifies a rule indicating that data having a destination port in the range of 4000-6000 should be compressed using compression dictionary A 710. It should be noted that the rule described above is merely exemplary and that any suitable network-based characteristic, such as flow, protocol, IP addresses, etc., may be used to identify and/or generate a corresponding compression dictionary.
Following this determination, a compression dictionary notification message 715 may be transmitted to network optimization device 150-C via WAN 170. As described in detail above, this message may be transmitted in either an out-of-band or in-band manner. Upon receipt of compression dictionary notification message 715, decompression component 430 and decompression dictionary identifier logic 435 may determine that compression dictionary A 730 is to be used to decompress data received from network optimization device 150-A. A return acknowledgement message 720 may be transmitted back to network optimization device 150-A, thus indicating that network optimization device 150-C is ready to receive compressed data.
Once network optimization device 150-A has received acknowledgement message 720, network optimization device 150-A compresses data 705 using compression dictionary A and transmits compressed data 725 to network optimization device 150-C. Upon receipt of compressed data 725, network optimization device 150-C decompresses data 725 using compression dictionary A 730 and forwards decompressed data 735 to file sever 120.
The techniques discussed above for compressing and sending data over a WAN may provide a number of advantages relative to existing techniques. For example, by using a number of customizable and dynamic compression libraries, the speed and efficiency with which compression of network traffic may be accomplished is significantly increased. This also increases bandwidth between devices on the network. Furthermore, by enabling for in-band or out-of-band exchange of a dictionary notification message, a sending device may efficiently notify a receiving device regarding the compression dictionary to be used and/or the underlying criteria on which the applied compression dictionary is based.
A network characteristic dependent compression technique is described for optimizing the performance of WAN data transfer. The technique uses a rules engine to determine or initiate creation of a compression dictionary based on user-defined or automatically defined network criteria associated with received data. A dictionary notification message may be sent to a receiving network optimizing device, to notify the receiving device about the compression dictionary to be used to compress the data.
While a series of acts have been described with regard to
It will be apparent that aspects described herein may be implemented in many different forms of software, firmware, and hardware in the implementations illustrated in the figures. The actual software code or specialized control hardware used to implement aspects does not limit the embodiments. Thus, the operation and behavior of the aspects were described without reference to the specific software code—it being understood that software and control hardware can be designed to implement the aspects based on the description herein.
Further, certain portions of the implementations have been described as “logic” that performs one or more functions. This logic may include hardware, such as a processor, an application specific integrated circuit, or a field programmable gate array, software, or a combination of hardware and software.
Even though particular combinations of features are recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the invention. In fact, many of these features may be combined in ways not specifically recited in the claims and/or disclosed in the specification.
No element, block, or instruction used in the present application should be construed as critical or essential to the implementations described herein unless explicitly described as such. Also, as used herein, the article “a” is intended to include one or more items. Where only one item is intended, the term “one” or similar language is used. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 12137999 | Jun 2008 | US |
| Child | 13091966 | US |
